{"ID":85264,"post_author":"9412100","post_date":"2019-07-01 12:54:43","post_date_gmt":"0000-00-00 00:00:00","post_content":"","post_title":"LIMSjournal - Summer 2019","post_excerpt":"","post_status":"draft","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"","to_ping":"","pinged":"","post_modified":"2019-07-01 12:54:43","post_modified_gmt":"2019-07-01 16:54:43","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=85264","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","_ebook_metadata":{"enabled":"on","private":"0","guid":"43EB956B-F609-4A90-A49E-AA1ABB08B3A6","title":"LIMSjournal - Summer 2019","subtitle":"Volume 5, Issue 2","cover_theme":"nico_21","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_21&subtitle=Volume+5%2C+Issue+2&editor=Shawn+Douglas&title=LIMSjournal+-+Summer+2019&title_image=https%3A%2F%2Fs3.limsforum.com%2Fwww.limsforum.com%2Fwp-content%2Fuploads%2FFig1_Schabacker_FrontBioengBiotech2019_7.jpg&publisher=LabLynx+Press","editor":"Shawn Douglas","publisher":"LabLynx Press","author_id":"26","image_url":"","items":{"f59eece05cdc9b79c1305c6bea6b72d6_type":"article","f59eece05cdc9b79c1305c6bea6b72d6_title":"DAQUA-MASS: An ISO 8000-61-based data quality management methodology for sensor data (Perez-Castillo et al. 2018)","f59eece05cdc9b79c1305c6bea6b72d6_url":"https:\/\/www.limswiki.org\/index.php\/Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data","f59eece05cdc9b79c1305c6bea6b72d6_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:DAQUA-MASS: An ISO 8000-61-based data quality management methodology for sensor data\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDAQUA-MASS: An ISO 8000-61-based data quality management methodology for sensor dataJournal\n \nSensorsAuthor(s)\n \nPerez-Castillo, Ricardo; Carretero, Ana G.; Caballero, Ismael; Rodriguez, Moises;\r\nPiattini, Mario; Mate, Alejandro; Kim, Sunho; Lee, DongwooAuthor affiliation(s)\n \nUniversity of Castilla-La Mancha, AQC Lab, University of Alicante, Myongji University, GTOne,Primary contact\n \nEmail: ricardo dot pdelcastillo @ uclm dot esYear published\n \n2018Volume and issue\n \n18(9)Page(s)\n \n3105DOI\n \n10.3390\/s18093105ISSN\n \n1424-8220Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.mdpi.com\/1424-8220\/18\/9\/3105\/htmDownload\n \nhttps:\/\/www.mdpi.com\/1424-8220\/18\/9\/3105\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Data quality challenges in SCP environments \n4 Related work \n\n4.1 Sensor data quality \n4.2 Data quality methodologies comparison \n\n\n5 DAQUA-Model: A data quality model \n6 DAQUA-MASS: A data quality management methodology for data sensors \n\n6.1 The \"plan\" phase \n\n6.1.1 P1. Characterization of the current state of the sensor data contexts \n6.1.2 P2. Assessment of the current state of the levels of data quality \n6.1.3 P3. Data quality enhancement plan definition \n\n\n6.2 The \"do\" phase \n\n6.2.1 D1. Execution of the data quality improvement plan \n\n\n6.3 The \"check\" phase \n\n6.3.1 C1. Testing the efficiency of the corrective actions \n6.3.2 C2. Review of data quality issues \n\n\n6.4 The \"act\" phase \n\n6.4.1 A1. Data quality issues prevention \n\n\n\n\n7 Conclusions \n8 Acknowledgements \n\n8.1 Author contributions \n8.2 Funding \n8.3 Conflict of interest \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nThe internet of things (IoT) introduces several technical and managerial challenges when it comes to the use of data generated and exchanged by and between various smart, connected products (SCPs) that are part of an IoT system (i.e., physical, intelligent devices with sensors and actuators). Added to the volume and the heterogeneous exchange and consumption of data, it is paramount to assure that data quality levels are maintained in every step of the data chain\/lifecycle. Otherwise, the system may fail to meet its expected function. While data quality (DQ) is a mature field, existing solutions are highly heterogeneous. Therefore, we propose that companies, developers, and vendors should align their data quality management mechanisms and artifacts with well-known best practices and standards, as for example, those provided by ISO 8000-61. This standard enables a process-approach to data quality management, overcoming the difficulties of isolated data quality activities. This paper introduces DAQUA-MASS, a methodology based on ISO 8000-61 for data quality management in sensor networks. The methodology consists of four steps according to the Plan-Do-Check-Act cycle by Deming.\nKeywords: data quality; data quality management processes; ISO 8000-61; data quality in sensors; internet of things; IoT; smart, connected products; SCPs\n\nIntroduction \n\u201cOur economy, society, and survival aren\u2019t based on ideas or information\u2014they\u2019re based on things.\u201d[1] This is one of the core foundations of the internet of things (IoT) as stated by Ashton, who coined the term. IoT is an emerging global internet-based information architecture facilitating the exchange of goods and services.[2] IoT systems are inherently built on data gathered from heterogeneous sources in which the volume, variety, and velocity of data generation, exchanging and processing are dramatically increasing.[3] Furthermore, there is a certain emergence of IoT semantic-oriented vision which needs ways to represent and manipulate the vast amount of raw data expected to be generated from and exchanged between the \u201cthings.\u201d[4]\nThe vast amount of data in IoT environments, gathered from a global-scale deployment of smart-things, is the basis for making intelligent decisions and providing better services (e.g., smart mobility, as presented by Zhang et al.[5]). In other words, data represents the bridge that connects cyber and physical worlds. Despite of its tremendous relevance, if data are of inadequate quality, decisions from both humans and other devices are likely to be unsound.[6][7] As a consequence, data quality (DQ) has become one of the key aspects in IoT.[6][8][9][10] IoT devices, and in particular smart, connected products (SCPs), have concrete characteristics that favor the apparition of problems due to inadequate levels of data quality. M\u00fchlh\u00e4user[11] defines SCPs as \u201centities (tangible object, software, or service) designed and made for self-organized embedding into different (smart) environments in the course of its lifecycle, providing improved simplicity and openness through improved connections.\u201d While some of the SCP-related characteristics might be considered omnipresent (i.e., uncertain, erroneous, noisy, distributed, and voluminous), other characteristics are more specific and highly dependent on the context and monitored phenomena (i.e., smooth variation, continuous, correlation, periodicity, or Markovian behavior).[6]\nAlso, outside of the IoT research area, DQ has been broadly studied during last years, and it has become a mature research area capturing the growing interest of the industry due to the different types of values that companies can extract from data.[12] This fact is reflected by the standardization efforts like ISO\/IEC 25000 series addressing systems and software quality requirements and evaluation (SQuaRE)[13] processes, and specific techniques for managing data concerns. We pose that such standards can be tailored and used within the IoT context, not only bring benefits standardizing solutions and enabling a better communication between partners. Also, the number of problems and system fails on the IoT environment is reduced, better decisions can be taken due to a better quality of data, all stakeholders are aligned and can take benefit of the advances on the standard used, and it is easier to apply data quality solutions in a global way because the heterogeneity is reduced.\nDue to the youth of IoT, and despite DQ standards, frameworks, management techniques, and tools proposed in the literature, DQ for IoT has not been yet widely studied. However, and prior to this research line, it is possible to cite some works that had addressed some DQ concerns in sensor wireless networks[8][14], or in data streaming[15][16] among other proposals.[6] However, these works have not considered the management of DQ in a holistic way in line with existing DQ-related standards. In our attempt to align the study of DQ in IoT to international standards, this paper provides practitioners and researchers with DAQUA-MASS, a methodology for managing data quality in SCP environments, which considers some of the DQ best practices for improving quality of data in SCP environments aligned to ISO 8000-61.[17] Due to the intrinsic distributed nature of IoT systems, using such standards will enable the various organizations to be aligned to the same foundations, and in the end, to work in a seamless way, what will undoubtedly improve the performance of the business processes.\nThe remainder of this paper is organized as follows: the next section presents the most challenging data quality management concerns in the context of the SCP environments; afterwards. related work is explored. Then the data quality model in which our methodology is based on is presented. The last two sections propose a methodology for managing data quality in SCP environments and discuss conclusions and implications of this work.\n\nData quality challenges in SCP environments \nThis section introduces some general ideas about SCPs and their operation as an essential part of IoT. In addition, some challenges related to DQ in SCP environments are also introduced.\nAccording to Cook and Das[18], a smart environment is a small world where all kinds of smart devices are continuously working to make inhabitants\u2019 lives more comfortable. According to M\u00fchlh\u00e4user[11], SCP provides intelligent actions through improved connections by means of context-awareness, semantic self-description, proactive behavior, multimodal natural interfaces, AI planning, and machine learning.\nSCPs have three main core components: physical, smart, and connectivity components. Smart components extend the capabilities and value of the physical components, while connectivity extends the capabilities and value of the smart components. This enables some smart components to exist outside the physical product itself, with a cycle of value improvement.[19]\nIoT and SCP can be confused in some contexts. However, IoT simply reflects the growing number of SCPs and highlights the new opportunities they can represent. IoT, which can involve people or things, is a means for interchanging information. What makes SCPs essentially different is not the \"internet,\" but the changing nature of the \u201cthings.\u201d[19] A product that is smart and connected to the cloud could become part of an interconnected management solution, and companies can therefore evolve from making products to offering more complex, higher-value services within a \u201csystem of systems.\u201d[20]\nSCPs include processors, sensors, software, and connectivity that allow data to be exchanged between the product and its environment. The data collected by sensors of these SCPs can be then analysed to inform decision-making, enable operational efficiencies, and continuously improve the performance of the product. This paper focuses on the data produced by such sensors, and how inadequate levels of data quality may affect the processing of the data, while smart and connectivity parts of SCPs are outside of the scope of this paper.\nSCPs can be connected in large, complex networks throughout three different layers[9]: acquisition, processing, and utilization layers (see Figure 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. Layers in SCP environments.\n\n\n\n The acquisition layer refers to the sensor data collection system where sensors, raw (or sensed) data, and pre-processed data are managed. This is the main focus of this paper.\n The processing layer involves data resulting from the data processing and management center, where energy, storage, and analysis capabilities are more significant.\n The utilization layer concerns delivered data (or post-processed data) exploited, for example, over a geographic information system (GIS) or combined with other services or applications.\nAs previously stated, the scope of the paper is limited to the data produced by SCPs\u2019 sensors. Hence, the proposal is mainly intended to be applied in the context of the acquisition layer. Nevertheless, the management of data quality in sensors can impact on how data is processed (processing layer) and how data may be used later (utilization layer).\nNetworking and management of SCP operations can generate the business intelligence needed to deliver smart services. Smart services are delivered to or via smart objects that feature awareness and connectivity.[21] SCP can carry out the following functions to support smart services[22]: status, diagnostics, upgrades, control and automation, profiling and behavior tracking, replenishment and commerce, location mapping. and logistics, among others.\nSCP operations enable new capabilities for companies, although new problems and challenges that arise must also be taken into account. On one hand, SCP operations require companies to build and support an entirely new technology infrastructure.[19] Technological layers in the new technology landscape include new product hardware, embedded software, connectivity, a product cloud running on remote servers, security tools, gateway for external information sources, and integration with enterprise business systems. On the other hand, SCP operations can provide competitive advantages, which are based on the operational effectiveness. Operation effectiveness requires the embrace of best practices along the value chain, including up-to-date product technologies, the latest production equipment, and state-of-the-art sales force methods, IT solutions, and so forth. Thus, SCP operations also creates new best practices across the value chain.[19]\nAccording to the different sources of data in these SCP environments, we can distinguish different types of aggregated data:\n\n Sensor data: data that is generated by sensors and digitized in a computer-readable format (for example, camera sensor readings)\n Device data: integrated by sensor data; observed metadata (metadata that characterizes the sensor data, e.g., timestamp of sensor data); and device meta data (metadata that characterizes the device, e.g., device model, sensor model, manufacturer, etc.), so device data, for example, can be data coming from the camera (device)\n General data: data related to\/or coming from devices which has been modified or computed to derive different data plus business data (i.e., data for business use such as operation, maintenance, service, customers, etc.)\n IoT data: general data plus device data\nA reduction in the levels of quality of these data due to different problems in SCP operations can threaten the success factors of SCP environments.[6] The quality of produced data is often affected by dysfunctional SCP devices and sensors, which are the sources providing data, and can potentially result in inadequate levels of quality that are only detected later on, when data are being processed and used. Therefore, while we can identify dysfunctional SCP devices through the analysis of sensor data by using data quality management techniques, it is noteworthy that these devices will impact the rest of the sensor network. According to[6], Table 1 summarizes some of these SCP factors that, in some cases, could condition or lead to data quality issues. In addition, the three columns on the right of Table 1 show (marked with a cross) the most critical layers affected in a greater extent by every SCP factor.\n\n\n\n\n\n\n\nTable 1. SCP factors that can finally affect the levels of DQ according to Karkouch et al.[6]\n\n\n\nSCP Factor\n\nSide Effect in Data Quality\n\nAcquisition\n\nProcessing\n\nUtilization\n\n\nDeployment scale\n\nSCPs are expected to be deployed on a global scale. This leads to a huge heterogeneity in data sources (not only computers but also daily objects). Also, the huge number of devices accumulates the chance of error occurrence.\n\nX\n\nX\n\n\n\n\nResource constraints\n\nFor example, computational and storage capabilities that do not allow complex operations due, in turn, to the battery-power constraints among others.\n\nX\n\nX\n\n\n\n\nNetwork\n\nIntermittent loss of connection in the IoT is recurrent. Things are only capable of transmitting small-sized messages due to their scarce resources.\n\n\n\nX\n\nX\n\n\nSensors\n\nEmbedded sensors may lack precision or suffer from loss of calibration or even low accuracy. Faulty sensors may also result in inconsistencies in data sensing.\n\nX\n\n\n\n\n\n\nEnvironment\n\nSCP devices will not be deployed only in tolerant and less aggressive environments. To monitor some phenomenon, sensors may be deployed in environments with extreme conditions. Data errors emerge when the sensor experiences the surrounding environment influences.[23]\n\nX\n\n\n\nX\n\n\nVandalism\n\nThings are generally defenseless from outside physical threats (both from humans and animals).\n\nX\n\n\n\nX\n\n\nFail-dirty\n\nA sensor node fails, but it keeps up reporting readings which are erroneous. It is a common problem for SCP networks and an important source of outlier readings.\n\nX\n\nX\n\n\n\n\nPrivacy\n\nPrivacy preservation processing, thus DQ could be intentionally reduced.\n\n\n\n\n\nX\n\n\nSecurity vulnerability\n\nSensor devices are vulnerable to attack, e.g., it is possible for a malicious entity to alter data in an SCP device.\n\nX\n\n\n\nX\n\n\nData stream processing\n\nData gathered by smart things are sent in the form of streams to the back-end pervasive applications which make use of them. Some stream processing operators could affect quality of the underlying data.[10] Other important factors are data granularity and variety.[24] Granularity concerns interpolation and spatio-temporal density while variety refers to interoperability and dynamic semantics.\n\nX\n\nX\n\n\n\n\n\nTilak et al.[23] provide a taxonomy of sensor errors. These errors are directly related to different data quality problems in the acquisition layer. The mentioned taxonomy distinguishes the following six types of data sensors errors (see Table 2). Apart from errors in isolated SCP devices, there are other communication errors which can happen at the SCP network level.[23] Table 3 summarizes the main types of communication errors: omission, crashes, delay, and message corruption. The table shows the DQ issue derived by each problem, the root cause, and possible solution.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 2. Sensors errors deriving DQ problems in SCP environments (adapted from Jesus et al.[8])\n\n\n\n\n\n\n\n\n\nTable 3. SCP Network Errors\n\n\n\nSensor fault\n\nDQ problem\n\nRoot cause\n\nSolution\n\n\nOmission faults\n\nAbsence of values\n\nMissing sensor\n\nNetwork reliability, retransmission\n\n\nCrash faults (fading\/intermittent)\n\nInaccuracy\/absence of values\n\nEnvironment interference\n\nRedundancy\/estimating with past values\n\n\nDelay faults\n\nInaccuracy\n\nTime domain\n\nTimeline solutions\n\n\nMessage corruption\n\nIntegrity\n\nCommunication\n\nIntegrity validation\n\n\n\nAll the mentioned SCP devices\/sensor errors will lead to different DQ problems in the three layers depicted in Figure 1. As previously mentioned, DQ problems can be represented as a degradation of some DQ characteristics that are especially important in different environments. Let us consider two groups of data quality characteristics:\n\n DQ characteristics to assess data quality in use within a specific context. This aspect considers selected criteria to estimate the quality of raw sensor data at the acquisition and processing layer. There are some DQ characteristics considered, which make it possible to estimate the quality on data sources, their context of acquisition, and their transmission to the data management and processing. These DQ characteristics are accuracy and completeness according to ISO\/IEC 25012[25] and reliability and communication reliability as proposed by Rodriguez and Servigne.[9] It is also related to the utilization layer and includes availability regarding ISO\/IEC 25012[25] plus timeliness and adequacy as defined by Rodriguez and Servigne.[9]\n DQ Characteristics aimed at managing internal data quality. The main goal of managing internal data quality is to avoid inconsistent data and maintain the temporality of sensor data at the processing layer. These characteristics are consistency and currency according to ISO\/IEC 25012[25] and volatility as proposed by Rodriguez and Servigne.[9]\nRelated work \nThe goal of this section is twofold. First, works related to the study of data quality in sensor networks and SCP environments in general are presented. Second, data quality methodologies are introduced and compared in order to draw the main contribution of the proposed methodology.\n\nSensor data quality \nThere are some published works in the literature that address the concerns related to data quality management in SCP and IoT environments. Karkouch et al.[6] presented a state-of-the-art survey for DQ in IoT. This survey presents IoT-related factors endangering the DQ and their impact on various DQ characteristics. Also, DQ problem manifestations are discussed (and their symptoms identified) as well as their impact in the context of IoT. Jesus et al.[8] provided a similar survey addressing the problem of not being able to ensure desired DQ levels for dependable monitoring when using wireless sensor networks. This work pays special attention to comprehension of which faults can affect sensors, how they can affect the quality of the information, and how this quality can be improved and quantified. Rodriguez and Servigne[9] also analysed data errors in sensor networks, in particular in environmental monitoring systems. In this paper, authors address the problem of uncertainty of data coming from sensors with an approach dedicated to providing environmental monitoring applications and users with data quality information. Badawy et al.[26] combined parametric and non-parametric signal processing and machine learning algorithms for automating sensor data quality control, which can identify those parts of the sensor data that are sufficiently reliable for further analysis and discards useless data.\nAnother research subarea of DQ in sensor networks is the DQ management in sensor data streams. Klein et al.[10] presented a quality-driven load shedding approach that screens the data stream to find and discard data items of minor quality. Thus, DQ of stream processing results is maximized under adverse conditions such as data overload. Campbell et al.[15] advocated for automated quality assurance and quality control procedures based on graphical and statistical summaries for review and track the provenance of the data in environmental sensor streams.\nOther works focus on data management from different viewpoints. For example, Al-Ruithe et al.[27] detailed roles, responsibilities, and policies in the context of IoT-cloud converged environments and provide a generic framework for data governance and security. Similarly, Qin et al.[14] provided a data management perspective on large-scale sensor environment applications posing non-functional requirements to meet the underlying timeliness, reliability, and accuracy needs in addition to the functional needs of data collection. Although all these approaches are interesting and provide a useful vision, they still do not address how to make available (e.g., institutionalize) best practices in data quality management to the entire organization. Such an approach has been proven to be more efficient when it comes to creating an organizational data quality culture. This vision is specifically important in the case of IoT, since the SCP operations can be executed across different networks belonging to different departments or even organizations. From our point of view, this is a critical aspect for IoT that must be covered in a holistic way.\n\nData quality methodologies comparison \nThere are some methodologies that can be used as drivers for assessing and managing DQ. First, Lee et al.[28] proposed AIMQ as a methodology that encompasses a model of data quality, a questionnaire to measure DQ, and analysis techniques for interpreting the DQ measures. This methodology is mainly used to analyse the gap between an organization and best practices, as well as to assess gaps between information systems professionals and data consumers. The application of this methodology is useful for determining the best area for DQ improvement activities. This methodology has not been widely used and in a greater extent it has been considered to be too theoretical and dependent on the domain. McGilvray[29] provides a practical approach for planning and managing information quality. In comparison with the methodology proposed by Lee et al., McGilvray provides a more pragmatic and practical approach to achieving the desired state of DQ within an organization. However, this methodology is still dependent on the domain of application. ISO\/TS 8000-150:2011[30] \u201cspecifies fundamental principles of master data quality management, and requirements for implementation, data exchange and provenance.\u201d This standard constitutes an informative framework that identifies processes for DQ management. This framework could be used in conjunction with, or independently of, quality management systems standards, for example, ISO 9001.[31]\nBatini et al.[32] provided a literature review about different methodologies for data quality assessment and improvement. Most of the methods and techniques included in such review cannot be considered as DQ management methodologies since they do not consider all the managerial concerns in a holistic manner. On the contrary, most of these methods are focused on DQ assessment or improvement in isolation. Similar to the mentioned review, a more recent study developed by Woodall et al.[33] classified most recent DQ assessment and improvement methods. This work suffers the same problem that the work of Batini et al. does. Apart from these examples, there is a lack of comprehensive methodologies for the assessment and improvement of DQ in the domain of SCP operations and their underlaying data.\nThe recent standard ISO 8000-61[17] provides a set of standard guidelines for managing DQ in a holistic way, which can be tailored for different domains. However, its main purpose is not to serve as a methodology for DQ management per se, but it simply provides a process reference model. In this sense, the standard is more descriptive than operative, what makes it not usable out-of-the-box. Aligned with this standard, this paper proposes the DAQUA-MASS methodology to deal with DQ in SCP environments. The main contribution of the DAQUA-MASS methodology is that it takes the standard best practices for depicting an operative way to manage DQ (as depicted in the processes of ISO 8000-61) and tailors these to the particular domain of SCP environments, and in particular, in sensor-related data.\n\nDAQUA-Model: A data quality model \nReviewing the literature, it is possible to find that the concept of \"data quality\" has been defined in different ways. The widest used definitions are aligned with the concept of \u201cfitness for use.\u201d Strong et al.[34] define data quality as \"data that is fit for use by data consumer. This means that usefulness and usability are important aspects of quality\u201d. Different stakeholders can have different perceptions of what quality means for the same data.[35] It largely depends on the context in which data is used. Thus, DQ in IoT environments must be adequately managed considering the very nature of the IoT systems. Typically, to improve data quality, a Plan-Do-Check-Act (PDCA) cycle specifically tailored for the context of usage is followed. In this sense, we think that adopting the processes of ISO 8000-61 which are deployed in the PDCA[36] order can largely help to provide a methodology for managing data quality in IoT environments. At the core of the PDCA cycle for IoT environments is the identification of a Data Quality Model (DQModel) which, being composed of several data quality characteristics suitable for the problem, is used to identify and represent the data quality requirements required in the context.[9]\nIn our case, and according to our philosophy of aligning with international standards, the DQ Model proposed is a specialization of the DQ Model introduced in ISO\/IEC 25012.[25] This DQ Model is widely accepted and used in the industry. Nevertheless, it has not been specifically developed for considering SCP aspects. In fact, the scope section of such standard literally states that it \u201cdoes not include data produced by embedded devices or real time sensors that are not retained for further processing or historical purposes.\u201d[25] Therefore, in order to complement the standard, we provide some orientation in this paper on how to specifically use ISO\/IEC 25012 in the context of SCP environments.\nThe DQ Model focuses on the quality of the data as part of an information system and defines quality characteristics for target data used by humans and systems (i.e., the data that the organization decides to analyse and validate through the model). This model categorizes quality attributes into 15 characteristics and considers three perspectives: inherent, system-dependent, and both jointly (see crosses in Table 4).\n\r\n\n\n\n\n\n\n\n\n\n\n Table 4. DQ Characteristics in ISO\/IEC 25012 that can be affected by sensor data errors\n\n\n\nAs part of the methodology, we propose the association of the DQ characteristics with some of the sensor data errors previously shown in Table 2. These relationships suggest that erroneous data generated due to some malfunctioning sensors could lead or affect some of the DQ characteristics. We distinguish these relationships in Table 4 with \u2018P\u2019 and \u2018S\u2019 respectively signifying that a sensor error can have a primary or secondary impact in DQ characteristics. Primary means a direct (or more critical) effect, while secondary is a collide (or less critical) impact.\nIn the following paragraphs, we introduce and summarize the fifteen DQ characteristics and, as a main contribution in this part, we provide a vision on how these DQ characteristics are tailored for SCPs operations. Along with the definitions, for the sake of understandability an example is also provided for each characteristic. These examples are specially provided for the interpretation of these DQ characteristics in the acquisition layer, which is the scope of this paper. However, it must be noted that all these DQ characteristics can be considered for all the SCP layers with different ways of assessment and interpretation.\nFrom the point of view of data quality management, it is quite important not to make the mistake of confusing the readings of dysfunctional sensors with inadequate levels of data quality, even when a dysfunctional sensor can produce data without having adequate levels of quality (i.e., not fitting the purpose of the use of data): the reason for making this distinction is that fixing errors due to dysfunctional sensors requires first fixing the sensors; on the other hand, if one can assure that the root cause is not grounded on a dysfunctional sensor, but on the data itself, then, to fix data quality errors, then data quality management techniques should be used since it should not be ignored what data means. The description of the data quality characteristics can be found in the following definitions:\n\nAccuracy \nThe degree to which data has attributes that correctly represent the true value of the intended attribute of a concept or event in a specific context of use - In SCP environments, a low degree of accuracy could be derived from devices that provide values that could differ from the value on the real world. For example, a low degree of accuracy can be when a humidity sensor reads a value of 30% and the real value is 50%. Low levels of accuracy could be directly related to sensor errors such as constant or offset, outlier errors and noise errors. Also, accuracy could be indirectly affected by continuous varying or drifting and trimming error (see Table 4).\nCompleteness \nThe degree to which subject data associated with an entity has values for all expected attributes and related entity instances in a specific context of use - In SCP environments, a low degree of completeness could be derived from devices that are reading and sending no values. For example, a low degree of completeness can happen when the records of sensor data have missing values. Low levels of completeness could be related directly to sensor errors such as crash or jammed errors and indirectly, trimming and noise errors (see Table 4).\nConsistency \nThe degree to which data has attributes that are free from contradiction and are coherent with other data in a specific context of use: either or both among data regarding one entity and across similar data for comparable entities - In SCP environments, a low degree of consistency could happen when two sensors produce contradictory data. For example, for proximity sensors that provide the relative distance to the sensor position, a consistency problem for a single sensor could be a negative distance value, while a consistency problem between two sensors in the same position could be two different values. Thus, low levels of consistency could be related with continuous varying\/drifting error and indirectly with constant or offset errors, trimming and noise error (see Table 4).\nCredibility \nThe degree to which data has attributes that are regarded as true and believable by users in a specific context of use - In SCP environments, a low degree of credibility could be derived from a single sensor placed somewhere and the data cannot be validated by another entity or even another sensor. For example, a credibility issue could happen when a sensor whose data is compared with another sensor placed nearby does not match. Low levels of credibility could be related directly to sensor errors such as outlier errors and indirectly with constant or offset error, continuous varying\/drifting error, and noise error (see Table 4).\nCurrentness \nThe degree to which data has attributes that are of the right age in a specific context of use - In SCP environments, a low degree of currentness could be derived from a sensor that can be indicating past values as current value (see Table 4). For example, if an irrigation sensor produces a value that indicates that the field must be irrigated, it has been irrigated and the data is not updated. The data indicates whether it is necessary irrigation, or it is already irrigated, so this would be data without sufficient currentness.\nAccessibility \nThe degree to which data can be accessed in a specific context of use, particularly by people who need supporting technology or special configuration because of some disability - In SCP environments, a low degree of accessibility could be derived due to the fact the necessary user is not allowed in at the precise moment. For example, data produced by a specific sensor is unreachable due to network issues.\nCompliance \nThe degree to which data has attributes that adhere to standards, conventions, or regulations in force and similar rules relating to data quality in a specific context of use - In SCP environments, a low degree of compliance could be derived from a data sensor that is not using the standard formats established by the organization. For example, if the organization establishes that for distance sensors the unit for values is meters, and if some sensors produce values expressed in meters and other in miles, these data have low compliance levels.\nConfidentiality \nThe degree to which data has attributes that ensure that it is only accessible and interpretable by authorized users in a specific context of use - In SCP environments, a low degree of confidentiality could be derived from inefficient security management of sensor data. For example, a confidentiality leak might happen when data produced by a sensor placed in a nuclear power plant can be freely accessed from external networks even when this data was marked as sensible and, therefore, confidential in order to prevent possible terrorist acts.\nEfficiency \nThe degree to which data has attributes that can be processed and provide the expected levels of performance by using the appropriate amounts and types of resources in a specific context of use - For example, a sensor send data about where is placed and send a code and a description every time the sensor sends o stores a record, it has low efficiency because only the code is enough to know all about the place. In SCP environments, a low degree of efficiency could be derived from the storage of duplicated data that could take more time and resources to send or manipulate the data.\nPrecision \nThe degree to which data has attributes that are exact or that provide discrimination in a specific context of use - In SCP environments, a low degree of precision could be derived from devices that are providing inexact values as in the next example. For example, sensor data that store weight with no decimals and it is required a minimum of three decimals. Low levels of consistency could be related directly with trimming errors, and indirectly with noise errors (see Table 4).\nTraceability \nThe degree to which data has attributes that provide an audit trail of access to the data and of any changes made to the data in a specific context of use - In SCP environments, a low degree of traceability could be derived from sensor data with no metadata. For example, data logs contain information about who has acceded to sensor data and operations made with them. Low levels of traceability could be related indirectly to crash or jammed errors as well as to temporal delay errors (see Table 4).\nUnderstandability \nThe degree to which data has attributes that enable it to be read and interpreted by users, and are expressed in appropriate languages, symbols, and units in a specific context of use - In SCP environments, a low degree of understandability could be derived from sensor data represented with codes instead of acronyms. For example, records of data about temperature on a car has an attribute to know the place of the sensor in the car. This attribute can be stored as a code like \u201cxkq1,\u201d but if it is stored as \u201cGasolineTank,\u201d it is supposed to have a higher level of understandability.\nAvailability \nThe degree to which data has attributes that enable it to be retrieved by authorized users and\/or applications in a specific context of use - In SCP environments, a low degree of availability could be derived from the insufficient resources of the system in which sensor data is stored. For example, to assure sensor data availability, sensor replication can be used to make it available even if there is some issue on a sensor. Low levels of availability could be related indirectly with temporal delay error and crash or jammed errors (see Table 4).\nPortability \nThe degree to which data has attributes that enable it to be installed, replaced, or moved from one system to another preserving the existing quality in a specific context of use - For example, sensor data is going to be shared with a concrete system or even other organization or department, data loss can occur. If this happens, for example, due to a data model mismatching or a problem with the data format, the reason is directly related to portability of data. In SCP environments, a low degree of portability could be derived from sensor data that does not follow a specific data model (see Table 4) or the format present some problems.\nRecoverability \nThe degree to which data has attributes that enable it to maintain and preserve a specified level of operations and quality, even in the event of failure, in a specific context of use. In SCP environments, a low degree of recoverability could be derived from devices that do not have a mechanism failure tolerant or backup. For example, when a device has a failure, data stored in that device should be recoverable. Low levels of recoverability could be related indirectly with temporal delay error and crash or jammed errors (see Table 4).\nAlthough our DQ Model considers initially all the DQ characteristics defined in ISO\/IEC 25012, it could be necessary to customize the DQ characteristics chosen to adapt them into the specific SCP context. This customization might depend on the concrete organization and how it applies the methodology to specific SCP contexts. The customized model will conform the data quality model for an organization with a specific SCP environment.\n\nDAQUA-MASS: A data quality management methodology for data sensors \nThis section presents DAQUA-MASS, an ISO 8000-61-based Data Quality Management methodology for data sensors. Steps provided in this methodology are based in some of the processes introduced by ISO 8000-61[17], which, as we previously said, gathers the best practices around data quality management by means of a process approach. Each process in this standard is described by means of a purpose, the outcomes, and the activities that are to be applied for the assurance of data quality. Thus, this standard mainly covers: (1) fundamental principles of data quality management; (2) the structure of the data quality management process; (3) definitions of the lower level processes for data quality management; (4) the relationship between data quality management and data governance; and (5) implementation requirements. This standard is used along with ISO 8000-62 to assess and improve organizational maturity when it comes to data quality management. However, in this paper, we consider the processes in ISO 8000-61 to rigorously depict DAQUA-MASS.\nSteps of DAQUA-MASS are grouped in four phases according to the Plan-Do-Check-Act cycle, as it is done in 8000-61. PDCA is implicitly a closed loop, signifying that the process is iterative, with the last phase of every iteration providing feedback on starting a new iteration. Many acknowledged models, such as the in IoT information fusion[37] or the JDL model[38], are all closed loops. As such, they can be self-improved to adjust the dynamic world to maximize the performance. The IoT is a dynamic system, and data quality management can be adapted to the changes in every loop of the methodology. Each loop serves to adapt to changes and new quality needs that may arise. The methodology is designed so that the complete cycle is iteratively executed depending on the goals, needs, and resources of the organization. The PDCA cycle will contribute to more effective and efficient data quality and consists of seven steps grouped in the following four phases:\n\n The \"plan\" phase establishes the strategy and the data quality improvement implementation plan as necessary to deliver results in accordance with data requirements;\n The \"do\" phase executes the data quality improvement implementation plan;\n The \"check\" phase monitors data quality and process performance against the strategy and data requirements and reports the results to validate the efficiency of the corrective actions; and finally\n The \"act\" phase takes actions to continually improve process performance.\nFigure 2 summarizes the step flow throughout the four phases. Every step is defined with a general description and set of concrete activities. Following sections present all the methodology steps grouped in the mentioned Plan-Do-Check-Act cycle. The definition for all the steps provides a table depicting all the expected input and output generated by each activity in the step as well as a RACIQ matrix (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer) which indicates the role of stakeholders involved in the activity.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2. Methodology DAQUA-MASS phases and steps\n\n\n\nWe consider the following list of stakeholders to be involved in the application of this methodology. It should be noted that there are alternative, similar (even overlapped) role definitions in the context of data quality management and SCP environments. However, we consider these to be the most common and well-known stakeholders, and other that are not mentioned can be easily assimilated to one of the proposed.\n\n Chief information officer (CIO): The most senior executive in an enterprise responsible for the traditional information technology and computer systems that support enterprise goals (executive level individual)\n Chief data officer (CDO): A corporate officer responsible for enterprise wide data governance and utilization of data as an asset, via data processing, analysis, data mining, information trading, and other means (executive level individual)\n Data governance manager (DGM): Oversees the enterprise data governance program development and is responsible for developing certain solutions and frameworks (strategic level individual)\n Data quality steward for SCP domain (SCP DQ steward): A DQ steward at the tactical level for the area of SCP environments by considering its implications in DQ\n Data quality steward (DQ Ssteward): Responsible for utilizing an organization\u2019s data quality governance processes to ensure fitness of data elements, including both the content and metadata (operational level individual) - Data stewards have a specialist role that incorporates processes, policies, guidelines, and responsibilities for administering organizations\u2019 entire data in compliance with DQ policies and procedures\n SCP technical architect (SCP arch): An SCP (and in general IoT) expert who makes high-level design choices and dictates technical standards, including SCP technology standards, tools, and platforms\nThe \"plan\" phase \nThe data quality planning phase establishes data requirements and objectives for data quality, creating plans to achieve the objectives and evaluating the performance of the plans. These plans balance current data quality levels, cost, resources, and capabilities across the organization for the assessment of data quality of sensor data. This phase is initiated based on needs and expectations of stakeholders or the feedback of the process improvements performed during data quality improvement (the act phase). The expected result of the execution of this phase is the complete planification for adequately collect sensor data. This includes requirements management, definition of policies for data, and device lifecycle management; it is important to highlight the definition of policies to retain the original unmanipulated data and all versions of the input data. Table 5 summarizes inputs, outputs, and responsibility distribution for this phase. Afterwards, a description of the steps grounded on the corresponding ISO 8000-61 process is introduced. Again, keep in mind that the scope of the methodology is on sensor data (see Figure 1).\n\r\n\n\n\n\n\n\n\n\n\n\n Table 5. Inputs, outputs, and RACIQ matrix for the Plan phase (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)\n\n\n\nP1. Characterization of the current state of the sensor data contexts \nThis first step is devoted to characterizing the current state of the SCP environments and all the specific sensor data contexts. This step has the following activities:\n\nP1-1. Sensor and sensor data lifecycle management specification - The lifecycle of sensors[2] and sensor data[39] have to be defined so that it can be managed. For sensors, the lifecycle starts when the device is obtained, and the last step is when it is necessary to replace o remove the sensor because it became useless. For sensor data, the lifecycle starts when sensors produce the data and ends when the data is eliminated. The end of the sensor data life cycle should not be confused with the moment in which the data goes from being operational to historical data. Operational sensor data is used on a day-to-day basis, but when data is no longer useful on a day-to-day basis due to its age, these data are separated and stored in another database and become historical data. It is important to highlight that sensor and sensor data lifecycle is used to better contextualize the environment of the data.\nP1-2. Management of quality policies, standards, and procedures for data quality management - Specify fundamental intentions and rules for data quality management. Ensure the data quality policies, standards, and procedures are appropriate for the data quality strategy, comply with data requirements, and establish the foundation for continual improvement of the effectiveness and efficiency of data quality management in all the key SCP operations. The purpose of data quality policy\/standards\/procedures management is to capture rules that apply to performing data quality control, data quality assurance, data quality improvement, data-related support, and resource provision consistently. Before the implementation and definition of the complete plan, the policies, standards, and procedures should be defined in order to define the implementation plan based on them.\nP1-3. Provision of sensor data and work specifications - Developing specifications that describe characteristics of data and work instructions for smart connected products enables data processing and data quality monitoring and control. To support the description of the provision of the sensor data work specifications, some metadata must be provided. Metadata can help with one of the biggest problems of SCP: interoperability. Interoperability refers to the ability for one or more smart connected products to communicate and exchange information.[15]\nP1-4. Identification, prioritization, and validation of sensor data requirements - Collect the needs and expectations related to sensor data from devices and stakeholders. Then, it is translated by identification, prioritization, and validation of data requirements. The purpose of requirements management is to establish the basis for creating or for refining a data quality strategy for SCP environments aligned to the needs and expectations of stakeholders. It is important to have well-defined and -implemented quality sensor data requirements to avoid problems from the start and to facilitate the collection and integration of sensor data.\nP2. Assessment of the current state of the levels of data quality \nThis step aims to evaluate the current state of the data quality levels in all the surrounding SCP environments. This step has the following activities:\n\nP2-1. Identification of the data quality characteristics representing quality requirements and determination and development of metrics and measurement methods - Develop or select the measurement indicators, corresponding metrics, and measurement methods used to measure the quality levels of data produced by devices during all the SCP operations.\nP2-2. Measurement and analysis of data quality levels - Measure the data quality levels by implementing the measurement plans and determining the measurement results. This means the application of the measurement algorithms defined and developed on the previous step (P2-1). Such algorithms strongly depend on the IT infrastructure landscape of every organization. Thereby, every organization often develop their own algorithms or use commercial solutions which are compliant with their own infrastructure. Merino et al.[7] presents algorithms and metrics for each data quality characteristic. After data quality levels have been measured, these can be quantitatively analysed to extract insights about the SCP environment being managed. As a result, a list of non-conformities can be elaborated in the next step, with which to make informed decisions.\nP3. Data quality enhancement plan definition \nData quality implementation planning identifies the resources and sequencing by which to perform DQ control, DQ Assurance, DQ Improvement, data-related support, and resource provision across the organization. At this point, after the collection of sensor data requirements and the definition of standards and policies, it is important to configure the devices to implement the plan and place each node following a defined strategy. Optimal node placement allows to obtain the optimal value through data. This step considers the following activities:\n\nP3-1. Analysis of root causes of data non-conformities - Analyzing the root causes of each data quality issue and assess the effect of the issue on business processes. The purpose of root cause analysis and solution development of non-solved data quality non-conformities is to establish, in accordance with the data quality strategy and with the priorities identified by data quality assurance, the basis on which to perform data cleansing and\/or process improvement for data non-conformity prevention.\nP3-2. Data quality risk assessment - Identify risks throughout the data life cycle, analyze the impact if each risk were to occur, and determine risk priorities to establish the basis for monitoring and control of processes and data. The purpose of data quality monitoring and control is, by following applicable work instructions, to identify and respond when data processing fails to deliver data that meet the requirements in the corresponding data specification. It allows for the control of what is happening on data or even on the SCP environment. For example, with the control of sensor data quality, it is possible to identify certain issues on devices. Data is often the most important thing to protect, because although the direct cost of losing it may be small compared with research data or intellectual property. If a sensor is not functioning any more for some reason, it can be replaced, but if a data loss incident occurs, it is difficult to recover from it or even impossible to recover. This can bring not only data loss but also be the root cause of other problems. In conclusion, data quality risks should be identified in order to avoid them. As Karkouch et al. note[6], the main factors affecting DQ in SCP or IoT environments are: deployment scale, resources constraints, network, sensors (as physical devices), environment, vandalism, fail-dirty, privacy preservation, security vulnerability, and data stream processing.\nP3-3. Development of improvement solutions to eliminate the root causes - Propose solutions to eliminate the root causes and prevent recurrence of non-conformities. Evaluate the feasibility of the proposed improvements through cost benefit analysis.\nP3-4. Definition of improvement targets - The purpose of this activity is to analyze possible improvement areas according to the business processes, risk catalog, and the data quality strategy; then it selects those that are more aligned with the data quality strategy and\/or are able to lead to greater data quality enhancement regarding previous detected risks. Specific areas or sub-nets of devices in the organization\u2019s SCP environments could also serve as a criterion to determine specific improvement targets.\nP3-5. Establishment of the data quality enhancement plan - Define the scope and target of data quality and prepare detailed implementation plans, defining and allocating the resources needed.\nThe \"do\" phase \nThis phase consists of the data quality control that is carried out based on the implementation plan established in data quality planning (see the \"plan\" phase). The phase, when completed, delivers data that meet the specified requirements. This phase involves creating, using, and updating data according to specified work instructions and monitoring quality by checking whether the data conform to predetermined specifications.\nIn this phase, it is important to address aspects such as the provision of complete metadata; the use of flags to convey information about data; the documentation of all sensor data processing; monitoring and control of data; maintain appropriate levels of human inspection; perform range, domain, and slope checks; implement an automated alert system for sensor network issues; or automate sensor data quality procedures. Table 6 summarizes inputs, outputs, and responsibility distribution for this phase.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 6. Inputs, outputs and RACIQ matrix for the Do phase. (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)\n\n\n\nD1. Execution of the data quality improvement plan \nThe purpose of provision of data specifications and work instructions is to establish the basis on which to perform data processing and data quality monitoring and control, taking account of the outcomes of the data quality planning, data-related support, and the resource provision processes. Having the plan obtained for the first phase, the first step is to provide data and work specification to collect data from devices on the defined way. This step considers the following activities:\n\nD1-1. Establish flags to convey information about the sensor data - Flags or qualifiers convey information about individual data values, typically using codes that are stored in a separate field to correspond with each value. Flags can be highly specific to individual studies and data sets or standardized across all data.\nD1-2. Definition of the optimal node placement plan - It is a challenging problem that has been proven to be NP-hard (non-deterministic polynomial-time hardness) for most of the formulations of sensor deployment.[40][41] To tackle such complexity, several heuristics have been proposed to find sub-optimal solutions.[42][43] However, the context of these optimization strategies is mainly static in the sense that assessing the quality of candidate positions is based on a structural quality metric such as distance, network connectivity, and\/or basing the analysis on a fixed topology. Also, application-level interest can vary over time, and the available network resources may change as new nodes join the network, or as existing nodes run out of energy.[44] Also, if we talk about node placement, we must also talk about node or sensor replication. Replicating data sensors is important for purposes of high availability and disaster recovery. Also, replication of this data on cloud storage needs to be implemented efficiently. Archiving is one way to recover lost or damaged data in primary storage space, but replicas of data repositories that are updated concurrently with the primary repositories can be used for sensitive systems with strong data availability requirements. Replication can be demanding in terms of storage and may degrade performance due to if a concurrent updates strategy is enforced.\nD1-3. Redesign the software or hardware that includes sensors to eliminate root causes - It is an alternative process to the optimal replacement of sensors. For example, redesign and reimplementing a fragment of a SCP\u2019s firmware could improve fault tolerance.\nD1-4. Data cleansing - The purpose of data cleansing is to ensure, in response to the results of root cause analysis and solution development, the organization can access data sets that contain no non-conformities capable of causing unacceptable disruption to the effectiveness and efficiency of decision making using those data. Also, the non-conformities are corrected implementing developed solutions and make a record of the corrections.\nD1-5. Force an appropriate level of human inspection - If performed by trained and experienced technicians, visual inspection is used to monitor the state of industrial equipment and to identify necessary repairs. There are also technologies used to assist with fault recognition or even to automate inspections. Shop floor workers are made responsible for basic maintenance including cleaning machines, visual inspection, and initial detection of machine degradation.\nD1-6. Implement an automated alert system to warn about potential sensor data quality issues - Having only human inspection can be a complex task for maintenance staff. It is necessary to implement a system in which some key indicators are constantly reading if sensor status is correct or not. If a sensor is not working properly, the root cause can be due to a hardware, network, or software failure and affects data quality. Furthermore, an automated data quality procedure might identify anomalous spikes in the data and flag them. However, it is almost always necessary to have human supervision, intervention, and inspection, as stated by Peppeler et al.[45] and [46]; the inclusion of automated quality is often an improvement, because it ensures consistency and reduces human bias. Automated data quality procedures are also more efficient at handling the vast quantities of data that are being generated by streaming sensor networks and reduces the amount of human inspection required.\nD1-7. Schedule sensor maintenance to minimize data quality issues - Sensors require routine maintenance and scheduled calibration that, in some cases, can be done only by the manufacturer. Ideally, maintenance and repairs are scheduled to minimize data loss (e.g., snow-depth sensors repaired during the summer) or staggered in such a way that data from a nearby sensor can be used to fill gaps. In cases in which unscheduled maintenance is required, stocking replacement parts on-site ensures that any part of the network can be replaced immediately.\nThe \"check\" phase \nThe check phase consists of the data quality assurance, which measures data quality levels and methodology steps' performance related to data non-conformities or other issues that have arisen as a result of data quality planning (see the \"plan\" phase) or data quality control (see the \"do\" phase). This measurement provides evidence by which to evaluate the impact of any identified poor levels of data quality on the effectiveness and efficiency of business processes. It is important during this phase to address aspects like scheduled sensor maintenance and repairs to minimize data loss, as well as measurement and evaluation of sensor data quality. Table 7 summarizes inputs, outputs, and responsibility distribution for this phase.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 7. Inputs, outputs, and RACIQ matrix for the Check phase (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)\n\n\n\nC1. Testing the efficiency of the corrective actions \nThe purpose of measurement of data quality is the provision of measurement criteria, to generate input for the evaluation of measurement results. This step considers the following activities:\n\nC1-1. Monitoring and control of the enhanced data - According to the identified risk priorities, monitor and measure conformity of data to the applicable specification. Monitoring and measuring takes place either at intervals or continuously and in accordance with applicable work instructions. This work instructions can be: perform range checks on numerical data, perform domain checks on categorical data, and perform slope and persistence checks on continuous data streams. If data non-conformities are found, then correct the data when viable and distribute to stakeholders a record of the viability and degree of success for each corrective action.\nC1-2. Definition of an interstice comparison plan - Create policies for comparing data with data from related sensors. If no replicate sensors exist, interstice comparisons are useful, whereby data from one location are compared with data from nearby identical sensors.\nC2. Review of data quality issues \nThe purpose of reviewing data quality issues is to identify the starting point for deciding to measure data quality levels and process performance with the potential to generate opportunities to improve data quality. The results of measurement and evaluation of data quality are analyzed and possible issues are identified. This step considers the following activities:\n\nC2-1. Issue analysis - Review non-solved non-conformities arising from data processing to identify those that are possibly connected to the reported issue that has triggered the need for data quality assurance. This review creates a set of related non-conformities. This set is the basis for further investigation through the measurement of data quality levels in SCP environments. Respond to the reporting of unresolved data non-conformities from within data quality control, indications of the recurrence of types of nonconformity, or other issues raised against the results of data quality planning or data quality control.\nThe \"act\" phase \nThe act phase consists of the data quality improvement that involves analyzing the root causes of data quality issues based on the assessment results derived from data quality assurance (see the \"check\" phase). To prevent future data non-conformities, the steps in this phase try to correct all the existing non-conformities and also transforms SCP operations as appropriate. On this phase, it is important to address aspects as analysis of root causes of sensor data problems, management, of sensor data problems, correction and prevention, make available ready access to replacement parts and anticipate common repairs, and maintain inventory replacement parts. Table 8 summarizes inputs, outputs, and responsibility distribution for this phase.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 8. Inputs, outputs, and RACIQ matrix for the Act phase (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)\n\n\n\nA1. Data quality issues prevention \nThe purpose of data quality issue prevention, resulting from root cause analysis and solution development, is to increase the extent to which the organization achieves a systematic and systemic approach to achieving data quality. This step tries to prevent issues on devices or, for example, to avoid data loss when repair or maintenance of devices of the SCP environment can be predicted. In order to prevent the recurrence of each actual or the occurrence of each potential data non-conformity or similar data non-conformities by refining and applying guidelines, rules, and procedures. To achieve this, the following activities should be conducted:'\n\nA1-1. Make available ready access to replacement parts - Schedule routine calibration of instruments and sensors based on manufacturer specifications. Maintaining additional calibrated sensors of the same make\/model can allow immediate replacement of sensors removed for calibration to avoid data loss. Otherwise, sensor calibrations can be scheduled at non-critical times or staggered such that a nearby sensor can be used as a proxy to fill gaps.\nA1-2. Update the strategy for node replacement - Controlled replacement is often pursued for only a selected subset of the employed nodes with the goal of structuring the network topology in a way that achieves the desired application requirements. In addition to coverage, the nodes\u2019 positions affect numerous network performance metrics such as energy consumption, delay, and throughput. For example, large distances between nodes weaken the communication links, lower the throughput, and increase energy consumption. Additionally, it can anticipate some common repairs and maintain inventory replacement parts. This means that sensors could be replaced before failure where sensor lifetimes are known or can be estimated.\nConclusions \nIn this paper, we have tackled the challenges of data quality problems in SCP environments. Many research and standardization efforts have been made in the DQ area over the years, and some interesting results have been already transferred to IoT as well. However, the approaches presented in the literature have two main drawbacks. On the one hand, these proposals do not take into account the nature of SCP environments and the concrete factors that affect the way in which DQ must be treated in such a context. On the other hand, these approaches do not consider existing DQ management standards that have not been tailored yet for IoT and, more specifically, to SCP contexts. Given the importance of institutionalizing best practices in SCP for DQ management, we consider it of paramount importance to provide practitioners and organizations with techniques aligned with standards, reducing their adaptation efforts and favoring systematic and holistic approaches to the problem. As a main contribution, we have provided in this paper a data quality management methodology for sensor data, named DAQUA-MASS, based on ISO 8000-61. The methodology is structured according to the PDCA cycle of continuous improvement. The methodology is composed of seven steps divided into several activities. Input and output products are already identified for each activity in the methodology. It is noteworthy to highlight the identification of the various roles involved in the management of data quality in sensor data.\nThe data quality model, along with the methodology, offers a unique framework to enable designers of IoT projects\u2014including sensor networks and practitioners in charge of exploiting IoT systems\u2014to assure that the business processes working over these systems can manage data with adequate levels of quality. Once discarded data quality problems or identified data quality problem root causes are identified, it will be easier to focus the attention exclusively on sensor networks of IoT systems. Working aligned to international open standards will enable organizations to speak the same language and devote the required efforts only to the proper working of the IoT systems, raising awareness of some other data quality concerns.\nFuture research lines are planned to be primarily focused on the empirical validation of this methodology. It will be used by different organizations in various SCP environments in order to ensure its applicability on a large scale. The mentioned case studies will allow us to provide different specializations of the data quality model (or recommendation on how to tailor it) to different organization depending on the domain. In parallel, other future research lines are in accordance with some limitations of the scope considered in this paper. Thus, data quality management issues are expected to be considered, as well as their implications in the processing and utilization layers apart from the acquisition layer (see Figure 1). While simplified data analysis is being performed on the devices in the acquisition layer to control the device, most analysis, diagnosis, and improvement of sensor data are being performed in the processing layer. This will force management of sensor data together with general-purpose data to achieve holistic data quality management in the processing layer.\n\nAcknowledgements \nAuthor contributions \nConceptualization, R.P.-C. and I.C.; Investigation, R.P.-C., A.G.C. and I.C.; Writing-Original Draft Preparation, R.P.-C., A.G.C. and I.C.; Writing-Review & Editing, R.P.-C., A.G.C.; I.C.; M.R., M.P., A.M., S.K. and D.L.; Supervision, I.C. and M.P.; Project Administration, R.P.-C. and I.C.; Funding Acquisition, I.C., M.P., S.K., D.L.\n\nFunding \nThis work was primarily funded by DQIoT project (Eureka program, E!11737; and CDTI (Centro Para el Desarrollo Tecnol\u00f3gico Industrial), INNO-20171086). Additionally, this work was partially funded by SEQUOIA project (TIN2015-63502-C3-1-R and TIN2015-63502-C3-3-R) (MINECO\/FEDER); GEMA SBPLY\/17\/180501\/000293, Consejer\u00eda de Educaci\u00f3n, Cultura y Deporte de la Direcci\u00f3n General de Universidades, Investigaci\u00f3n e Innovaci\u00f3n de la JCCM); ECD project (Evaluaci\u00f3n y Certificaci\u00f3n de la Calidad de Datos) (PTQ-16-08504) (Torres Quevedo Program, MINECO). Finally, it was also supported through a grant to Ricardo P\u00e9rez-Castillo enjoys from JCCM within the initiatives for talent retention and return in line with RIS3 goals.\n\nConflict of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 Ashton, K. (2009). \"That 'Internet of Things' Thing\". RFID Journal 22: 97\u2013114.   \n\n\u2191 2.0 2.1 Weber, R.H. (2013). \"Internet of things \u2013 Governance quo vadis?\". Computer Law & Security Review 29 (4): 341-347. doi:10.1016\/j.clsr.2013.05.010.   \n\n\u2191 Hassanein, H.S.; Oteafy, S.M.A. (2017). \"Big Sensed Data Challenges in the Internet of Things\". Proceedings from the 13th International Conference on Distributed Computing in Sensor Systems: 207\u20138. doi:10.1109\/DCOSS.2017.35.   \n\n\u2191 Atzori, L.; Iera, A.; Morabito, G. (2010). \"The Internet of Things: A survey\". Computer Networks 54 (15): 2787-2805. doi:10.1016\/j.comnet.2010.05.010.   \n\n\u2191 Zhang, W.; Zhang, Z.; Chao, H.-C. (2017). \"Cooperative Fog Computing for Dealing with Big Data in the Internet of Vehicles: Architecture and Hierarchical Resource Management\". IEEE Communications Magazine 55 (12): 60\u20137. doi:10.1109\/MCOM.2017.1700208.   \n\n\u2191 6.0 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 Karkouch, A.; Mousannif, H.; Al Moatassime, H. et al. (2016). \"Data quality in internet of things: A state-of-the-art survey\". Journal of Network and Computer Applications 73: 57\u201381. doi:10.1016\/j.jnca.2016.08.002.   \n\n\u2191 7.0 7.1 Merino, J.; Caballero, I.; Rivas, B. et al. (2016). \"A Data Quality in Use model for Big Data\". Future Generation Computer Systems 63: 123\u201330. doi:10.1016\/j.future.2015.11.024.   \n\n\u2191 8.0 8.1 8.2 8.3 Jesus, G.; Casimiro, A.; Oliveira, A. (2017). \"A Survey on Data Quality for Dependable Monitoring in Wireless Sensor Networks\". Sensors 17 (9): E2010. doi:10.3390\/s17092010. PMC PMC5620495. PMID 28869505. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5620495 .   \n\n\u2191 9.0 9.1 9.2 9.3 9.4 9.5 9.6 Rodr\u00edguez, C.C.G.; Servigne, S. (2017). \"Managing Sensor Data Uncertainty: A Data Quality Approach\". International Journal of Agricultural and Environmental Information Systems 4 (1): 3. doi:10.4018\/jaeis.2013010103.   \n\n\u2191 10.0 10.1 10.2 Klein, A.; Hackenbroich, G.; Lehner, W. (2009). \"How to Screen a Data Stream - Quality-Driven Load Shedding in Sensor Data Streams\". Proceedings of the 2009 International Conference on Information Quality: 1\u201315. http:\/\/mitiq.mit.edu\/iciqpapers.aspx?iciqyear=2009 .   \n\n\u2191 11.0 11.1 M\u00fchlh\u00e4user, M. (2007). \"Smart Products: An Introduction\". Proceedings from the 2007 European Conference on Ambient Intelligence: 158\u201364. doi:10.1007\/978-3-540-85379-4_20.   \n\n\u2191 Laney, Douglas B. (2017). Infonomics: How to Monetize, Manage, and Measure Information as an Asset for Competitive Advantage. Routledge. ISBN 9781138090385.   \n\n\u2191 \"ISO\/IEC 25000:2014\". International Organization for Standardization. March 2014. https:\/\/www.iso.org\/standard\/64764.html . Retrieved 13 September 2018 .   \n\n\u2191 14.0 14.1 Qin, z.; Han, Q.; Mehrotra, S. et al. (2014). \"Quality-Aware Sensor Data Management\". In Ammari, H.M.. The Art of Wireless Sensor Networks. Springer. pp. 429\u201364. ISBN 9783642400094.   \n\n\u2191 15.0 15.1 15.2 Campbell, J.L.; Rustad, L.E.; Porter, J.H. et al. (2013). \"Quantity is Nothing without Quality: Automated QA\/QC for Streaming Environmental Sensor Data\". BioScience 63 (7): 574\u201385. doi:10.1525\/bio.2013.63.7.10.   \n\n\u2191 Klein, A.; Lehner, W. (2009). \"Representing Data Quality in Sensor Data Streaming Environments\". Journal of Data and Information Quality 1 (2): 10. doi:10.1145\/1577840.1577845.   \n\n\u2191 17.0 17.1 17.2 \"ISO 8000-61:2016\". International Organization for Standardization. November 2016. https:\/\/www.iso.org\/standard\/63086.html . Retrieved 13 September 2018 .   \n\n\u2191 Cook, D.; Das, S.K. (2004). Smart Environments: Technology, Protocols and Applications (1st ed.). Wiley-Interscience. ISBN 9780471544487.   \n\n\u2191 19.0 19.1 19.2 19.3 Porter, M.E.; Heppelmann, J.E. (2014). \"How Smart, Connected Products Are Transforming Competition\". Harvard Business Review 92: 64\u201388. https:\/\/hbr.org\/2014\/11\/how-smart-connected-products-are-transforming-competition .   \n\n\u2191 Ostrower, D. (2014). \"Smart Connected Products: Killing Industries, Boosting Innovation\". Wired. https:\/\/www.wired.com\/insights\/2014\/11\/smart-connected-products\/ . Retrieved 13 September 2019 .   \n\n\u2191 Wuenderlich, N.V.; Heinonen, K.; Ostrom, A.L. et al. (2015). \"\u201cFuturizing\u201d smart service: Implications for service researchers and managers\". Journal of Services Marketing 29 (6\/7): 442\u201347. doi:10.1108\/JSM-01-2015-0040.   \n\n\u2191 Allmendinger, G.; Lombreglia, R. (2005). \"Four Strategies for the Age of Smart Services\". Harvard Business Review 83: 131. https:\/\/hbr.org\/2005\/10\/four-strategies-for-the-age-of-smart-services .   \n\n\u2191 23.0 23.1 23.2 Tilak, S.; Abu-Ghazaleh, N.B.; Heinzelman, W. (2002). \"A taxonomy of wireless micro-sensor network models\". ACM SIGMOBILE Mobile Computing and Communications Review 6 (2): 28\u201336. doi:10.1145\/565702.565708.   \n\n\u2191 Barnaghi, P.; Bermudez-Edo, M.; T\u00f6njes, R. (2015). \"Challenges for Quality of Data in Smart Cities\". Journal of Data and Information Quality 6 (2\u20133): 6. doi:10.1145\/2747881.   \n\n\u2191 25.0 25.1 25.2 25.3 25.4 \"ISO\/IEC 25012:2008\". International Organization for Standardization. December 2008. https:\/\/www.iso.org\/standard\/35736.html . Retrieved 13 September 2018 .   \n\n\u2191 Badawy, R.; Raykov, Y.P.; Evers, L.J.W. et al. (2018). \"Automated Quality Control for Sensor Based Symptom Measurement Performed Outside the Lab\". Sensors 18 (4): E1215. doi:10.3390\/s18041215. PMC PMC5948536. PMID 29659528. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5948536 .   \n\n\u2191 Al-Ruithe, M.; Mthunzi, S.; Benkhelifa, E. (2016). \"Data governance for security in IoT & cloud converged environments\". Proceedings from the 13th International Conference of Computer Systems and Applications: 1\u20138. doi:10.1109\/AICCSA.2016.7945737.   \n\n\u2191 Lee, Y.W.; Strong, D.M.; Kahn, B.K. et al. (2002). \"AIMQ: A methodology for information quality assessment\". Information & Management: 133-46. doi:10.1016\/S0378-7206(02)00043-5.   \n\n\u2191 McGilvray, D. (2008). Executing Data Quality Projects: Ten Steps to Quality Data and Trusted Information (TM). Elsevier. ISBN 9780080951584.   \n\n\u2191 \"ISO\/TS 8000-150:2011\". International Organization for Standardization. December 2011. https:\/\/www.iso.org\/standard\/54579.html . Retrieved 13 September 2018 .   \n\n\u2191 \"ISO 9001:2015\". International Organization for Standardization. September 2015. https:\/\/www.iso.org\/standard\/62085.html . Retrieved 13 September 2018 .   \n\n\u2191 Batini, C.; Cappiello, C.; Francalanci, C. et al. (2009). \"Methodologies for data quality assessment and improvement\". ACM Computing Surveys 41 (3): 16. doi:10.1145\/1541880.1541883.   \n\n\u2191 Woodall, P.; Oberhofer, M.; Borek, A. (2014). \"A classification of data quality assessment and improvement methods\". International Journal of Information Quality 3 (4): 298\u2013321. doi:10.1504\/IJIQ.2014.068656.   \n\n\u2191 Strong, D.M.; Lee, Y.W.; Wang, R.Y. (1997). \"Data quality in context\". Communications of the ACM 40 (5): 103\u201310. doi:10.1145\/253769.253804.   \n\n\u2191 Wang, R.Y. (1998). \"A product perspective on total data quality management\". Communications of the ACM 41 (2): 58\u201365. doi:10.1145\/269012.269022.   \n\n\u2191 Srivannaboon, S. (2009). \"Achieving Competitive Advantage through the use of Project Management under the Plan-do-Check-act Concept\". Journal of General Management 34 (3): 1\u201320. doi:10.1177\/030630700903400301.   \n\n\u2191 Chen, P.-Y.; Cheng, S.-M.; Chen, K.-C. (2014). \"Information Fusion to Defend Intentional Attack in Internet of Things\". IEEE Internet of Things Journal 1 (4): 337\u201348. doi:10.1109\/JIOT.2014.2337018.   \n\n\u2191 Blasch, E.; Steinberg, A.; Das, S. et al. (2013). \"Revisiting the JDL model for information exploitation\". Proceedings of the 16th International Conference on Information Fusion: 129\u201336. http:\/\/ieeexplore.ieee.org\/stamp\/stamp.jsp?tp=&arnumber=6641155&isnumber=6641065 .   \n\n\u2191 Pastorello Jr., G.Z. (19 December 2008). \"Gerenciamento do Ciclo de Vida de Dados de Sensores: da Produ\u00b8c\u02dcao ao Consumo\" (PDF). https:\/\/lis-unicamp.github.io\/wp-content\/uploads\/2014\/09\/tese_GZPastorelloJr.pdf . Retrieved 13 September 2018 .   \n\n\u2191 Cerpa, A.; Estrin, D. (2004). \"ASCENT: Adaptive self-configuring sensor networks topologies\". IEEE Transactions on Mobile Computing 3 (3): 272\u201385. doi:10.1109\/TMC.2004.16.   \n\n\u2191 Cheng, X.; Du, D.-Z.; Wang, L. et al. (2008). \"Relay sensor placement in wireless sensor networks\". Wireless Networks 14 (3): 347\u2013355. doi:10.1007\/s11276-006-0724-8.   \n\n\u2191 Dhillon, S.S.; Chakrabarty, K. (2003). \"Sensor placement for effective coverage and surveillance in distributed sensor networks\". Proceedings from 2003 IEEE Wireless Communications and Networking 3: 1609-1614. doi:10.1109\/WCNC.2003.1200627.   \n\n\u2191 Pan, J.; Cai, L.; Hou, Y.T. et al. (2005). \"Optimal base-station locations in two-tiered wireless sensor networks\". IEEE Transactions on Mobile Computing 4 (5): 458\u201373. doi:10.1109\/TMC.2005.68.   \n\n\u2191 Wu, J.; Yang, S. (2005). \"SMART: A scan-based movement-assisted sensor deployment method in wireless sensor networks\". Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies 4: 2313-2324. doi:10.1109\/INFCOM.2005.1498518.   \n\n\u2191 Peppler, R.A.; Long, C.N.; Sisterson, D.L. et al. (2008). \"An Overview of ARM Program Climate Research Facility Data Quality Assurance\". The Open Atmospheric Science Journal 2: 192\u2013216. doi:10.2174\/1874282300802010192.   \n\n\u2191 Fiebrich, C.A.; Grimsley, D.L.; McPherson, R.A. et al. (2006). \"The Value of Routine Site Visits in Managing and Maintaining Quality Data from the Oklahoma Mesonet\". Journal of Atmospheric and Oceanic Technology 23: 406\u201316. doi:10.1175\/JTECH1852.1.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\">https:\/\/www.limswiki.org\/index.php\/Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on data qualityLIMSwiki journal articles on internet of thingsLIMSwiki journal articles on sensor networks\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 2 April 2019, at 19:42.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 228 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","f59eece05cdc9b79c1305c6bea6b72d6_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_DAQUA-MASS_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:DAQUA-MASS: An ISO 8000-61-based data quality management methodology for sensor data<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The <a href=\"https:\/\/www.limswiki.org\/index.php\/Internet_of_things\" title=\"Internet of things\" class=\"wiki-link\" data-key=\"13e0b826fa1770fe4bea72e3cb942f0f\">internet of things<\/a> (IoT) introduces several technical and managerial challenges when it comes to the use of data generated and exchanged by and between various smart, connected products (SCPs) that are part of an IoT system (i.e., physical, intelligent devices with sensors and actuators). Added to the volume and the heterogeneous exchange and consumption of data, it is paramount to <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_assurance\" title=\"Quality assurance\" class=\"wiki-link\" data-key=\"2ede4490f0ea707b14456f44439c0984\">assure<\/a> that data quality levels are maintained in every step of the data chain\/lifecycle. Otherwise, the system may fail to meet its expected function. While data quality (DQ) is a mature field, existing solutions are highly heterogeneous. Therefore, we propose that companies, developers, and vendors should align their data quality management mechanisms and artifacts with well-known best practices and <a href=\"https:\/\/www.limswiki.org\/index.php\/Specification_(technical_standard)\" title=\"Specification (technical standard)\" class=\"wiki-link\" data-key=\"dc2050725de4ea7ca520800e4f969b20\">standards<\/a>, as for example, those provided by ISO 8000-61. This standard enables a process-approach to data quality management, overcoming the difficulties of isolated data quality activities. This paper introduces DAQUA-MASS, a methodology based on ISO 8000-61 for data quality management in sensor networks. The methodology consists of four steps according to the Plan-Do-Check-Act cycle by Deming.\n<\/p><p><b>Keywords<\/b>: data quality; data quality management processes; ISO 8000-61; data quality in sensors; internet of things; IoT; smart, connected products; SCPs\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>\u201cOur economy, society, and survival aren\u2019t based on ideas or information\u2014they\u2019re based on things.\u201d<sup id=\"rdp-ebb-cite_ref-AshtonThatInt09_1-0\" class=\"reference\"><a href=\"#cite_note-AshtonThatInt09-1\">[1]<\/a><\/sup> This is one of the core foundations of the internet of things (IoT) as stated by Ashton, who coined the term. IoT is an emerging global internet-based <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> architecture facilitating the exchange of goods and services.<sup id=\"rdp-ebb-cite_ref-WeberInternet13_2-0\" class=\"reference\"><a href=\"#cite_note-WeberInternet13-2\">[2]<\/a><\/sup> IoT systems are inherently built on data gathered from heterogeneous sources in which the volume, variety, and velocity of data generation, exchanging and processing are dramatically increasing.<sup id=\"rdp-ebb-cite_ref-HassaneinBig17_3-0\" class=\"reference\"><a href=\"#cite_note-HassaneinBig17-3\">[3]<\/a><\/sup> Furthermore, there is a certain emergence of IoT semantic-oriented vision which needs ways to represent and manipulate the vast amount of raw data expected to be generated from and exchanged between the \u201cthings.\u201d<sup id=\"rdp-ebb-cite_ref-AtzoriTheInt10_4-0\" class=\"reference\"><a href=\"#cite_note-AtzoriTheInt10-4\">[4]<\/a><\/sup>\n<\/p><p>The vast amount of data in IoT environments, gathered from a global-scale deployment of smart-things, is the basis for making intelligent decisions and providing better services (e.g., smart mobility, as presented by Zhang <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-ZhangCoop17_5-0\" class=\"reference\"><a href=\"#cite_note-ZhangCoop17-5\">[5]<\/a><\/sup>). In other words, data represents the bridge that connects cyber and physical worlds. Despite of its tremendous relevance, if data are of inadequate quality, decisions from both humans and other devices are likely to be unsound.<sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-0\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MerinoAData16_7-0\" class=\"reference\"><a href=\"#cite_note-MerinoAData16-7\">[7]<\/a><\/sup> As a consequence, data quality (DQ) has become one of the key aspects in IoT.<sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-1\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JesusAsurv17_8-0\" class=\"reference\"><a href=\"#cite_note-JesusAsurv17-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RodriguezManaging13_9-0\" class=\"reference\"><a href=\"#cite_note-RodriguezManaging13-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KleinHowTo09_10-0\" class=\"reference\"><a href=\"#cite_note-KleinHowTo09-10\">[10]<\/a><\/sup> IoT devices, and in particular smart, connected products (SCPs), have concrete characteristics that favor the apparition of problems due to inadequate levels of data quality. M\u00fchlh\u00e4user<sup id=\"rdp-ebb-cite_ref-M.C3.BChlh.C3.A4userSmart07_11-0\" class=\"reference\"><a href=\"#cite_note-M.C3.BChlh.C3.A4userSmart07-11\">[11]<\/a><\/sup> defines SCPs as \u201centities (tangible object, software, or service) designed and made for self-organized embedding into different (smart) environments in the course of its lifecycle, providing improved simplicity and openness through improved connections.\u201d While some of the SCP-related characteristics might be considered omnipresent (i.e., uncertain, erroneous, noisy, distributed, and voluminous), other characteristics are more specific and highly dependent on the context and monitored phenomena (i.e., smooth variation, continuous, correlation, periodicity, or Markovian behavior).<sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-2\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup>\n<\/p><p>Also, outside of the IoT research area, DQ has been broadly studied during last years, and it has become a mature research area capturing the growing interest of the industry due to the different types of values that companies can extract from data.<sup id=\"rdp-ebb-cite_ref-LaneyInfo17_12-0\" class=\"reference\"><a href=\"#cite_note-LaneyInfo17-12\">[12]<\/a><\/sup> This fact is reflected by the standardization efforts like ISO\/IEC 25000 series addressing systems and software quality requirements and evaluation (SQuaRE)<sup id=\"rdp-ebb-cite_ref-ISO_25000_13-0\" class=\"reference\"><a href=\"#cite_note-ISO_25000-13\">[13]<\/a><\/sup> processes, and specific techniques for managing data concerns. We pose that such standards can be tailored and used within the IoT context, not only bring benefits standardizing solutions and enabling a better communication between partners. Also, the number of problems and system fails on the IoT environment is reduced, better decisions can be taken due to a better quality of data, all stakeholders are aligned and can take benefit of the advances on the standard used, and it is easier to apply data quality solutions in a global way because the heterogeneity is reduced.\n<\/p><p>Due to the youth of IoT, and despite DQ standards, frameworks, management techniques, and tools proposed in the literature, DQ for IoT has not been yet widely studied. However, and prior to this research line, it is possible to cite some works that had addressed some DQ concerns in sensor wireless networks<sup id=\"rdp-ebb-cite_ref-JesusAsurv17_8-1\" class=\"reference\"><a href=\"#cite_note-JesusAsurv17-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-QinQuality14_14-0\" class=\"reference\"><a href=\"#cite_note-QinQuality14-14\">[14]<\/a><\/sup>, or in data streaming<sup id=\"rdp-ebb-cite_ref-CampbellQuant13_15-0\" class=\"reference\"><a href=\"#cite_note-CampbellQuant13-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KleinRepresent09_16-0\" class=\"reference\"><a href=\"#cite_note-KleinRepresent09-16\">[16]<\/a><\/sup> among other proposals.<sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-3\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup> However, these works have not considered the management of DQ in a holistic way in line with existing DQ-related standards. In our attempt to align the study of DQ in IoT to international standards, this paper provides practitioners and researchers with DAQUA-MASS, a methodology for managing data quality in SCP environments, which considers some of the DQ best practices for improving quality of data in SCP environments aligned to ISO 8000-61.<sup id=\"rdp-ebb-cite_ref-ISO_8000-61_17-0\" class=\"reference\"><a href=\"#cite_note-ISO_8000-61-17\">[17]<\/a><\/sup> Due to the intrinsic distributed nature of IoT systems, using such standards will enable the various organizations to be aligned to the same foundations, and in the end, to work in a seamless way, what will undoubtedly improve the performance of the business processes.\n<\/p><p>The remainder of this paper is organized as follows: the next section presents the most challenging data quality management concerns in the context of the SCP environments; afterwards. related work is explored. Then the data quality model in which our methodology is based on is presented. The last two sections propose a methodology for managing data quality in SCP environments and discuss conclusions and implications of this work.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Data_quality_challenges_in_SCP_environments\">Data quality challenges in SCP environments<\/span><\/h2>\n<p>This section introduces some general ideas about SCPs and their operation as an essential part of IoT. In addition, some challenges related to DQ in SCP environments are also introduced.\n<\/p><p>According to Cook and Das<sup id=\"rdp-ebb-cite_ref-CookSmart04_18-0\" class=\"reference\"><a href=\"#cite_note-CookSmart04-18\">[18]<\/a><\/sup>, a smart environment is a small world where all kinds of smart devices are continuously working to make inhabitants\u2019 lives more comfortable. According to M\u00fchlh\u00e4user<sup id=\"rdp-ebb-cite_ref-M.C3.BChlh.C3.A4userSmart07_11-1\" class=\"reference\"><a href=\"#cite_note-M.C3.BChlh.C3.A4userSmart07-11\">[11]<\/a><\/sup>, SCP provides intelligent actions through improved connections by means of context-awareness, semantic self-description, proactive behavior, multimodal natural interfaces, AI planning, and machine learning.\n<\/p><p>SCPs have three main core components: physical, smart, and connectivity components. Smart components extend the capabilities and value of the physical components, while connectivity extends the capabilities and value of the smart components. This enables some smart components to exist outside the physical product itself, with a cycle of value improvement.<sup id=\"rdp-ebb-cite_ref-PorterHowSmart14_19-0\" class=\"reference\"><a href=\"#cite_note-PorterHowSmart14-19\">[19]<\/a><\/sup>\n<\/p><p>IoT and SCP can be confused in some contexts. However, IoT simply reflects the growing number of SCPs and highlights the new opportunities they can represent. IoT, which can involve people or things, is a means for interchanging information. What makes SCPs essentially different is not the \"internet,\" but the changing nature of the \u201cthings.\u201d<sup id=\"rdp-ebb-cite_ref-PorterHowSmart14_19-1\" class=\"reference\"><a href=\"#cite_note-PorterHowSmart14-19\">[19]<\/a><\/sup> A product that is smart and connected to the cloud could become part of an interconnected management solution, and companies can therefore evolve from making products to offering more complex, higher-value services within a \u201csystem of systems.\u201d<sup id=\"rdp-ebb-cite_ref-OstrowerSmart14_20-0\" class=\"reference\"><a href=\"#cite_note-OstrowerSmart14-20\">[20]<\/a><\/sup>\n<\/p><p>SCPs include processors, sensors, software, and connectivity that allow data to be exchanged between the product and its environment. The data collected by sensors of these SCPs can be then analysed to inform decision-making, enable operational efficiencies, and continuously improve the performance of the product. This paper focuses on the data produced by such sensors, and how inadequate levels of data quality may affect the processing of the data, while smart and connectivity parts of SCPs are outside of the scope of this paper.\n<\/p><p>SCPs can be connected in large, complex networks throughout three different layers<sup id=\"rdp-ebb-cite_ref-RodriguezManaging13_9-1\" class=\"reference\"><a href=\"#cite_note-RodriguezManaging13-9\">[9]<\/a><\/sup>: acquisition, processing, and utilization layers (see Figure 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"bf37de6e3785feae79c1de0fa3bcd1ca\"><img alt=\"Fig1 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/8a\/Fig1_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> Layers in SCP environments.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<ul><li> The acquisition layer refers to the sensor data collection system where sensors, raw (or sensed) data, and pre-processed data are managed. This is the main focus of this paper.<\/li>\n<li> The processing layer involves data resulting from the data processing and management center, where energy, storage, and analysis capabilities are more significant.<\/li>\n<li> The utilization layer concerns delivered data (or post-processed data) exploited, for example, over a <a href=\"https:\/\/www.limswiki.org\/index.php\/Geographic_information_system\" title=\"Geographic information system\" class=\"wiki-link\" data-key=\"8981ab93f8ebf0730c3b38949b39ad99\">geographic information system<\/a> (GIS) or combined with other services or applications.<\/li><\/ul>\n<p>As previously stated, the scope of the paper is limited to the data produced by SCPs\u2019 sensors. Hence, the proposal is mainly intended to be applied in the context of the acquisition layer. Nevertheless, the management of data quality in sensors can impact on how data is processed (processing layer) and how data may be used later (utilization layer).\n<\/p><p>Networking and management of SCP operations can generate the business intelligence needed to deliver smart services. Smart services are delivered to or via smart objects that feature awareness and connectivity.<sup id=\"rdp-ebb-cite_ref-WuenderlichFutur15_21-0\" class=\"reference\"><a href=\"#cite_note-WuenderlichFutur15-21\">[21]<\/a><\/sup> SCP can carry out the following functions to support smart services<sup id=\"rdp-ebb-cite_ref-AllmendingerFour05_22-0\" class=\"reference\"><a href=\"#cite_note-AllmendingerFour05-22\">[22]<\/a><\/sup>: status, diagnostics, upgrades, control and automation, profiling and behavior tracking, replenishment and commerce, location mapping. and logistics, among others.\n<\/p><p>SCP operations enable new capabilities for companies, although new problems and challenges that arise must also be taken into account. On one hand, SCP operations require companies to build and support an entirely new technology infrastructure.<sup id=\"rdp-ebb-cite_ref-PorterHowSmart14_19-2\" class=\"reference\"><a href=\"#cite_note-PorterHowSmart14-19\">[19]<\/a><\/sup> Technological layers in the new technology landscape include new product hardware, embedded software, connectivity, a product cloud running on remote servers, security tools, gateway for external information sources, and integration with enterprise business systems. On the other hand, SCP operations can provide competitive advantages, which are based on the operational effectiveness. Operation effectiveness requires the embrace of best practices along the value chain, including up-to-date product technologies, the latest production equipment, and state-of-the-art sales force methods, IT solutions, and so forth. Thus, SCP operations also creates new best practices across the value chain.<sup id=\"rdp-ebb-cite_ref-PorterHowSmart14_19-3\" class=\"reference\"><a href=\"#cite_note-PorterHowSmart14-19\">[19]<\/a><\/sup>\n<\/p><p>According to the different sources of data in these SCP environments, we can distinguish different types of aggregated data:\n<\/p>\n<ul><li> Sensor data: data that is generated by sensors and digitized in a computer-readable format (for example, camera sensor readings)<\/li>\n<li> Device data: integrated by sensor data; observed metadata (metadata that characterizes the sensor data, e.g., timestamp of sensor data); and device meta data (metadata that characterizes the device, e.g., device model, sensor model, manufacturer, etc.), so device data, for example, can be data coming from the camera (device)<\/li>\n<li> General data: data related to\/or coming from devices which has been modified or computed to derive different data plus business data (i.e., data for business use such as operation, maintenance, service, customers, etc.)<\/li>\n<li> IoT data: general data plus device data<\/li><\/ul>\n<p>A reduction in the levels of quality of these data due to different problems in SCP operations can threaten the success factors of SCP environments.<sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-4\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup> The quality of produced data is often affected by dysfunctional SCP devices and sensors, which are the sources providing data, and can potentially result in inadequate levels of quality that are only detected later on, when data are being processed and used. Therefore, while we can identify dysfunctional SCP devices through the analysis of sensor data by using data quality management techniques, it is noteworthy that these devices will impact the rest of the sensor network. According to<sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-5\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup>, Table 1 summarizes some of these SCP factors that, in some cases, could condition or lead to data quality issues. In addition, the three columns on the right of Table 1 show (marked with a cross) the most critical layers affected in a greater extent by every SCP factor.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> SCP factors that can finally affect the levels of DQ according to Karkouch <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-6\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup>\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">SCP Factor\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Side Effect in Data Quality\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Acquisition\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Processing\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Utilization\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Deployment scale\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SCPs are expected to be deployed on a global scale. This leads to a huge heterogeneity in data sources (not only computers but also daily objects). Also, the huge number of devices accumulates the chance of error occurrence.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Resource constraints\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">For example, computational and storage capabilities that do not allow complex operations due, in turn, to the battery-power constraints among others.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Network\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Intermittent loss of connection in the IoT is recurrent. Things are only capable of transmitting small-sized messages due to their scarce resources.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sensors\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Embedded sensors may lack precision or suffer from loss of calibration or even low accuracy. Faulty sensors may also result in inconsistencies in data sensing.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Environment\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SCP devices will not be deployed only in tolerant and less aggressive environments. To monitor some phenomenon, sensors may be deployed in environments with extreme conditions. Data errors emerge when the sensor experiences the surrounding environment influences.<sup id=\"rdp-ebb-cite_ref-TilakATax02_23-0\" class=\"reference\"><a href=\"#cite_note-TilakATax02-23\">[23]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Vandalism\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Things are generally defenseless from outside physical threats (both from humans and animals).\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fail-dirty\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A sensor node fails, but it keeps up reporting readings which are erroneous. It is a common problem for SCP networks and an important source of outlier readings.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Privacy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Privacy preservation processing, thus DQ could be intentionally reduced.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security vulnerability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sensor devices are vulnerable to attack, e.g., it is possible for a malicious entity to alter data in an SCP device.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data stream processing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data gathered by smart things are sent in the form of streams to the back-end pervasive applications which make use of them. Some stream processing operators could affect quality of the underlying data.<sup id=\"rdp-ebb-cite_ref-KleinHowTo09_10-1\" class=\"reference\"><a href=\"#cite_note-KleinHowTo09-10\">[10]<\/a><\/sup> Other important factors are data granularity and variety.<sup id=\"rdp-ebb-cite_ref-BarnaghiChallenges15_24-0\" class=\"reference\"><a href=\"#cite_note-BarnaghiChallenges15-24\">[24]<\/a><\/sup> Granularity concerns interpolation and spatio-temporal density while variety refers to interoperability and dynamic semantics.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Tilak <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TilakATax02_23-1\" class=\"reference\"><a href=\"#cite_note-TilakATax02-23\">[23]<\/a><\/sup> provide a taxonomy of sensor errors. These errors are directly related to different data quality problems in the acquisition layer. The mentioned taxonomy distinguishes the following six types of data sensors errors (see Table 2). Apart from errors in isolated SCP devices, there are other communication errors which can happen at the SCP network level.<sup id=\"rdp-ebb-cite_ref-TilakATax02_23-2\" class=\"reference\"><a href=\"#cite_note-TilakATax02-23\">[23]<\/a><\/sup> Table 3 summarizes the main types of communication errors: omission, crashes, delay, and message corruption. The table shows the DQ issue derived by each problem, the root cause, and possible solution.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab2_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"cff2ffd82bd199fd60bfd68fb85743d9\"><img alt=\"Tab2 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/8c\/Tab2_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 2.<\/b> Sensors errors deriving DQ problems in SCP environments (adapted from Jesus <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-JesusAsurv17_8-2\" class=\"reference\"><a href=\"#cite_note-JesusAsurv17-8\">[8]<\/a><\/sup>)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 3.<\/b> SCP Network Errors\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Sensor fault\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">DQ problem\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Root cause\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Solution\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Omission faults\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Absence of values\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Missing sensor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Network reliability, retransmission\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Crash faults (fading\/intermittent)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Inaccuracy\/absence of values\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Environment interference\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Redundancy\/estimating with past values\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Delay faults\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Inaccuracy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Time domain\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Timeline solutions\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Message corruption\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Integrity\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Communication\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Integrity validation\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>All the mentioned SCP devices\/sensor errors will lead to different DQ problems in the three layers depicted in Figure 1. As previously mentioned, DQ problems can be represented as a degradation of some DQ characteristics that are especially important in different environments. Let us consider two groups of data quality characteristics:\n<\/p>\n<ul><li> DQ characteristics to assess data quality in use within a specific context. This aspect considers selected criteria to estimate the quality of raw sensor data at the acquisition and processing layer. There are some DQ characteristics considered, which make it possible to estimate the quality on data sources, their context of acquisition, and their transmission to the data management and processing. These DQ characteristics are accuracy and completeness according to ISO\/IEC 25012<sup id=\"rdp-ebb-cite_ref-ISO_25012_25-0\" class=\"reference\"><a href=\"#cite_note-ISO_25012-25\">[25]<\/a><\/sup> and reliability and communication reliability as proposed by Rodriguez and Servigne.<sup id=\"rdp-ebb-cite_ref-RodriguezManaging13_9-2\" class=\"reference\"><a href=\"#cite_note-RodriguezManaging13-9\">[9]<\/a><\/sup> It is also related to the utilization layer and includes availability regarding ISO\/IEC 25012<sup id=\"rdp-ebb-cite_ref-ISO_25012_25-1\" class=\"reference\"><a href=\"#cite_note-ISO_25012-25\">[25]<\/a><\/sup> plus timeliness and adequacy as defined by Rodriguez and Servigne.<sup id=\"rdp-ebb-cite_ref-RodriguezManaging13_9-3\" class=\"reference\"><a href=\"#cite_note-RodriguezManaging13-9\">[9]<\/a><\/sup><\/li>\n<li> DQ Characteristics aimed at managing internal data quality. The main goal of managing internal data quality is to avoid inconsistent data and maintain the temporality of sensor data at the processing layer. These characteristics are consistency and currency according to ISO\/IEC 25012<sup id=\"rdp-ebb-cite_ref-ISO_25012_25-2\" class=\"reference\"><a href=\"#cite_note-ISO_25012-25\">[25]<\/a><\/sup> and volatility as proposed by Rodriguez and Servigne.<sup id=\"rdp-ebb-cite_ref-RodriguezManaging13_9-4\" class=\"reference\"><a href=\"#cite_note-RodriguezManaging13-9\">[9]<\/a><\/sup><\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Related_work\">Related work<\/span><\/h2>\n<p>The goal of this section is twofold. First, works related to the study of data quality in sensor networks and SCP environments in general are presented. Second, data quality methodologies are introduced and compared in order to draw the main contribution of the proposed methodology.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sensor_data_quality\">Sensor data quality<\/span><\/h3>\n<p>There are some published works in the literature that address the concerns related to data quality management in SCP and IoT environments. Karkouch <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-7\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup> presented a state-of-the-art survey for DQ in IoT. This survey presents IoT-related factors endangering the DQ and their impact on various DQ characteristics. Also, DQ problem manifestations are discussed (and their symptoms identified) as well as their impact in the context of IoT. Jesus <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-JesusAsurv17_8-3\" class=\"reference\"><a href=\"#cite_note-JesusAsurv17-8\">[8]<\/a><\/sup> provided a similar survey addressing the problem of not being able to ensure desired DQ levels for dependable monitoring when using wireless sensor networks. This work pays special attention to comprehension of which faults can affect sensors, how they can affect the quality of the information, and how this quality can be improved and quantified. Rodriguez and Servigne<sup id=\"rdp-ebb-cite_ref-RodriguezManaging13_9-5\" class=\"reference\"><a href=\"#cite_note-RodriguezManaging13-9\">[9]<\/a><\/sup> also analysed data errors in sensor networks, in particular in environmental monitoring systems. In this paper, authors address the problem of uncertainty of data coming from sensors with an approach dedicated to providing environmental monitoring applications and users with data quality information. Badawy <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BadawyAuto18_26-0\" class=\"reference\"><a href=\"#cite_note-BadawyAuto18-26\">[26]<\/a><\/sup> combined parametric and non-parametric signal processing and machine learning algorithms for automating sensor data quality control, which can identify those parts of the sensor data that are sufficiently reliable for further analysis and discards useless data.\n<\/p><p>Another research subarea of DQ in sensor networks is the DQ management in sensor data streams. Klein <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-KleinHowTo09_10-2\" class=\"reference\"><a href=\"#cite_note-KleinHowTo09-10\">[10]<\/a><\/sup> presented a quality-driven load shedding approach that screens the data stream to find and discard data items of minor quality. Thus, DQ of stream processing results is maximized under adverse conditions such as data overload. Campbell <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CampbellQuant13_15-1\" class=\"reference\"><a href=\"#cite_note-CampbellQuant13-15\">[15]<\/a><\/sup> advocated for automated quality assurance and quality control procedures based on graphical and statistical summaries for review and track the provenance of the data in environmental sensor streams.\n<\/p><p>Other works focus on data management from different viewpoints. For example, Al-Ruithe <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Al-RuitheData16_27-0\" class=\"reference\"><a href=\"#cite_note-Al-RuitheData16-27\">[27]<\/a><\/sup> detailed roles, responsibilities, and policies in the context of IoT-cloud converged environments and provide a generic framework for data governance and security. Similarly, Qin <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-QinQuality14_14-1\" class=\"reference\"><a href=\"#cite_note-QinQuality14-14\">[14]<\/a><\/sup> provided a data management perspective on large-scale sensor environment applications posing non-functional requirements to meet the underlying timeliness, reliability, and accuracy needs in addition to the functional needs of data collection. Although all these approaches are interesting and provide a useful vision, they still do not address how to make available (e.g., institutionalize) best practices in data quality management to the entire organization. Such an approach has been proven to be more efficient when it comes to creating an organizational data quality culture. This vision is specifically important in the case of IoT, since the SCP operations can be executed across different networks belonging to different departments or even organizations. From our point of view, this is a critical aspect for IoT that must be covered in a holistic way.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_quality_methodologies_comparison\">Data quality methodologies comparison<\/span><\/h3>\n<p>There are some methodologies that can be used as drivers for assessing and managing DQ. First, Lee <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-LeeAIMQ02_28-0\" class=\"reference\"><a href=\"#cite_note-LeeAIMQ02-28\">[28]<\/a><\/sup> proposed AIMQ as a methodology that encompasses a model of data quality, a questionnaire to measure DQ, and analysis techniques for interpreting the DQ measures. This methodology is mainly used to analyse the gap between an organization and best practices, as well as to assess gaps between information systems professionals and data consumers. The application of this methodology is useful for determining the best area for DQ improvement activities. This methodology has not been widely used and in a greater extent it has been considered to be too theoretical and dependent on the domain. McGilvray<sup id=\"rdp-ebb-cite_ref-McGilvrayExecuting08_29-0\" class=\"reference\"><a href=\"#cite_note-McGilvrayExecuting08-29\">[29]<\/a><\/sup> provides a practical approach for planning and managing information quality. In comparison with the methodology proposed by Lee <i>et al.<\/i>, McGilvray provides a more pragmatic and practical approach to achieving the desired state of DQ within an organization. However, this methodology is still dependent on the domain of application. ISO\/TS 8000-150:2011<sup id=\"rdp-ebb-cite_ref-ISO_8000_30-0\" class=\"reference\"><a href=\"#cite_note-ISO_8000-30\">[30]<\/a><\/sup> \u201cspecifies fundamental principles of master data quality management, and requirements for implementation, data exchange and provenance.\u201d This standard constitutes an informative framework that identifies processes for DQ management. This framework could be used in conjunction with, or independently of, quality management systems standards, for example, ISO 9001.<sup id=\"rdp-ebb-cite_ref-ISO_9001_31-0\" class=\"reference\"><a href=\"#cite_note-ISO_9001-31\">[31]<\/a><\/sup>\n<\/p><p>Batini <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BatiniMethod09_32-0\" class=\"reference\"><a href=\"#cite_note-BatiniMethod09-32\">[32]<\/a><\/sup> provided a literature review about different methodologies for data quality assessment and improvement. Most of the methods and techniques included in such review cannot be considered as DQ management methodologies since they do not consider all the managerial concerns in a holistic manner. On the contrary, most of these methods are focused on DQ assessment or improvement in isolation. Similar to the mentioned review, a more recent study developed by Woodall <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-WoodallAClass14_33-0\" class=\"reference\"><a href=\"#cite_note-WoodallAClass14-33\">[33]<\/a><\/sup> classified most recent DQ assessment and improvement methods. This work suffers the same problem that the work of Batini <i>et al.<\/i> does. Apart from these examples, there is a lack of comprehensive methodologies for the assessment and improvement of DQ in the domain of SCP operations and their underlaying data.\n<\/p><p>The recent standard ISO 8000-61<sup id=\"rdp-ebb-cite_ref-ISO_8000-61_17-1\" class=\"reference\"><a href=\"#cite_note-ISO_8000-61-17\">[17]<\/a><\/sup> provides a set of standard guidelines for managing DQ in a holistic way, which can be tailored for different domains. However, its main purpose is not to serve as a methodology for DQ management per se, but it simply provides a process reference model. In this sense, the standard is more descriptive than operative, what makes it not usable out-of-the-box. Aligned with this standard, this paper proposes the DAQUA-MASS methodology to deal with DQ in SCP environments. The main contribution of the DAQUA-MASS methodology is that it takes the standard best practices for depicting an operative way to manage DQ (as depicted in the processes of ISO 8000-61) and tailors these to the particular domain of SCP environments, and in particular, in sensor-related data.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"DAQUA-Model:_A_data_quality_model\">DAQUA-Model: A data quality model<\/span><\/h2>\n<p>Reviewing the literature, it is possible to find that the concept of \"data quality\" has been defined in different ways. The widest used definitions are aligned with the concept of \u201cfitness for use.\u201d Strong <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-StrongData97_34-0\" class=\"reference\"><a href=\"#cite_note-StrongData97-34\">[34]<\/a><\/sup> define data quality as \"data that is fit for use by data consumer. This means that usefulness and usability are important aspects of quality\u201d. Different stakeholders can have different perceptions of what quality means for the same data.<sup id=\"rdp-ebb-cite_ref-WangAProd98_35-0\" class=\"reference\"><a href=\"#cite_note-WangAProd98-35\">[35]<\/a><\/sup> It largely depends on the context in which data is used. Thus, DQ in IoT environments must be adequately managed considering the very nature of the IoT systems. Typically, to improve data quality, a Plan-Do-Check-Act (PDCA) cycle specifically tailored for the context of usage is followed. In this sense, we think that adopting the processes of ISO 8000-61 which are deployed in the PDCA<sup id=\"rdp-ebb-cite_ref-SrivannaboonAchiev09_36-0\" class=\"reference\"><a href=\"#cite_note-SrivannaboonAchiev09-36\">[36]<\/a><\/sup> order can largely help to provide a methodology for managing data quality in IoT environments. At the core of the PDCA cycle for IoT environments is the identification of a Data Quality Model (DQModel) which, being composed of several data quality characteristics suitable for the problem, is used to identify and represent the data quality requirements required in the context.<sup id=\"rdp-ebb-cite_ref-RodriguezManaging13_9-6\" class=\"reference\"><a href=\"#cite_note-RodriguezManaging13-9\">[9]<\/a><\/sup>\n<\/p><p>In our case, and according to our philosophy of aligning with international standards, the DQ Model proposed is a specialization of the DQ Model introduced in ISO\/IEC 25012.<sup id=\"rdp-ebb-cite_ref-ISO_25012_25-3\" class=\"reference\"><a href=\"#cite_note-ISO_25012-25\">[25]<\/a><\/sup> This DQ Model is widely accepted and used in the industry. Nevertheless, it has not been specifically developed for considering SCP aspects. In fact, the scope section of such standard literally states that it \u201cdoes not include data produced by embedded devices or real time sensors that are not retained for further processing or historical purposes.\u201d<sup id=\"rdp-ebb-cite_ref-ISO_25012_25-4\" class=\"reference\"><a href=\"#cite_note-ISO_25012-25\">[25]<\/a><\/sup> Therefore, in order to complement the standard, we provide some orientation in this paper on how to specifically use ISO\/IEC 25012 in the context of SCP environments.\n<\/p><p>The DQ Model focuses on the quality of the data as part of an information system and defines quality characteristics for target data used by humans and systems (i.e., the data that the organization decides to analyse and validate through the model). This model categorizes quality attributes into 15 characteristics and considers three perspectives: inherent, system-dependent, and both jointly (see crosses in Table 4).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab4_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"1db8d5212c600023ec2fc714dbb09929\"><img alt=\"Tab4 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/0\/03\/Tab4_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 4.<\/b> DQ Characteristics in ISO\/IEC 25012 that can be affected by sensor data errors<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As part of the methodology, we propose the association of the DQ characteristics with some of the sensor data errors previously shown in Table 2. These relationships suggest that erroneous data generated due to some malfunctioning sensors could lead or affect some of the DQ characteristics. We distinguish these relationships in Table 4 with \u2018P\u2019 and \u2018S\u2019 respectively signifying that a sensor error can have a primary or secondary impact in DQ characteristics. Primary means a direct (or more critical) effect, while secondary is a collide (or less critical) impact.\n<\/p><p>In the following paragraphs, we introduce and summarize the fifteen DQ characteristics and, as a main contribution in this part, we provide a vision on how these DQ characteristics are tailored for SCPs operations. Along with the definitions, for the sake of understandability an example is also provided for each characteristic. These examples are specially provided for the interpretation of these DQ characteristics in the acquisition layer, which is the scope of this paper. However, it must be noted that all these DQ characteristics can be considered for all the SCP layers with different ways of assessment and interpretation.\n<\/p><p>From the point of view of data quality management, it is quite important not to make the mistake of confusing the readings of dysfunctional sensors with inadequate levels of data quality, even when a dysfunctional sensor can produce data without having adequate levels of quality (i.e., not fitting the purpose of the use of data): the reason for making this distinction is that fixing errors due to dysfunctional sensors requires first fixing the sensors; on the other hand, if one can assure that the root cause is not grounded on a dysfunctional sensor, but on the data itself, then, to fix data quality errors, then data quality management techniques should be used since it should not be ignored what data means. The description of the data quality characteristics can be found in the following definitions:\n<\/p>\n<dl><dt>Accuracy <\/dt>\n<dd>The degree to which data has attributes that correctly represent the true value of the intended attribute of a concept or event in a specific context of use - In SCP environments, a low degree of accuracy could be derived from devices that provide values that could differ from the value on the real world. For example, a low degree of accuracy can be when a humidity sensor reads a value of 30% and the real value is 50%. Low levels of accuracy could be directly related to sensor errors such as constant or offset, outlier errors and noise errors. Also, accuracy could be indirectly affected by continuous varying or drifting and trimming error (see Table 4).<\/dd><\/dl>\n<dl><dt>Completeness <\/dt>\n<dd>The degree to which subject data associated with an entity has values for all expected attributes and related entity instances in a specific context of use - In SCP environments, a low degree of completeness could be derived from devices that are reading and sending no values. For example, a low degree of completeness can happen when the records of sensor data have missing values. Low levels of completeness could be related directly to sensor errors such as crash or jammed errors and indirectly, trimming and noise errors (see Table 4).<\/dd><\/dl>\n<dl><dt>Consistency <\/dt>\n<dd>The degree to which data has attributes that are free from contradiction and are coherent with other data in a specific context of use: either or both among data regarding one entity and across similar data for comparable entities - In SCP environments, a low degree of consistency could happen when two sensors produce contradictory data. For example, for proximity sensors that provide the relative distance to the sensor position, a consistency problem for a single sensor could be a negative distance value, while a consistency problem between two sensors in the same position could be two different values. Thus, low levels of consistency could be related with continuous varying\/drifting error and indirectly with constant or offset errors, trimming and noise error (see Table 4).<\/dd><\/dl>\n<dl><dt>Credibility <\/dt>\n<dd>The degree to which data has attributes that are regarded as true and believable by users in a specific context of use - In SCP environments, a low degree of credibility could be derived from a single sensor placed somewhere and the data cannot be validated by another entity or even another sensor. For example, a credibility issue could happen when a sensor whose data is compared with another sensor placed nearby does not match. Low levels of credibility could be related directly to sensor errors such as outlier errors and indirectly with constant or offset error, continuous varying\/drifting error, and noise error (see Table 4).<\/dd><\/dl>\n<dl><dt>Currentness <\/dt>\n<dd>The degree to which data has attributes that are of the right age in a specific context of use - In SCP environments, a low degree of currentness could be derived from a sensor that can be indicating past values as current value (see Table 4). For example, if an irrigation sensor produces a value that indicates that the field must be irrigated, it has been irrigated and the data is not updated. The data indicates whether it is necessary irrigation, or it is already irrigated, so this would be data without sufficient currentness.<\/dd><\/dl>\n<dl><dt>Accessibility <\/dt>\n<dd>The degree to which data can be accessed in a specific context of use, particularly by people who need supporting technology or special configuration because of some disability - In SCP environments, a low degree of accessibility could be derived due to the fact the necessary user is not allowed in at the precise moment. For example, data produced by a specific sensor is unreachable due to network issues.<\/dd><\/dl>\n<dl><dt>Compliance <\/dt>\n<dd>The degree to which data has attributes that adhere to standards, conventions, or regulations in force and similar rules relating to data quality in a specific context of use - In SCP environments, a low degree of compliance could be derived from a data sensor that is not using the standard formats established by the organization. For example, if the organization establishes that for distance sensors the unit for values is meters, and if some sensors produce values expressed in meters and other in miles, these data have low compliance levels.<\/dd><\/dl>\n<dl><dt>Confidentiality <\/dt>\n<dd>The degree to which data has attributes that ensure that it is only accessible and interpretable by authorized users in a specific context of use - In SCP environments, a low degree of confidentiality could be derived from inefficient security management of sensor data. For example, a confidentiality leak might happen when data produced by a sensor placed in a nuclear power plant can be freely accessed from external networks even when this data was marked as sensible and, therefore, confidential in order to prevent possible terrorist acts.<\/dd><\/dl>\n<dl><dt>Efficiency <\/dt>\n<dd>The degree to which data has attributes that can be processed and provide the expected levels of performance by using the appropriate amounts and types of resources in a specific context of use - For example, a sensor send data about where is placed and send a code and a description every time the sensor sends o stores a record, it has low efficiency because only the code is enough to know all about the place. In SCP environments, a low degree of efficiency could be derived from the storage of duplicated data that could take more time and resources to send or manipulate the data.<\/dd><\/dl>\n<dl><dt>Precision <\/dt>\n<dd>The degree to which data has attributes that are exact or that provide discrimination in a specific context of use - In SCP environments, a low degree of precision could be derived from devices that are providing inexact values as in the next example. For example, sensor data that store weight with no decimals and it is required a minimum of three decimals. Low levels of consistency could be related directly with trimming errors, and indirectly with noise errors (see Table 4).<\/dd><\/dl>\n<dl><dt>Traceability <\/dt>\n<dd>The degree to which data has attributes that provide an <a href=\"https:\/\/www.limswiki.org\/index.php\/Audit_trail\" title=\"Audit trail\" class=\"wiki-link\" data-key=\"96a617b543c5b2f26617288ba923c0f0\">audit trail<\/a> of access to the data and of any changes made to the data in a specific context of use - In SCP environments, a low degree of traceability could be derived from sensor data with no metadata. For example, data logs contain information about who has acceded to sensor data and operations made with them. Low levels of traceability could be related indirectly to crash or jammed errors as well as to temporal delay errors (see Table 4).<\/dd><\/dl>\n<dl><dt>Understandability <\/dt>\n<dd>The degree to which data has attributes that enable it to be read and interpreted by users, and are expressed in appropriate languages, symbols, and units in a specific context of use - In SCP environments, a low degree of understandability could be derived from sensor data represented with codes instead of acronyms. For example, records of data about temperature on a car has an attribute to know the place of the sensor in the car. This attribute can be stored as a code like \u201cxkq1,\u201d but if it is stored as \u201cGasolineTank,\u201d it is supposed to have a higher level of understandability.<\/dd><\/dl>\n<dl><dt>Availability <\/dt>\n<dd>The degree to which data has attributes that enable it to be retrieved by authorized users and\/or applications in a specific context of use - In SCP environments, a low degree of availability could be derived from the insufficient resources of the system in which sensor data is stored. For example, to assure sensor data availability, sensor replication can be used to make it available even if there is some issue on a sensor. Low levels of availability could be related indirectly with temporal delay error and crash or jammed errors (see Table 4).<\/dd><\/dl>\n<dl><dt>Portability <\/dt>\n<dd>The degree to which data has attributes that enable it to be installed, replaced, or moved from one system to another preserving the existing quality in a specific context of use - For example, sensor data is going to be shared with a concrete system or even other organization or department, data loss can occur. If this happens, for example, due to a data model mismatching or a problem with the data format, the reason is directly related to portability of data. In SCP environments, a low degree of portability could be derived from sensor data that does not follow a specific data model (see Table 4) or the format present some problems.<\/dd><\/dl>\n<dl><dt>Recoverability <\/dt>\n<dd>The degree to which data has attributes that enable it to maintain and preserve a specified level of operations and quality, even in the event of failure, in a specific context of use. In SCP environments, a low degree of recoverability could be derived from devices that do not have a mechanism failure tolerant or backup. For example, when a device has a failure, data stored in that device should be recoverable. Low levels of recoverability could be related indirectly with temporal delay error and crash or jammed errors (see Table 4).<\/dd><\/dl>\n<p>Although our DQ Model considers initially all the DQ characteristics defined in ISO\/IEC 25012, it could be necessary to customize the DQ characteristics chosen to adapt them into the specific SCP context. This customization might depend on the concrete organization and how it applies the methodology to specific SCP contexts. The customized model will conform the data quality model for an organization with a specific SCP environment.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"DAQUA-MASS:_A_data_quality_management_methodology_for_data_sensors\">DAQUA-MASS: A data quality management methodology for data sensors<\/span><\/h2>\n<p>This section presents DAQUA-MASS, an ISO 8000-61-based Data Quality Management methodology for data sensors. Steps provided in this methodology are based in some of the processes introduced by ISO 8000-61<sup id=\"rdp-ebb-cite_ref-ISO_8000-61_17-2\" class=\"reference\"><a href=\"#cite_note-ISO_8000-61-17\">[17]<\/a><\/sup>, which, as we previously said, gathers the best practices around data quality management by means of a process approach. Each process in this standard is described by means of a purpose, the outcomes, and the activities that are to be applied for the assurance of data quality. Thus, this standard mainly covers: (1) fundamental principles of data quality management; (2) the structure of the data quality management process; (3) definitions of the lower level processes for data quality management; (4) the relationship between data quality management and data governance; and (5) implementation requirements. This standard is used along with ISO 8000-62 to assess and improve organizational maturity when it comes to data quality management. However, in this paper, we consider the processes in ISO 8000-61 to rigorously depict DAQUA-MASS.\n<\/p><p>Steps of DAQUA-MASS are grouped in four phases according to the Plan-Do-Check-Act cycle, as it is done in 8000-61. PDCA is implicitly a closed loop, signifying that the process is iterative, with the last phase of every iteration providing feedback on starting a new iteration. Many acknowledged models, such as the in IoT information fusion<sup id=\"rdp-ebb-cite_ref-ChenInfo14_37-0\" class=\"reference\"><a href=\"#cite_note-ChenInfo14-37\">[37]<\/a><\/sup> or the JDL model<sup id=\"rdp-ebb-cite_ref-BlaschRevisit13_38-0\" class=\"reference\"><a href=\"#cite_note-BlaschRevisit13-38\">[38]<\/a><\/sup>, are all closed loops. As such, they can be self-improved to adjust the dynamic world to maximize the performance. The IoT is a dynamic system, and data quality management can be adapted to the changes in every loop of the methodology. Each loop serves to adapt to changes and new quality needs that may arise. The methodology is designed so that the complete cycle is iteratively executed depending on the goals, needs, and resources of the organization. The PDCA cycle will contribute to more effective and efficient data quality and consists of seven steps grouped in the following four phases:\n<\/p>\n<ul><li> The \"plan\" phase establishes the strategy and the data quality improvement implementation plan as necessary to deliver results in accordance with data requirements;<\/li>\n<li> The \"do\" phase executes the data quality improvement implementation plan;<\/li>\n<li> The \"check\" phase monitors data quality and process performance against the strategy and data requirements and reports the results to validate the efficiency of the corrective actions; and finally<\/li>\n<li> The \"act\" phase takes actions to continually improve process performance.<\/li><\/ul>\n<p>Figure 2 summarizes the step flow throughout the four phases. Every step is defined with a general description and set of concrete activities. Following sections present all the methodology steps grouped in the mentioned Plan-Do-Check-Act cycle. The definition for all the steps provides a table depicting all the expected input and output generated by each activity in the step as well as a RACIQ matrix (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer) which indicates the role of stakeholders involved in the activity.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"f29fd7885c1f9048c9dc135d72dd866d\"><img alt=\"Fig2 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/4\/41\/Fig2_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Methodology DAQUA-MASS phases and steps<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We consider the following list of stakeholders to be involved in the application of this methodology. It should be noted that there are alternative, similar (even overlapped) role definitions in the context of data quality management and SCP environments. However, we consider these to be the most common and well-known stakeholders, and other that are not mentioned can be easily assimilated to one of the proposed.\n<\/p>\n<ul><li> Chief information officer (CIO): The most senior executive in an enterprise responsible for the traditional information technology and computer systems that support enterprise goals (executive level individual)<\/li>\n<li> Chief data officer (CDO): A corporate officer responsible for enterprise wide data governance and utilization of data as an asset, via data processing, analysis, data mining, information trading, and other means (executive level individual)<\/li>\n<li> Data governance manager (DGM): Oversees the enterprise data governance program development and is responsible for developing certain solutions and frameworks (strategic level individual)<\/li>\n<li> Data quality steward for SCP domain (SCP DQ steward): A DQ steward at the tactical level for the area of SCP environments by considering its implications in DQ<\/li>\n<li> Data quality steward (DQ Ssteward): Responsible for utilizing an organization\u2019s data quality governance processes to ensure fitness of data elements, including both the content and metadata (operational level individual) - Data stewards have a specialist role that incorporates processes, policies, guidelines, and responsibilities for administering organizations\u2019 entire data in compliance with DQ policies and procedures<\/li>\n<li> SCP technical architect (SCP arch): An SCP (and in general IoT) expert who makes high-level design choices and dictates technical standards, including SCP technology standards, tools, and platforms<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"The_.22plan.22_phase\">The \"plan\" phase<\/span><\/h3>\n<p>The data quality planning phase establishes data requirements and objectives for data quality, creating plans to achieve the objectives and evaluating the performance of the plans. These plans balance current data quality levels, cost, resources, and capabilities across the organization for the assessment of data quality of sensor data. This phase is initiated based on needs and expectations of stakeholders or the feedback of the process improvements performed during data quality improvement (the act phase). The expected result of the execution of this phase is the complete planification for adequately collect sensor data. This includes requirements management, definition of policies for data, and device lifecycle management; it is important to highlight the definition of policies to retain the original unmanipulated data and all versions of the input data. Table 5 summarizes inputs, outputs, and responsibility distribution for this phase. Afterwards, a description of the steps grounded on the corresponding ISO 8000-61 process is introduced. Again, keep in mind that the scope of the methodology is on sensor data (see Figure 1).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab5_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"bef87bc43e216b4c28eb8e9c5d506e0d\"><img alt=\"Tab5 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/6\/65\/Tab5_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 5.<\/b> Inputs, outputs, and RACIQ matrix for the Plan phase (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"P1._Characterization_of_the_current_state_of_the_sensor_data_contexts\">P1. Characterization of the current state of the sensor data contexts<\/span><\/h4>\n<p>This first step is devoted to characterizing the current state of the SCP environments and all the specific sensor data contexts. This step has the following activities:\n<\/p>\n<dl><dd><b>P1-1.<\/b> Sensor and sensor data lifecycle management specification - The lifecycle of sensors<sup id=\"rdp-ebb-cite_ref-WeberInternet13_2-1\" class=\"reference\"><a href=\"#cite_note-WeberInternet13-2\">[2]<\/a><\/sup> and sensor data<sup id=\"rdp-ebb-cite_ref-PastorelloGerencia08_39-0\" class=\"reference\"><a href=\"#cite_note-PastorelloGerencia08-39\">[39]<\/a><\/sup> have to be defined so that it can be managed. For sensors, the lifecycle starts when the device is obtained, and the last step is when it is necessary to replace o remove the sensor because it became useless. For sensor data, the lifecycle starts when sensors produce the data and ends when the data is eliminated. The end of the sensor data life cycle should not be confused with the moment in which the data goes from being operational to historical data. Operational sensor data is used on a day-to-day basis, but when data is no longer useful on a day-to-day basis due to its age, these data are separated and stored in another database and become historical data. It is important to highlight that sensor and sensor data lifecycle is used to better contextualize the environment of the data.<\/dd><\/dl>\n<dl><dd><b>P1-2.<\/b> Management of quality policies, standards, and procedures for data quality management - Specify fundamental intentions and rules for data quality management. Ensure the data quality policies, standards, and procedures are appropriate for the data quality strategy, comply with data requirements, and establish the foundation for continual improvement of the effectiveness and efficiency of data quality management in all the key SCP operations. The purpose of data quality policy\/standards\/procedures management is to capture rules that apply to performing data quality control, data quality assurance, data quality improvement, data-related support, and resource provision consistently. Before the implementation and definition of the complete plan, the policies, standards, and procedures should be defined in order to define the implementation plan based on them.<\/dd><\/dl>\n<dl><dd><b>P1-3.<\/b> Provision of sensor data and work specifications - Developing specifications that describe characteristics of data and work instructions for smart connected products enables data processing and data quality monitoring and control. To support the description of the provision of the sensor data work specifications, some metadata must be provided. Metadata can help with one of the biggest problems of SCP: interoperability. Interoperability refers to the ability for one or more smart connected products to communicate and exchange information.<sup id=\"rdp-ebb-cite_ref-CampbellQuant13_15-2\" class=\"reference\"><a href=\"#cite_note-CampbellQuant13-15\">[15]<\/a><\/sup><\/dd><\/dl>\n<dl><dd><b>P1-4.<\/b> Identification, prioritization, and validation of sensor data requirements - Collect the needs and expectations related to sensor data from devices and stakeholders. Then, it is translated by identification, prioritization, and validation of data requirements. The purpose of requirements management is to establish the basis for creating or for refining a data quality strategy for SCP environments aligned to the needs and expectations of stakeholders. It is important to have well-defined and -implemented quality sensor data requirements to avoid problems from the start and to facilitate the collection and integration of sensor data.<\/dd><\/dl>\n<h4><span class=\"mw-headline\" id=\"P2._Assessment_of_the_current_state_of_the_levels_of_data_quality\">P2. Assessment of the current state of the levels of data quality<\/span><\/h4>\n<p>This step aims to evaluate the current state of the data quality levels in all the surrounding SCP environments. This step has the following activities:\n<\/p>\n<dl><dd><b>P2-1.<\/b> Identification of the data quality characteristics representing quality requirements and determination and development of metrics and measurement methods - Develop or select the measurement indicators, corresponding metrics, and measurement methods used to measure the quality levels of data produced by devices during all the SCP operations.<\/dd><\/dl>\n<dl><dd><b>P2-2.<\/b> Measurement and analysis of data quality levels - Measure the data quality levels by implementing the measurement plans and determining the measurement results. This means the application of the measurement algorithms defined and developed on the previous step (P2-1). Such algorithms strongly depend on the IT infrastructure landscape of every organization. Thereby, every organization often develop their own algorithms or use commercial solutions which are compliant with their own infrastructure. Merino <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MerinoAData16_7-1\" class=\"reference\"><a href=\"#cite_note-MerinoAData16-7\">[7]<\/a><\/sup> presents algorithms and metrics for each data quality characteristic. After data quality levels have been measured, these can be quantitatively analysed to extract insights about the SCP environment being managed. As a result, a list of non-conformities can be elaborated in the next step, with which to make informed decisions.<\/dd><\/dl>\n<h4><span class=\"mw-headline\" id=\"P3._Data_quality_enhancement_plan_definition\">P3. Data quality enhancement plan definition<\/span><\/h4>\n<p>Data quality implementation planning identifies the resources and sequencing by which to perform DQ control, DQ Assurance, DQ Improvement, data-related support, and resource provision across the organization. At this point, after the collection of sensor data requirements and the definition of standards and policies, it is important to configure the devices to implement the plan and place each node following a defined strategy. Optimal node placement allows to obtain the optimal value through data. This step considers the following activities:\n<\/p>\n<dl><dd><b>P3-1.<\/b> Analysis of root causes of data non-conformities - Analyzing the root causes of each data quality issue and assess the effect of the issue on business processes. The purpose of root cause analysis and solution development of non-solved data quality non-conformities is to establish, in accordance with the data quality strategy and with the priorities identified by data quality assurance, the basis on which to perform data cleansing and\/or process improvement for data non-conformity prevention.<\/dd><\/dl>\n<dl><dd><b>P3-2.<\/b> Data quality risk assessment - Identify risks throughout the data life cycle, analyze the impact if each risk were to occur, and determine risk priorities to establish the basis for monitoring and control of processes and data. The purpose of data quality monitoring and control is, by following applicable work instructions, to identify and respond when data processing fails to deliver data that meet the requirements in the corresponding data specification. It allows for the control of what is happening on data or even on the SCP environment. For example, with the control of sensor data quality, it is possible to identify certain issues on devices. Data is often the most important thing to protect, because although the direct cost of losing it may be small compared with research data or intellectual property. If a sensor is not functioning any more for some reason, it can be replaced, but if a data loss incident occurs, it is difficult to recover from it or even impossible to recover. This can bring not only data loss but also be the root cause of other problems. In conclusion, data quality risks should be identified in order to avoid them. As Karkouch <i>et al.<\/i> note<sup id=\"rdp-ebb-cite_ref-KarkouchData16_6-8\" class=\"reference\"><a href=\"#cite_note-KarkouchData16-6\">[6]<\/a><\/sup>, the main factors affecting DQ in SCP or IoT environments are: deployment scale, resources constraints, network, sensors (as physical devices), environment, vandalism, fail-dirty, privacy preservation, security vulnerability, and data stream processing.<\/dd><\/dl>\n<dl><dd><b>P3-3.<\/b> Development of improvement solutions to eliminate the root causes - Propose solutions to eliminate the root causes and prevent recurrence of non-conformities. Evaluate the feasibility of the proposed improvements through cost benefit analysis.<\/dd><\/dl>\n<dl><dd><b>P3-4.<\/b> Definition of improvement targets - The purpose of this activity is to analyze possible improvement areas according to the business processes, risk catalog, and the data quality strategy; then it selects those that are more aligned with the data quality strategy and\/or are able to lead to greater data quality enhancement regarding previous detected risks. Specific areas or sub-nets of devices in the organization\u2019s SCP environments could also serve as a criterion to determine specific improvement targets.<\/dd><\/dl>\n<dl><dd><b>P3-5.<\/b> Establishment of the data quality enhancement plan - Define the scope and target of data quality and prepare detailed implementation plans, defining and allocating the resources needed.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"The_.22do.22_phase\">The \"do\" phase<\/span><\/h3>\n<p>This phase consists of the data quality control that is carried out based on the implementation plan established in data quality planning (see the \"plan\" phase). The phase, when completed, delivers data that meet the specified requirements. This phase involves creating, using, and updating data according to specified work instructions and monitoring quality by checking whether the data conform to predetermined specifications.\n<\/p><p>In this phase, it is important to address aspects such as the provision of complete metadata; the use of flags to convey information about data; the documentation of all sensor data processing; monitoring and control of data; maintain appropriate levels of human inspection; perform range, domain, and slope checks; implement an automated alert system for sensor network issues; or automate sensor data quality procedures. Table 6 summarizes inputs, outputs, and responsibility distribution for this phase.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab6_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"f3fec5d1e185c9a7a8e9b326817b19aa\"><img alt=\"Tab6 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/4\/40\/Tab6_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 6.<\/b> Inputs, outputs and RACIQ matrix for the Do phase. (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"D1._Execution_of_the_data_quality_improvement_plan\">D1. Execution of the data quality improvement plan<\/span><\/h4>\n<p>The purpose of provision of data specifications and work instructions is to establish the basis on which to perform data processing and data quality monitoring and control, taking account of the outcomes of the data quality planning, data-related support, and the resource provision processes. Having the plan obtained for the first phase, the first step is to provide data and work specification to collect data from devices on the defined way. This step considers the following activities:\n<\/p>\n<dl><dd><b>D1-1.<\/b> Establish flags to convey information about the sensor data - Flags or qualifiers convey information about individual data values, typically using codes that are stored in a separate field to correspond with each value. Flags can be highly specific to individual studies and data sets or standardized across all data.<\/dd><\/dl>\n<dl><dd><b>D1-2.<\/b> Definition of the optimal node placement plan - It is a challenging problem that has been proven to be NP-hard (non-deterministic polynomial-time hardness) for most of the formulations of sensor deployment.<sup id=\"rdp-ebb-cite_ref-CerpaASCENT04_40-0\" class=\"reference\"><a href=\"#cite_note-CerpaASCENT04-40\">[40]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChengRelay08_41-0\" class=\"reference\"><a href=\"#cite_note-ChengRelay08-41\">[41]<\/a><\/sup> To tackle such complexity, several heuristics have been proposed to find sub-optimal solutions.<sup id=\"rdp-ebb-cite_ref-DhillonSensor03_42-0\" class=\"reference\"><a href=\"#cite_note-DhillonSensor03-42\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PanOptimal05_43-0\" class=\"reference\"><a href=\"#cite_note-PanOptimal05-43\">[43]<\/a><\/sup> However, the context of these optimization strategies is mainly static in the sense that assessing the quality of candidate positions is based on a structural quality metric such as distance, network connectivity, and\/or basing the analysis on a fixed topology. Also, application-level interest can vary over time, and the available network resources may change as new nodes join the network, or as existing nodes run out of energy.<sup id=\"rdp-ebb-cite_ref-WuSMART05_44-0\" class=\"reference\"><a href=\"#cite_note-WuSMART05-44\">[44]<\/a><\/sup> Also, if we talk about node placement, we must also talk about node or sensor replication. Replicating data sensors is important for purposes of high availability and disaster recovery. Also, replication of this data on cloud storage needs to be implemented efficiently. Archiving is one way to recover lost or damaged data in primary storage space, but replicas of data repositories that are updated concurrently with the primary repositories can be used for sensitive systems with strong data availability requirements. Replication can be demanding in terms of storage and may degrade performance due to if a concurrent updates strategy is enforced.<\/dd><\/dl>\n<dl><dd><b>D1-3.<\/b> Redesign the software or hardware that includes sensors to eliminate root causes - It is an alternative process to the optimal replacement of sensors. For example, redesign and reimplementing a fragment of a SCP\u2019s firmware could improve fault tolerance.<\/dd><\/dl>\n<dl><dd><b>D1-4.<\/b> Data cleansing - The purpose of data cleansing is to ensure, in response to the results of root cause analysis and solution development, the organization can access data sets that contain no non-conformities capable of causing unacceptable disruption to the effectiveness and efficiency of decision making using those data. Also, the non-conformities are corrected implementing developed solutions and make a record of the corrections.<\/dd><\/dl>\n<dl><dd><b>D1-5.<\/b> Force an appropriate level of human inspection - If performed by trained and experienced technicians, visual inspection is used to monitor the state of industrial equipment and to identify necessary repairs. There are also technologies used to assist with fault recognition or even to automate inspections. Shop floor workers are made responsible for basic maintenance including cleaning machines, visual inspection, and initial detection of machine degradation.<\/dd><\/dl>\n<dl><dd><b>D1-6.<\/b> Implement an automated alert system to warn about potential sensor data quality issues - Having only human inspection can be a complex task for maintenance staff. It is necessary to implement a system in which some key indicators are constantly reading if sensor status is correct or not. If a sensor is not working properly, the root cause can be due to a hardware, network, or software failure and affects data quality. Furthermore, an automated data quality procedure might identify anomalous spikes in the data and flag them. However, it is almost always necessary to have human supervision, intervention, and inspection, as stated by Peppeler <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-PepplerAnOver08_45-0\" class=\"reference\"><a href=\"#cite_note-PepplerAnOver08-45\">[45]<\/a><\/sup> and <sup id=\"rdp-ebb-cite_ref-FiebrichTheValue06_46-0\" class=\"reference\"><a href=\"#cite_note-FiebrichTheValue06-46\">[46]<\/a><\/sup>; the inclusion of automated quality is often an improvement, because it ensures consistency and reduces human bias. Automated data quality procedures are also more efficient at handling the vast quantities of data that are being generated by streaming sensor networks and reduces the amount of human inspection required.<\/dd><\/dl>\n<dl><dd><b>D1-7.<\/b> Schedule sensor maintenance to minimize data quality issues - Sensors require routine maintenance and scheduled calibration that, in some cases, can be done only by the manufacturer. Ideally, maintenance and repairs are scheduled to minimize data loss (e.g., snow-depth sensors repaired during the summer) or staggered in such a way that data from a nearby sensor can be used to fill gaps. In cases in which unscheduled maintenance is required, stocking replacement parts on-site ensures that any part of the network can be replaced immediately.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"The_.22check.22_phase\">The \"check\" phase<\/span><\/h3>\n<p>The check phase consists of the data quality assurance, which measures data quality levels and methodology steps' performance related to data non-conformities or other issues that have arisen as a result of data quality planning (see the \"plan\" phase) or data quality control (see the \"do\" phase). This measurement provides evidence by which to evaluate the impact of any identified poor levels of data quality on the effectiveness and efficiency of business processes. It is important during this phase to address aspects like scheduled sensor maintenance and repairs to minimize data loss, as well as measurement and evaluation of sensor data quality. Table 7 summarizes inputs, outputs, and responsibility distribution for this phase.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab7_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"849c54a0def523720025c7be1a18e4cb\"><img alt=\"Tab7 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/0\/0f\/Tab7_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 7.<\/b> Inputs, outputs, and RACIQ matrix for the Check phase (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"C1._Testing_the_efficiency_of_the_corrective_actions\">C1. Testing the efficiency of the corrective actions<\/span><\/h4>\n<p>The purpose of measurement of data quality is the provision of measurement criteria, to generate input for the evaluation of measurement results. This step considers the following activities:\n<\/p>\n<dl><dd><b>C1-1.<\/b> Monitoring and control of the enhanced data - According to the identified risk priorities, monitor and measure conformity of data to the applicable specification. Monitoring and measuring takes place either at intervals or continuously and in accordance with applicable work instructions. This work instructions can be: perform range checks on numerical data, perform domain checks on categorical data, and perform slope and persistence checks on continuous data streams. If data non-conformities are found, then correct the data when viable and distribute to stakeholders a record of the viability and degree of success for each corrective action.<\/dd><\/dl>\n<dl><dd><b>C1-2.<\/b> Definition of an interstice comparison plan - Create policies for comparing data with data from related sensors. If no replicate sensors exist, interstice comparisons are useful, whereby data from one location are compared with data from nearby identical sensors.<\/dd><\/dl>\n<h4><span class=\"mw-headline\" id=\"C2._Review_of_data_quality_issues\">C2. Review of data quality issues<\/span><\/h4>\n<p>The purpose of reviewing data quality issues is to identify the starting point for deciding to measure data quality levels and process performance with the potential to generate opportunities to improve data quality. The results of measurement and evaluation of data quality are analyzed and possible issues are identified. This step considers the following activities:\n<\/p>\n<dl><dd><b>C2-1.<\/b> Issue analysis - Review non-solved non-conformities arising from data processing to identify those that are possibly connected to the reported issue that has triggered the need for data quality assurance. This review creates a set of related non-conformities. This set is the basis for further investigation through the measurement of data quality levels in SCP environments. Respond to the reporting of unresolved data non-conformities from within data quality control, indications of the recurrence of types of nonconformity, or other issues raised against the results of data quality planning or data quality control.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"The_.22act.22_phase\">The \"act\" phase<\/span><\/h3>\n<p>The act phase consists of the data quality improvement that involves analyzing the root causes of data quality issues based on the assessment results derived from data quality assurance (see the \"check\" phase). To prevent future data non-conformities, the steps in this phase try to correct all the existing non-conformities and also transforms SCP operations as appropriate. On this phase, it is important to address aspects as analysis of root causes of sensor data problems, management, of sensor data problems, correction and prevention, make available ready access to replacement parts and anticipate common repairs, and maintain inventory replacement parts. Table 8 summarizes inputs, outputs, and responsibility distribution for this phase.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab8_Perez-Castillo_Sensors2018_18-9.png\" class=\"image wiki-link\" data-key=\"e6ea84b9943486c282e50a4dce1f3df8\"><img alt=\"Tab8 Perez-Castillo Sensors2018 18-9.png\" src=\"https:\/\/www.limswiki.org\/images\/5\/55\/Tab8_Perez-Castillo_Sensors2018_18-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 8.<\/b> Inputs, outputs, and RACIQ matrix for the Act phase (R\u2014Responsible (works on); A\u2014Accountable, C\u2014Consulted; I\u2014Informed; Q\u2014Quality Reviewer)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"A1._Data_quality_issues_prevention\">A1. Data quality issues prevention<\/span><\/h4>\n<p>The purpose of data quality issue prevention, resulting from root cause analysis and solution development, is to increase the extent to which the organization achieves a systematic and systemic approach to achieving data quality. This step tries to prevent issues on devices or, for example, to avoid data loss when repair or maintenance of devices of the SCP environment can be predicted. In order to prevent the recurrence of each actual or the occurrence of each potential data non-conformity or similar data non-conformities by refining and applying guidelines, rules, and procedures. To achieve this, the following activities should be conducted:'\n<\/p>\n<dl><dd><b>A1-1.<\/b> Make available ready access to replacement parts - Schedule routine calibration of instruments and sensors based on manufacturer specifications. Maintaining additional calibrated sensors of the same make\/model can allow immediate replacement of sensors removed for calibration to avoid data loss. Otherwise, sensor calibrations can be scheduled at non-critical times or staggered such that a nearby sensor can be used as a proxy to fill gaps.<\/dd><\/dl>\n<dl><dd><b>A1-2.<\/b> Update the strategy for node replacement - Controlled replacement is often pursued for only a selected subset of the employed nodes with the goal of structuring the network topology in a way that achieves the desired application requirements. In addition to coverage, the nodes\u2019 positions affect numerous network performance metrics such as energy consumption, delay, and throughput. For example, large distances between nodes weaken the communication links, lower the throughput, and increase energy consumption. Additionally, it can anticipate some common repairs and maintain inventory replacement parts. This means that sensors could be replaced before failure where sensor lifetimes are known or can be estimated.<\/dd><\/dl>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>In this paper, we have tackled the challenges of data quality problems in SCP environments. Many research and standardization efforts have been made in the DQ area over the years, and some interesting results have been already transferred to IoT as well. However, the approaches presented in the literature have two main drawbacks. On the one hand, these proposals do not take into account the nature of SCP environments and the concrete factors that affect the way in which DQ must be treated in such a context. On the other hand, these approaches do not consider existing DQ management standards that have not been tailored yet for IoT and, more specifically, to SCP contexts. Given the importance of institutionalizing best practices in SCP for DQ management, we consider it of paramount importance to provide practitioners and organizations with techniques aligned with standards, reducing their adaptation efforts and favoring systematic and holistic approaches to the problem. As a main contribution, we have provided in this paper a data quality management methodology for sensor data, named DAQUA-MASS, based on ISO 8000-61. The methodology is structured according to the PDCA cycle of continuous improvement. The methodology is composed of seven steps divided into several activities. Input and output products are already identified for each activity in the methodology. It is noteworthy to highlight the identification of the various roles involved in the management of data quality in sensor data.\n<\/p><p>The data quality model, along with the methodology, offers a unique framework to enable designers of IoT projects\u2014including sensor networks and practitioners in charge of exploiting IoT systems\u2014to assure that the business processes working over these systems can manage data with adequate levels of quality. Once discarded data quality problems or identified data quality problem root causes are identified, it will be easier to focus the attention exclusively on sensor networks of IoT systems. Working aligned to international open standards will enable organizations to speak the same language and devote the required efforts only to the proper working of the IoT systems, raising awareness of some other data quality concerns.\n<\/p><p>Future research lines are planned to be primarily focused on the empirical validation of this methodology. It will be used by different organizations in various SCP environments in order to ensure its applicability on a large scale. The mentioned case studies will allow us to provide different specializations of the data quality model (or recommendation on how to tailor it) to different organization depending on the domain. In parallel, other future research lines are in accordance with some limitations of the scope considered in this paper. Thus, data quality management issues are expected to be considered, as well as their implications in the processing and utilization layers apart from the acquisition layer (see Figure 1). While simplified data analysis is being performed on the devices in the acquisition layer to control the device, most analysis, diagnosis, and improvement of sensor data are being performed in the processing layer. This will force management of sensor data together with general-purpose data to achieve holistic data quality management in the processing layer.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Conceptualization, R.P.-C. and I.C.; Investigation, R.P.-C., A.G.C. and I.C.; Writing-Original Draft Preparation, R.P.-C., A.G.C. and I.C.; Writing-Review & Editing, R.P.-C., A.G.C.; I.C.; M.R., M.P., A.M., S.K. and D.L.; Supervision, I.C. and M.P.; Project Administration, R.P.-C. and I.C.; Funding Acquisition, I.C., M.P., S.K., D.L.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was primarily funded by DQIoT project (Eureka program, E!11737; and CDTI (Centro Para el Desarrollo Tecnol\u00f3gico Industrial), INNO-20171086). Additionally, this work was partially funded by SEQUOIA project (TIN2015-63502-C3-1-R and TIN2015-63502-C3-3-R) (MINECO\/FEDER); GEMA SBPLY\/17\/180501\/000293, Consejer\u00eda de Educaci\u00f3n, Cultura y Deporte de la Direcci\u00f3n General de Universidades, Investigaci\u00f3n e Innovaci\u00f3n de la JCCM); ECD project (Evaluaci\u00f3n y Certificaci\u00f3n de la Calidad de Datos) (PTQ-16-08504) (Torres Quevedo Program, MINECO). Finally, it was also supported through a grant to Ricardo P\u00e9rez-Castillo enjoys from JCCM within the initiatives for talent retention and return in line with RIS3 goals.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-AshtonThatInt09-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AshtonThatInt09_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ashton, K. (2009). \"That 'Internet of Things' Thing\". <i>RFID Journal<\/i> <b>22<\/b>: 97\u2013114.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=That+%27Internet+of+Things%27+Thing&rft.jtitle=RFID+Journal&rft.aulast=Ashton%2C+K.&rft.au=Ashton%2C+K.&rft.date=2009&rft.volume=22&rft.pages=97%E2%80%93114&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeberInternet13-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WeberInternet13_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-WeberInternet13_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Weber, R.H. (2013). \"Internet of things \u2013 Governance quo vadis?\". <i>Computer Law & Security Review<\/i> <b>29<\/b> (4): 341-347. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.clsr.2013.05.010\" target=\"_blank\">10.1016\/j.clsr.2013.05.010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Internet+of+things+%E2%80%93+Governance+quo+vadis%3F&rft.jtitle=Computer+Law+%26+Security+Review&rft.aulast=Weber%2C+R.H.&rft.au=Weber%2C+R.H.&rft.date=2013&rft.volume=29&rft.issue=4&rft.pages=341-347&rft_id=info:doi\/10.1016%2Fj.clsr.2013.05.010&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HassaneinBig17-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HassaneinBig17_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hassanein, H.S.; Oteafy, S.M.A. (2017). \"Big Sensed Data Challenges in the Internet of Things\". <i>Proceedings from the 13th International Conference on Distributed Computing in Sensor Systems<\/i>: 207\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FDCOSS.2017.35\" target=\"_blank\">10.1109\/DCOSS.2017.35<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+Sensed+Data+Challenges+in+the+Internet+of+Things&rft.jtitle=Proceedings+from+the+13th+International+Conference+on+Distributed+Computing+in+Sensor+Systems&rft.aulast=Hassanein%2C+H.S.%3B+Oteafy%2C+S.M.A.&rft.au=Hassanein%2C+H.S.%3B+Oteafy%2C+S.M.A.&rft.date=2017&rft.pages=207%E2%80%938&rft_id=info:doi\/10.1109%2FDCOSS.2017.35&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AtzoriTheInt10-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AtzoriTheInt10_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Atzori, L.; Iera, A.; Morabito, G. (2010). \"The Internet of Things: A survey\". <i>Computer Networks<\/i> <b>54<\/b> (15): 2787-2805. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.comnet.2010.05.010\" target=\"_blank\">10.1016\/j.comnet.2010.05.010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Internet+of+Things%3A+A+survey&rft.jtitle=Computer+Networks&rft.aulast=Atzori%2C+L.%3B+Iera%2C+A.%3B+Morabito%2C+G.&rft.au=Atzori%2C+L.%3B+Iera%2C+A.%3B+Morabito%2C+G.&rft.date=2010&rft.volume=54&rft.issue=15&rft.pages=2787-2805&rft_id=info:doi\/10.1016%2Fj.comnet.2010.05.010&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangCoop17-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhangCoop17_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, W.; Zhang, Z.; Chao, H.-C. (2017). \"Cooperative Fog Computing for Dealing with Big Data in the Internet of Vehicles: Architecture and Hierarchical Resource Management\". <i>IEEE Communications Magazine<\/i> <b>55<\/b> (12): 60\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMCOM.2017.1700208\" target=\"_blank\">10.1109\/MCOM.2017.1700208<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cooperative+Fog+Computing+for+Dealing+with+Big+Data+in+the+Internet+of+Vehicles%3A+Architecture+and+Hierarchical+Resource+Management&rft.jtitle=IEEE+Communications+Magazine&rft.aulast=Zhang%2C+W.%3B+Zhang%2C+Z.%3B+Chao%2C+H.-C.&rft.au=Zhang%2C+W.%3B+Zhang%2C+Z.%3B+Chao%2C+H.-C.&rft.date=2017&rft.volume=55&rft.issue=12&rft.pages=60%E2%80%937&rft_id=info:doi\/10.1109%2FMCOM.2017.1700208&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KarkouchData16-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KarkouchData16_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-2\">6.2<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-3\">6.3<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-4\">6.4<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-5\">6.5<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-6\">6.6<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-7\">6.7<\/a><\/sup> <sup><a href=\"#cite_ref-KarkouchData16_6-8\">6.8<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Karkouch, A.; Mousannif, H.; Al Moatassime, H. et al. (2016). \"Data quality in internet of things: A state-of-the-art survey\". <i>Journal of Network and Computer Applications<\/i> <b>73<\/b>: 57\u201381. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jnca.2016.08.002\" target=\"_blank\">10.1016\/j.jnca.2016.08.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+quality+in+internet+of+things%3A+A+state-of-the-art+survey&rft.jtitle=Journal+of+Network+and+Computer+Applications&rft.aulast=Karkouch%2C+A.%3B+Mousannif%2C+H.%3B+Al+Moatassime%2C+H.+et+al.&rft.au=Karkouch%2C+A.%3B+Mousannif%2C+H.%3B+Al+Moatassime%2C+H.+et+al.&rft.date=2016&rft.volume=73&rft.pages=57%E2%80%9381&rft_id=info:doi\/10.1016%2Fj.jnca.2016.08.002&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MerinoAData16-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MerinoAData16_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-MerinoAData16_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Merino, J.; Caballero, I.; Rivas, B. et al. (2016). \"A Data Quality in Use model for Big Data\". <i>Future Generation Computer Systems<\/i> <b>63<\/b>: 123\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.future.2015.11.024\" target=\"_blank\">10.1016\/j.future.2015.11.024<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Data+Quality+in+Use+model+for+Big+Data&rft.jtitle=Future+Generation+Computer+Systems&rft.aulast=Merino%2C+J.%3B+Caballero%2C+I.%3B+Rivas%2C+B.+et+al.&rft.au=Merino%2C+J.%3B+Caballero%2C+I.%3B+Rivas%2C+B.+et+al.&rft.date=2016&rft.volume=63&rft.pages=123%E2%80%9330&rft_id=info:doi\/10.1016%2Fj.future.2015.11.024&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JesusAsurv17-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JesusAsurv17_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-JesusAsurv17_8-1\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-JesusAsurv17_8-2\">8.2<\/a><\/sup> <sup><a href=\"#cite_ref-JesusAsurv17_8-3\">8.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jesus, G.; Casimiro, A.; Oliveira, A. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5620495\" target=\"_blank\">\"A Survey on Data Quality for Dependable Monitoring in Wireless Sensor Networks\"<\/a>. <i>Sensors<\/i> <b>17<\/b> (9): E2010. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs17092010\" target=\"_blank\">10.3390\/s17092010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5620495\/\" target=\"_blank\">PMC5620495<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28869505\" target=\"_blank\">28869505<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5620495\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5620495<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Survey+on+Data+Quality+for+Dependable+Monitoring+in+Wireless+Sensor+Networks&rft.jtitle=Sensors&rft.aulast=Jesus%2C+G.%3B+Casimiro%2C+A.%3B+Oliveira%2C+A.&rft.au=Jesus%2C+G.%3B+Casimiro%2C+A.%3B+Oliveira%2C+A.&rft.date=2017&rft.volume=17&rft.issue=9&rft.pages=E2010&rft_id=info:doi\/10.3390%2Fs17092010&rft_id=info:pmc\/PMC5620495&rft_id=info:pmid\/28869505&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5620495&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RodriguezManaging13-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RodriguezManaging13_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-RodriguezManaging13_9-1\">9.1<\/a><\/sup> <sup><a href=\"#cite_ref-RodriguezManaging13_9-2\">9.2<\/a><\/sup> <sup><a href=\"#cite_ref-RodriguezManaging13_9-3\">9.3<\/a><\/sup> <sup><a href=\"#cite_ref-RodriguezManaging13_9-4\">9.4<\/a><\/sup> <sup><a href=\"#cite_ref-RodriguezManaging13_9-5\">9.5<\/a><\/sup> <sup><a href=\"#cite_ref-RodriguezManaging13_9-6\">9.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rodr\u00edguez, C.C.G.; Servigne, S. (2017). \"Managing Sensor Data Uncertainty: A Data Quality Approach\". <i>International Journal of Agricultural and Environmental Information Systems<\/i> <b>4<\/b> (1): 3. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4018%2Fjaeis.2013010103\" target=\"_blank\">10.4018\/jaeis.2013010103<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Managing+Sensor+Data+Uncertainty%3A+A+Data+Quality+Approach&rft.jtitle=International+Journal+of+Agricultural+and+Environmental+Information+Systems&rft.aulast=Rodr%C3%ADguez%2C+C.C.G.%3B+Servigne%2C+S.&rft.au=Rodr%C3%ADguez%2C+C.C.G.%3B+Servigne%2C+S.&rft.date=2017&rft.volume=4&rft.issue=1&rft.pages=3&rft_id=info:doi\/10.4018%2Fjaeis.2013010103&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KleinHowTo09-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KleinHowTo09_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-KleinHowTo09_10-1\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-KleinHowTo09_10-2\">10.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Klein, A.; Hackenbroich, G.; Lehner, W. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/mitiq.mit.edu\/iciqpapers.aspx?iciqyear=2009\" target=\"_blank\">\"How to Screen a Data Stream - Quality-Driven Load Shedding in Sensor Data Streams\"<\/a>. <i>Proceedings of the 2009 International Conference on Information Quality<\/i>: 1\u201315<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mitiq.mit.edu\/iciqpapers.aspx?iciqyear=2009\" target=\"_blank\">http:\/\/mitiq.mit.edu\/iciqpapers.aspx?iciqyear=2009<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+to+Screen+a+Data+Stream+-+Quality-Driven+Load+Shedding+in+Sensor+Data+Streams&rft.jtitle=Proceedings+of+the+2009+International+Conference+on+Information+Quality&rft.aulast=Klein%2C+A.%3B+Hackenbroich%2C+G.%3B+Lehner%2C+W.&rft.au=Klein%2C+A.%3B+Hackenbroich%2C+G.%3B+Lehner%2C+W.&rft.date=2009&rft.pages=1%E2%80%9315&rft_id=http%3A%2F%2Fmitiq.mit.edu%2Ficiqpapers.aspx%3Ficiqyear%3D2009&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-M.C3.BChlh.C3.A4userSmart07-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-M.C3.BChlh.C3.A4userSmart07_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-M.C3.BChlh.C3.A4userSmart07_11-1\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">M\u00fchlh\u00e4user, M. (2007). \"Smart Products: An Introduction\". <i>Proceedings from the 2007 European Conference on Ambient Intelligence<\/i>: 158\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-540-85379-4_20\" target=\"_blank\">10.1007\/978-3-540-85379-4_20<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Smart+Products%3A+An+Introduction&rft.jtitle=Proceedings+from+the+2007+European+Conference+on+Ambient+Intelligence&rft.aulast=M%C3%BChlh%C3%A4user%2C+M.&rft.au=M%C3%BChlh%C3%A4user%2C+M.&rft.date=2007&rft.pages=158%E2%80%9364&rft_id=info:doi\/10.1007%2F978-3-540-85379-4_20&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LaneyInfo17-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LaneyInfo17_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Laney, Douglas B. (2017). <i>Infonomics: How to Monetize, Manage, and Measure Information as an Asset for Competitive Advantage<\/i>. Routledge. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781138090385.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Infonomics%3A+How+to+Monetize%2C+Manage%2C+and+Measure+Information+as+an+Asset+for+Competitive+Advantage&rft.aulast=Laney%2C+Douglas+B.&rft.au=Laney%2C+Douglas+B.&rft.date=2017&rft.pub=Routledge&rft.isbn=9781138090385&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO_25000-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISO_25000_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/64764.html\" target=\"_blank\">\"ISO\/IEC 25000:2014\"<\/a>. International Organization for Standardization. March 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/64764.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/64764.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO%2FIEC+25000%3A2014&rft.atitle=&rft.date=March+2014&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F64764.html&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-QinQuality14-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-QinQuality14_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-QinQuality14_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Qin, z.; Han, Q.; Mehrotra, S. et al. (2014). \"Quality-Aware Sensor Data Management\". In Ammari, H.M.. <i>The Art of Wireless Sensor Networks<\/i>. Springer. pp. 429\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783642400094.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Quality-Aware+Sensor+Data+Management&rft.atitle=The+Art+of+Wireless+Sensor+Networks&rft.aulast=Qin%2C+z.%3B+Han%2C+Q.%3B+Mehrotra%2C+S.+et+al.&rft.au=Qin%2C+z.%3B+Han%2C+Q.%3B+Mehrotra%2C+S.+et+al.&rft.date=2014&rft.pages=pp.%26nbsp%3B429%E2%80%9364&rft.pub=Springer&rft.isbn=9783642400094&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CampbellQuant13-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CampbellQuant13_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-CampbellQuant13_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-CampbellQuant13_15-2\">15.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Campbell, J.L.; Rustad, L.E.; Porter, J.H. et al. (2013). \"Quantity is Nothing without Quality: Automated QA\/QC for Streaming Environmental Sensor Data\". <i>BioScience<\/i> <b>63<\/b> (7): 574\u201385. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1525%2Fbio.2013.63.7.10\" target=\"_blank\">10.1525\/bio.2013.63.7.10<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quantity+is+Nothing+without+Quality%3A+Automated+QA%2FQC+for+Streaming+Environmental+Sensor+Data&rft.jtitle=BioScience&rft.aulast=Campbell%2C+J.L.%3B+Rustad%2C+L.E.%3B+Porter%2C+J.H.+et+al.&rft.au=Campbell%2C+J.L.%3B+Rustad%2C+L.E.%3B+Porter%2C+J.H.+et+al.&rft.date=2013&rft.volume=63&rft.issue=7&rft.pages=574%E2%80%9385&rft_id=info:doi\/10.1525%2Fbio.2013.63.7.10&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KleinRepresent09-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KleinRepresent09_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Klein, A.; Lehner, W. (2009). \"Representing Data Quality in Sensor Data Streaming Environments\". <i>Journal of Data and Information Quality<\/i> <b>1<\/b> (2): 10. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1577840.1577845\" target=\"_blank\">10.1145\/1577840.1577845<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Representing+Data+Quality+in+Sensor+Data+Streaming+Environments&rft.jtitle=Journal+of+Data+and+Information+Quality&rft.aulast=Klein%2C+A.%3B+Lehner%2C+W.&rft.au=Klein%2C+A.%3B+Lehner%2C+W.&rft.date=2009&rft.volume=1&rft.issue=2&rft.pages=10&rft_id=info:doi\/10.1145%2F1577840.1577845&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO_8000-61-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISO_8000-61_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISO_8000-61_17-1\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-ISO_8000-61_17-2\">17.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/63086.html\" target=\"_blank\">\"ISO 8000-61:2016\"<\/a>. International Organization for Standardization. November 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/63086.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/63086.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO+8000-61%3A2016&rft.atitle=&rft.date=November+2016&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F63086.html&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CookSmart04-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CookSmart04_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Cook, D.; Das, S.K. (2004). <i>Smart Environments: Technology, Protocols and Applications<\/i> (1st ed.). Wiley-Interscience. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780471544487.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Smart+Environments%3A+Technology%2C+Protocols+and+Applications&rft.aulast=Cook%2C+D.%3B+Das%2C+S.K.&rft.au=Cook%2C+D.%3B+Das%2C+S.K.&rft.date=2004&rft.edition=1st&rft.pub=Wiley-Interscience&rft.isbn=9780471544487&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PorterHowSmart14-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PorterHowSmart14_19-0\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-PorterHowSmart14_19-1\">19.1<\/a><\/sup> <sup><a href=\"#cite_ref-PorterHowSmart14_19-2\">19.2<\/a><\/sup> <sup><a href=\"#cite_ref-PorterHowSmart14_19-3\">19.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Porter, M.E.; Heppelmann, J.E. (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/hbr.org\/2014\/11\/how-smart-connected-products-are-transforming-competition\" target=\"_blank\">\"How Smart, Connected Products Are Transforming Competition\"<\/a>. <i>Harvard Business Review<\/i> <b>92<\/b>: 64\u201388<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/hbr.org\/2014\/11\/how-smart-connected-products-are-transforming-competition\" target=\"_blank\">https:\/\/hbr.org\/2014\/11\/how-smart-connected-products-are-transforming-competition<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+Smart%2C+Connected+Products+Are+Transforming+Competition&rft.jtitle=Harvard+Business+Review&rft.aulast=Porter%2C+M.E.%3B+Heppelmann%2C+J.E.&rft.au=Porter%2C+M.E.%3B+Heppelmann%2C+J.E.&rft.date=2014&rft.volume=92&rft.pages=64%E2%80%9388&rft_id=https%3A%2F%2Fhbr.org%2F2014%2F11%2Fhow-smart-connected-products-are-transforming-competition&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OstrowerSmart14-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OstrowerSmart14_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Ostrower, D. (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wired.com\/insights\/2014\/11\/smart-connected-products\/\" target=\"_blank\">\"Smart Connected Products: Killing Industries, Boosting Innovation\"<\/a>. <i>Wired<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wired.com\/insights\/2014\/11\/smart-connected-products\/\" target=\"_blank\">https:\/\/www.wired.com\/insights\/2014\/11\/smart-connected-products\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Smart+Connected+Products%3A+Killing+Industries%2C+Boosting+Innovation&rft.atitle=Wired&rft.aulast=Ostrower%2C+D.&rft.au=Ostrower%2C+D.&rft.date=2014&rft_id=https%3A%2F%2Fwww.wired.com%2Finsights%2F2014%2F11%2Fsmart-connected-products%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WuenderlichFutur15-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WuenderlichFutur15_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wuenderlich, N.V.; Heinonen, K.; Ostrom, A.L. et al. (2015). \"\u201cFuturizing\u201d smart service: Implications for service researchers and managers\". <i>Journal of Services Marketing<\/i> <b>29<\/b> (6\/7): 442\u201347. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2FJSM-01-2015-0040\" target=\"_blank\">10.1108\/JSM-01-2015-0040<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%E2%80%9CFuturizing%E2%80%9D+smart+service%3A+Implications+for+service+researchers+and+managers&rft.jtitle=Journal+of+Services+Marketing&rft.aulast=Wuenderlich%2C+N.V.%3B+Heinonen%2C+K.%3B+Ostrom%2C+A.L.+et+al.&rft.au=Wuenderlich%2C+N.V.%3B+Heinonen%2C+K.%3B+Ostrom%2C+A.L.+et+al.&rft.date=2015&rft.volume=29&rft.issue=6%2F7&rft.pages=442%E2%80%9347&rft_id=info:doi\/10.1108%2FJSM-01-2015-0040&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AllmendingerFour05-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AllmendingerFour05_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Allmendinger, G.; Lombreglia, R. (2005). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/hbr.org\/2005\/10\/four-strategies-for-the-age-of-smart-services\" target=\"_blank\">\"Four Strategies for the Age of Smart Services\"<\/a>. <i>Harvard Business Review<\/i> <b>83<\/b>: 131<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/hbr.org\/2005\/10\/four-strategies-for-the-age-of-smart-services\" target=\"_blank\">https:\/\/hbr.org\/2005\/10\/four-strategies-for-the-age-of-smart-services<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Four+Strategies+for+the+Age+of+Smart+Services&rft.jtitle=Harvard+Business+Review&rft.aulast=Allmendinger%2C+G.%3B+Lombreglia%2C+R.&rft.au=Allmendinger%2C+G.%3B+Lombreglia%2C+R.&rft.date=2005&rft.volume=83&rft.pages=131&rft_id=https%3A%2F%2Fhbr.org%2F2005%2F10%2Ffour-strategies-for-the-age-of-smart-services&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TilakATax02-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TilakATax02_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-TilakATax02_23-1\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-TilakATax02_23-2\">23.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tilak, S.; Abu-Ghazaleh, N.B.; Heinzelman, W. (2002). \"A taxonomy of wireless micro-sensor network models\". <i>ACM SIGMOBILE Mobile Computing and Communications Review<\/i> <b>6<\/b> (2): 28\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F565702.565708\" target=\"_blank\">10.1145\/565702.565708<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+taxonomy+of+wireless+micro-sensor+network+models&rft.jtitle=ACM+SIGMOBILE+Mobile+Computing+and+Communications+Review&rft.aulast=Tilak%2C+S.%3B+Abu-Ghazaleh%2C+N.B.%3B+Heinzelman%2C+W.&rft.au=Tilak%2C+S.%3B+Abu-Ghazaleh%2C+N.B.%3B+Heinzelman%2C+W.&rft.date=2002&rft.volume=6&rft.issue=2&rft.pages=28%E2%80%9336&rft_id=info:doi\/10.1145%2F565702.565708&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarnaghiChallenges15-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BarnaghiChallenges15_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barnaghi, P.; Bermudez-Edo, M.; T\u00f6njes, R. (2015). \"Challenges for Quality of Data in Smart Cities\". <i>Journal of Data and Information Quality<\/i> <b>6<\/b> (2\u20133): 6. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2747881\" target=\"_blank\">10.1145\/2747881<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Challenges+for+Quality+of+Data+in+Smart+Cities&rft.jtitle=Journal+of+Data+and+Information+Quality&rft.aulast=Barnaghi%2C+P.%3B+Bermudez-Edo%2C+M.%3B+T%C3%B6njes%2C+R.&rft.au=Barnaghi%2C+P.%3B+Bermudez-Edo%2C+M.%3B+T%C3%B6njes%2C+R.&rft.date=2015&rft.volume=6&rft.issue=2%E2%80%933&rft.pages=6&rft_id=info:doi\/10.1145%2F2747881&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO_25012-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISO_25012_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISO_25012_25-1\">25.1<\/a><\/sup> <sup><a href=\"#cite_ref-ISO_25012_25-2\">25.2<\/a><\/sup> <sup><a href=\"#cite_ref-ISO_25012_25-3\">25.3<\/a><\/sup> <sup><a href=\"#cite_ref-ISO_25012_25-4\">25.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/35736.html\" target=\"_blank\">\"ISO\/IEC 25012:2008\"<\/a>. International Organization for Standardization. December 2008<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/35736.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/35736.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO%2FIEC+25012%3A2008&rft.atitle=&rft.date=December+2008&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F35736.html&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BadawyAuto18-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BadawyAuto18_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Badawy, R.; Raykov, Y.P.; Evers, L.J.W. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5948536\" target=\"_blank\">\"Automated Quality Control for Sensor Based Symptom Measurement Performed Outside the Lab\"<\/a>. <i>Sensors<\/i> <b>18<\/b> (4): E1215. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fs18041215\" target=\"_blank\">10.3390\/s18041215<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5948536\/\" target=\"_blank\">PMC5948536<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29659528\" target=\"_blank\">29659528<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5948536\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5948536<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automated+Quality+Control+for+Sensor+Based+Symptom+Measurement+Performed+Outside+the+Lab&rft.jtitle=Sensors&rft.aulast=Badawy%2C+R.%3B+Raykov%2C+Y.P.%3B+Evers%2C+L.J.W.+et+al.&rft.au=Badawy%2C+R.%3B+Raykov%2C+Y.P.%3B+Evers%2C+L.J.W.+et+al.&rft.date=2018&rft.volume=18&rft.issue=4&rft.pages=E1215&rft_id=info:doi\/10.3390%2Fs18041215&rft_id=info:pmc\/PMC5948536&rft_id=info:pmid\/29659528&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5948536&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Al-RuitheData16-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Al-RuitheData16_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Al-Ruithe, M.; Mthunzi, S.; Benkhelifa, E. (2016). \"Data governance for security in IoT & cloud converged environments\". <i>Proceedings from the 13th International Conference of Computer Systems and Applications<\/i>: 1\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FAICCSA.2016.7945737\" target=\"_blank\">10.1109\/AICCSA.2016.7945737<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+governance+for+security+in+IoT+%26+cloud+converged+environments&rft.jtitle=Proceedings+from+the+13th+International+Conference+of+Computer+Systems+and+Applications&rft.aulast=Al-Ruithe%2C+M.%3B+Mthunzi%2C+S.%3B+Benkhelifa%2C+E.&rft.au=Al-Ruithe%2C+M.%3B+Mthunzi%2C+S.%3B+Benkhelifa%2C+E.&rft.date=2016&rft.pages=1%E2%80%938&rft_id=info:doi\/10.1109%2FAICCSA.2016.7945737&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeeAIMQ02-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeeAIMQ02_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lee, Y.W.; Strong, D.M.; Kahn, B.K. et al. (2002). \"AIMQ: A methodology for information quality assessment\". <i>Information & Management<\/i>: 133-46. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0378-7206%2802%2900043-5\" target=\"_blank\">10.1016\/S0378-7206(02)00043-5<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=AIMQ%3A+A+methodology+for+information+quality+assessment&rft.jtitle=Information+%26+Management&rft.aulast=Lee%2C+Y.W.%3B+Strong%2C+D.M.%3B+Kahn%2C+B.K.+et+al.&rft.au=Lee%2C+Y.W.%3B+Strong%2C+D.M.%3B+Kahn%2C+B.K.+et+al.&rft.date=2002&rft.pages=133-46&rft_id=info:doi\/10.1016%2FS0378-7206%2802%2900043-5&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McGilvrayExecuting08-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McGilvrayExecuting08_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">McGilvray, D. (2008). <i>Executing Data Quality Projects: Ten Steps to Quality Data and Trusted Information (TM)<\/i>. Elsevier. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780080951584.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Executing+Data+Quality+Projects%3A+Ten+Steps+to+Quality+Data+and+Trusted+Information+%28TM%29&rft.aulast=McGilvray%2C+D.&rft.au=McGilvray%2C+D.&rft.date=2008&rft.pub=Elsevier&rft.isbn=9780080951584&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO_8000-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISO_8000_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/54579.html\" target=\"_blank\">\"ISO\/TS 8000-150:2011\"<\/a>. International Organization for Standardization. December 2011<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/54579.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/54579.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO%2FTS+8000-150%3A2011&rft.atitle=&rft.date=December+2011&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F54579.html&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO_9001-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ISO_9001_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.iso.org\/standard\/62085.html\" target=\"_blank\">\"ISO 9001:2015\"<\/a>. International Organization for Standardization. September 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.iso.org\/standard\/62085.html\" target=\"_blank\">https:\/\/www.iso.org\/standard\/62085.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO+9001%3A2015&rft.atitle=&rft.date=September+2015&rft.pub=International+Organization+for+Standardization&rft_id=https%3A%2F%2Fwww.iso.org%2Fstandard%2F62085.html&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BatiniMethod09-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BatiniMethod09_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Batini, C.; Cappiello, C.; Francalanci, C. et al. (2009). \"Methodologies for data quality assessment and improvement\". <i>ACM Computing Surveys<\/i> <b>41<\/b> (3): 16. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1541880.1541883\" target=\"_blank\">10.1145\/1541880.1541883<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Methodologies+for+data+quality+assessment+and+improvement&rft.jtitle=ACM+Computing+Surveys&rft.aulast=Batini%2C+C.%3B+Cappiello%2C+C.%3B+Francalanci%2C+C.+et+al.&rft.au=Batini%2C+C.%3B+Cappiello%2C+C.%3B+Francalanci%2C+C.+et+al.&rft.date=2009&rft.volume=41&rft.issue=3&rft.pages=16&rft_id=info:doi\/10.1145%2F1541880.1541883&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoodallAClass14-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WoodallAClass14_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Woodall, P.; Oberhofer, M.; Borek, A. (2014). \"A classification of data quality assessment and improvement methods\". <i>International Journal of Information Quality<\/i> <b>3<\/b> (4): 298\u2013321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1504%2FIJIQ.2014.068656\" target=\"_blank\">10.1504\/IJIQ.2014.068656<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+classification+of+data+quality+assessment+and+improvement+methods&rft.jtitle=International+Journal+of+Information+Quality&rft.aulast=Woodall%2C+P.%3B+Oberhofer%2C+M.%3B+Borek%2C+A.&rft.au=Woodall%2C+P.%3B+Oberhofer%2C+M.%3B+Borek%2C+A.&rft.date=2014&rft.volume=3&rft.issue=4&rft.pages=298%E2%80%93321&rft_id=info:doi\/10.1504%2FIJIQ.2014.068656&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StrongData97-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StrongData97_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Strong, D.M.; Lee, Y.W.; Wang, R.Y. (1997). \"Data quality in context\". <i>Communications of the ACM<\/i> <b>40<\/b> (5): 103\u201310. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F253769.253804\" target=\"_blank\">10.1145\/253769.253804<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+quality+in+context&rft.jtitle=Communications+of+the+ACM&rft.aulast=Strong%2C+D.M.%3B+Lee%2C+Y.W.%3B+Wang%2C+R.Y.&rft.au=Strong%2C+D.M.%3B+Lee%2C+Y.W.%3B+Wang%2C+R.Y.&rft.date=1997&rft.volume=40&rft.issue=5&rft.pages=103%E2%80%9310&rft_id=info:doi\/10.1145%2F253769.253804&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangAProd98-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangAProd98_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, R.Y. (1998). \"A product perspective on total data quality management\". <i>Communications of the ACM<\/i> <b>41<\/b> (2): 58\u201365. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F269012.269022\" target=\"_blank\">10.1145\/269012.269022<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+product+perspective+on+total+data+quality+management&rft.jtitle=Communications+of+the+ACM&rft.aulast=Wang%2C+R.Y.&rft.au=Wang%2C+R.Y.&rft.date=1998&rft.volume=41&rft.issue=2&rft.pages=58%E2%80%9365&rft_id=info:doi\/10.1145%2F269012.269022&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SrivannaboonAchiev09-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SrivannaboonAchiev09_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Srivannaboon, S. (2009). \"Achieving Competitive Advantage through the use of Project Management under the Plan-do-Check-act Concept\". <i>Journal of General Management<\/i> <b>34<\/b> (3): 1\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F030630700903400301\" target=\"_blank\">10.1177\/030630700903400301<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Achieving+Competitive+Advantage+through+the+use+of+Project+Management+under+the+Plan-do-Check-act+Concept&rft.jtitle=Journal+of+General+Management&rft.aulast=Srivannaboon%2C+S.&rft.au=Srivannaboon%2C+S.&rft.date=2009&rft.volume=34&rft.issue=3&rft.pages=1%E2%80%9320&rft_id=info:doi\/10.1177%2F030630700903400301&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChenInfo14-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChenInfo14_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chen, P.-Y.; Cheng, S.-M.; Chen, K.-C. (2014). \"Information Fusion to Defend Intentional Attack in Internet of Things\". <i>IEEE Internet of Things Journal<\/i> <b>1<\/b> (4): 337\u201348. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FJIOT.2014.2337018\" target=\"_blank\">10.1109\/JIOT.2014.2337018<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+Fusion+to+Defend+Intentional+Attack+in+Internet+of+Things&rft.jtitle=IEEE+Internet+of+Things+Journal&rft.aulast=Chen%2C+P.-Y.%3B+Cheng%2C+S.-M.%3B+Chen%2C+K.-C.&rft.au=Chen%2C+P.-Y.%3B+Cheng%2C+S.-M.%3B+Chen%2C+K.-C.&rft.date=2014&rft.volume=1&rft.issue=4&rft.pages=337%E2%80%9348&rft_id=info:doi\/10.1109%2FJIOT.2014.2337018&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlaschRevisit13-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlaschRevisit13_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blasch, E.; Steinberg, A.; Das, S. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ieeexplore.ieee.org\/stamp\/stamp.jsp?tp=&arnumber=6641155&isnumber=6641065\" target=\"_blank\">\"Revisiting the JDL model for information exploitation\"<\/a>. <i>Proceedings of the 16th International Conference on Information Fusion<\/i>: 129\u201336<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ieeexplore.ieee.org\/stamp\/stamp.jsp?tp=&arnumber=6641155&isnumber=6641065\" target=\"_blank\">http:\/\/ieeexplore.ieee.org\/stamp\/stamp.jsp?tp=&arnumber=6641155&isnumber=6641065<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Revisiting+the+JDL+model+for+information+exploitation&rft.jtitle=Proceedings+of+the+16th+International+Conference+on+Information+Fusion&rft.aulast=Blasch%2C+E.%3B+Steinberg%2C+A.%3B+Das%2C+S.+et+al.&rft.au=Blasch%2C+E.%3B+Steinberg%2C+A.%3B+Das%2C+S.+et+al.&rft.date=2013&rft.pages=129%E2%80%9336&rft_id=http%3A%2F%2Fieeexplore.ieee.org%2Fstamp%2Fstamp.jsp%3Ftp%3D%26arnumber%3D6641155%26isnumber%3D6641065&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PastorelloGerencia08-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PastorelloGerencia08_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pastorello Jr., G.Z. (19 December 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/lis-unicamp.github.io\/wp-content\/uploads\/2014\/09\/tese_GZPastorelloJr.pdf\" target=\"_blank\">\"Gerenciamento do Ciclo de Vida de Dados de Sensores: da Produ\u00b8c\u02dcao ao Consumo\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/lis-unicamp.github.io\/wp-content\/uploads\/2014\/09\/tese_GZPastorelloJr.pdf\" target=\"_blank\">https:\/\/lis-unicamp.github.io\/wp-content\/uploads\/2014\/09\/tese_GZPastorelloJr.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Gerenciamento+do+Ciclo+de+Vida+de+Dados+de+Sensores%3A+da+Produ%C2%B8c%CB%9Cao+ao+Consumo&rft.atitle=&rft.aulast=Pastorello+Jr.%2C+G.Z.&rft.au=Pastorello+Jr.%2C+G.Z.&rft.date=19+December+2008&rft_id=https%3A%2F%2Flis-unicamp.github.io%2Fwp-content%2Fuploads%2F2014%2F09%2Ftese_GZPastorelloJr.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CerpaASCENT04-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CerpaASCENT04_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cerpa, A.; Estrin, D. (2004). \"ASCENT: Adaptive self-configuring sensor networks topologies\". <i>IEEE Transactions on Mobile Computing<\/i> <b>3<\/b> (3): 272\u201385. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTMC.2004.16\" target=\"_blank\">10.1109\/TMC.2004.16<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ASCENT%3A+Adaptive+self-configuring+sensor+networks+topologies&rft.jtitle=IEEE+Transactions+on+Mobile+Computing&rft.aulast=Cerpa%2C+A.%3B+Estrin%2C+D.&rft.au=Cerpa%2C+A.%3B+Estrin%2C+D.&rft.date=2004&rft.volume=3&rft.issue=3&rft.pages=272%E2%80%9385&rft_id=info:doi\/10.1109%2FTMC.2004.16&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChengRelay08-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChengRelay08_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cheng, X.; Du, D.-Z.; Wang, L. et al. (2008). \"Relay sensor placement in wireless sensor networks\". <i>Wireless Networks<\/i> <b>14<\/b> (3): 347\u2013355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11276-006-0724-8\" target=\"_blank\">10.1007\/s11276-006-0724-8<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Relay+sensor+placement+in+wireless+sensor+networks&rft.jtitle=Wireless+Networks&rft.aulast=Cheng%2C+X.%3B+Du%2C+D.-Z.%3B+Wang%2C+L.+et+al.&rft.au=Cheng%2C+X.%3B+Du%2C+D.-Z.%3B+Wang%2C+L.+et+al.&rft.date=2008&rft.volume=14&rft.issue=3&rft.pages=347%E2%80%93355&rft_id=info:doi\/10.1007%2Fs11276-006-0724-8&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DhillonSensor03-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DhillonSensor03_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dhillon, S.S.; Chakrabarty, K. (2003). \"Sensor placement for effective coverage and surveillance in distributed sensor networks\". <i>Proceedings from 2003 IEEE Wireless Communications and Networking<\/i> <b>3<\/b>: 1609-1614. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FWCNC.2003.1200627\" target=\"_blank\">10.1109\/WCNC.2003.1200627<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sensor+placement+for+effective+coverage+and+surveillance+in+distributed+sensor+networks&rft.jtitle=Proceedings+from+2003+IEEE+Wireless+Communications+and+Networking&rft.aulast=Dhillon%2C+S.S.%3B+Chakrabarty%2C+K.&rft.au=Dhillon%2C+S.S.%3B+Chakrabarty%2C+K.&rft.date=2003&rft.volume=3&rft.pages=1609-1614&rft_id=info:doi\/10.1109%2FWCNC.2003.1200627&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PanOptimal05-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PanOptimal05_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pan, J.; Cai, L.; Hou, Y.T. et al. (2005). \"Optimal base-station locations in two-tiered wireless sensor networks\". <i>IEEE Transactions on Mobile Computing<\/i> <b>4<\/b> (5): 458\u201373. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTMC.2005.68\" target=\"_blank\">10.1109\/TMC.2005.68<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Optimal+base-station+locations+in+two-tiered+wireless+sensor+networks&rft.jtitle=IEEE+Transactions+on+Mobile+Computing&rft.aulast=Pan%2C+J.%3B+Cai%2C+L.%3B+Hou%2C+Y.T.+et+al.&rft.au=Pan%2C+J.%3B+Cai%2C+L.%3B+Hou%2C+Y.T.+et+al.&rft.date=2005&rft.volume=4&rft.issue=5&rft.pages=458%E2%80%9373&rft_id=info:doi\/10.1109%2FTMC.2005.68&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WuSMART05-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WuSMART05_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wu, J.; Yang, S. (2005). \"SMART: A scan-based movement-assisted sensor deployment method in wireless sensor networks\". <i>Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies<\/i> <b>4<\/b>: 2313-2324. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FINFCOM.2005.1498518\" target=\"_blank\">10.1109\/INFCOM.2005.1498518<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SMART%3A+A+scan-based+movement-assisted+sensor+deployment+method+in+wireless+sensor+networks&rft.jtitle=Proceedings+IEEE+24th+Annual+Joint+Conference+of+the+IEEE+Computer+and+Communications+Societies&rft.aulast=Wu%2C+J.%3B+Yang%2C+S.&rft.au=Wu%2C+J.%3B+Yang%2C+S.&rft.date=2005&rft.volume=4&rft.pages=2313-2324&rft_id=info:doi\/10.1109%2FINFCOM.2005.1498518&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PepplerAnOver08-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PepplerAnOver08_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peppler, R.A.; Long, C.N.; Sisterson, D.L. et al. (2008). \"An Overview of ARM Program Climate Research Facility Data Quality Assurance\". <i>The Open Atmospheric Science Journal<\/i> <b>2<\/b>: 192\u2013216. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2174%2F1874282300802010192\" target=\"_blank\">10.2174\/1874282300802010192<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Overview+of+ARM+Program+Climate+Research+Facility+Data+Quality+Assurance&rft.jtitle=The+Open+Atmospheric+Science+Journal&rft.aulast=Peppler%2C+R.A.%3B+Long%2C+C.N.%3B+Sisterson%2C+D.L.+et+al.&rft.au=Peppler%2C+R.A.%3B+Long%2C+C.N.%3B+Sisterson%2C+D.L.+et+al.&rft.date=2008&rft.volume=2&rft.pages=192%E2%80%93216&rft_id=info:doi\/10.2174%2F1874282300802010192&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FiebrichTheValue06-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FiebrichTheValue06_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fiebrich, C.A.; Grimsley, D.L.; McPherson, R.A. et al. (2006). \"The Value of Routine Site Visits in Managing and Maintaining Quality Data from the Oklahoma Mesonet\". <i>Journal of Atmospheric and Oceanic Technology<\/i> <b>23<\/b>: 406\u201316. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1175%2FJTECH1852.1\" target=\"_blank\">10.1175\/JTECH1852.1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Value+of+Routine+Site+Visits+in+Managing+and+Maintaining+Quality+Data+from+the+Oklahoma+Mesonet&rft.jtitle=Journal+of+Atmospheric+and+Oceanic+Technology&rft.aulast=Fiebrich%2C+C.A.%3B+Grimsley%2C+D.L.%3B+McPherson%2C+R.A.+et+al.&rft.au=Fiebrich%2C+C.A.%3B+Grimsley%2C+D.L.%3B+McPherson%2C+R.A.+et+al.&rft.date=2006&rft.volume=23&rft.pages=406%E2%80%9316&rft_id=info:doi\/10.1175%2FJTECH1852.1&rfr_id=info:sid\/en.wikipedia.org:Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Grammar was cleaned up for smoother reading. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165501\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.086 seconds\nReal time usage: 1.138 seconds\nPreprocessor visited node count: 33801\/1000000\nPreprocessor generated node count: 41166\/1000000\nPost\u2010expand include size: 217359\/2097152 bytes\nTemplate argument size: 71876\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 1050.195 1 - -total\n 85.27% 895.499 1 - Template:Reflist\n 74.68% 784.301 46 - Template:Citation\/core\n 61.78% 648.824 35 - Template:Cite_journal\n 10.39% 109.067 7 - Template:Cite_web\n 7.80% 81.878 1 - Template:Infobox_journal_article\n 7.47% 78.416 1 - Template:Infobox\n 7.01% 73.594 4 - Template:Cite_book\n 5.37% 56.409 38 - Template:Citation\/identifier\n 4.17% 43.773 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11004-0!*!0!!en!5!* and timestamp 20190701165500 and revision id 35380\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data\">https:\/\/www.limswiki.org\/index.php\/Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","f59eece05cdc9b79c1305c6bea6b72d6_images":["https:\/\/www.limswiki.org\/images\/8\/8a\/Fig1_Perez-Castillo_Sensors2018_18-9.png","https:\/\/www.limswiki.org\/images\/8\/8c\/Tab2_Perez-Castillo_Sensors2018_18-9.png","https:\/\/www.limswiki.org\/images\/0\/03\/Tab4_Perez-Castillo_Sensors2018_18-9.png","https:\/\/www.limswiki.org\/images\/4\/41\/Fig2_Perez-Castillo_Sensors2018_18-9.png","https:\/\/www.limswiki.org\/images\/6\/65\/Tab5_Perez-Castillo_Sensors2018_18-9.png","https:\/\/www.limswiki.org\/images\/4\/40\/Tab6_Perez-Castillo_Sensors2018_18-9.png","https:\/\/www.limswiki.org\/images\/0\/0f\/Tab7_Perez-Castillo_Sensors2018_18-9.png","https:\/\/www.limswiki.org\/images\/5\/55\/Tab8_Perez-Castillo_Sensors2018_18-9.png"],"f59eece05cdc9b79c1305c6bea6b72d6_timestamp":1562000100,"4be2f1cd180a132f1a4ce3ca4fde9d6e_type":"article","4be2f1cd180a132f1a4ce3ca4fde9d6e_title":"What is the meaning of sharing: Informing, being informed or information overload? (Haugsbakken 2018)","4be2f1cd180a132f1a4ce3ca4fde9d6e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F","4be2f1cd180a132f1a4ce3ca4fde9d6e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:What is the meaning of sharing: Informing, being informed or information overload?\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nWhat is the meaning of sharing: Informing, being informed or information overload?Journal\n \nNordic Journal of Science and Technology StudiesAuthor(s)\n \nHaugsbakken, HalvdanAuthor affiliation(s)\n \nNorwegian University of Science and TechnologyPrimary contact\n \nEmail: Halvdan dot Haugsbakken at ntnu dot noYear published\n \n2018Volume and issue\n \n6(1)Page(s)\n \n46\u201358DOI\n \n10.5324\/njsts.v6i1.2546ISSN\n \n1894-4647Distribution license\n \nCreative Commons Attribution-ShareAlike 4.0 InternationalWebsite\n \nhttps:\/\/www.ntnu.no\/ojs\/index.php\/njsts\/article\/view\/2546Download\n \nhttps:\/\/www.ntnu.no\/ojs\/index.php\/njsts\/article\/view\/2546\/2796 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Theoretical perspective \n4 About the case and methods \n5 Data analysis \n\n5.1 Theme 1: Sharing as a facilitator of organizational change \n5.2 Theme 2: Sharing as a trigger for self-censorship and risk-taking \n5.3 Theme 3: Sharing in separate digital ecosystems \n5.4 Theme 4: Sharing practice as an individual informing strategy \n\n\n6 Discussion \n7 Conclusion \n8 References \n9 Notes \n\n\n\nAbstract \nIn recent years, several Norwegian public organizations have introduced enterprise social media platforms (ESMPs). The rationale for their implementation pertains to a goal of improving internal communications and work processes in organizational life. Such objectives can be attained on the condition that employees adopt the platform and embrace the practice of sharing. Although sharing work on ESMPs can bring benefits, making sense of the practice of sharing constitutes a challenge. In this regard, the paper performs an analysis on a case whereby an ESMP was introduced in a Norwegian public organization. The analytical focus is on the challenges and experiences of making sense of the practice of sharing. The research results show that users faced challenges in making sense of sharing. The paper indicates that sharing is interpreted and performed as an informing practice, which results in an information overload problem and causes users to become disengaged. The study suggests a continued need for the application of theoretical lenses that emphasize interpretation and practice in the implementation of new digital technologies in organizations.\nKeywords: enterprise social media, sharing, public organizations, Norway\n\nIntroduction \nIn the last decade, many private and public organizations have started to take great interest in enterprise social media platforms (ESMPs).[1] Implying an expansion of Enterprise 2.0[2], the term refers to a platform used for internal communication in organizations. ESMPs contain a range of features that are used to share and organize information, such as tagging systems, user profiles, search engines, follower features, discussion boards, and group features. Known examples of ESMPs are Yammer and Facebook@ work. The platforms are assumed to bring a range of benefits for organizations and for the organization of work processes. These benefits can include enhancement of the quality of internal communications and workflows. A central practice related to the successful use of ESMPs is active engagement by users or employees through the sharing or co-creation of content, although the workplace principle is not always easy to put into practice.\nSince the end of the 2000s, several large Norwegian private and public organizations have introduced ESMPs to their employees. The incentive for their acquisition is motivated by various goals. For example, they can reduce internal organizational barriers, enhance organizational communications, and cut down on time spent sending e-mails. In this way, one can attain a greater overview of organizational activities and the competencies of employees. In this regard, ESMPs are presented as a solution that can contribute to solving traditional management challenges that are faced daily by public organizations. In the wake of this development, discourses focusing on the importance of sharing in organizations emerge. Top and middle managers stress the sharing of work and engagement via ESMPs as means of bringing about organizational change and unity and the use of digital technologies in work life. Surfing on the top of such management discourses is an emphasis that employees embrace a \u201csharing culture.\u201d Such developments substantiate the importance of analyzing the meaning of \"sharing\" through social constructionist research perspectives regarding the use of technology in organizations.\nIn 2012, a Norwegian County Authority decided to upgrade its intranet to become an ESMP, an effort initiated by the top management. The goal was to simplify the workload because the employees previously worked across separate forms of information and communications technologies (ICTs). A further objective was to transfer work practices from e-mail and local storage to the newly acquired platform by sharing. Although the technical implementation of the ESMP was successful, top management found that employees were not sharing work as intended. By using a practice perspective on technology and the organization of work, as well as related research on enterprise social media[3][4], this paper questions how a group of employees working in the County Authority interpret the meaning of sharing and put it into practice through the ESMP. The use of a practice perspective indicates that employees face challenges in interpreting the meaning of sharing. Sharing is interpreted and performed as an informing practice, which results in an information overload problem and disengaged users. \nIn order to tackle the research question, the paper is divided into different parts. The following section addresses the scholarly discussion upon which the study is based. Thereafter, the research strategies used to complete this study are outlined. The research findings are subsequently presented, before the research results are discussed in relation to the relevant research horizon. The final part concludes the paper.\n\nTheoretical perspective \nSharing has emerged as a significant social action performed by billions of social media users worldwide. In general, sharing brings with it a range of claimed unintended consequences[5], and it can be defined as a practice that originates in reconstituting dynamics and reciprocal relationships between the material properties of social media and social action. As such, it has affected the organization of social life. For example, what people share on social media draws media attention and is predicted by traditional media as having positive and negative effects on our well-being. The sharing of experiences can create community awareness on civic matters that are important to society, but also accusations of egocentric behavior. We also see that ongoing online socialization may lead to new mediated practices such as phubbing and digital detox. \"Phubbing\" is defined as the act of ignoring a person\u2019s surroundings through the use of a cellphone, which is deemed an impolite action. \"Digital detox\" is understood as a period in which a person stops using electronic, web-connected devices such as smart phones and tablets. These indicate that the organization of communicative practices in the digital sphere can become unmanageable and chaotic. Although research on social media and sharing has proliferated, organization researchers have yet to fully frame the impact of sharing on organizational life.\nIn consequence, such dynamics call for the development of a research perspective that discusses the meaning of sharing in organizations by use of ESMPs, especially where sharing assumes a different role than that intended. This argument is valid for several reasons. Surprisingly, organizational scholars who study knowledge-sharing processes by use of knowledge management systems (for example) claim that what is actually shared by users on platforms for the sharing of work has yet to be adequately framed.[6] In particular, a knowledge gap seems to exist regarding the formation of sharing processes and how this is related to emergent properties coming from the use of recursive technology in work processes.[7] Instead, the knowledge management research stream has examined predefined assumptions of sharing[8][9] and conditions that prevent the sharing of knowledge in virtual communities.[6][10][11]\nWith the advent of ESMPs in organizations, there is an urgent need to formulate and facilitate a new and much broader research agenda. This has been seen in organizational research, which has introduced new definitions of platforms and has criticized existing definitions of social media for their shortcomings. An example of a new definition is enterprise social media, which is defined as: \u201cweb-based platforms that allow workers to (1) communicate messages with specific coworkers or broadcast messages to everyone in the organization; (2) explicitly indicate or implicitly reveal particular coworkers as communication partners; (3) post, edit, and sort text and files linked to themselves or others; and (4) view the messages, connections, text, and files communicated, posted, edited, and sorted by anyone else in the organization at any time of their choosing.\u201d[1] This definition is a modified version of Kaplan and Haenlein\u2019s[12], who define social media as a \u201cgroup of Internet-based applications that build on the ideological and technological foundations of Web 2.0, and that allow the creation and exchange of user-generated content.\u201d\nCurrent definitions of social media are criticized for failing to adequately explain organizational processes.[13] To accommodate these limitations, we find an emerging body of research studying ESMPs that uses a social constructionist research lens on technology and the organization of work.[14] This research stream has (among others) developed an \"affordance\" perspective[15] to grasp the impact of ESMPs in organizations. Affordance[15] stresses the advantages of technologies. It argues that technologies can be perceived as beneficial in performing activities without paying attention to what an object \u201cis,\u201d that is, to ask what it can afford.[13] The focus on perception means to put emphasis on an object\u2019s utility, and affordance provides the possibility of understanding action potential and the capabilities of a technology, and how it can be linked to processes in organizations.[13] The affordance lens is used to place greater emphasis on the meaning of materiality, which is said to have diminished in value as other concepts dominate the research agenda, such as sociomateriality.[16] Affordance is linked to critical theory, which is deemed as providing new innovative ways of addressing the relationships between materiality and immateriality.[17] A critical theory approach assumes the existence of multiple realities that operate interchangeably and independently of one another, with the implication that actors and objects are self-contained entities that influence each other through impacts or social interaction.[17]\nIn contrast, sociomateriality[18], which draws on influences from Actor-Network-Theory[19][20][21], has emerged as an alternative sub-research stream to understanding the impact of social media in organizational studies. Sociomateriality, which assumes that \u201cmateriality is intrinsic to everyday activities and relations,\u201d[22] provides an alternative approach to understanding the meaning of technology and therefore what the potential consequences of social media might entail. Orlikowski argues that previous understandings of materiality in management studies were framed around an ontology of separateness[23], in which one theorized that the material and the immaterial are separate entities and realities.[24] Sociomateriality instead argues that they should be seen as linked, equal, and inseparable, which means addressing a relational ontology. Therefore, future organizational researchers with an interest in social media could study technological artifacts \u201csymmetrically to the humans, and as equivalent participants in a network of humans and non-humans that (temporarily) align to achieve particular effects.\u201d[24] In this regard, Orlikowski and Scott[25] apply a sociomaterial practice perspective in one of their latest research works on valuation regimes. They demonstrate that online evaluations of hotels performed by users on social media have a drastic impact on the domestic travel industry.[25]\nHowever, an important argument running through the above literature is the requirement for more theorizing. Current research focuses on a particular platform or features, leading to a claim that researchers are incapable of making inferences about the consequences of the material for organizing.[1] Current definitions of ESMPs are too application-focused and overlook the social dynamics and reciprocal relationships between the material and the immaterial.[1] Treem and Leonardi[13] argue that this causes scholars to fail to possess sufficient terminology to explain the ways in which ESMPs can influence social behavior and to generalize matters to organizations across contexts. Here, the affordance lens offers researchers the possibility of making interesting analyses regarding the ways in which ESMPs influence organizational processes such as socialization and power aspects in organizations. Research has suggested various affordances that ESMPs can give for organizational processes. For example, Treem and Leonardi[13] suggest that ESMPs can enable four affordances: visibility, edibility, persistence, and association. A case in point highlighting the meaning of a singular affordance is Treem and Leonardi\u2019s[13] argument that the affordance visibility is seen when employees use an ESMP to make their behavior, knowledge, preferences, and commutation network visible to others. They argue that actions like posting updates, showing a list of friends, and writing personal profiles are beneficial and enable the visualization of work to third parties. Leonardi[26][27] illustrates this point by showing that work interaction on an ESMP platform is pivotal for knowledge work and for the transfer of knowledge in a large organization. Based on a study of a financial service in the United States of America, Leonardi shows that the use of a company\u2019s ESMP assists third-users to enhance awareness of meta knowledge, as one learns about the competencies of co-workers and the matters on which they are working. ESMPs can be used to make accurate interferences of people\u2019s meta knowledge and the sharing of co-workers\u2019 communication activities, and communicating via messenger software can offer innovative products and avoid the duplication of work. In a related study, Leonardi and Meyer[28] develop the above claim in a study of a communications business unit in a telecommunication unit. Leonardi and Meyer test out a set of hypotheses and instances of knowledge transfer to show that when knowledge workers are exposed to communication activities on an ESMP, internal communication can be enhanced.\nBeyond these works, researchers have theorized affordance in two other principle directions. First, we can identify works that conceive of affordance at a conceptual level. Second, researchers develop the term empirically through case study designs. An example of the former is a study by Majchrzak, Faraj, Kane, and Azad[29], who demonstrate how ESMPs have four affordances to inspire engagement of visible knowledge conversations in organizations. These include met voicing, triggered attending, network-informed associating, and generative role-taking. For example, met voicing would mean that an ESMP has the action capability to enable users to react to others\u2019 presence, profiles, content, and activities. Ellison, Gibbs, and Weber[30] develop a collective affordance and affordances for organizing, and explore the role of organizational affordances in light of the fact that organizations become distributed entities. \nThe affordance lens needs to be broadened beyond the context of individual uses, which has been seen in many analyses. Ellison et al.[30] note that the affordances of ESMPs can include concepts such as social capital dynamics, identity formation, context collapse, and networked organizational structures. Fulk and Yuan[31] argue that ESMPs have the affordance to solve organizational challenges and represent a preferable platform for organizing knowledge sharing in comparison to older knowledge management systems. Fulk and Yan argue that by combining transitive memory theory, public good theory, and social capital theory, ESMPs have the affordance to deal with three associated challenges in the sharing of organizational knowledge. These include knowledge of the location of expertise in the organization, motivation to share knowledge, and the development and maintenance of relationships with knowledge providers. In considering work that uses the affordance lens for cases from organizational life, however, Vaast and Kaganer[32] explore how organizations react to employees\u2019 adoption of ESMPs. Based on a sample of corporate policy documents, Vaast and Kaganer find that organizations view ESMPs as more of a risk than an asset. Oostervink, Agterberg, and Huysman[33] have undertaken a study connecting enactment to the affordance lens, as affordances are enacted in practice and institutional forces in an organization can shape how ESMPs are used by employees. Oostervink et al. point out that the institutional logics of a corporation and employees\u2019 professional expertise shape the knowledge that employees share on ESMPs. Although the affordance of visibility and associability are assumed to enhance knowledge sharing in organizations, Gibbs, Rozaidi and Eisenberg[34] find the opposite effect. They performed a study among a group of engineers and noted that engaging EMPSs create contradictions in workplace interactions. For example, constantly remaining accessible and open to other suggestions is a hassle, thus causing employees to feel that they need to hide certain behaviors from others. Based on this finding, Gibbs et al.[34] have suggested that scholars theorize affordance in terms of dichotomies, not singular affordance concepts. They establish three affordances with which users interact when they use an ESMP. These include visibility-invisibility tension, engagement-disengagement tension, and sharing-control tension.\nOrganizational researchers have also explored ESMPs from other angles. Research shows that employees are receptive to ESMPs in certain organizations: those that make and sell the technology, being IT companies and organizations with the resources to research the technology in large projects. In this regard, IBM\u2019s Beehive project is groundbreaking. One can read in numerous research papers the ways in which Beehive has been implemented and tested on IBM employees, as researchers have documented basic user behaviors. Researchers have focused on an entire ESMP[35] or on features such as tagging systems[36] and user profiles[37] Beehive research papers often use a social capital perspective[38] to establish links between ESMP uses, and connecting strategies constitute a recurring theme. Researchers have identified that IBM employees use Beehive as a platform to expand their professional networks, using it to communicate with colleagues across organizational levels.[39] IBM employees undertake a range of search and retrieving practices[40] and use Beehive as a knowledge repository.[41] Other case studies on ESMPs in organizations other than IBM exist, but have yielded limited insights. They show that employees use ESMPs to streamline their online behavior to work practices and organizational affiliation.[42] Researchers have examined the challenges of adopting an organizational ESMPs. It is not uncommon to come across findings that highlight how employees continue to prefer to communicate via e-mail and chat software, and silently monitor news streams.[43][44][45] Consequently, one finds a pattern that a core group adopts SNS and maintains network activities, while a larger group uses \"older\" forms of ICTs.\nTherefore, the research horizon described appears limited and somewhat inchoate. Scholars have predominantly focused on the material properties of ESMPs and have contributed through experimental theorizing. Absent from the research literature is the specific role that interpretation and practice take in employees\u2019 recursive use, with ESMPs a crucial aspect of work processes. Moreover, it appears that the research field has yet to adequately frame whether sharing can take on a different role than that intended, and what it means when technology is used differently in an organizational setting. This means that the research field can advance a research lens focusing on situations and enacting with emergent properties that come from the use of recursive technology, hence placing clearer emphasis on what people do with an ESMP. Thus, one can use a practice perspective on technology.[3] A practice lens on ESMPs can also be used to fill a knowledge gap regarding the formation of knowledge-sharing processes in the use of ESMPs, facilitating our understanding of the unintended consequences of technology use in organizations. Here, Orlikowski and Gash\u2019s[4] technological frames can be of assistance. Technological frames are defined as: \u201cthat subset of members\u2019 organizational frames that concern the assumptions, expectations, and knowledge they use to understand technology in organizations. This includes not only the nature and role of the technology itself, but the specific conditions, applications, and consequences of that technology in particular contexts.\u201d[4] The concept can enable us to frame how individuals and social groups in organizations alike make sense of a technology to determine their actions, allowing us to move beyond conceptualizing a technology\u2019s mere value and perception among users. Technological frames problematize the \"taken for granted\" notions of a technology and can facilitate an understanding of how individuals and groups in organizations develop particular assumptions, expectations and knowledge of a new technology in an organizational setting. Orlikowski and Gash[4] have illustrated technological frames in their study of the implementation of the groupware Notes in a consultant company. The researchers interviewed the implementers and adopters, grouping them into \"technologists\" and \"users.\" \"Technologists\" was used to refer to technology staff, whereas \"users\" comprised the organization\u2019s consultants. Orlikowski and Gash demonstrated a large set of differences in terms of expectations and actions, which they attributed to differences in technological frames. For example, the technologists viewed Notes as an enabler for information sharing, electronic communication, document management, and online discussion, which they believed could contribute to collaboration. The users\u2019 interpretation was different, viewing Notes\u2019 electronic e-mail features as a potential substitute for existing communication technologies such as fax and telephone. The technologists therefore framed Notes as a collaborative technology, whereas the users used it as a means for individual and personal communications.\n\nAbout the case and methods \nNorway is divided into nineteen large administrative units, called counties, and roughly 350 municipalities. Each county is governed by a County Authority (CA), rendering this form of governance the first form of subdivision in the country. The CA where the data for this case study were collected consists of a political structure, an administrative body, and welfare units. The political structure is an elected body consisting of the County Council, the County Executive Board, the County Principal Standing Committees, and the County Mayor. The County Council is supported by an administrative body, the County Administration, which implements and administers policies. The County Administration is organized into eight administrative units and an executive secretariat board. Other welfare units also exist, which play a role for citizens and produce services. Among others, these consist of high schools, libraries, dental services, and transportation. A large body of the CA workforce includes high school teachers, and in total the CA contains approximately 2,800 employees.\nThe study made use of an exploratory qualitative research strategy. This approach was used to facilitate an in-depth investigation of the ways in which public employees working predominantly in the County Administration used the ESMP and interpreted sharing in an organizational setting. The study is primarily based on qualitative research interviews. Written documentation was collected, but it is not used as part of this paper and is thus excluded from the data analysis. However, the research design started with an informal approach to employees in the CA who had been responsible for the public procurement and implementation of the ESMP. They agreed to be part of the exploratory qualitative study and recruited the informants. Eight informants were recruited, and they worked in different departments and holding positions, predominantly as advisors in the County Administration. The criterion for selection was that they were all users of or involved in the implementation of ESMP. In sum, eight semi-structured qualitative interviews with the use of a guide were completed. The interviews were undertaken one-to-one, meaning that only the researcher and the informant were present in the interview setting. Each interview lasted approximately one hour and focused on two main themes related to sharing: previous user sharing experience on social media, and how the individual used the ESMP to organize the sharing of work. Each interview was recorded using a digital audio recorder. The data were collected over two periods, from August to September 2013 and in February 2014. The study was based on informed consent and the informants were anonymized. The background of the informants is displayed in Table 1.\n\n\n\n\n\n\n\nTable 1. The background of informants\n\n\n\nGender\n\nNumber\n\nPosition\n\nDuration\n\nDate\n\n\nF\n\nI-1\n\nAdvisor\n\n1 hour\n\n27 Aug. 2013\n\n\nF\n\nI-2\n\nMiddle Manager\n\n1 hour\n\n30 Aug. 2013\n\n\nF\n\nI-3\n\nAdvisor\n\n1 hour\n\n5 Sep. 2013\n\n\nF\n\nI-4\n\nExe. Director\n\n1 hour\n\n12 Feb. 2014\n\n\nM\n\nI-5\n\nAdvisor\n\n1 hour\n\n10 Feb. 2014\n\n\nM\n\nI-6\n\nAdvisor\n\n1 hour\n\n17 Feb. 2014\n\n\nF\n\nI-7\n\nConsultant\n\n1 hour\n\n18 Feb. 2014\n\n\nM\n\nI-8\n\nAdvisor\n\n1 hour\n\n18 Feb. 2014\n\n\n\nOnce the interviews were completed, they were transcribed. The data analysis was inspired by an open coding strategy of the interview data. Here, the main focus was on finding emerging patterns, which consisted of grouping and comparing the informants\u2019 perceptions, user patterns, and experiences of sharing. The informants\u2019 answers were grouped into four broad themes. In order to offer the informants a voice, direct quotations are used in the data analysis.\n\nData analysis \nThis section presents the data analysis and seeks to answer the research question addressed earlier in the paper: how does a group of employees in the CA interpret the meaning of sharing? The emphasis here is on breaking down this notion by presenting four emergent themes from the open coding of the interview data. Each theme represents an adoption of the new workplace principle of sharing, showing how individuals develop particular assumptions, expectations, and knowledge about its definition. Performing such an analysis can facilitate a clearer understanding of the meaning of \"sharing.\" The four themes are displayed in Table 2.\n\n\n\n\n\n\n\nTable 2. Emergent themes from the data analysis\n\n\n\nTheme no.\n\nName of theme\n\n\n1\n\nSharing as a facilitator of organizational change\n\n\n2\n\nSharing as a trigger for self-censorship and risk-taking\n\n\n3\n\nSharing in separate digital ecosystems\n\n\n4\n\nSharing as an individual informing strategy\n\n\n\nTheme 1: Sharing as a facilitator of organizational change \nThe first theme pertains to expectations of sharing as a facilitator of organizational change and represents an approach whereby top managers want their employees to work in a different way. This interpretation comes as little surprise, because the CA\u2019s top management was the initiator behind the ESMP. Sharing represents a \u201cproblem-solver\u201d and can produce organizational belonging in the face of internal forces that may contradict unity. Certain professions, such as teachers\u2014a large profession\u2014are assumed to identify with the high schools in which they work and with their professional identity, rather than with a feeling of belonging to the CA. However, the motivation for implementing the social ESMP was related to disentangling a common problem with which most organizations struggle to cope today: escaping the meeting culture, e-mail overload, and the use of too many forms of IT:\n\nThere was a need to create an ESMP that considered the fact that we worked with various work surfaces. You had to open each system one at a time, just to approve an invoice. Our challenge was also to escape the \"hell of e-mail.\" (I-4)\nThe top management aimed to simplify employees\u2019 workload. This was related to the fact that employees worked across several non-integrated ICTs and stored information in different places, making it challenging to create an overview. In response, the top management argued that a single site that could work as the central access point connecting all the employees was required. This would be realized by replacing the intranet with the ESMP. Therefore, a project group was created to work with various drafts of a new interface design, which would break an established work pattern in the CA. While the intranet was run as an internal website on which a group colleague wrote internal news stories, the new design suggested that the ESMP should be the main site opened by employees each day, with embedded sharing features and URL links to each internal IT system. In this way, the ESM would be the melting pot where everybody talked about work. Afterwards, an organizational discourse emphasizing the importance of a sharing culture emerged:\n\nIt was clear that we needed something that could enable us to work with the culture across [the organization], knowledge of each other\u2019s work. My responsibility has been to legitimize sharing in the management structure. Parallel to that, we made attempts to raise discussion about organizational culture and work processes internally. Should we establish a sharing culture, in the sense that people can easily participate in and reinforce each other\u2019s work, or take part in reports, or take part in other kinds of things, take part in the knowledge we have, this requires a culture where [people] actively participate. (I-4)\nHowever, translating and making sense of sharing into a manageable practice proved challenging, as it surfaced as ambiguous:\n\nIt sounds very good. It has a great positivity to it when it\u2019s presented, but not so great when you try it out in practice. At an early stage, there was a positive feel. You didn\u2019t know exactly what it was. There was this belief that we should change the work culture. (I-5)\nLater, this awareness amplified as the initiators realized that the employees rarely started a work process by beginning from scratch\u2014by creating a document in which everyone can engage, for example\u2014but instead viewed sharing as an informing practice of circulating ready-made documents. A recurring theme was how sharing was directly linked to previous publishing habits. The employees were accustomed to an article format, meaning that postings had an \"internal story\" label attached to them. Participation involved performing simple tasks, like writing status updates, following colleagues, and updating profiles. The ESMP was an information channel where information was pushed out, not a platform in which one engaged in two-way dialogue. Furthermore, the employees fulfilled activities that required little commitment, such as posting a profile picture, writing status updates, tagging competence, or uploading completed documents. Beyond these actions there was little evidence that the users aimed to participate in activities requiring the performance of reciprocal actions:\n\nNinety percent of the information posted on the ESMP is not something that we\u2019ve published. It\u2019s made by the organization. It has become a place where items are shared. It\u2019s divided between heavy and light documents. People share when documents are finished. You don\u2019t see many examples where people collaborate on a document, which is part of a work process. We haven\u2019t gone any further in changing work culture and the ways we work. (I-5)\nTheme 2: Sharing as a trigger for self-censorship and risk-taking \nThe second theme emerging from the data analysis is that sharing includes a high degree of self-censorship and is associated with risk-taking. This pattern was seen among employees who use the ESMP and who were not part of the actual implementation process, who work across different departments and who are affiliated further down in the management structure. However, an interesting pattern consists of the ways in which earlier and alternative private social media platforms shape perceptions of appropriate net behavior. For example, the informants registered on social media services that became mainstream in the 2000s (the informants\u2019 use is displayed in Table 3). The data indicates that the informants had a strong passive and critical approach to participation. They saw the benefits of sharing, but demonstrated a \"reading and textbook\" approach consisting of monitoring others\u2019 actions and only frugally sharing about themselves. This molded a view that sharing was seldom regarded as a two-way communicative process between two parties:\n\nI don\u2019t share information about what I\u2019ve eaten for dinner, what I do during my evenings. I share if it is appropriate and relevant, not just one of those private things. Sometimes I post a picture of a mountain summit on Facebook. I have a pretty high threshold that the summit should be a little more interesting for others to see. (I-2)\n\n\n\n\n\n\nTable 3. The informants\u2019 use of social media platforms\n\n\n\nGender\n\nNumber\n\nPosition\n\nFacebook\n\nTwitter\n\nLinkedIn\n\nInstagram\n\nGoogle Drive\n\n\nF\n\nI-1\n\nAdvisor\n\nX\n\n-\n\nX\n\n-\n\n-\n\n\nF\n\nI-2\n\nMiddle Manager\n\nX\n\nX\n\nX\n\nX\n\n-\n\n\nF\n\nI-3\n\nAdvisor\n\nX\n\n-\n\n-\n\n-\n\n-\n\n\nF\n\nI-4\n\nExe. Director\n\nX\n\nX\n\nX\n\n-\n\n-\n\n\nM\n\nI-5\n\nAdvisor\n\nX\n\nX\n\nX\n\n-\n\nX\n\n\nM\n\nI-6\n\nAdvisor\n\nX\n\n-\n\n-\n\n-\n\n-\n\n\nF\n\nI-7\n\nConsultant\n\n-\n\n-\n\n-\n\n-\n\n-\n\n\nM\n\nI-8\n\nAdvisor\n\nX\n\nX\n\nX\n\n-\n\nX\n\n\n\nTherefore, it is more accurate to maintain that communicative practices are based on being informed and to inform, which ignores how a goal is to engage with an equal to create knowledge. This is reflected in beliefs about socializing and the means of ascribing social media with personal labels. For example, Facebook belongs to the private sphere and is used for \u201cscrolling after fun stuff and setting likes.\u201d LinkedIn is a \u201cCV database,\u201d while Twitter is a medium where \u201cone only sends URL links to news you have already read.\u201d Such beliefs hint at what is acceptable to share on which platform:\n\nThere are people who write things that shouldn\u2019t be shouted out loud. One gets the impression that, \u201cI\u2019m sad today\u201d. If anyone had a nice trip, which is worth writing about, then you can do it, by all means. It\u2019s easy to explain it, but when you see it, it gets difficult to say it with words. There are some things that are just a bit \"intimate,\" very personal stuff. It doesn\u2019t belong on social media, because it doesn\u2019t concern everybody. (I-6)\nGrooming and gossiping are disregarded, and publishing of overly personal information evokes intimacy and is taboo. Instead, the informants had clear perceptions that what should be shared had to be interesting and relevant, meaning that the value of what can be shared has to be informative and of high quality. This creates boundaries delineating how one should engage on the ESMP, which are manifested in the form of distinctions. For example, the most common trait was to draw distinctions between work and non-work-related use, and external and internal use. Here, Twitter is work-related and is used because politicians interact in the Twittersphere, and it forms an arena for public debate. Some informants see Twitter as a \u201clistening post\u201d where one can monitor what is going on and to stay up-to-date with current events:\n\nI use Twitter because it gives me something related to my work and because I follow public debates. So, I started paying attention to what was going on Twitter. In our department, we follow public debates. (I-8)\nConsequently, one could expect that engaging on the ESMP should represent a challenge. Yet, the data suggest otherwise, as sharing on the ESMP was connected with risk-taking, expressed in the informants\u2019 views of their willingness to make a work process transparent to others. Here, the informants placed themselves on a scale from relatively open to very restricted in what they shared:\n\nI try to set an example. When I create a document, I publish it right away. It says it\u2019s a document in progress, which we\u2019re working on. (I-2)\nI don\u2019t have a problem with posting something that is not one hundred percent complete. I would have made it clear that this is \u2018work in progress\u2019 which I want feedback on. (I-5)\nHowever, we also find examples illustrating the risk-taking associated with publishing work in progress. This is related to the idea that informants assume that they can be criticized, meaning that\npublished and unfinished work can create misunderstandings:\n\nOne thing is that some us find it a bit uncomfortable to share things that are not finished, because then we get criticized. It becomes uncomfortable when it\u2019s not completed. If things are just published and not finished, it can cause more harm, because it creates sanctions on something that it was not intended to be. We have specific discussions within our work areas, documents concerning the management side and on the political aspects, which we publish. When things are at a certain stage, a working document, it is not intended that everybody should see it. If there are many who use it, they can abuse it in a number of contexts. (I-2)\nThe data also show that informants seek \u2018approval\u2019 from their closest manager to publish content on the ESMP. Rather than deciding independently, for example, as the basis for sharing a document, informants enforce a quality-safety practice where they ask permission from an authority in the management structure:\n\nThings that are unfinished and not approved can create panic when it is a different figure from what you think is going to be on paper. If we begin to rewrite the CA\u2019s economy and everyone can read that, there will be something new to most people. Many people absorb it, even when it is wrong. It creates a lot of \"storm\" in your organization if it is not correct. I can take an example from the corporate governance program, which has an indicator called \"financial statements and budgets,\" which shows how much of a deficit\/surplus we have to date. It is an indicator that gets its numbers straight from our accounting system. When we updated the financial system, the indicator \"froze\" itself in Corporater and showed figures from November 2013. This is a completely wrong figure. We have notified about that on the ESMP, but still I keep getting phone calls that the figures are wrong. (I-6)\nAnother informant gave a similar example:\n\nIf I am to work on a case, I want to have the final answer before I publish it. I can give you an example, which applies to the CA\u2019s dental clinics. When they want to send over a thing, in the process, things that go to debt collection arrive on my desk. I have not posted anything during the process because I wanted my manager to look at the draft along with other managers. It\u2019s the way that I work, the way I think: the routine should be completed before any dentist gets access. Considering that you want to have a unified management involved, they have to see the final result first. Then time passes, and we have a routine: we end up with draft C and D, until we finally land on something. (I-7)\nTheme 3: Sharing in separate digital ecosystems \nThe third theme from the interview data illustrates the ways in which informants adopt social media services and construct knowledge-sharing processes that form part of a work process. Thus, it makes sense to state that the informants create separate digital ecosystems that are used when they perceive that the ordinary ICTs provided by the CA are insufficient to performing their work, which influences employees to look for alternatives, a technology-adoption that occurs \"under the radar\" of the IT department.\nLooking at particular practices, an informant explained how they combined Dropbox and Google Drive as part of a work process that was used to complete the organization of a public procurement. In several cases, the CA works together with the neighboring municipalities. As part of the process, the CA assumes the lead role as the public buyer and lead organizer, meaning that the CA acts on behalf of many municipalities to achieve greater benefits for all. This work requires collaboration with colleagues in other municipalities. In that regard, one can expect that colleagues in different municipalities have diverse needs and competencies, hence many persons voice different opinions and needs. This will lead to long e-mail exchanges and numerous attached documents, with the effect that one quickly loses the overview. Instead of sending back and forth large numbers of e-mails with large documents attached, the respondents used Dropbox or Google Drive to increase the efficiency and economy of the work process for everybody:\n\nWe created a Dropbox account because we don\u2019t have the same e-mail system or share the same case management system. And it\u2019s challenging. You don\u2019t get Dropbox solutions on the PCs here. The IT department thinks it\u2019s unsecure, [lacking] information security. We need tools to do our job, so we ended up defying that a bit and we downloaded the software to our PCs. Sometimes it happens that we use Google Drive when working with external partners. I used Google Drive to share documents more efficiently than by e-mail, before they get too large. (I-8)\nAnother practice is how Facebook groups are used either as information repositories or for external communication with particular groups who use the welfare services provided by the CA. Here, one does not find examples of practices that demonstrate knowledge sharing between several parties, but merely how Facebook groups are used as public bulletin boards, where online sharing again represents an informing practice. For instance, an employee was a representative in one of the CA\u2019s worker unions and interacted with representatives from other CAs. In the process, they created a Facebook group that enabled them to stay in contact and inform one another:\n\nIt was part of a different role, which was part of a task I had here in the CA. I had contact with others with the same role in other CAs. We used the Facebook group to share information that was more or less of the same nature. It was a way to share knowledge on issues of health and safety at work. (I-1)\nAnother public employee explained how they created a Facebook group to communicate with a group of citizens the CA serves directly, students in high schools. The Facebook group was created on the assumption that students would contact the CA there; given that students are in the social media landscape, they concluded that the CA also needed to be present in a similar capacity. After some years of use, the Facebook group has roughly 300 \"likers,\" but expectations have not turned into reality. Relatively few requests from students have been seen. Instead, it has turned into more of a public bulletin board where information is posted:\n\nIt runs every day. We don\u2019t get many requests. We publish when we have specific information. We were unsure whether it would be an active user channel. I think it\u2019s going to become that in the long run. We intend to continue to use it and improve its uses, and even get more users. (I-3)\nTheme 4: Sharing practice as an individual informing strategy \nThe fourth theme emerging from the analysis demonstrates how the employees turn sharing into individual informing strategies, which arguably fulfill a goal of complying with an overall objective to share on the ESMP. The fourth theme additionally demonstrates the challenges associated with performing sharing in an internal organizational setting, and translation problems connected with practicing sharing as a two-way communication act, given that it again becomes an informing and pushing strategy of camera-made information to an audience that does not respond. This is illustrated by scrutinizing a particular feature deemed as being important for creating the conditions for sharing on the ESMP, so-called \"rooms.\" Rooms can best be described as Facebook groups or information repositories that operate as spaces for cooperation. Within them, users can upload and download documents, follow people, and receive messages about recent activities. The rooms have members and were grouped according to the CA\u2019s department structure and across departmental borders.\nIn considering particular overall user experiences, the data show that the informants adopted the rooms. They registered members and followed rooms, uploaded documents and so forth, such that it was common to follow between two and five rooms. Afterwards, the informants experienced challenges, illustrating the disadvantages and benefits associated with sharing. First, the informants created rooms and registered members who worked in the same department or in the same field as themselves. Second, the findings indicate that upload documents consisted of re-published information that was already stored in other places. A lack of data exists that shows that employees created new documents and began to co-write them in real-time; rather, they uploaded approved and ready-made documents that were only read for notification purposes. This indicates that sharing is an informing practice to a large audience, which does not invite a two-way communication process. Third, all informants reported that little interaction (such as participation and reading discussions) occurred in the rooms. In sum, users framed the rooms as information repositories rather than as sites for collaboration. Thus, we see the pattern that employees with super-user status\u2014users who enjoyed an administrative role in the rooms\u2014tried to stimulate increased engagement, which represented an outcome of a lack of responses to their informing practices. In order to reach the top management goal of sharing, super-users adopted particular roles and strategies to promote participation, which in turn illustrates the challenges pertaining to sharing.\nIn reviewing these practices, it can be noted that one super-user would adopt an \"online gardener\" strategy and attempt to encourage co-workers to engage in the rooms she administrated. This is not dissimilar to an automated e-mail notification feature, which is generated following a period of interactivity in a knowledge repository. She assumed the role of a sharer and pusher of information, which consisted of sending friendly e-mail reminders when she uploaded new documents:\n\nI send an e-mail to everyone who has an interest in the room and then I share information with them that it\u2019s posted on the ESMP. Then I invite them to follow the room, because there is information there that is relevant to them. I say that it will only be posted there. That I\u2019ve done for about a year. (I-1)\nThis means informing across multiple channels, turning sharing into a practice of double in-forming. Afterwards, the user questioned the value of sharing and was uncertain about the extent to which her efforts were worthwhile, a thought shared by another informant:\n\nI note that there are not many who follow the rooms, after many invitations and reminders to others who I think might have an interest in it. And the thoughts come. Do we spend unnecessary time on posting information that people do not read anyway? (I-2)\nThis raises the question of whether the room members post material and use the rooms as intended. For example, after uploading, an informant also received phone calls to ask if the same documents could be sent by e-mail:\n\nI often get the question, if I can send them an e-mail when there is new information in the rooms. We have decided on that\u2014no, we don\u2019t send an extra e-mail. We put it out there, and then people must seek it out themselves. I feel that people don\u2019t pay attention to all that is posted in the rooms. They would have paid attention if we had sent it in an e-mail. But we have made a conscious choice on that. I think that people read it if they get an e-mail because it\u2019s a direct message aimed at them, rather than having to search for the information themselves. (I-2)\nThis experience shows the start of a disengagement regarding sharing, as it vanishes and becomes overtaken by other assignments whose completion is deemed more important:\n\nWe have two rooms. I post a lot of information in them. I try to ensure that new information is posted. But I do not use the opportunity to follow other rooms on the ESMP, for example, as I had hoped and thought I would. It disappears into my daily work life. When I need information, I don\u2019t find it with the search mechanisms that we have today as we had with the intranet, although there is more information out there now. But now, I think it is harder to find information. (I-2)\nInforming over a long period of time creates an awareness that attached to sharing is an embedded information overload problem. This is illustrated by repeatedly performing an informing practice wherein users redirect information that is stored elsewhere. For example, while information is stored locally on hardware or in a local folder, such information is exposed and redistributed multiple times in the rooms. Hence, making information available to create transparency led to other consequences:\n\nThe intention with the ESMP was that we should move away from local storage of information in our own local folder structures. Everything was to be stored on the ESMP. I\u2019m skeptical of it, because it is such a vast amount of information that it makes it difficult to identify what is relevant. We end up with huge hits when we search, and we spend a lot of time on finding out what is relevant. And when we do not have the rigid old structure, which we had under the intranet, we spend a lot of time looking among all the hits we get. I think we would have wanted it to be a little stricter on what should be stored. Things should be deleted, if they are considered [ir]relevant. I\u2019m also skeptical that we use the ESMP as a primary storage source for everything. I\u2019m also concerned because we could forget the formal filing and procedural rules that we have to deal with. When we publish on the ESMP, we think we preserve it forever, and that\u2019s not right. There are some formal things that make me skeptical. The most concerning thing, however, is that it has become such a huge volume of documentation. And when it comes to relevant and non-relevant information on individual characteristics, I am a bit skeptical of that Facebook style of writing status updates. I think it\u2019s nice to have colleagues, but it\u2019s more interesting knowing how we are professionally connected. We are a large organization. If all of us post information that we were sick, and that we are looking forward to the weekend, there is a limit to how much of that information I want to see. I think it takes another turn and we\u2019re moving towards that side. That part I\u2019m not thrilled over. I\u2019m one of those who think that when I\u2019m at work, I\u2019m at work. People can tell me interesting things that are useful or fun for me to learn at work. I don\u2019t want a lot of private information, which I\u2019m not related to. (I-2)\nExposure to excessive information instigates users to enforce a personal filter mechanism and to return to old work habits such as using e-mail. This results in the creation of distance from knowledge formation processes and the prevalence of disengaged users:\n\nIn the start, when it was brand new, I tried to make use of any opportunity, which was not in the intranet. We had the possibility to create rooms. I did that and invited people. I joined other rooms. But, afterwards, I failed to follow up all that. In neither of the rooms I administer now, did I manage to develop anything. I\u2019m rarely there and don\u2019t check the rooms that I am a member of. (I-3)\nThis user saw the rooms as an opportunity to improve conditions for interaction with the high schools with which she has frequent contact as part of her work. Much of the daily contact consists of sending general information. Instead of sending all of this information via e-mail, it could be transferred to the rooms, but she did not manage to uphold the goal of sharing:\n\nI haven\u2019t had time to prioritize the room. They come far down on my priority list. My workday is packed with to-do tasks. To sit down to try to use the possibilities and communicate in the rooms has instead led me not doing that. Now, I don\u2019t bother checking notifications from the rooms I administer or follow or what my colleagues have written in their status updates. I skip that very fast and I go directly to check my e-mails. (I-3)\nThis pattern of disengagement was seen in another experience. An informant explained that the challenges of generating engagement were related to aspects of the user interface itself. For example, it was difficult to ascertain whether the rooms were used by others as no panel to show numbers of visitors existed. The male user argued that the information shared in the rooms was already available and ready-made in other spaces, which meant that it was stored by co-workers in their e-mail inboxes. However, other matters drew attention:\n\nThe challenge with the ESMP is that there are too many rooms. It\u2019s almost like we have a room for each employee. You have to click on a link to get to somewhere. And then you have to go back again and click on a new link again, so it will be many rounds, just to get hold of the information you\u2019re looking for. (I-6)\nAlthough a number of the informants were uncertain as to the extent to which their sharing in the rooms was of benefit, another user shared a quite different opinion. A female user working with accounting explained that the rooms represent a type of \"manual\" She was an active user and saw the benefit of retrieving and finding information that had been shared by others:\n\nFor example, I\u2019m working in the accounting system and I find out that I need to get hold of a manual or retrieve information on an account. I go on the ESMP. There, I locate documents or things that are written about the case I\u2019m working on. I\u2019m a member of all the rooms that have something to do with accounting, a factor allowing me to know what we\u2019ve posted and what others ask about. (I-7)\nThe rooms are beneficial in different ways. For example, they are information depositories, where one can find quick answers, as they narrow down the need for searching. Alternatively, she would have to search for the same information in larger web-based databases.\n\nSince they exist, they are easy available. They are part of a knowledge you can easily use. They are there if you need to be reminded about something. For example, in accounting, there are clear definitions, clear rules for use; there\u2019s a clear date of notice for certain things. Things that are not so relevant one day, I often get information about in advance. But then I get questions from colleagues working in other departments, who ask about a deadline. What date is set as a deadline for the final reporting? Now, I know where I can quickly get and give an answer back on that. It\u2019s not necessarily that I have that knowledge in my head, but now I have good knowledge of where the answer is located. (I-7)\nDiscussion \nOrlikowski and Gash[4] utilized a practice and interpretation perspective to conjecture that Notes can be interpreted in diverse ways, revealing differences in intent and actual use. Their analysis highlighted the notion that implementers see Notes as a tool for organizational change and collaboration, whereas end-users interpret it as a means of individual and personal communication. Using a similar research lens with alternative empirical material\u2014the implementation of an ESMP in a Norwegian public organization\u2014what clues are provided that help to answer the paper\u2019s research question?\nThe most important finding is the contradictory nature of a top management initiative intended at simplifying employees\u2019 workload in a public organization that seems to have had the opposite outcome in terms of end-users\u2019 use and action. Sharing, introduced as a new workplace principle, was expected to create transparency and enhance the flow of internal communication, but when the end-users attempted to translate sharing into a manageable practice\u2014as the basis for participation in a knowledge formation process\u2014they interpreted sharing as a complicated work practice, with the larger consequence of producing disengaged users. This is primarily related to the fact that users are not performing a sharing practice, that is, a two-way communication process whereby knowledge is created through collaboration. Rather, this data analysis shows that the users engaged in informing practices to fulfill the goal of sharing, an aspect that has been demonstrated throughout the data analysis section. This informing practice\u2014which represents an essential ingredient in creating a knowledge-sharing process\u2014is performed on the premise of informing an audience and of being informed. Moreover, the informing practice is seldom the start of a knowledge process where two users exchange information to create knowledge by reflection on action, for example. Instead, sharing is carried out by re-publishing ready-made and approved official documents found elsewhere in the CA, creating an unmanageable information overload problem that encapsulates the challenges in forming a sensible knowledge-sharing process in practice. Furthermore, clues are provided regarding what is actually shared, which in this exploratory case study pertains to information that is already known. Sharing proves to be problematic and is associated with risk-taking for those involved, leading to the enforcement of self-censorship and the construction of separate and private workplaces that the informants deem beneficial to completing their work. In contrast, the users institute personal filters and return to work practices they believe work, which in most cases is e-mail. In other words, sharing in this case study is interpreted and performed as informing or as being informed.\n\nConclusion \nThe main outcomes of this exploratory case study have been an examination of the term \"sharing,\" and demonstrating the challenges involved in introducing it as a workplace principle in a public organization. Moreover, when public employees attempt to perform and make sense of \u2018sharing\u2019 in practice, a two-way communication practice emerges that can be misidentified and performed as a practice based on informing an audience and of being informed, hence causing an information overload problem and the prevalence of disengaged users in organizational life.\n\nReferences \n\n\n\u2191 1.0 1.1 1.2 1.3 Leonardi, P.M.; Huysman, M.; Steinfield, C. (2013). \"Enterprise Social Media: Definition, History, and Prospects for the Study of Social Technologies in Organizations\". Journal of Computer-Mediated Communication 19 (1): 1\u201319. doi:10.1111\/jcc4.12029.   \n\n\u2191 McAfee, A. (September 2006). \"Enterprise 2.0 Inclusionists and Deletionists\". AndrewMcAfee.org. http:\/\/andrewmcafee.org\/2006\/09\/enterprise_20_inclusionists_and_deletionists\/ .   \n\n\u2191 3.0 3.1 Orlikowski, W.J. (2000). \"Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations\". Organization Science 11 (4): 367\u2013472. doi:10.1287\/orsc.11.4.404.14600.   \n\n\u2191 4.0 4.1 4.2 4.3 4.4 Orlikowski, W.J.; Gash, D.C. (1994). \"Technological frames: making sense of information technology in organizations\". ACM Transactions on Information Systems 12 (2): 174\u2013207. doi:10.1145\/196734.196745.   \n\n\u2191 Merton, R.K. (1936). \"The Unanticipated Consequences of Purposive Social Action\". American Sociological Review 1 (6): 894\u2013904. doi:10.2307\/2084615.   \n\n\u2191 6.0 6.1 Ardichvili, A.; Page, V.; Wentling, T. (2003). \"Motivation and barriers to participation in virtual knowledge \u2010 Sharing communities of practice\". Journal of Knowledge Management 7 (1): 64\u201377. doi:10.1108\/13673270310463626.   \n\n\u2191 Kosonen, M. (2009). \"Knowledge sharing in virtual communities \u2013 A review of the empirical research\". International Journal of Web Based Communities 5 (2): 144\u201363. doi:10.1504\/IJWBC.2009.023962.   \n\n\u2191 Chen, C.-J.; Hung, S.-W. (2010). \"To give or to receive? Factors influencing members\u2019 knowledge sharing and community promotion in professional virtual communities\". Information & Management 47 (4): 226-236. doi:10.1016\/j.im.2010.03.001.   \n\n\u2191 Wasko, M.M.; Faraj, S. (2005). \"Why Should I Share? Examining Social Capital and Knowledge Contribution in Electronic Networks of Practice\". MIS Quarterly 29 (1): 35-57. doi:10.2307\/25148667.   \n\n\u2191 Ardichvili, A. (2008). \"Learning and Knowledge Sharing in Virtual Communities of Practice: Motivators, Barriers, and Enablers\". Advances in Developing Human Resources 10 (4): 541\u201344. doi:10.1177\/1523422308319536.   \n\n\u2191 Ardichvili, A.; Maurer, M.; Li, W. et al. (2006). \"Cultural influences on knowledge sharing through online communities of practice\". Journal of Knowledge Management 10 (1): 94\u2013107. doi:10.1108\/13673270610650139.   \n\n\u2191 Kaplan, A.M.; Haenlein, M. (2010). \"Users of the world, unite! The challenges and opportunities of Social Media\". Business Horizons 53 (1): 59\u201368. doi:10.1016\/j.bushor.2009.09.003.   \n\n\u2191 13.0 13.1 13.2 13.3 13.4 13.5 Treem, J.W.; Leonardi, P.M. (2012). \"Social Media Use in Organizations: Exploring the Affordances of Visibility, Editability, Persistence, and Association\". Communication Yearbook 36: 143\u201389. doi:10.2139\/ssrn.2129853.   \n\n\u2191 Leonardi, P.M.; Barley, S.R. (2010). \"What\u2019s Under Construction Here? Social Action, Materiality, and Power in Constructivist Studies of Technology and Organizing\". Academy of Management Annals 4 (1): 1\u201351. doi:10.5465\/19416521003654160.   \n\n\u2191 15.0 15.1 Gibson, J.J. (1986). The Ecological Approach to Visual Perception. Lawrence Erlbaum Associates. ISBN 0898599598.   \n\n\u2191 Mutch, A. (2013). \"Sociomateriality \u2014 Taking the wrong turning?\". Information and Organization 23 (1): 28\u201340. doi:10.1016\/j.infoandorg.2013.02.001.   \n\n\u2191 17.0 17.1 Leonardi, P.M. (2013). \"Theoretical foundations for the study of sociomateriality\". Information and Organization 23 (2): 59\u201376. doi:10.1016\/j.infoandorg.2013.02.002.   \n\n\u2191 Orlikowski, W.J. (2007). \"Sociomaterial Practices: Exploring Technology at Work\". Organization Studies 28 (9): 1435-1448. doi:10.1177\/0170840607081138.   \n\n\u2191 Callon, M. (1986). \"Some elements of a sociology of translation: domestication of the scallops and the fishermen of St Brieuc Bay\". In Law, J.. Power, action, and belief: A new sociology of knowledge?. Routledge & Kegan Paul. ISBN 9780710208026.   \n\n\u2191 Latour, B. (1988). Science in Action: How to Follow Scientists and Engineers through Society. Harvard University Press. ISBN 9780674792913.   \n\n\u2191 Latour, B. (2005). Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford University Press. ISBN 9780199256051.   \n\n\u2191 Orlikowski, W.J.; Scott, S.V. (2008). \"Sociomateriality: Challenging the Separation of Technology, Work and Organization\". Academy of Management Annals 2 (1): 433\u201374. doi:10.5465\/19416520802211644.   \n\n\u2191 Suchman, L. (2007). Human-Machine Reconfigurations: Plans and Situated Actions (2nd ed.). Cambridge University Press. ISBN 9780521858915.   \n\n\u2191 24.0 24.1 Orlikowski, W.J. (2010). \"The sociomateriality of organisational life: Considering technology in management research\". Cambridge Journal of Economics 34 (1): 125\u201341. doi:10.1093\/cje\/bep058.   \n\n\u2191 25.0 25.1 Orlikowski, W.J.; Scott, S.V. (2014). \"What Happens When Evaluation Goes Online? Exploring Apparatuses of Valuation in the Travel Sector\". Organization Science 25 (3): 868\u201391. doi:10.1287\/orsc.2013.0877.   \n\n\u2191 Leonardi, P.M. (2014). \"Social Media, Knowledge Sharing, and Innovation: Toward a Theory of Communication Visibility\". Information Systems Research 25 (4): 796\u2013816. doi:10.1287\/isre.2014.0536.   \n\n\u2191 Leonardi, P.M. (2015). \"Ambient Awareness and Knowledge Acquisition: Using Social Media to Learn \"Who Knows What\" and \"Who Knows Whom\"\". MIS Quarterly 39 (4): 747\u201362. https:\/\/misq.org\/ambient-awareness-and-knowledge-acquisition-using-social-media-to-learn-who-knows-what-and-who-knows-whom.html .   \n\n\u2191 Leonardi, P.M.; Meyer, S.R. (2015). \"Social Media as Social Lubricant: How Ambient Awareness Eases Knowledge Transfer\". American Behavioral Scientist 59 (1): 10\u201334. doi:10.1177\/0002764214540509.   \n\n\u2191 Majchrzak, A.; Faraj, S.; Kane, G.C.; Azad, B. (2013). \"The Contradictory Influence of Social Media Affordances on Online Communal Knowledge Sharing\". Journal of Computer-Mediated Communication 19 (1): 38\u201355. doi:10.1111\/jcc4.12030.   \n\n\u2191 30.0 30.1 Ellison, N.B.; Gibbs, J.L.; Weber, M.S. (2015). \"The Use of Enterprise Social Network Sites for Knowledge Sharing in Distributed Organizations: The Role of Organizational Affordances\". American Behavioral Scientist 59 (1): 103\u2013123. doi:10.1177\/0002764214540510.   \n\n\u2191 Fulk, J.; Yuan, Y.C. (2013). \"Location, Motivation, and Social Capitalization via Enterprise Social Networking\". Journal of Computer-Mediated Communication 19 (1): 20\u201337. doi:10.1111\/jcc4.12033.   \n\n\u2191 Vaast, E.; Kaganer, E. (2013). \"Social media affordances and governance in the workplace: An examination of organizational policies\". Journal of Computer-Mediated Communication 19 (1): 78\u2013101. doi:10.1111\/jcc4.12032.   \n\n\u2191 Oostervink, N.; Agerberg, M.; Huysman, M. (2013). \"Knowledge Sharing on Enterprise Social Media: Practices to Cope With Institutional Complexity\". Journal of Computer-Mediated Communication 21 (2): 156\u201376. doi:10.1111\/jcc4.12153.   \n\n\u2191 34.0 34.1 Gibbs, J.L., Rozaidi, N.H.; Eisenberg, J. (2013). \"Overcoming the \u201cIdeology of Openness\u201d: Probing the Affordances of Social Media for Organizational Knowledge Sharing\". Journal of Computer-Mediated Communication 19 (1): 102\u201320. doi:10.1111\/jcc4.12034.   \n\n\u2191 Ehrlich, K.; Lin, C.-Y.; Griffiths-Fisher, V. (2007). \"Searching for experts in the enterprise: combining text and social network analysis\". Proceedings of the 2007 International ACM Conference on Supporting Group Work: 117\u201326. doi:10.1145\/1316624.1316642.   \n\n\u2191 Thom-Santelli; J.; Muller, M.J.; Millen, D.R. (2008). \"Social tagging roles: Publishers, evangelists, leaders\". Proceedings of the 2008 SIGCHI Conference on Human Factors in Computing Systems: 1041\u201344. doi:10.1145\/1357054.1357215.   \n\n\u2191 Dugan, C.; Geyer, W.; Muller, M. et al. (2008). \"It's all 'about you': Diversity in online profiles\". Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work: 703\u201306. doi:10.1145\/1460563.1460672.   \n\n\u2191 Steinfield, C.; DiMicco, J.M.; Ellison, N.B. et al. (2009). \"Bowling online: Social networking and social capital within the organization\". Proceedings of the Fourth International Conference on Communities and Technologies: 245\u201354. doi:10.1145\/1556460.1556496.   \n\n\u2191 Wu, A.; DiMicco, J.M.; Millen, D.R. (2010). \"Detecting professional versus personal closeness using an enterprise social network site\". Proceedings of the 2010 SIGCHI Conference on Human Factors in Computing Systems: 1955\u201364. doi:10.1145\/1753326.1753622.   \n\n\u2191 Thom-Santelli, J.; Millen, D.R.; MiMicco, J.M. (2010). \"Characterizing global participation in an enterprise SNS\". Proceedings of the 3rd International Conference on Intercultural Collaboration: 251\u201354. doi:10.1145\/1841853.1841900.   \n\n\u2191 Thom-Santelli, J.; Millen, D.R.; Gergle, D. (2011). \"Organizational acculturation and social networking\". Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work: 313\u201316. doi:10.1145\/1958824.1958871.   \n\n\u2191 Zhang, J.; Qu, Y.; Cody, J.; Wu, Y. (2010). \"A case study of micro-blogging in the enterprise: Use, value, and related issues\". Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: 123\u201332. doi:10.1145\/1753326.1753346.   \n\n\u2191 L\u00fcders, M. (2013). \"Networking and notworking in social intranets: User archetypes and participatory divides\". First Monday 18 (8). doi:10.5210\/fm.v18i8.4693.   \n\n\u2191 Pettersen, L. (2014). \"From Mass Production to Mass Collaboration: Institutionalized Hindrances to Social Platforms in the Workplace\". Nordic Journal of Science and Technology Studies 2 (2): 29\u201340. doi:10.5324\/njsts.v2i2.2146.   \n\n\u2191 Pettersen, L. (2016). \"The Role of Offline Places for Communication and Social Interaction in Online and Virtual Spaces in the Multinational Workplace\". Nordicom Review 37 (Special Issue): 131\u201346. doi:10.1515\/nor-2016-0028.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\">https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on data management and sharing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 29 April 2019, at 23:03.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 225 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","4be2f1cd180a132f1a4ce3ca4fde9d6e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_What_is_the_meaning_of_sharing_Informing_being_informed_or_information_overload skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:What is the meaning of sharing: Informing, being informed or information overload?<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>In recent years, several Norwegian public organizations have introduced enterprise social media platforms (ESMPs). The rationale for their implementation pertains to a goal of improving internal communications and work processes in organizational life. Such objectives can be attained on the condition that employees adopt the platform and embrace the practice of sharing. Although sharing work on ESMPs can bring benefits, making sense of the practice of sharing constitutes a challenge. In this regard, the paper performs an analysis on a case whereby an ESMP was introduced in a Norwegian public organization. The analytical focus is on the challenges and experiences of making sense of the practice of sharing. The research results show that users faced challenges in making sense of sharing. The paper indicates that sharing is interpreted and performed as an informing practice, which results in an information overload problem and causes users to become disengaged. The study suggests a continued need for the application of theoretical lenses that emphasize interpretation and practice in the implementation of new digital technologies in organizations.\n<\/p><p><b>Keywords<\/b>: enterprise social media, sharing, public organizations, Norway\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>In the last decade, many private and public organizations have started to take great interest in enterprise social media platforms (ESMPs).<sup id=\"rdp-ebb-cite_ref-LeonardiEnter13_1-0\" class=\"reference\"><a href=\"#cite_note-LeonardiEnter13-1\">[1]<\/a><\/sup> Implying an expansion of Enterprise 2.0<sup id=\"rdp-ebb-cite_ref-McAfeeEnter06_2-0\" class=\"reference\"><a href=\"#cite_note-McAfeeEnter06-2\">[2]<\/a><\/sup>, the term refers to a platform used for internal communication in organizations. ESMPs contain a range of features that are used to share and organize <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, such as tagging systems, user profiles, search engines, follower features, discussion boards, and group features. Known examples of ESMPs are Yammer and Facebook@ work. The platforms are assumed to bring a range of benefits for organizations and for the organization of work processes. These benefits can include enhancement of the quality of internal communications and workflows. A central practice related to the successful use of ESMPs is active engagement by users or employees through the sharing or co-creation of content, although the workplace principle is not always easy to put into practice.\n<\/p><p>Since the end of the 2000s, several large Norwegian private and public organizations have introduced ESMPs to their employees. The incentive for their acquisition is motivated by various goals. For example, they can reduce internal organizational barriers, enhance organizational communications, and cut down on time spent sending e-mails. In this way, one can attain a greater overview of organizational activities and the competencies of employees. In this regard, ESMPs are presented as a solution that can contribute to solving traditional management challenges that are faced daily by public organizations. In the wake of this development, discourses focusing on the importance of sharing in organizations emerge. Top and middle managers stress the sharing of work and engagement via ESMPs as means of bringing about organizational change and unity and the use of digital technologies in work life. Surfing on the top of such management discourses is an emphasis that employees embrace a \u201csharing culture.\u201d Such developments substantiate the importance of analyzing the meaning of \"sharing\" through social constructionist research perspectives regarding the use of technology in organizations.\n<\/p><p>In 2012, a Norwegian County Authority decided to upgrade its intranet to become an ESMP, an effort initiated by the top management. The goal was to simplify the workload because the employees previously worked across separate forms of information and communications technologies (ICTs). A further objective was to transfer work practices from e-mail and local storage to the newly acquired platform by sharing. Although the technical implementation of the ESMP was successful, top management found that employees were not sharing work as intended. By using a practice perspective on technology and the organization of work, as well as related research on enterprise social media<sup id=\"rdp-ebb-cite_ref-OrlikowskiUsing00_3-0\" class=\"reference\"><a href=\"#cite_note-OrlikowskiUsing00-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OrlikowskiTechno94_4-0\" class=\"reference\"><a href=\"#cite_note-OrlikowskiTechno94-4\">[4]<\/a><\/sup>, this paper questions how a group of employees working in the County Authority interpret the meaning of sharing and put it into practice through the ESMP. The use of a practice perspective indicates that employees face challenges in interpreting the meaning of sharing. Sharing is interpreted and performed as an informing practice, which results in an information overload problem and disengaged users. \n<\/p><p>In order to tackle the research question, the paper is divided into different parts. The following section addresses the scholarly discussion upon which the study is based. Thereafter, the research strategies used to complete this study are outlined. The research findings are subsequently presented, before the research results are discussed in relation to the relevant research horizon. The final part concludes the paper.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Theoretical_perspective\">Theoretical perspective<\/span><\/h2>\n<p>Sharing has emerged as a significant social action performed by billions of social media users worldwide. In general, sharing brings with it a range of claimed unintended consequences<sup id=\"rdp-ebb-cite_ref-MertonTheUnant36_5-0\" class=\"reference\"><a href=\"#cite_note-MertonTheUnant36-5\">[5]<\/a><\/sup>, and it can be defined as a practice that originates in reconstituting dynamics and reciprocal relationships between the material properties of social media and social action. As such, it has affected the organization of social life. For example, what people share on social media draws media attention and is predicted by traditional media as having positive and negative effects on our well-being. The sharing of experiences can create community awareness on civic matters that are important to society, but also accusations of egocentric behavior. We also see that ongoing online socialization may lead to new mediated practices such as phubbing and digital detox. \"Phubbing\" is defined as the act of ignoring a person\u2019s surroundings through the use of a cellphone, which is deemed an impolite action. \"Digital detox\" is understood as a period in which a person stops using electronic, web-connected devices such as smart phones and tablets. These indicate that the organization of communicative practices in the digital sphere can become unmanageable and chaotic. Although research on social media and sharing has proliferated, organization researchers have yet to fully frame the impact of sharing on organizational life.\n<\/p><p>In consequence, such dynamics call for the development of a research perspective that discusses the meaning of sharing in organizations by use of ESMPs, especially where sharing assumes a different role than that intended. This argument is valid for several reasons. Surprisingly, organizational scholars who study knowledge-sharing processes by use of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">knowledge management<\/a> systems (for example) claim that what is actually shared by users on platforms for the sharing of work has yet to be adequately framed.<sup id=\"rdp-ebb-cite_ref-ArdichviliMotiv03_6-0\" class=\"reference\"><a href=\"#cite_note-ArdichviliMotiv03-6\">[6]<\/a><\/sup> In particular, a knowledge gap seems to exist regarding the formation of sharing processes and how this is related to emergent properties coming from the use of recursive technology in work processes.<sup id=\"rdp-ebb-cite_ref-KosonenKnowl09_7-0\" class=\"reference\"><a href=\"#cite_note-KosonenKnowl09-7\">[7]<\/a><\/sup> Instead, the knowledge management research stream has examined predefined assumptions of sharing<sup id=\"rdp-ebb-cite_ref-ChenToGive10_8-0\" class=\"reference\"><a href=\"#cite_note-ChenToGive10-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WaskoWhyShould05_9-0\" class=\"reference\"><a href=\"#cite_note-WaskoWhyShould05-9\">[9]<\/a><\/sup> and conditions that prevent the sharing of knowledge in virtual communities.<sup id=\"rdp-ebb-cite_ref-ArdichviliMotiv03_6-1\" class=\"reference\"><a href=\"#cite_note-ArdichviliMotiv03-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArdichviliLearn08_10-0\" class=\"reference\"><a href=\"#cite_note-ArdichviliLearn08-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArdichviliCultural06_11-0\" class=\"reference\"><a href=\"#cite_note-ArdichviliCultural06-11\">[11]<\/a><\/sup>\n<\/p><p>With the advent of ESMPs in organizations, there is an urgent need to formulate and facilitate a new and much broader research agenda. This has been seen in organizational research, which has introduced new definitions of platforms and has criticized existing definitions of social media for their shortcomings. An example of a new definition is enterprise social media, which is defined as: \u201cweb-based platforms that allow workers to (1) communicate messages with specific coworkers or broadcast messages to everyone in the organization; (2) explicitly indicate or implicitly reveal particular coworkers as communication partners; (3) post, edit, and sort text and files linked to themselves or others; and (4) view the messages, connections, text, and files communicated, posted, edited, and sorted by anyone else in the organization at any time of their choosing.\u201d<sup id=\"rdp-ebb-cite_ref-LeonardiEnter13_1-1\" class=\"reference\"><a href=\"#cite_note-LeonardiEnter13-1\">[1]<\/a><\/sup> This definition is a modified version of Kaplan and Haenlein\u2019s<sup id=\"rdp-ebb-cite_ref-KaplanUsers10_12-0\" class=\"reference\"><a href=\"#cite_note-KaplanUsers10-12\">[12]<\/a><\/sup>, who define social media as a \u201cgroup of Internet-based applications that build on the ideological and technological foundations of Web 2.0, and that allow the creation and exchange of user-generated content.\u201d\n<\/p><p>Current definitions of social media are criticized for failing to adequately explain organizational processes.<sup id=\"rdp-ebb-cite_ref-TreemSocial12_13-0\" class=\"reference\"><a href=\"#cite_note-TreemSocial12-13\">[13]<\/a><\/sup> To accommodate these limitations, we find an emerging body of research studying ESMPs that uses a social constructionist research lens on technology and the organization of work.<sup id=\"rdp-ebb-cite_ref-LeonardiWhatsUnder10_14-0\" class=\"reference\"><a href=\"#cite_note-LeonardiWhatsUnder10-14\">[14]<\/a><\/sup> This research stream has (among others) developed an \"affordance\" perspective<sup id=\"rdp-ebb-cite_ref-GibsonTheEco86_15-0\" class=\"reference\"><a href=\"#cite_note-GibsonTheEco86-15\">[15]<\/a><\/sup> to grasp the impact of ESMPs in organizations. Affordance<sup id=\"rdp-ebb-cite_ref-GibsonTheEco86_15-1\" class=\"reference\"><a href=\"#cite_note-GibsonTheEco86-15\">[15]<\/a><\/sup> stresses the advantages of technologies. It argues that technologies can be perceived as beneficial in performing activities without paying attention to what an object \u201cis,\u201d that is, to ask what it can afford.<sup id=\"rdp-ebb-cite_ref-TreemSocial12_13-1\" class=\"reference\"><a href=\"#cite_note-TreemSocial12-13\">[13]<\/a><\/sup> The focus on perception means to put emphasis on an object\u2019s utility, and affordance provides the possibility of understanding action potential and the capabilities of a technology, and how it can be linked to processes in organizations.<sup id=\"rdp-ebb-cite_ref-TreemSocial12_13-2\" class=\"reference\"><a href=\"#cite_note-TreemSocial12-13\">[13]<\/a><\/sup> The affordance lens is used to place greater emphasis on the meaning of materiality, which is said to have diminished in value as other concepts dominate the research agenda, such as sociomateriality.<sup id=\"rdp-ebb-cite_ref-MutchSocio13_16-0\" class=\"reference\"><a href=\"#cite_note-MutchSocio13-16\">[16]<\/a><\/sup> Affordance is linked to critical theory, which is deemed as providing new innovative ways of addressing the relationships between materiality and immateriality.<sup id=\"rdp-ebb-cite_ref-LeonardiTheor13_17-0\" class=\"reference\"><a href=\"#cite_note-LeonardiTheor13-17\">[17]<\/a><\/sup> A critical theory approach assumes the existence of multiple realities that operate interchangeably and independently of one another, with the implication that actors and objects are self-contained entities that influence each other through impacts or social interaction.<sup id=\"rdp-ebb-cite_ref-LeonardiTheor13_17-1\" class=\"reference\"><a href=\"#cite_note-LeonardiTheor13-17\">[17]<\/a><\/sup>\n<\/p><p>In contrast, sociomateriality<sup id=\"rdp-ebb-cite_ref-OrlikowskiSocio07_18-0\" class=\"reference\"><a href=\"#cite_note-OrlikowskiSocio07-18\">[18]<\/a><\/sup>, which draws on influences from Actor-Network-Theory<sup id=\"rdp-ebb-cite_ref-CallonSome86_19-0\" class=\"reference\"><a href=\"#cite_note-CallonSome86-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LatourScience88_20-0\" class=\"reference\"><a href=\"#cite_note-LatourScience88-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LatourReass05_21-0\" class=\"reference\"><a href=\"#cite_note-LatourReass05-21\">[21]<\/a><\/sup>, has emerged as an alternative sub-research stream to understanding the impact of social media in organizational studies. Sociomateriality, which assumes that \u201cmateriality is intrinsic to everyday activities and relations,\u201d<sup id=\"rdp-ebb-cite_ref-OrlikowskiSocio08_22-0\" class=\"reference\"><a href=\"#cite_note-OrlikowskiSocio08-22\">[22]<\/a><\/sup> provides an alternative approach to understanding the meaning of technology and therefore what the potential consequences of social media might entail. Orlikowski argues that previous understandings of materiality in management studies were framed around an ontology of separateness<sup id=\"rdp-ebb-cite_ref-SuchmanHuman07_23-0\" class=\"reference\"><a href=\"#cite_note-SuchmanHuman07-23\">[23]<\/a><\/sup>, in which one theorized that the material and the immaterial are separate entities and realities.<sup id=\"rdp-ebb-cite_ref-OrlikowskiTheSocio10_24-0\" class=\"reference\"><a href=\"#cite_note-OrlikowskiTheSocio10-24\">[24]<\/a><\/sup> Sociomateriality instead argues that they should be seen as linked, equal, and inseparable, which means addressing a relational ontology. Therefore, future organizational researchers with an interest in social media could study technological artifacts \u201csymmetrically to the humans, and as equivalent participants in a network of humans and non-humans that (temporarily) align to achieve particular effects.\u201d<sup id=\"rdp-ebb-cite_ref-OrlikowskiTheSocio10_24-1\" class=\"reference\"><a href=\"#cite_note-OrlikowskiTheSocio10-24\">[24]<\/a><\/sup> In this regard, Orlikowski and Scott<sup id=\"rdp-ebb-cite_ref-OrlikowskiWhatHappens14_25-0\" class=\"reference\"><a href=\"#cite_note-OrlikowskiWhatHappens14-25\">[25]<\/a><\/sup> apply a sociomaterial practice perspective in one of their latest research works on valuation regimes. They demonstrate that online evaluations of hotels performed by users on social media have a drastic impact on the domestic travel industry.<sup id=\"rdp-ebb-cite_ref-OrlikowskiWhatHappens14_25-1\" class=\"reference\"><a href=\"#cite_note-OrlikowskiWhatHappens14-25\">[25]<\/a><\/sup>\n<\/p><p>However, an important argument running through the above literature is the requirement for more theorizing. Current research focuses on a particular platform or features, leading to a claim that researchers are incapable of making inferences about the consequences of the material for organizing.<sup id=\"rdp-ebb-cite_ref-LeonardiEnter13_1-2\" class=\"reference\"><a href=\"#cite_note-LeonardiEnter13-1\">[1]<\/a><\/sup> Current definitions of ESMPs are too application-focused and overlook the social dynamics and reciprocal relationships between the material and the immaterial.<sup id=\"rdp-ebb-cite_ref-LeonardiEnter13_1-3\" class=\"reference\"><a href=\"#cite_note-LeonardiEnter13-1\">[1]<\/a><\/sup> Treem and Leonardi<sup id=\"rdp-ebb-cite_ref-TreemSocial12_13-3\" class=\"reference\"><a href=\"#cite_note-TreemSocial12-13\">[13]<\/a><\/sup> argue that this causes scholars to fail to possess sufficient terminology to explain the ways in which ESMPs can influence social behavior and to generalize matters to organizations across contexts. Here, the affordance lens offers researchers the possibility of making interesting analyses regarding the ways in which ESMPs influence organizational processes such as socialization and power aspects in organizations. Research has suggested various affordances that ESMPs can give for organizational processes. For example, Treem and Leonardi<sup id=\"rdp-ebb-cite_ref-TreemSocial12_13-4\" class=\"reference\"><a href=\"#cite_note-TreemSocial12-13\">[13]<\/a><\/sup> suggest that ESMPs can enable four affordances: visibility, edibility, persistence, and association. A case in point highlighting the meaning of a singular affordance is Treem and Leonardi\u2019s<sup id=\"rdp-ebb-cite_ref-TreemSocial12_13-5\" class=\"reference\"><a href=\"#cite_note-TreemSocial12-13\">[13]<\/a><\/sup> argument that the affordance visibility is seen when employees use an ESMP to make their behavior, knowledge, preferences, and commutation network visible to others. They argue that actions like posting updates, showing a list of friends, and writing personal profiles are beneficial and enable the visualization of work to third parties. Leonardi<sup id=\"rdp-ebb-cite_ref-LeonardiSocial14_26-0\" class=\"reference\"><a href=\"#cite_note-LeonardiSocial14-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeonardiAmbient15_27-0\" class=\"reference\"><a href=\"#cite_note-LeonardiAmbient15-27\">[27]<\/a><\/sup> illustrates this point by showing that work interaction on an ESMP platform is pivotal for knowledge work and for the transfer of knowledge in a large organization. Based on a study of a financial service in the United States of America, Leonardi shows that the use of a company\u2019s ESMP assists third-users to enhance awareness of meta knowledge, as one learns about the competencies of co-workers and the matters on which they are working. ESMPs can be used to make accurate interferences of people\u2019s meta knowledge and the sharing of co-workers\u2019 communication activities, and communicating via messenger software can offer innovative products and avoid the duplication of work. In a related study, Leonardi and Meyer<sup id=\"rdp-ebb-cite_ref-LeonardiSocialMedia15_28-0\" class=\"reference\"><a href=\"#cite_note-LeonardiSocialMedia15-28\">[28]<\/a><\/sup> develop the above claim in a study of a communications business unit in a telecommunication unit. Leonardi and Meyer test out a set of hypotheses and instances of knowledge transfer to show that when knowledge workers are exposed to communication activities on an ESMP, internal communication can be enhanced.\n<\/p><p>Beyond these works, researchers have theorized affordance in two other principle directions. First, we can identify works that conceive of affordance at a conceptual level. Second, researchers develop the term empirically through case study designs. An example of the former is a study by Majchrzak, Faraj, Kane, and Azad<sup id=\"rdp-ebb-cite_ref-MajchrzakTheContra13_29-0\" class=\"reference\"><a href=\"#cite_note-MajchrzakTheContra13-29\">[29]<\/a><\/sup>, who demonstrate how ESMPs have four affordances to inspire engagement of visible knowledge conversations in organizations. These include met voicing, triggered attending, network-informed associating, and generative role-taking. For example, met voicing would mean that an ESMP has the action capability to enable users to react to others\u2019 presence, profiles, content, and activities. Ellison, Gibbs, and Weber<sup id=\"rdp-ebb-cite_ref-EllisonTheUse15_30-0\" class=\"reference\"><a href=\"#cite_note-EllisonTheUse15-30\">[30]<\/a><\/sup> develop a collective affordance and affordances for organizing, and explore the role of organizational affordances in light of the fact that organizations become distributed entities. \n<\/p><p>The affordance lens needs to be broadened beyond the context of individual uses, which has been seen in many analyses. Ellison <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-EllisonTheUse15_30-1\" class=\"reference\"><a href=\"#cite_note-EllisonTheUse15-30\">[30]<\/a><\/sup> note that the affordances of ESMPs can include concepts such as social capital dynamics, identity formation, context collapse, and networked organizational structures. Fulk and Yuan<sup id=\"rdp-ebb-cite_ref-FulkLocation13_31-0\" class=\"reference\"><a href=\"#cite_note-FulkLocation13-31\">[31]<\/a><\/sup> argue that ESMPs have the affordance to solve organizational challenges and represent a preferable platform for organizing knowledge sharing in comparison to older knowledge management systems. Fulk and Yan argue that by combining transitive memory theory, public good theory, and social capital theory, ESMPs have the affordance to deal with three associated challenges in the sharing of organizational knowledge. These include knowledge of the location of expertise in the organization, motivation to share knowledge, and the development and maintenance of relationships with knowledge providers. In considering work that uses the affordance lens for cases from organizational life, however, Vaast and Kaganer<sup id=\"rdp-ebb-cite_ref-VaastSocial13_32-0\" class=\"reference\"><a href=\"#cite_note-VaastSocial13-32\">[32]<\/a><\/sup> explore how organizations react to employees\u2019 adoption of ESMPs. Based on a sample of corporate policy documents, Vaast and Kaganer find that organizations view ESMPs as more of a risk than an asset. Oostervink, Agterberg, and Huysman<sup id=\"rdp-ebb-cite_ref-OostervinkKnowl13_33-0\" class=\"reference\"><a href=\"#cite_note-OostervinkKnowl13-33\">[33]<\/a><\/sup> have undertaken a study connecting enactment to the affordance lens, as affordances are enacted in practice and institutional forces in an organization can shape how ESMPs are used by employees. Oostervink <i>et al.<\/i> point out that the institutional logics of a corporation and employees\u2019 professional expertise shape the knowledge that employees share on ESMPs. Although the affordance of visibility and associability are assumed to enhance knowledge sharing in organizations, Gibbs, Rozaidi and Eisenberg<sup id=\"rdp-ebb-cite_ref-GibbsOver13_34-0\" class=\"reference\"><a href=\"#cite_note-GibbsOver13-34\">[34]<\/a><\/sup> find the opposite effect. They performed a study among a group of engineers and noted that engaging EMPSs create contradictions in workplace interactions. For example, constantly remaining accessible and open to other suggestions is a hassle, thus causing employees to feel that they need to hide certain behaviors from others. Based on this finding, Gibbs <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-GibbsOver13_34-1\" class=\"reference\"><a href=\"#cite_note-GibbsOver13-34\">[34]<\/a><\/sup> have suggested that scholars theorize affordance in terms of dichotomies, not singular affordance concepts. They establish three affordances with which users interact when they use an ESMP. These include visibility-invisibility tension, engagement-disengagement tension, and sharing-control tension.\n<\/p><p>Organizational researchers have also explored ESMPs from other angles. Research shows that employees are receptive to ESMPs in certain organizations: those that make and sell the technology, being IT companies and organizations with the resources to research the technology in large projects. In this regard, IBM\u2019s Beehive project is groundbreaking. One can read in numerous research papers the ways in which Beehive has been implemented and tested on IBM employees, as researchers have documented basic user behaviors. Researchers have focused on an entire ESMP<sup id=\"rdp-ebb-cite_ref-EhrlichSearching07_35-0\" class=\"reference\"><a href=\"#cite_note-EhrlichSearching07-35\">[35]<\/a><\/sup> or on features such as tagging systems<sup id=\"rdp-ebb-cite_ref-Thom-SantelliSocial08_36-0\" class=\"reference\"><a href=\"#cite_note-Thom-SantelliSocial08-36\">[36]<\/a><\/sup> and user profiles<sup id=\"rdp-ebb-cite_ref-DuganItsAll08_37-0\" class=\"reference\"><a href=\"#cite_note-DuganItsAll08-37\">[37]<\/a><\/sup> Beehive research papers often use a social capital perspective<sup id=\"rdp-ebb-cite_ref-SteinfieldBowling09_38-0\" class=\"reference\"><a href=\"#cite_note-SteinfieldBowling09-38\">[38]<\/a><\/sup> to establish links between ESMP uses, and connecting strategies constitute a recurring theme. Researchers have identified that IBM employees use Beehive as a platform to expand their professional networks, using it to communicate with colleagues across organizational levels.<sup id=\"rdp-ebb-cite_ref-WuDetecting10_39-0\" class=\"reference\"><a href=\"#cite_note-WuDetecting10-39\">[39]<\/a><\/sup> IBM employees undertake a range of search and retrieving practices<sup id=\"rdp-ebb-cite_ref-Thom-SantelliChar10_40-0\" class=\"reference\"><a href=\"#cite_note-Thom-SantelliChar10-40\">[40]<\/a><\/sup> and use Beehive as a knowledge repository.<sup id=\"rdp-ebb-cite_ref-Thom-SantelliOrg11_41-0\" class=\"reference\"><a href=\"#cite_note-Thom-SantelliOrg11-41\">[41]<\/a><\/sup> Other case studies on ESMPs in organizations other than IBM exist, but have yielded limited insights. They show that employees use ESMPs to streamline their online behavior to work practices and organizational affiliation.<sup id=\"rdp-ebb-cite_ref-ZhangACase10_42-0\" class=\"reference\"><a href=\"#cite_note-ZhangACase10-42\">[42]<\/a><\/sup> Researchers have examined the challenges of adopting an organizational ESMPs. It is not uncommon to come across findings that highlight how employees continue to prefer to communicate via e-mail and chat software, and silently monitor news streams.<sup id=\"rdp-ebb-cite_ref-L.C3.BCdersNetwork13_43-0\" class=\"reference\"><a href=\"#cite_note-L.C3.BCdersNetwork13-43\">[43]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PettersonFromMass14_44-0\" class=\"reference\"><a href=\"#cite_note-PettersonFromMass14-44\">[44]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PettersonTheRole16_45-0\" class=\"reference\"><a href=\"#cite_note-PettersonTheRole16-45\">[45]<\/a><\/sup> Consequently, one finds a pattern that a core group adopts SNS and maintains network activities, while a larger group uses \"older\" forms of ICTs.\n<\/p><p>Therefore, the research horizon described appears limited and somewhat inchoate. Scholars have predominantly focused on the material properties of ESMPs and have contributed through experimental theorizing. Absent from the research literature is the specific role that interpretation and practice take in employees\u2019 recursive use, with ESMPs a crucial aspect of work processes. Moreover, it appears that the research field has yet to adequately frame whether sharing can take on a different role than that intended, and what it means when technology is used differently in an organizational setting. This means that the research field can advance a research lens focusing on situations and enacting with emergent properties that come from the use of recursive technology, hence placing clearer emphasis on what people do with an ESMP. Thus, one can use a practice perspective on technology.<sup id=\"rdp-ebb-cite_ref-OrlikowskiUsing00_3-1\" class=\"reference\"><a href=\"#cite_note-OrlikowskiUsing00-3\">[3]<\/a><\/sup> A practice lens on ESMPs can also be used to fill a knowledge gap regarding the formation of knowledge-sharing processes in the use of ESMPs, facilitating our understanding of the unintended consequences of technology use in organizations. Here, Orlikowski and Gash\u2019s<sup id=\"rdp-ebb-cite_ref-OrlikowskiTechno94_4-1\" class=\"reference\"><a href=\"#cite_note-OrlikowskiTechno94-4\">[4]<\/a><\/sup> technological frames can be of assistance. Technological frames are defined as: \u201cthat subset of members\u2019 organizational frames that concern the assumptions, expectations, and knowledge they use to understand technology in organizations. This includes not only the nature and role of the technology itself, but the specific conditions, applications, and consequences of that technology in particular contexts.\u201d<sup id=\"rdp-ebb-cite_ref-OrlikowskiTechno94_4-2\" class=\"reference\"><a href=\"#cite_note-OrlikowskiTechno94-4\">[4]<\/a><\/sup> The concept can enable us to frame how individuals and social groups in organizations alike make sense of a technology to determine their actions, allowing us to move beyond conceptualizing a technology\u2019s mere value and perception among users. Technological frames problematize the \"taken for granted\" notions of a technology and can facilitate an understanding of how individuals and groups in organizations develop particular assumptions, expectations and knowledge of a new technology in an organizational setting. Orlikowski and Gash<sup id=\"rdp-ebb-cite_ref-OrlikowskiTechno94_4-3\" class=\"reference\"><a href=\"#cite_note-OrlikowskiTechno94-4\">[4]<\/a><\/sup> have illustrated technological frames in their study of the implementation of the groupware Notes in a consultant company. The researchers interviewed the implementers and adopters, grouping them into \"technologists\" and \"users.\" \"Technologists\" was used to refer to technology staff, whereas \"users\" comprised the organization\u2019s consultants. Orlikowski and Gash demonstrated a large set of differences in terms of expectations and actions, which they attributed to differences in technological frames. For example, the technologists viewed Notes as an enabler for information sharing, electronic communication, document management, and online discussion, which they believed could contribute to collaboration. The users\u2019 interpretation was different, viewing Notes\u2019 electronic e-mail features as a potential substitute for existing communication technologies such as fax and telephone. The technologists therefore framed Notes as a collaborative technology, whereas the users used it as a means for individual and personal communications.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"About_the_case_and_methods\">About the case and methods<\/span><\/h2>\n<p>Norway is divided into nineteen large administrative units, called counties, and roughly 350 municipalities. Each county is governed by a County Authority (CA), rendering this form of governance the first form of subdivision in the country. The CA where the data for this case study were collected consists of a political structure, an administrative body, and welfare units. The political structure is an elected body consisting of the County Council, the County Executive Board, the County Principal Standing Committees, and the County Mayor. The County Council is supported by an administrative body, the County Administration, which implements and administers policies. The County Administration is organized into eight administrative units and an executive secretariat board. Other welfare units also exist, which play a role for citizens and produce services. Among others, these consist of high schools, libraries, dental services, and transportation. A large body of the CA workforce includes high school teachers, and in total the CA contains approximately 2,800 employees.\n<\/p><p>The study made use of an exploratory qualitative research strategy. This approach was used to facilitate an in-depth investigation of the ways in which public employees working predominantly in the County Administration used the ESMP and interpreted sharing in an organizational setting. The study is primarily based on qualitative research interviews. Written documentation was collected, but it is not used as part of this paper and is thus excluded from the data analysis. However, the research design started with an informal approach to employees in the CA who had been responsible for the public procurement and implementation of the ESMP. They agreed to be part of the exploratory qualitative study and recruited the informants. Eight informants were recruited, and they worked in different departments and holding positions, predominantly as advisors in the County Administration. The criterion for selection was that they were all users of or involved in the implementation of ESMP. In sum, eight semi-structured qualitative interviews with the use of a guide were completed. The interviews were undertaken one-to-one, meaning that only the researcher and the informant were present in the interview setting. Each interview lasted approximately one hour and focused on two main themes related to sharing: previous user sharing experience on social media, and how the individual used the ESMP to organize the sharing of work. Each interview was recorded using a digital audio recorder. The data were collected over two periods, from August to September 2013 and in February 2014. The study was based on informed consent and the informants were anonymized. The background of the informants is displayed in Table 1.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> The background of informants\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Gender\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Number\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Position\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Duration\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Date\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">27 Aug. 2013\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Middle Manager\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30 Aug. 2013\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5 Sep. 2013\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Exe. Director\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12 Feb. 2014\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">M\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10 Feb. 2014\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">M\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17 Feb. 2014\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Consultant\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 Feb. 2014\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">M\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 hour\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 Feb. 2014\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Once the interviews were completed, they were transcribed. The data analysis was inspired by an open coding strategy of the interview data. Here, the main focus was on finding emerging patterns, which consisted of grouping and comparing the informants\u2019 perceptions, user patterns, and experiences of sharing. The informants\u2019 answers were grouped into four broad themes. In order to offer the informants a voice, direct quotations are used in the data analysis.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Data_analysis\">Data analysis<\/span><\/h2>\n<p>This section presents the data analysis and seeks to answer the research question addressed earlier in the paper: how does a group of employees in the CA interpret the meaning of sharing? The emphasis here is on breaking down this notion by presenting four emergent themes from the open coding of the interview data. Each theme represents an adoption of the new workplace principle of sharing, showing how individuals develop particular assumptions, expectations, and knowledge about its definition. Performing such an analysis can facilitate a clearer understanding of the meaning of \"sharing.\" The four themes are displayed in Table 2.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 2.<\/b> Emergent themes from the data analysis\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Theme no.\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Name of theme\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sharing as a facilitator of organizational change\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sharing as a trigger for self-censorship and risk-taking\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sharing in separate digital ecosystems\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sharing as an individual informing strategy\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Theme_1:_Sharing_as_a_facilitator_of_organizational_change\">Theme 1: Sharing as a facilitator of organizational change<\/span><\/h3>\n<p>The first theme pertains to expectations of sharing as a facilitator of organizational change and represents an approach whereby top managers want their employees to work in a different way. This interpretation comes as little surprise, because the CA\u2019s top management was the initiator behind the ESMP. Sharing represents a \u201cproblem-solver\u201d and can produce organizational belonging in the face of internal forces that may contradict unity. Certain professions, such as teachers\u2014a large profession\u2014are assumed to identify with the high schools in which they work and with their professional identity, rather than with a feeling of belonging to the CA. However, the motivation for implementing the social ESMP was related to disentangling a common problem with which most organizations struggle to cope today: escaping the meeting culture, e-mail overload, and the use of too many forms of IT:\n<\/p>\n<blockquote>There was a need to create an ESMP that considered the fact that we worked with various work surfaces. You had to open each system one at a time, just to approve an invoice. Our challenge was also to escape the \"hell of e-mail.\" (I-4)<\/blockquote>\n<p>The top management aimed to simplify employees\u2019 workload. This was related to the fact that employees worked across several non-integrated ICTs and stored information in different places, making it challenging to create an overview. In response, the top management argued that a single site that could work as the central access point connecting all the employees was required. This would be realized by replacing the intranet with the ESMP. Therefore, a project group was created to work with various drafts of a new interface design, which would break an established work pattern in the CA. While the intranet was run as an internal website on which a group colleague wrote internal news stories, the new design suggested that the ESMP should be the main site opened by employees each day, with embedded sharing features and URL links to each internal IT system. In this way, the ESM would be the melting pot where everybody talked about work. Afterwards, an organizational discourse emphasizing the importance of a sharing culture emerged:\n<\/p>\n<blockquote>It was clear that we needed something that could enable us to work with the culture across [the organization], knowledge of each other\u2019s work. My responsibility has been to legitimize sharing in the management structure. Parallel to that, we made attempts to raise discussion about organizational culture and work processes internally. Should we establish a sharing culture, in the sense that people can easily participate in and reinforce each other\u2019s work, or take part in reports, or take part in other kinds of things, take part in the knowledge we have, this requires a culture where [people] actively participate. (I-4)<\/blockquote>\n<p>However, translating and making sense of sharing into a manageable practice proved challenging, as it surfaced as ambiguous:\n<\/p>\n<blockquote>It sounds very good. It has a great positivity to it when it\u2019s presented, but not so great when you try it out in practice. At an early stage, there was a positive feel. You didn\u2019t know exactly what it was. There was this belief that we should change the work culture. (I-5)<\/blockquote>\n<p>Later, this awareness amplified as the initiators realized that the employees rarely started a work process by beginning from scratch\u2014by creating a document in which everyone can engage, for example\u2014but instead viewed sharing as an informing practice of circulating ready-made documents. A recurring theme was how sharing was directly linked to previous publishing habits. The employees were accustomed to an article format, meaning that postings had an \"internal story\" label attached to them. Participation involved performing simple tasks, like writing status updates, following colleagues, and updating profiles. The ESMP was an information channel where information was pushed out, not a platform in which one engaged in two-way dialogue. Furthermore, the employees fulfilled activities that required little commitment, such as posting a profile picture, writing status updates, tagging competence, or uploading completed documents. Beyond these actions there was little evidence that the users aimed to participate in activities requiring the performance of reciprocal actions:\n<\/p>\n<blockquote>Ninety percent of the information posted on the ESMP is not something that we\u2019ve published. It\u2019s made by the organization. It has become a place where items are shared. It\u2019s divided between heavy and light documents. People share when documents are finished. You don\u2019t see many examples where people collaborate on a document, which is part of a work process. We haven\u2019t gone any further in changing work culture and the ways we work. (I-5)<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Theme_2:_Sharing_as_a_trigger_for_self-censorship_and_risk-taking\">Theme 2: Sharing as a trigger for self-censorship and risk-taking<\/span><\/h3>\n<p>The second theme emerging from the data analysis is that sharing includes a high degree of self-censorship and is associated with risk-taking. This pattern was seen among employees who use the ESMP and who were not part of the actual implementation process, who work across different departments and who are affiliated further down in the management structure. However, an interesting pattern consists of the ways in which earlier and alternative private social media platforms shape perceptions of appropriate net behavior. For example, the informants registered on social media services that became mainstream in the 2000s (the informants\u2019 use is displayed in Table 3). The data indicates that the informants had a strong passive and critical approach to participation. They saw the benefits of sharing, but demonstrated a \"reading and textbook\" approach consisting of monitoring others\u2019 actions and only frugally sharing about themselves. This molded a view that sharing was seldom regarded as a two-way communicative process between two parties:\n<\/p>\n<blockquote>I don\u2019t share information about what I\u2019ve eaten for dinner, what I do during my evenings. I share if it is appropriate and relevant, not just one of those private things. Sometimes I post a picture of a mountain summit on Facebook. I have a pretty high threshold that the summit should be a little more interesting for others to see. (I-2)<\/blockquote>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 3.<\/b> The informants\u2019 use of social media platforms\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Gender\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Number\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Position\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Facebook\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Twitter\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">LinkedIn\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Instagram\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Google Drive\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Middle Manager\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Exe. Director\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">M\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">M\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">F\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Consultant\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">M\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I-8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Advisor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">X\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Therefore, it is more accurate to maintain that communicative practices are based on being informed and to inform, which ignores how a goal is to engage with an equal to create knowledge. This is reflected in beliefs about socializing and the means of ascribing social media with personal labels. For example, Facebook belongs to the private sphere and is used for \u201cscrolling after fun stuff and setting likes.\u201d LinkedIn is a \u201cCV database,\u201d while Twitter is a medium where \u201cone only sends URL links to news you have already read.\u201d Such beliefs hint at what is acceptable to share on which platform:\n<\/p>\n<blockquote>There are people who write things that shouldn\u2019t be shouted out loud. One gets the impression that, \u201cI\u2019m sad today\u201d. If anyone had a nice trip, which is worth writing about, then you can do it, by all means. It\u2019s easy to explain it, but when you see it, it gets difficult to say it with words. There are some things that are just a bit \"intimate,\" very personal stuff. It doesn\u2019t belong on social media, because it doesn\u2019t concern everybody. (I-6)<\/blockquote>\n<p>Grooming and gossiping are disregarded, and publishing of overly personal information evokes intimacy and is taboo. Instead, the informants had clear perceptions that what should be shared had to be interesting and relevant, meaning that the value of what can be shared has to be informative and of high quality. This creates boundaries delineating how one should engage on the ESMP, which are manifested in the form of distinctions. For example, the most common trait was to draw distinctions between work and non-work-related use, and external and internal use. Here, Twitter is work-related and is used because politicians interact in the Twittersphere, and it forms an arena for public debate. Some informants see Twitter as a \u201clistening post\u201d where one can monitor what is going on and to stay up-to-date with current events:\n<\/p>\n<blockquote>I use Twitter because it gives me something related to my work and because I follow public debates. So, I started paying attention to what was going on Twitter. In our department, we follow public debates. (I-8)<\/blockquote>\n<p>Consequently, one could expect that engaging on the ESMP should represent a challenge. Yet, the data suggest otherwise, as sharing on the ESMP was connected with risk-taking, expressed in the informants\u2019 views of their willingness to make a work process transparent to others. Here, the informants placed themselves on a scale from relatively open to very restricted in what they shared:\n<\/p>\n<blockquote>I try to set an example. When I create a document, I publish it right away. It says it\u2019s a document in progress, which we\u2019re working on. (I-2)<\/blockquote>\n<blockquote>I don\u2019t have a problem with posting something that is not one hundred percent complete. I would have made it clear that this is \u2018work in progress\u2019 which I want feedback on. (I-5)<\/blockquote>\n<p>However, we also find examples illustrating the risk-taking associated with publishing work in progress. This is related to the idea that informants assume that they can be criticized, meaning that\npublished and unfinished work can create misunderstandings:\n<\/p>\n<blockquote>One thing is that some us find it a bit uncomfortable to share things that are not finished, because then we get criticized. It becomes uncomfortable when it\u2019s not completed. If things are just published and not finished, it can cause more harm, because it creates sanctions on something that it was not intended to be. We have specific discussions within our work areas, documents concerning the management side and on the political aspects, which we publish. When things are at a certain stage, a working document, it is not intended that everybody should see it. If there are many who use it, they can abuse it in a number of contexts. (I-2)<\/blockquote>\n<p>The data also show that informants seek \u2018approval\u2019 from their closest manager to publish content on the ESMP. Rather than deciding independently, for example, as the basis for sharing a document, informants enforce a quality-safety practice where they ask permission from an authority in the management structure:\n<\/p>\n<blockquote>Things that are unfinished and not approved can create panic when it is a different figure from what you think is going to be on paper. If we begin to rewrite the CA\u2019s economy and everyone can read that, there will be something new to most people. Many people absorb it, even when it is wrong. It creates a lot of \"storm\" in your organization if it is not correct. I can take an example from the corporate governance program, which has an indicator called \"financial statements and budgets,\" which shows how much of a deficit\/surplus we have to date. It is an indicator that gets its numbers straight from our accounting system. When we updated the financial system, the indicator \"froze\" itself in Corporater and showed figures from November 2013. This is a completely wrong figure. We have notified about that on the ESMP, but still I keep getting phone calls that the figures are wrong. (I-6)<\/blockquote>\n<p>Another informant gave a similar example:\n<\/p>\n<blockquote>If I am to work on a case, I want to have the final answer before I publish it. I can give you an example, which applies to the CA\u2019s dental clinics. When they want to send over a thing, in the process, things that go to debt collection arrive on my desk. I have not posted anything during the process because I wanted my manager to look at the draft along with other managers. It\u2019s the way that I work, the way I think: the routine should be completed before any dentist gets access. Considering that you want to have a unified management involved, they have to see the final result first. Then time passes, and we have a routine: we end up with draft C and D, until we finally land on something. (I-7)<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Theme_3:_Sharing_in_separate_digital_ecosystems\">Theme 3: Sharing in separate digital ecosystems<\/span><\/h3>\n<p>The third theme from the interview data illustrates the ways in which informants adopt social media services and construct knowledge-sharing processes that form part of a work process. Thus, it makes sense to state that the informants create separate digital ecosystems that are used when they perceive that the ordinary ICTs provided by the CA are insufficient to performing their work, which influences employees to look for alternatives, a technology-adoption that occurs \"under the radar\" of the IT department.\n<\/p><p>Looking at particular practices, an informant explained how they combined Dropbox and Google Drive as part of a work process that was used to complete the organization of a public procurement. In several cases, the CA works together with the neighboring municipalities. As part of the process, the CA assumes the lead role as the public buyer and lead organizer, meaning that the CA acts on behalf of many municipalities to achieve greater benefits for all. This work requires collaboration with colleagues in other municipalities. In that regard, one can expect that colleagues in different municipalities have diverse needs and competencies, hence many persons voice different opinions and needs. This will lead to long e-mail exchanges and numerous attached documents, with the effect that one quickly loses the overview. Instead of sending back and forth large numbers of e-mails with large documents attached, the respondents used Dropbox or Google Drive to increase the efficiency and economy of the work process for everybody:\n<\/p>\n<blockquote>We created a Dropbox account because we don\u2019t have the same e-mail system or share the same case management system. And it\u2019s challenging. You don\u2019t get Dropbox solutions on the PCs here. The IT department thinks it\u2019s unsecure, [lacking] information security. We need tools to do our job, so we ended up defying that a bit and we downloaded the software to our PCs. Sometimes it happens that we use Google Drive when working with external partners. I used Google Drive to share documents more efficiently than by e-mail, before they get too large. (I-8)<\/blockquote>\n<p>Another practice is how Facebook groups are used either as information repositories or for external communication with particular groups who use the welfare services provided by the CA. Here, one does not find examples of practices that demonstrate knowledge sharing between several parties, but merely how Facebook groups are used as public bulletin boards, where online sharing again represents an informing practice. For instance, an employee was a representative in one of the CA\u2019s worker unions and interacted with representatives from other CAs. In the process, they created a Facebook group that enabled them to stay in contact and inform one another:\n<\/p>\n<blockquote>It was part of a different role, which was part of a task I had here in the CA. I had contact with others with the same role in other CAs. We used the Facebook group to share information that was more or less of the same nature. It was a way to share knowledge on issues of health and safety at work. (I-1)<\/blockquote>\n<p>Another public employee explained how they created a Facebook group to communicate with a group of citizens the CA serves directly, students in high schools. The Facebook group was created on the assumption that students would contact the CA there; given that students are in the social media landscape, they concluded that the CA also needed to be present in a similar capacity. After some years of use, the Facebook group has roughly 300 \"likers,\" but expectations have not turned into reality. Relatively few requests from students have been seen. Instead, it has turned into more of a public bulletin board where information is posted:\n<\/p>\n<blockquote>It runs every day. We don\u2019t get many requests. We publish when we have specific information. We were unsure whether it would be an active user channel. I think it\u2019s going to become that in the long run. We intend to continue to use it and improve its uses, and even get more users. (I-3)<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Theme_4:_Sharing_practice_as_an_individual_informing_strategy\">Theme 4: Sharing practice as an individual informing strategy<\/span><\/h3>\n<p>The fourth theme emerging from the analysis demonstrates how the employees turn sharing into individual informing strategies, which arguably fulfill a goal of complying with an overall objective to share on the ESMP. The fourth theme additionally demonstrates the challenges associated with performing sharing in an internal organizational setting, and translation problems connected with practicing sharing as a two-way communication act, given that it again becomes an informing and pushing strategy of camera-made information to an audience that does not respond. This is illustrated by scrutinizing a particular feature deemed as being important for creating the conditions for sharing on the ESMP, so-called \"rooms.\" Rooms can best be described as Facebook groups or information repositories that operate as spaces for cooperation. Within them, users can upload and download documents, follow people, and receive messages about recent activities. The rooms have members and were grouped according to the CA\u2019s department structure and across departmental borders.\n<\/p><p>In considering particular overall user experiences, the data show that the informants adopted the rooms. They registered members and followed rooms, uploaded documents and so forth, such that it was common to follow between two and five rooms. Afterwards, the informants experienced challenges, illustrating the disadvantages and benefits associated with sharing. First, the informants created rooms and registered members who worked in the same department or in the same field as themselves. Second, the findings indicate that upload documents consisted of re-published information that was already stored in other places. A lack of data exists that shows that employees created new documents and began to co-write them in real-time; rather, they uploaded approved and ready-made documents that were only read for notification purposes. This indicates that sharing is an informing practice to a large audience, which does not invite a two-way communication process. Third, all informants reported that little interaction (such as participation and reading discussions) occurred in the rooms. In sum, users framed the rooms as information repositories rather than as sites for collaboration. Thus, we see the pattern that employees with super-user status\u2014users who enjoyed an administrative role in the rooms\u2014tried to stimulate increased engagement, which represented an outcome of a lack of responses to their informing practices. In order to reach the top management goal of sharing, super-users adopted particular roles and strategies to promote participation, which in turn illustrates the challenges pertaining to sharing.\n<\/p><p>In reviewing these practices, it can be noted that one super-user would adopt an \"online gardener\" strategy and attempt to encourage co-workers to engage in the rooms she administrated. This is not dissimilar to an automated e-mail notification feature, which is generated following a period of interactivity in a knowledge repository. She assumed the role of a sharer and pusher of information, which consisted of sending friendly e-mail reminders when she uploaded new documents:\n<\/p>\n<blockquote>I send an e-mail to everyone who has an interest in the room and then I share information with them that it\u2019s posted on the ESMP. Then I invite them to follow the room, because there is information there that is relevant to them. I say that it will only be posted there. That I\u2019ve done for about a year. (I-1)<\/blockquote>\n<p>This means informing across multiple channels, turning sharing into a practice of double in-forming. Afterwards, the user questioned the value of sharing and was uncertain about the extent to which her efforts were worthwhile, a thought shared by another informant:\n<\/p>\n<blockquote>I note that there are not many who follow the rooms, after many invitations and reminders to others who I think might have an interest in it. And the thoughts come. Do we spend unnecessary time on posting information that people do not read anyway? (I-2)<\/blockquote>\n<p>This raises the question of whether the room members post material and use the rooms as intended. For example, after uploading, an informant also received phone calls to ask if the same documents could be sent by e-mail:\n<\/p>\n<blockquote>I often get the question, if I can send them an e-mail when there is new information in the rooms. We have decided on that\u2014no, we don\u2019t send an extra e-mail. We put it out there, and then people must seek it out themselves. I feel that people don\u2019t pay attention to all that is posted in the rooms. They would have paid attention if we had sent it in an e-mail. But we have made a conscious choice on that. I think that people read it if they get an e-mail because it\u2019s a direct message aimed at them, rather than having to search for the information themselves. (I-2)<\/blockquote>\n<p>This experience shows the start of a disengagement regarding sharing, as it vanishes and becomes overtaken by other assignments whose completion is deemed more important:\n<\/p>\n<blockquote>We have two rooms. I post a lot of information in them. I try to ensure that new information is posted. But I do not use the opportunity to follow other rooms on the ESMP, for example, as I had hoped and thought I would. It disappears into my daily work life. When I need information, I don\u2019t find it with the search mechanisms that we have today as we had with the intranet, although there is more information out there now. But now, I think it is harder to find information. (I-2)<\/blockquote>\n<p>Informing over a long period of time creates an awareness that attached to sharing is an embedded information overload problem. This is illustrated by repeatedly performing an informing practice wherein users redirect information that is stored elsewhere. For example, while information is stored locally on hardware or in a local folder, such information is exposed and redistributed multiple times in the rooms. Hence, making information available to create transparency led to other consequences:\n<\/p>\n<blockquote>The intention with the ESMP was that we should move away from local storage of information in our own local folder structures. Everything was to be stored on the ESMP. I\u2019m skeptical of it, because it is such a vast amount of information that it makes it difficult to identify what is relevant. We end up with huge hits when we search, and we spend a lot of time on finding out what is relevant. And when we do not have the rigid old structure, which we had under the intranet, we spend a lot of time looking among all the hits we get. I think we would have wanted it to be a little stricter on what should be stored. Things should be deleted, if they are considered [ir]relevant. I\u2019m also skeptical that we use the ESMP as a primary storage source for everything. I\u2019m also concerned because we could forget the formal filing and procedural rules that we have to deal with. When we publish on the ESMP, we think we preserve it forever, and that\u2019s not right. There are some formal things that make me skeptical. The most concerning thing, however, is that it has become such a huge volume of documentation. And when it comes to relevant and non-relevant information on individual characteristics, I am a bit skeptical of that Facebook style of writing status updates. I think it\u2019s nice to have colleagues, but it\u2019s more interesting knowing how we are professionally connected. We are a large organization. If all of us post information that we were sick, and that we are looking forward to the weekend, there is a limit to how much of that information I want to see. I think it takes another turn and we\u2019re moving towards that side. That part I\u2019m not thrilled over. I\u2019m one of those who think that when I\u2019m at work, I\u2019m at work. People can tell me interesting things that are useful or fun for me to learn at work. I don\u2019t want a lot of private information, which I\u2019m not related to. (I-2)<\/blockquote>\n<p>Exposure to excessive information instigates users to enforce a personal filter mechanism and to return to old work habits such as using e-mail. This results in the creation of distance from knowledge formation processes and the prevalence of disengaged users:\n<\/p>\n<blockquote>In the start, when it was brand new, I tried to make use of any opportunity, which was not in the intranet. We had the possibility to create rooms. I did that and invited people. I joined other rooms. But, afterwards, I failed to follow up all that. In neither of the rooms I administer now, did I manage to develop anything. I\u2019m rarely there and don\u2019t check the rooms that I am a member of. (I-3)<\/blockquote>\n<p>This user saw the rooms as an opportunity to improve conditions for interaction with the high schools with which she has frequent contact as part of her work. Much of the daily contact consists of sending general information. Instead of sending all of this information via e-mail, it could be transferred to the rooms, but she did not manage to uphold the goal of sharing:\n<\/p>\n<blockquote>I haven\u2019t had time to prioritize the room. They come far down on my priority list. My workday is packed with to-do tasks. To sit down to try to use the possibilities and communicate in the rooms has instead led me not doing that. Now, I don\u2019t bother checking notifications from the rooms I administer or follow or what my colleagues have written in their status updates. I skip that very fast and I go directly to check my e-mails. (I-3)<\/blockquote>\n<p>This pattern of disengagement was seen in another experience. An informant explained that the challenges of generating engagement were related to aspects of the user interface itself. For example, it was difficult to ascertain whether the rooms were used by others as no panel to show numbers of visitors existed. The male user argued that the information shared in the rooms was already available and ready-made in other spaces, which meant that it was stored by co-workers in their e-mail inboxes. However, other matters drew attention:\n<\/p>\n<blockquote>The challenge with the ESMP is that there are too many rooms. It\u2019s almost like we have a room for each employee. You have to click on a link to get to somewhere. And then you have to go back again and click on a new link again, so it will be many rounds, just to get hold of the information you\u2019re looking for. (I-6)<\/blockquote>\n<p>Although a number of the informants were uncertain as to the extent to which their sharing in the rooms was of benefit, another user shared a quite different opinion. A female user working with accounting explained that the rooms represent a type of \"manual\" She was an active user and saw the benefit of retrieving and finding information that had been shared by others:\n<\/p>\n<blockquote>For example, I\u2019m working in the accounting system and I find out that I need to get hold of a manual or retrieve information on an account. I go on the ESMP. There, I locate documents or things that are written about the case I\u2019m working on. I\u2019m a member of all the rooms that have something to do with accounting, a factor allowing me to know what we\u2019ve posted and what others ask about. (I-7)<\/blockquote>\n<p>The rooms are beneficial in different ways. For example, they are information depositories, where one can find quick answers, as they narrow down the need for searching. Alternatively, she would have to search for the same information in larger web-based databases.\n<\/p>\n<blockquote>Since they exist, they are easy available. They are part of a knowledge you can easily use. They are there if you need to be reminded about something. For example, in accounting, there are clear definitions, clear rules for use; there\u2019s a clear date of notice for certain things. Things that are not so relevant one day, I often get information about in advance. But then I get questions from colleagues working in other departments, who ask about a deadline. What date is set as a deadline for the final reporting? Now, I know where I can quickly get and give an answer back on that. It\u2019s not necessarily that I have that knowledge in my head, but now I have good knowledge of where the answer is located. (I-7)<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Orlikowski and Gash<sup id=\"rdp-ebb-cite_ref-OrlikowskiTechno94_4-4\" class=\"reference\"><a href=\"#cite_note-OrlikowskiTechno94-4\">[4]<\/a><\/sup> utilized a practice and interpretation perspective to conjecture that Notes can be interpreted in diverse ways, revealing differences in intent and actual use. Their analysis highlighted the notion that implementers see Notes as a tool for organizational change and collaboration, whereas end-users interpret it as a means of individual and personal communication. Using a similar research lens with alternative empirical material\u2014the implementation of an ESMP in a Norwegian public organization\u2014what clues are provided that help to answer the paper\u2019s research question?\n<\/p><p>The most important finding is the contradictory nature of a top management initiative intended at simplifying employees\u2019 workload in a public organization that seems to have had the opposite outcome in terms of end-users\u2019 use and action. Sharing, introduced as a new workplace principle, was expected to create transparency and enhance the flow of internal communication, but when the end-users attempted to translate sharing into a manageable practice\u2014as the basis for participation in a knowledge formation process\u2014they interpreted sharing as a complicated work practice, with the larger consequence of producing disengaged users. This is primarily related to the fact that users are not performing a sharing practice, that is, a two-way communication process whereby knowledge is created through collaboration. Rather, this data analysis shows that the users engaged in informing practices to fulfill the goal of sharing, an aspect that has been demonstrated throughout the data analysis section. This informing practice\u2014which represents an essential ingredient in creating a knowledge-sharing process\u2014is performed on the premise of informing an audience and of being informed. Moreover, the informing practice is seldom the start of a knowledge process where two users exchange information to create knowledge by reflection on action, for example. Instead, sharing is carried out by re-publishing ready-made and approved official documents found elsewhere in the CA, creating an unmanageable information overload problem that encapsulates the challenges in forming a sensible knowledge-sharing process in practice. Furthermore, clues are provided regarding what is actually shared, which in this exploratory case study pertains to information that is already known. Sharing proves to be problematic and is associated with risk-taking for those involved, leading to the enforcement of self-censorship and the construction of separate and private workplaces that the informants deem beneficial to completing their work. In contrast, the users institute personal filters and return to work practices they believe work, which in most cases is e-mail. In other words, sharing in this case study is interpreted and performed as informing or as being informed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The main outcomes of this exploratory case study have been an examination of the term \"sharing,\" and demonstrating the challenges involved in introducing it as a workplace principle in a public organization. Moreover, when public employees attempt to perform and make sense of \u2018sharing\u2019 in practice, a two-way communication practice emerges that can be misidentified and performed as a practice based on informing an audience and of being informed, hence causing an information overload problem and the prevalence of disengaged users in organizational life.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-LeonardiEnter13-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LeonardiEnter13_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-LeonardiEnter13_1-1\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-LeonardiEnter13_1-2\">1.2<\/a><\/sup> <sup><a href=\"#cite_ref-LeonardiEnter13_1-3\">1.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonardi, P.M.; Huysman, M.; Steinfield, C. (2013). \"Enterprise Social Media: Definition, History, and Prospects for the Study of Social Technologies in Organizations\". <i>Journal of Computer-Mediated Communication<\/i> <b>19<\/b> (1): 1\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjcc4.12029\" target=\"_blank\">10.1111\/jcc4.12029<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enterprise+Social+Media%3A+Definition%2C+History%2C+and+Prospects+for+the+Study+of+Social+Technologies+in+Organizations&rft.jtitle=Journal+of+Computer-Mediated+Communication&rft.aulast=Leonardi%2C+P.M.%3B+Huysman%2C+M.%3B+Steinfield%2C+C.&rft.au=Leonardi%2C+P.M.%3B+Huysman%2C+M.%3B+Steinfield%2C+C.&rft.date=2013&rft.volume=19&rft.issue=1&rft.pages=1%E2%80%9319&rft_id=info:doi\/10.1111%2Fjcc4.12029&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McAfeeEnter06-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McAfeeEnter06_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">McAfee, A. (September 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/andrewmcafee.org\/2006\/09\/enterprise_20_inclusionists_and_deletionists\/\" target=\"_blank\">\"Enterprise 2.0 Inclusionists and Deletionists\"<\/a>. <i>AndrewMcAfee.org<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/andrewmcafee.org\/2006\/09\/enterprise_20_inclusionists_and_deletionists\/\" target=\"_blank\">http:\/\/andrewmcafee.org\/2006\/09\/enterprise_20_inclusionists_and_deletionists\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Enterprise+2.0+Inclusionists+and+Deletionists&rft.atitle=AndrewMcAfee.org&rft.aulast=McAfee%2C+A.&rft.au=McAfee%2C+A.&rft.date=September+2006&rft_id=http%3A%2F%2Fandrewmcafee.org%2F2006%2F09%2Fenterprise_20_inclusionists_and_deletionists%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OrlikowskiUsing00-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OrlikowskiUsing00_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-OrlikowskiUsing00_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Orlikowski, W.J. (2000). \"Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations\". <i>Organization Science<\/i> <b>11<\/b> (4): 367\u2013472. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Forsc.11.4.404.14600\" target=\"_blank\">10.1287\/orsc.11.4.404.14600<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+Technology+and+Constituting+Structures%3A+A+Practice+Lens+for+Studying+Technology+in+Organizations&rft.jtitle=Organization+Science&rft.aulast=Orlikowski%2C+W.J.&rft.au=Orlikowski%2C+W.J.&rft.date=2000&rft.volume=11&rft.issue=4&rft.pages=367%E2%80%93472&rft_id=info:doi\/10.1287%2Forsc.11.4.404.14600&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OrlikowskiTechno94-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OrlikowskiTechno94_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-OrlikowskiTechno94_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-OrlikowskiTechno94_4-2\">4.2<\/a><\/sup> <sup><a href=\"#cite_ref-OrlikowskiTechno94_4-3\">4.3<\/a><\/sup> <sup><a href=\"#cite_ref-OrlikowskiTechno94_4-4\">4.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Orlikowski, W.J.; Gash, D.C. (1994). \"Technological frames: making sense of information technology in organizations\". <i>ACM Transactions on Information Systems<\/i> <b>12<\/b> (2): 174\u2013207. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F196734.196745\" target=\"_blank\">10.1145\/196734.196745<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Technological+frames%3A+making+sense+of+information+technology+in+organizations&rft.jtitle=ACM+Transactions+on+Information+Systems&rft.aulast=Orlikowski%2C+W.J.%3B+Gash%2C+D.C.&rft.au=Orlikowski%2C+W.J.%3B+Gash%2C+D.C.&rft.date=1994&rft.volume=12&rft.issue=2&rft.pages=174%E2%80%93207&rft_id=info:doi\/10.1145%2F196734.196745&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MertonTheUnant36-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MertonTheUnant36_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Merton, R.K. (1936). \"The Unanticipated Consequences of Purposive Social Action\". <i>American Sociological Review<\/i> <b>1<\/b> (6): 894\u2013904. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2307%2F2084615\" target=\"_blank\">10.2307\/2084615<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Unanticipated+Consequences+of+Purposive+Social+Action&rft.jtitle=American+Sociological+Review&rft.aulast=Merton%2C+R.K.&rft.au=Merton%2C+R.K.&rft.date=1936&rft.volume=1&rft.issue=6&rft.pages=894%E2%80%93904&rft_id=info:doi\/10.2307%2F2084615&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArdichviliMotiv03-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ArdichviliMotiv03_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-ArdichviliMotiv03_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ardichvili, A.; Page, V.; Wentling, T. (2003). \"Motivation and barriers to participation in virtual knowledge \u2010 Sharing communities of practice\". <i>Journal of Knowledge Management<\/i> <b>7<\/b> (1): 64\u201377. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2F13673270310463626\" target=\"_blank\">10.1108\/13673270310463626<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Motivation+and+barriers+to+participation+in+virtual+knowledge+%E2%80%90+Sharing+communities+of+practice&rft.jtitle=Journal+of+Knowledge+Management&rft.aulast=Ardichvili%2C+A.%3B+Page%2C+V.%3B+Wentling%2C+T.&rft.au=Ardichvili%2C+A.%3B+Page%2C+V.%3B+Wentling%2C+T.&rft.date=2003&rft.volume=7&rft.issue=1&rft.pages=64%E2%80%9377&rft_id=info:doi\/10.1108%2F13673270310463626&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KosonenKnowl09-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KosonenKnowl09_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kosonen, M. (2009). \"Knowledge sharing in virtual communities \u2013 A review of the empirical research\". <i>International Journal of Web Based Communities<\/i> <b>5<\/b> (2): 144\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1504%2FIJWBC.2009.023962\" target=\"_blank\">10.1504\/IJWBC.2009.023962<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Knowledge+sharing+in+virtual+communities+%E2%80%93+A+review+of+the+empirical+research&rft.jtitle=International+Journal+of+Web+Based+Communities&rft.aulast=Kosonen%2C+M.&rft.au=Kosonen%2C+M.&rft.date=2009&rft.volume=5&rft.issue=2&rft.pages=144%E2%80%9363&rft_id=info:doi\/10.1504%2FIJWBC.2009.023962&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChenToGive10-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChenToGive10_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chen, C.-J.; Hung, S.-W. (2010). \"To give or to receive? Factors influencing members\u2019 knowledge sharing and community promotion in professional virtual communities\". <i>Information & Management<\/i> <b>47<\/b> (4): 226-236. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.im.2010.03.001\" target=\"_blank\">10.1016\/j.im.2010.03.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=To+give+or+to+receive%3F+Factors+influencing+members%E2%80%99+knowledge+sharing+and+community+promotion+in+professional+virtual+communities&rft.jtitle=Information+%26+Management&rft.aulast=Chen%2C+C.-J.%3B+Hung%2C+S.-W.&rft.au=Chen%2C+C.-J.%3B+Hung%2C+S.-W.&rft.date=2010&rft.volume=47&rft.issue=4&rft.pages=226-236&rft_id=info:doi\/10.1016%2Fj.im.2010.03.001&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WaskoWhyShould05-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WaskoWhyShould05_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wasko, M.M.; Faraj, S. (2005). \"Why Should I Share? Examining Social Capital and Knowledge Contribution in Electronic Networks of Practice\". <i>MIS Quarterly<\/i> <b>29<\/b> (1): 35-57. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2307%2F25148667\" target=\"_blank\">10.2307\/25148667<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Why+Should+I+Share%3F+Examining+Social+Capital+and+Knowledge+Contribution+in+Electronic+Networks+of+Practice&rft.jtitle=MIS+Quarterly&rft.aulast=Wasko%2C+M.M.%3B+Faraj%2C+S.&rft.au=Wasko%2C+M.M.%3B+Faraj%2C+S.&rft.date=2005&rft.volume=29&rft.issue=1&rft.pages=35-57&rft_id=info:doi\/10.2307%2F25148667&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArdichviliLearn08-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArdichviliLearn08_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ardichvili, A. (2008). \"Learning and Knowledge Sharing in Virtual Communities of Practice: Motivators, Barriers, and Enablers\". <i>Advances in Developing Human Resources<\/i> <b>10<\/b> (4): 541\u201344. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1523422308319536\" target=\"_blank\">10.1177\/1523422308319536<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Learning+and+Knowledge+Sharing+in+Virtual+Communities+of+Practice%3A+Motivators%2C+Barriers%2C+and+Enablers&rft.jtitle=Advances+in+Developing+Human+Resources&rft.aulast=Ardichvili%2C+A.&rft.au=Ardichvili%2C+A.&rft.date=2008&rft.volume=10&rft.issue=4&rft.pages=541%E2%80%9344&rft_id=info:doi\/10.1177%2F1523422308319536&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArdichviliCultural06-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArdichviliCultural06_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ardichvili, A.; Maurer, M.; Li, W. et al. (2006). \"Cultural influences on knowledge sharing through online communities of practice\". <i>Journal of Knowledge Management<\/i> <b>10<\/b> (1): 94\u2013107. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2F13673270610650139\" target=\"_blank\">10.1108\/13673270610650139<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cultural+influences+on+knowledge+sharing+through+online+communities+of+practice&rft.jtitle=Journal+of+Knowledge+Management&rft.aulast=Ardichvili%2C+A.%3B+Maurer%2C+M.%3B+Li%2C+W.+et+al.&rft.au=Ardichvili%2C+A.%3B+Maurer%2C+M.%3B+Li%2C+W.+et+al.&rft.date=2006&rft.volume=10&rft.issue=1&rft.pages=94%E2%80%93107&rft_id=info:doi\/10.1108%2F13673270610650139&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KaplanUsers10-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KaplanUsers10_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kaplan, A.M.; Haenlein, M. (2010). \"Users of the world, unite! The challenges and opportunities of Social Media\". <i>Business Horizons<\/i> <b>53<\/b> (1): 59\u201368. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.bushor.2009.09.003\" target=\"_blank\">10.1016\/j.bushor.2009.09.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Users+of+the+world%2C+unite%21+The+challenges+and+opportunities+of+Social+Media&rft.jtitle=Business+Horizons&rft.aulast=Kaplan%2C+A.M.%3B+Haenlein%2C+M.&rft.au=Kaplan%2C+A.M.%3B+Haenlein%2C+M.&rft.date=2010&rft.volume=53&rft.issue=1&rft.pages=59%E2%80%9368&rft_id=info:doi\/10.1016%2Fj.bushor.2009.09.003&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TreemSocial12-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TreemSocial12_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-TreemSocial12_13-1\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-TreemSocial12_13-2\">13.2<\/a><\/sup> <sup><a href=\"#cite_ref-TreemSocial12_13-3\">13.3<\/a><\/sup> <sup><a href=\"#cite_ref-TreemSocial12_13-4\">13.4<\/a><\/sup> <sup><a href=\"#cite_ref-TreemSocial12_13-5\">13.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Treem, J.W.; Leonardi, P.M. (2012). \"Social Media Use in Organizations: Exploring the Affordances of Visibility, Editability, Persistence, and Association\". <i>Communication Yearbook<\/i> <b>36<\/b>: 143\u201389. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2139%2Fssrn.2129853\" target=\"_blank\">10.2139\/ssrn.2129853<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Social+Media+Use+in+Organizations%3A+Exploring+the+Affordances+of+Visibility%2C+Editability%2C+Persistence%2C+and+Association&rft.jtitle=Communication+Yearbook&rft.aulast=Treem%2C+J.W.%3B+Leonardi%2C+P.M.&rft.au=Treem%2C+J.W.%3B+Leonardi%2C+P.M.&rft.date=2012&rft.volume=36&rft.pages=143%E2%80%9389&rft_id=info:doi\/10.2139%2Fssrn.2129853&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonardiWhatsUnder10-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonardiWhatsUnder10_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonardi, P.M.; Barley, S.R. (2010). \"What\u2019s Under Construction Here? Social Action, Materiality, and Power in Constructivist Studies of Technology and Organizing\". <i>Academy of Management Annals<\/i> <b>4<\/b> (1): 1\u201351. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5465%2F19416521003654160\" target=\"_blank\">10.5465\/19416521003654160<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What%E2%80%99s+Under+Construction+Here%3F+Social+Action%2C+Materiality%2C+and+Power+in+Constructivist+Studies+of+Technology+and+Organizing&rft.jtitle=Academy+of+Management+Annals&rft.aulast=Leonardi%2C+P.M.%3B+Barley%2C+S.R.&rft.au=Leonardi%2C+P.M.%3B+Barley%2C+S.R.&rft.date=2010&rft.volume=4&rft.issue=1&rft.pages=1%E2%80%9351&rft_id=info:doi\/10.5465%2F19416521003654160&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GibsonTheEco86-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GibsonTheEco86_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-GibsonTheEco86_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Gibson, J.J. (1986). <i>The Ecological Approach to Visual Perception<\/i>. Lawrence Erlbaum Associates. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 0898599598.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Ecological+Approach+to+Visual+Perception&rft.aulast=Gibson%2C+J.J.&rft.au=Gibson%2C+J.J.&rft.date=1986&rft.pub=Lawrence+Erlbaum+Associates&rft.isbn=0898599598&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MutchSocio13-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MutchSocio13_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mutch, A. (2013). \"Sociomateriality \u2014 Taking the wrong turning?\". <i>Information and Organization<\/i> <b>23<\/b> (1): 28\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.infoandorg.2013.02.001\" target=\"_blank\">10.1016\/j.infoandorg.2013.02.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sociomateriality+%E2%80%94+Taking+the+wrong+turning%3F&rft.jtitle=Information+and+Organization&rft.aulast=Mutch%2C+A.&rft.au=Mutch%2C+A.&rft.date=2013&rft.volume=23&rft.issue=1&rft.pages=28%E2%80%9340&rft_id=info:doi\/10.1016%2Fj.infoandorg.2013.02.001&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonardiTheor13-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LeonardiTheor13_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-LeonardiTheor13_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonardi, P.M. (2013). \"Theoretical foundations for the study of sociomateriality\". <i>Information and Organization<\/i> <b>23<\/b> (2): 59\u201376. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.infoandorg.2013.02.002\" target=\"_blank\">10.1016\/j.infoandorg.2013.02.002<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Theoretical+foundations+for+the+study+of+sociomateriality&rft.jtitle=Information+and+Organization&rft.aulast=Leonardi%2C+P.M.&rft.au=Leonardi%2C+P.M.&rft.date=2013&rft.volume=23&rft.issue=2&rft.pages=59%E2%80%9376&rft_id=info:doi\/10.1016%2Fj.infoandorg.2013.02.002&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OrlikowskiSocio07-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OrlikowskiSocio07_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Orlikowski, W.J. (2007). \"Sociomaterial Practices: Exploring Technology at Work\". <i>Organization Studies<\/i> <b>28<\/b> (9): 1435-1448. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0170840607081138\" target=\"_blank\">10.1177\/0170840607081138<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sociomaterial+Practices%3A+Exploring+Technology+at+Work&rft.jtitle=Organization+Studies&rft.aulast=Orlikowski%2C+W.J.&rft.au=Orlikowski%2C+W.J.&rft.date=2007&rft.volume=28&rft.issue=9&rft.pages=1435-1448&rft_id=info:doi\/10.1177%2F0170840607081138&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CallonSome86-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CallonSome86_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Callon, M. (1986). \"Some elements of a sociology of translation: domestication of the scallops and the fishermen of St Brieuc Bay\". In Law, J.. <i>Power, action, and belief: A new sociology of knowledge?<\/i>. Routledge & Kegan Paul. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780710208026.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Some+elements+of+a+sociology+of+translation%3A+domestication+of+the+scallops+and+the+fishermen+of+St+Brieuc+Bay&rft.atitle=Power%2C+action%2C+and+belief%3A+A+new+sociology+of+knowledge%3F&rft.aulast=Callon%2C+M.&rft.au=Callon%2C+M.&rft.date=1986&rft.pub=Routledge+%26+Kegan+Paul&rft.isbn=9780710208026&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LatourScience88-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LatourScience88_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Latour, B. (1988). <i>Science in Action: How to Follow Scientists and Engineers through Society<\/i>. Harvard University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780674792913.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Science+in+Action%3A+How+to+Follow+Scientists+and+Engineers+through+Society&rft.aulast=Latour%2C+B.&rft.au=Latour%2C+B.&rft.date=1988&rft.pub=Harvard+University+Press&rft.isbn=9780674792913&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LatourReass05-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LatourReass05_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Latour, B. (2005). <i>Reassembling the Social: An Introduction to Actor-Network-Theory<\/i>. Oxford University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780199256051.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Reassembling+the+Social%3A+An+Introduction+to+Actor-Network-Theory&rft.aulast=Latour%2C+B.&rft.au=Latour%2C+B.&rft.date=2005&rft.pub=Oxford+University+Press&rft.isbn=9780199256051&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OrlikowskiSocio08-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OrlikowskiSocio08_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Orlikowski, W.J.; Scott, S.V. (2008). \"Sociomateriality: Challenging the Separation of Technology, Work and Organization\". <i>Academy of Management Annals<\/i> <b>2<\/b> (1): 433\u201374. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5465%2F19416520802211644\" target=\"_blank\">10.5465\/19416520802211644<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sociomateriality%3A+Challenging+the+Separation+of+Technology%2C+Work+and+Organization&rft.jtitle=Academy+of+Management+Annals&rft.aulast=Orlikowski%2C+W.J.%3B+Scott%2C+S.V.&rft.au=Orlikowski%2C+W.J.%3B+Scott%2C+S.V.&rft.date=2008&rft.volume=2&rft.issue=1&rft.pages=433%E2%80%9374&rft_id=info:doi\/10.5465%2F19416520802211644&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SuchmanHuman07-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SuchmanHuman07_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Suchman, L. (2007). <i>Human-Machine Reconfigurations: Plans and Situated Actions<\/i> (2nd ed.). Cambridge University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780521858915.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Human-Machine+Reconfigurations%3A+Plans+and+Situated+Actions&rft.aulast=Suchman%2C+L.&rft.au=Suchman%2C+L.&rft.date=2007&rft.edition=2nd&rft.pub=Cambridge+University+Press&rft.isbn=9780521858915&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OrlikowskiTheSocio10-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OrlikowskiTheSocio10_24-0\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-OrlikowskiTheSocio10_24-1\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Orlikowski, W.J. (2010). \"The sociomateriality of organisational life: Considering technology in management research\". <i>Cambridge Journal of Economics<\/i> <b>34<\/b> (1): 125\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fcje%2Fbep058\" target=\"_blank\">10.1093\/cje\/bep058<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+sociomateriality+of+organisational+life%3A+Considering+technology+in+management+research&rft.jtitle=Cambridge+Journal+of+Economics&rft.aulast=Orlikowski%2C+W.J.&rft.au=Orlikowski%2C+W.J.&rft.date=2010&rft.volume=34&rft.issue=1&rft.pages=125%E2%80%9341&rft_id=info:doi\/10.1093%2Fcje%2Fbep058&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OrlikowskiWhatHappens14-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OrlikowskiWhatHappens14_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-OrlikowskiWhatHappens14_25-1\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Orlikowski, W.J.; Scott, S.V. (2014). \"What Happens When Evaluation Goes Online? Exploring Apparatuses of Valuation in the Travel Sector\". <i>Organization Science<\/i> <b>25<\/b> (3): 868\u201391. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Forsc.2013.0877\" target=\"_blank\">10.1287\/orsc.2013.0877<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+Happens+When+Evaluation+Goes+Online%3F+Exploring+Apparatuses+of+Valuation+in+the+Travel+Sector&rft.jtitle=Organization+Science&rft.aulast=Orlikowski%2C+W.J.%3B+Scott%2C+S.V.&rft.au=Orlikowski%2C+W.J.%3B+Scott%2C+S.V.&rft.date=2014&rft.volume=25&rft.issue=3&rft.pages=868%E2%80%9391&rft_id=info:doi\/10.1287%2Forsc.2013.0877&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonardiSocial14-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonardiSocial14_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonardi, P.M. (2014). \"Social Media, Knowledge Sharing, and Innovation: Toward a Theory of Communication Visibility\". <i>Information Systems Research<\/i> <b>25<\/b> (4): 796\u2013816. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Fisre.2014.0536\" target=\"_blank\">10.1287\/isre.2014.0536<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Social+Media%2C+Knowledge+Sharing%2C+and+Innovation%3A+Toward+a+Theory+of+Communication+Visibility&rft.jtitle=Information+Systems+Research&rft.aulast=Leonardi%2C+P.M.&rft.au=Leonardi%2C+P.M.&rft.date=2014&rft.volume=25&rft.issue=4&rft.pages=796%E2%80%93816&rft_id=info:doi\/10.1287%2Fisre.2014.0536&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonardiAmbient15-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonardiAmbient15_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonardi, P.M. (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/misq.org\/ambient-awareness-and-knowledge-acquisition-using-social-media-to-learn-who-knows-what-and-who-knows-whom.html\" target=\"_blank\">\"Ambient Awareness and Knowledge Acquisition: Using Social Media to Learn \"Who Knows What\" and \"Who Knows Whom\"\"<\/a>. <i>MIS Quarterly<\/i> <b>39<\/b> (4): 747\u201362<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/misq.org\/ambient-awareness-and-knowledge-acquisition-using-social-media-to-learn-who-knows-what-and-who-knows-whom.html\" target=\"_blank\">https:\/\/misq.org\/ambient-awareness-and-knowledge-acquisition-using-social-media-to-learn-who-knows-what-and-who-knows-whom.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ambient+Awareness+and+Knowledge+Acquisition%3A+Using+Social+Media+to+Learn+%22Who+Knows+What%22+and+%22Who+Knows+Whom%22&rft.jtitle=MIS+Quarterly&rft.aulast=Leonardi%2C+P.M.&rft.au=Leonardi%2C+P.M.&rft.date=2015&rft.volume=39&rft.issue=4&rft.pages=747%E2%80%9362&rft_id=https%3A%2F%2Fmisq.org%2Fambient-awareness-and-knowledge-acquisition-using-social-media-to-learn-who-knows-what-and-who-knows-whom.html&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeonardiSocialMedia15-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeonardiSocialMedia15_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leonardi, P.M.; Meyer, S.R. (2015). \"Social Media as Social Lubricant: How Ambient Awareness Eases Knowledge Transfer\". <i>American Behavioral Scientist<\/i> <b>59<\/b> (1): 10\u201334. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0002764214540509\" target=\"_blank\">10.1177\/0002764214540509<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Social+Media+as+Social+Lubricant%3A+How+Ambient+Awareness+Eases+Knowledge+Transfer&rft.jtitle=American+Behavioral+Scientist&rft.aulast=Leonardi%2C+P.M.%3B+Meyer%2C+S.R.&rft.au=Leonardi%2C+P.M.%3B+Meyer%2C+S.R.&rft.date=2015&rft.volume=59&rft.issue=1&rft.pages=10%E2%80%9334&rft_id=info:doi\/10.1177%2F0002764214540509&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MajchrzakTheContra13-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MajchrzakTheContra13_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Majchrzak, A.; Faraj, S.; Kane, G.C.; Azad, B. (2013). \"The Contradictory Influence of Social Media Affordances on Online Communal Knowledge Sharing\". <i>Journal of Computer-Mediated Communication<\/i> <b>19<\/b> (1): 38\u201355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjcc4.12030\" target=\"_blank\">10.1111\/jcc4.12030<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Contradictory+Influence+of+Social+Media+Affordances+on+Online+Communal+Knowledge+Sharing&rft.jtitle=Journal+of+Computer-Mediated+Communication&rft.aulast=Majchrzak%2C+A.%3B+Faraj%2C+S.%3B+Kane%2C+G.C.%3B+Azad%2C+B.&rft.au=Majchrzak%2C+A.%3B+Faraj%2C+S.%3B+Kane%2C+G.C.%3B+Azad%2C+B.&rft.date=2013&rft.volume=19&rft.issue=1&rft.pages=38%E2%80%9355&rft_id=info:doi\/10.1111%2Fjcc4.12030&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EllisonTheUse15-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-EllisonTheUse15_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-EllisonTheUse15_30-1\">30.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ellison, N.B.; Gibbs, J.L.; Weber, M.S. (2015). \"The Use of Enterprise Social Network Sites for Knowledge Sharing in Distributed Organizations: The Role of Organizational Affordances\". <i>American Behavioral Scientist<\/i> <b>59<\/b> (1): 103\u2013123. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F0002764214540510\" target=\"_blank\">10.1177\/0002764214540510<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Use+of+Enterprise+Social+Network+Sites+for+Knowledge+Sharing+in+Distributed+Organizations%3A+The+Role+of+Organizational+Affordances&rft.jtitle=American+Behavioral+Scientist&rft.aulast=Ellison%2C+N.B.%3B+Gibbs%2C+J.L.%3B+Weber%2C+M.S.&rft.au=Ellison%2C+N.B.%3B+Gibbs%2C+J.L.%3B+Weber%2C+M.S.&rft.date=2015&rft.volume=59&rft.issue=1&rft.pages=103%E2%80%93123&rft_id=info:doi\/10.1177%2F0002764214540510&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FulkLocation13-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FulkLocation13_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fulk, J.; Yuan, Y.C. (2013). \"Location, Motivation, and Social Capitalization via Enterprise Social Networking\". <i>Journal of Computer-Mediated Communication<\/i> <b>19<\/b> (1): 20\u201337. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjcc4.12033\" target=\"_blank\">10.1111\/jcc4.12033<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Location%2C+Motivation%2C+and+Social+Capitalization+via+Enterprise+Social+Networking&rft.jtitle=Journal+of+Computer-Mediated+Communication&rft.aulast=Fulk%2C+J.%3B+Yuan%2C+Y.C.&rft.au=Fulk%2C+J.%3B+Yuan%2C+Y.C.&rft.date=2013&rft.volume=19&rft.issue=1&rft.pages=20%E2%80%9337&rft_id=info:doi\/10.1111%2Fjcc4.12033&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VaastSocial13-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VaastSocial13_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vaast, E.; Kaganer, E. (2013). \"Social media affordances and governance in the workplace: An examination of organizational policies\". <i>Journal of Computer-Mediated Communication<\/i> <b>19<\/b> (1): 78\u2013101. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjcc4.12032\" target=\"_blank\">10.1111\/jcc4.12032<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Social+media+affordances+and+governance+in+the+workplace%3A+An+examination+of+organizational+policies&rft.jtitle=Journal+of+Computer-Mediated+Communication&rft.aulast=Vaast%2C+E.%3B+Kaganer%2C+E.&rft.au=Vaast%2C+E.%3B+Kaganer%2C+E.&rft.date=2013&rft.volume=19&rft.issue=1&rft.pages=78%E2%80%93101&rft_id=info:doi\/10.1111%2Fjcc4.12032&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OostervinkKnowl13-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OostervinkKnowl13_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Oostervink, N.; Agerberg, M.; Huysman, M. (2013). \"Knowledge Sharing on Enterprise Social Media: Practices to Cope With Institutional Complexity\". <i>Journal of Computer-Mediated Communication<\/i> <b>21<\/b> (2): 156\u201376. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjcc4.12153\" target=\"_blank\">10.1111\/jcc4.12153<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Knowledge+Sharing+on+Enterprise+Social+Media%3A+Practices+to+Cope+With+Institutional+Complexity&rft.jtitle=Journal+of+Computer-Mediated+Communication&rft.aulast=Oostervink%2C+N.%3B+Agerberg%2C+M.%3B+Huysman%2C+M.&rft.au=Oostervink%2C+N.%3B+Agerberg%2C+M.%3B+Huysman%2C+M.&rft.date=2013&rft.volume=21&rft.issue=2&rft.pages=156%E2%80%9376&rft_id=info:doi\/10.1111%2Fjcc4.12153&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GibbsOver13-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GibbsOver13_34-0\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-GibbsOver13_34-1\">34.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gibbs, J.L., Rozaidi, N.H.; Eisenberg, J. (2013). \"Overcoming the \u201cIdeology of Openness\u201d: Probing the Affordances of Social Media for Organizational Knowledge Sharing\". <i>Journal of Computer-Mediated Communication<\/i> <b>19<\/b> (1): 102\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjcc4.12034\" target=\"_blank\">10.1111\/jcc4.12034<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Overcoming+the+%E2%80%9CIdeology+of+Openness%E2%80%9D%3A+Probing+the+Affordances+of+Social+Media+for+Organizational+Knowledge+Sharing&rft.jtitle=Journal+of+Computer-Mediated+Communication&rft.aulast=Gibbs%2C+J.L.%2C+Rozaidi%2C+N.H.%3B+Eisenberg%2C+J.&rft.au=Gibbs%2C+J.L.%2C+Rozaidi%2C+N.H.%3B+Eisenberg%2C+J.&rft.date=2013&rft.volume=19&rft.issue=1&rft.pages=102%E2%80%9320&rft_id=info:doi\/10.1111%2Fjcc4.12034&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EhrlichSearching07-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EhrlichSearching07_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ehrlich, K.; Lin, C.-Y.; Griffiths-Fisher, V. (2007). \"Searching for experts in the enterprise: combining text and social network analysis\". <i>Proceedings of the 2007 International ACM Conference on Supporting Group Work<\/i>: 117\u201326. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1316624.1316642\" target=\"_blank\">10.1145\/1316624.1316642<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Searching+for+experts+in+the+enterprise%3A+combining+text+and+social+network+analysis&rft.jtitle=Proceedings+of+the+2007+International+ACM+Conference+on+Supporting+Group+Work&rft.aulast=Ehrlich%2C+K.%3B+Lin%2C+C.-Y.%3B+Griffiths-Fisher%2C+V.&rft.au=Ehrlich%2C+K.%3B+Lin%2C+C.-Y.%3B+Griffiths-Fisher%2C+V.&rft.date=2007&rft.pages=117%E2%80%9326&rft_id=info:doi\/10.1145%2F1316624.1316642&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Thom-SantelliSocial08-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Thom-SantelliSocial08_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thom-Santelli; J.; Muller, M.J.; Millen, D.R. (2008). \"Social tagging roles: Publishers, evangelists, leaders\". <i>Proceedings of the 2008 SIGCHI Conference on Human Factors in Computing Systems<\/i>: 1041\u201344. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1357054.1357215\" target=\"_blank\">10.1145\/1357054.1357215<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Social+tagging+roles%3A+Publishers%2C+evangelists%2C+leaders&rft.jtitle=Proceedings+of+the+2008+SIGCHI+Conference+on+Human+Factors+in+Computing+Systems&rft.aulast=Thom-Santelli%3B+J.%3B+Muller%2C+M.J.%3B+Millen%2C+D.R.&rft.au=Thom-Santelli%3B+J.%3B+Muller%2C+M.J.%3B+Millen%2C+D.R.&rft.date=2008&rft.pages=1041%E2%80%9344&rft_id=info:doi\/10.1145%2F1357054.1357215&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DuganItsAll08-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DuganItsAll08_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dugan, C.; Geyer, W.; Muller, M. et al. (2008). \"It's all 'about you': Diversity in online profiles\". <i>Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work<\/i>: 703\u201306. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1460563.1460672\" target=\"_blank\">10.1145\/1460563.1460672<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=It%27s+all+%27about+you%27%3A+Diversity+in+online+profiles&rft.jtitle=Proceedings+of+the+2008+ACM+Conference+on+Computer+Supported+Cooperative+Work&rft.aulast=Dugan%2C+C.%3B+Geyer%2C+W.%3B+Muller%2C+M.+et+al.&rft.au=Dugan%2C+C.%3B+Geyer%2C+W.%3B+Muller%2C+M.+et+al.&rft.date=2008&rft.pages=703%E2%80%9306&rft_id=info:doi\/10.1145%2F1460563.1460672&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SteinfieldBowling09-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SteinfieldBowling09_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Steinfield, C.; DiMicco, J.M.; Ellison, N.B. et al. (2009). \"Bowling online: Social networking and social capital within the organization\". <i>Proceedings of the Fourth International Conference on Communities and Technologies<\/i>: 245\u201354. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1556460.1556496\" target=\"_blank\">10.1145\/1556460.1556496<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bowling+online%3A+Social+networking+and+social+capital+within+the+organization&rft.jtitle=Proceedings+of+the+Fourth+International+Conference+on+Communities+and+Technologies&rft.aulast=Steinfield%2C+C.%3B+DiMicco%2C+J.M.%3B+Ellison%2C+N.B.+et+al.&rft.au=Steinfield%2C+C.%3B+DiMicco%2C+J.M.%3B+Ellison%2C+N.B.+et+al.&rft.date=2009&rft.pages=245%E2%80%9354&rft_id=info:doi\/10.1145%2F1556460.1556496&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WuDetecting10-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WuDetecting10_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wu, A.; DiMicco, J.M.; Millen, D.R. (2010). \"Detecting professional versus personal closeness using an enterprise social network site\". <i>Proceedings of the 2010 SIGCHI Conference on Human Factors in Computing Systems<\/i>: 1955\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1753326.1753622\" target=\"_blank\">10.1145\/1753326.1753622<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Detecting+professional+versus+personal+closeness+using+an+enterprise+social+network+site&rft.jtitle=Proceedings+of+the+2010+SIGCHI+Conference+on+Human+Factors+in+Computing+Systems&rft.aulast=Wu%2C+A.%3B+DiMicco%2C+J.M.%3B+Millen%2C+D.R.&rft.au=Wu%2C+A.%3B+DiMicco%2C+J.M.%3B+Millen%2C+D.R.&rft.date=2010&rft.pages=1955%E2%80%9364&rft_id=info:doi\/10.1145%2F1753326.1753622&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Thom-SantelliChar10-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Thom-SantelliChar10_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thom-Santelli, J.; Millen, D.R.; MiMicco, J.M. (2010). \"Characterizing global participation in an enterprise SNS\". <i>Proceedings of the 3rd International Conference on Intercultural Collaboration<\/i>: 251\u201354. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1841853.1841900\" target=\"_blank\">10.1145\/1841853.1841900<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Characterizing+global+participation+in+an+enterprise+SNS&rft.jtitle=Proceedings+of+the+3rd+International+Conference+on+Intercultural+Collaboration&rft.aulast=Thom-Santelli%2C+J.%3B+Millen%2C+D.R.%3B+MiMicco%2C+J.M.&rft.au=Thom-Santelli%2C+J.%3B+Millen%2C+D.R.%3B+MiMicco%2C+J.M.&rft.date=2010&rft.pages=251%E2%80%9354&rft_id=info:doi\/10.1145%2F1841853.1841900&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Thom-SantelliOrg11-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Thom-SantelliOrg11_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thom-Santelli, J.; Millen, D.R.; Gergle, D. (2011). \"Organizational acculturation and social networking\". <i>Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work<\/i>: 313\u201316. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1958824.1958871\" target=\"_blank\">10.1145\/1958824.1958871<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Organizational+acculturation+and+social+networking&rft.jtitle=Proceedings+of+the+ACM+2011+Conference+on+Computer+Supported+Cooperative+Work&rft.aulast=Thom-Santelli%2C+J.%3B+Millen%2C+D.R.%3B+Gergle%2C+D.&rft.au=Thom-Santelli%2C+J.%3B+Millen%2C+D.R.%3B+Gergle%2C+D.&rft.date=2011&rft.pages=313%E2%80%9316&rft_id=info:doi\/10.1145%2F1958824.1958871&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhangACase10-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhangACase10_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhang, J.; Qu, Y.; Cody, J.; Wu, Y. (2010). \"A case study of micro-blogging in the enterprise: Use, value, and related issues\". <i>Proceedings of the SIGCHI Conference on Human Factors in Computing Systems<\/i>: 123\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1753326.1753346\" target=\"_blank\">10.1145\/1753326.1753346<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+case+study+of+micro-blogging+in+the+enterprise%3A+Use%2C+value%2C+and+related+issues&rft.jtitle=Proceedings+of+the+SIGCHI+Conference+on+Human+Factors+in+Computing+Systems&rft.aulast=Zhang%2C+J.%3B+Qu%2C+Y.%3B+Cody%2C+J.%3B+Wu%2C+Y.&rft.au=Zhang%2C+J.%3B+Qu%2C+Y.%3B+Cody%2C+J.%3B+Wu%2C+Y.&rft.date=2010&rft.pages=123%E2%80%9332&rft_id=info:doi\/10.1145%2F1753326.1753346&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-L.C3.BCdersNetwork13-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-L.C3.BCdersNetwork13_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">L\u00fcders, M. (2013). \"Networking and notworking in social intranets: User archetypes and participatory divides\". <i>First Monday<\/i> <b>18<\/b> (8). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5210%2Ffm.v18i8.4693\" target=\"_blank\">10.5210\/fm.v18i8.4693<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Networking+and+notworking+in+social+intranets%3A+User+archetypes+and+participatory+divides&rft.jtitle=First+Monday&rft.aulast=L%C3%BCders%2C+M.&rft.au=L%C3%BCders%2C+M.&rft.date=2013&rft.volume=18&rft.issue=8&rft_id=info:doi\/10.5210%2Ffm.v18i8.4693&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PettersonFromMass14-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PettersonFromMass14_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pettersen, L. (2014). \"From Mass Production to Mass Collaboration: Institutionalized Hindrances to Social Platforms in the Workplace\". <i>Nordic Journal of Science and Technology Studies<\/i> <b>2<\/b> (2): 29\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5324%2Fnjsts.v2i2.2146\" target=\"_blank\">10.5324\/njsts.v2i2.2146<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+Mass+Production+to+Mass+Collaboration%3A+Institutionalized+Hindrances+to+Social+Platforms+in+the+Workplace&rft.jtitle=Nordic+Journal+of+Science+and+Technology+Studies&rft.aulast=Pettersen%2C+L.&rft.au=Pettersen%2C+L.&rft.date=2014&rft.volume=2&rft.issue=2&rft.pages=29%E2%80%9340&rft_id=info:doi\/10.5324%2Fnjsts.v2i2.2146&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PettersonTheRole16-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PettersonTheRole16_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pettersen, L. (2016). \"The Role of Offline Places for Communication and Social Interaction in Online and Virtual Spaces in the Multinational Workplace\". <i>Nordicom Review<\/i> <b>37<\/b> (Special Issue): 131\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1515%2Fnor-2016-0028\" target=\"_blank\">10.1515\/nor-2016-0028<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Role+of+Offline+Places+for+Communication+and+Social+Interaction+in+Online+and+Virtual+Spaces+in+the+Multinational+Workplace&rft.jtitle=Nordicom+Review&rft.aulast=Pettersen%2C+L.&rft.au=Pettersen%2C+L.&rft.date=2016&rft.volume=37&rft.issue=Special+Issue&rft.pages=131%E2%80%9346&rft_id=info:doi\/10.1515%2Fnor-2016-0028&rfr_id=info:sid\/en.wikipedia.org:Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version\u2014by design\u2014lists them in order of appearance.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165500\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.002 seconds\nReal time usage: 1.032 seconds\nPreprocessor visited node count: 33399\/1000000\nPreprocessor generated node count: 40026\/1000000\nPost\u2010expand include size: 217434\/2097152 bytes\nTemplate argument size: 72194\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 999.864 1 - -total\n 87.79% 877.758 1 - Template:Reflist\n 77.42% 774.112 45 - Template:Citation\/core\n 70.85% 708.432 39 - Template:Cite_journal\n 8.97% 89.700 5 - Template:Cite_book\n 6.59% 65.928 1 - Template:Infobox_journal_article\n 6.32% 63.169 1 - Template:Infobox\n 6.08% 60.832 43 - Template:Citation\/identifier\n 4.03% 40.267 47 - Template:Citation\/make_link\n 3.83% 38.332 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11037-0!*!0!!en!*!* and timestamp 20190701165459 and revision id 35529\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F\">https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","4be2f1cd180a132f1a4ce3ca4fde9d6e_images":[],"4be2f1cd180a132f1a4ce3ca4fde9d6e_timestamp":1562000099,"97b51c665da19cc225b05dd3ff621e96_type":"article","97b51c665da19cc225b05dd3ff621e96_title":"Developing workforce capacity in public health informatics: Core competencies and curriculum design (Wholey et al. 2019)","97b51c665da19cc225b05dd3ff621e96_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design","97b51c665da19cc225b05dd3ff621e96_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Developing workforce capacity in public health informatics: Core competencies and curriculum design\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDeveloping workforce capacity in public health informatics: Core competencies and curriculum designJournal\n \nFrontiers in Public HealthAuthor(s)\n \nWholey, Douglas R.; LaVenture, Martin; Rajamani, Sripriya; Kreiger, Rob.; Hedberg, Craig; Kenyon, CynthiaAuthor affiliation(s)\n \nUniversity of Minnesota, Minnesota Department of Health, AllinaHealthPrimary contact\n \nwhole001 at umn dot eduEditors\n \nCaron, Rosemary M.Year published\n \n2019Volume and issue\n \n6Page(s)\n \n124DOI\n \n10.3389\/fpubh.2018.00124ISSN\n \n2296-2565Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fpubh.2018.00124\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fpubh.2018.00124\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 PHI program design objectives and assumptions \n4 PHI program design challenges \n5 PHI competencies \n\n5.1 Core competencies \n5.2 Declarative and procedural competencies \n5.3 Electives \n\n\n6 Organizing competencies into courses \n7 Implications for policy and practice \n8 Conclusion \n9 Acknowledgements \n\n9.1 Author contributions \n9.2 Conflict of interest \n\n\n10 References \n11 Notes \n\n\n\nAbstract \nWe describe a master\u2019s level public health informatics (PHI) curriculum to support workforce development. Public health decision-making requires intensive information management to organize responses to health threats and develop effective health education and promotion. PHI competencies prepare the public health workforce to design and implement these information systems. The objective for a master's and certificate in PHI is to prepare public health informaticians with the competencies to work collaboratively with colleagues in public health and other health professions to design and develop information systems that support population health improvement. The PHI competencies are drawn from computer, information, and organizational sciences. A curriculum is proposed to deliver the competencies, and the results of a pilot PHI program are presented. Since the public health workforce needs to use information technology effectively to improve population health, it is essential for public health academic institutions to develop and implement PHI workforce training programs.\nKeywords: public health informatics, public health practice, public health workforce, systems analysis, systems design\n\nIntroduction \nWith the increasing use of electronic data collection and storage, there is an increasing focus on information and knowledge management systems.[1] This results in public health workforce needs for public health informatics (PHI) professionals skilled in designing and implementing these systems. PHI professionals are those who \u201cwork in practice, research, or academia and whose primary work function is to use informatics to improve the health of populations.\u201d[2] Recent studies highlight the need for informatics training in the public health workforce.[3][4] Similarly, the Council on Education for Public Health (CEPH) includes informatics as a foundational competency for accreditation of public health programs.[5][6] We build on the literature for developing informatics' savvy public health departments[7][8][9][10] and PHI core competency descriptions[2][11] to design a curriculum for a master's program in public health (MPH) and a certificate in PHI. The proposed curriculum differs from many existing health informatics programs by emphasizing the public health informatician's architectural role as a systems analyst linking public health users and information technology specialists by guiding the analysis and design of information systems. The focus is on: (a) problem definition for public health information systems (PHInfSys); (b) analyzing, designing, and implementing effective PHInfSys, such as surveillance, community health assessment, and population health management systems;[2][12] (c) encoding, collecting, curating, storing, retrieving, and analyzing data to create information; (d) assuring system and data governance, management, integration, confidentiality, and security; (e) creating and managing information technologies; and (f) collaborating in and leading inter-disciplinary and cross-cutting teams.\nThe foundation of public health is information, collected across a variety of sources, with the goal of generating and disseminating new knowledge and instituting actions to improve the public\u2019s health. The need for public health education programs to be informed by current public health needs and employment opportunities has been long noted[13], as has the importance of establishing academic-practice links with public health agencies and training students in practice situations. In 2001, the need for PHI training was noted in an American Medical Informatics Association (AMIA) conference on developing a PHI agenda.[14] In 2012, AMIA revisited and updated these recommendations.[15] Both reports covered technical topics, such as information architecture and standards; governance and policy, including confidentiality and privacy; and workforce training. These needs are addressed by the PHI training program we describe.\nThe 2012 AMIA report noted that the \u201c[t]he public health value chain is composed of business processes and use cases that describe the flow of data and information. Business processes describe how data, information, and knowledge are used by creating a framework that relates work activities to domains of public health function.\u201d[15] PHI is a foundation for modernizing public health value chains, including public health department business processes[16], public health surveillance[17][18], public health emergency response, population health science[19], population health management[20], learning health systems[21], and public health research. PHI is an essential specialization that bridges the digital gap public health agencies face given growing expectations of service and preparedness. PHI integrates knowledge and skills from the information sciences (computer, information, organizational, and systems sciences) with public health expertise and \u201cincludes the conceptualization, design, development, deployment, refinement, maintenance, and evaluation of communication, surveillance, and information systems relevant to public health.\u201d[22]\nWhile PHI shares some core skills with other health informatics domains, such as bioinformatics or clinical and nursing informatics, it is distinct in necessitating integration of core informatics skills with public health systems, such as surveillance systems, community health assessment, disease\/condition registries, and prevention programs in an overall framework of population health science. Public health informaticians (PHInf) differ from information technologists (ITs) because \u201c[t]he focus of IT is to implement and operate information systems (hardware and software) that meet programmatic needs. In contrast, PHInf have a strategic and systems view of how information systems and technology can impact public health, such as how information systems can support public health decision-making. Unlike IT specialists, PHInf work within the larger context of how information systems function within the political, cultural, economic, and social environment and evaluate their impact within the broad sphere of public health. Thus, PHInf are in the unique position to understand how information systems can improve the practice and science of public health while contributing to the evidence-based practice of public health informatics.\u201d[2]\nOur paper focuses on training PHInf through MPH and certificate programs to provide them a foundation for a PHI professional career. The professional role is one of four related PHI roles identified by the PHI Institute, the Association of State and Territorial Health Officials, and the National Association of County and City Health Officials (NACCHO): executive; manager; professional; and clinical.[23] The curriculum design highlighted here focuses on the professional role because it is a frontline position that is the foundation for a career path through management to executive leadership.[24]\nOur paper describes program design objectives and assumptions, design challenges, and core competencies. This overview provides context and prioritization for the program design.\n\nPHI program design objectives and assumptions \nThe objective of PHI training is to educate PHInf with public health and PHI knowledge and skills that enable them to design and implement effective PHInfSys and have a successful PHI career. The design assumes that PHInf require competencies in (a) public health core competencies[25], functions, and systems (e.g., disease and environmental surveillance[18], population health management[20], and community health assessment[20][26][27]); (b) design thinking and systems analysis[28][29][30]; (c) computer, information, organizational, and systems sciences[22][31]; (d) evaluating information systems effectiveness[32]; and (e) teamwork and project management.\nThe core public health functions are assurance, assessment, and advocacy, which are used to maximize population health.[33] The PHI program design focuses on understanding how information systems achieve these functions rather than having a sub-goal focus on technological, computer science, or data mining aspects of PHI. Because PHI focuses on designing PHInfSys it has a foundation in design thinking.[28][29][30] Design thinking focuses on identifying problems as gaps in population health that information systems can help reduce by determining the root causes of those gaps and designing and implementing information systems to reduce population health gaps. Systems analysts are the architects who bridge the gap between users and technologies by understanding and translating user needs to system specifications and collaborating with technologists to implement those systems. The goal of systems analysis is to minimize the frequent disconnects between information system development and the usefulness of the resulting information systems[34][35] by avoiding simply automating existing inefficient information pathways.\nTo address these disconnects, effective systems analysis requires that PHInf are \u201cwell grounded in the fundamentals of organization theory, decision-making, teamwork and leadership, and research methods as well as current and emerging information systems technologies.\u201d[36] PHInf also need the ability to design, implement, and evaluate information systems in multiple contexts, using approaches such as realistic evaluation and context-mechanism-outcome configurations (CMOc).[32]\nFinally, PHInf need strong teamwork and project management skills because \u201cPHI is cross-cutting. \u2026 Informaticians see the \u2018big picture\u2019 and \u2018connects the dots\u2019 across all of the other fields related to public health.\u201d[12] Because information systems span diverse domains, the public health informatician requires the knowledge and skills to collaborate with diverse clients, lead systems analysis and development, and coordinate the diverse specialists implementing information systems, data use and privacy agreements, data and vocabulary standards, evaluation, training, and more.\nIn sum, PHInf require competencies in (a) public health core concepts and systems; (b) systems thinking and systems analysis; (c) computer, information, organizational, and systems sciences; (d) evaluating information systems effectiveness; and (e) the ability to work in and lead teams. These competencies provide the foundation for successfully progressing from being a PHI professional to PHI leadership positions.\n\nPHI program design challenges \nPublic health informatics requires many different competencies. This can result in a laundry list of competencies that are not feasible to implement in a credit-constrained curriculum. Implementing a constrained educational program requires competency prioritization and trade-offs. There are three major challenges.\n1. Content: PHI requires competencies in distinct domains\u2014public health, health informatics, computer, information, and organizational sciences.[2][11][31] These include required core competencies for public health practice.[25] Health informatics and computer science focus on technical competencies. Information, systems, and organizational science competencies support solving population health problems because they provide systems analysis and design competencies needed to: (1) understand and translate user needs into system requirements and (2) create designs that can be implemented by ITs. Computer science competencies, such as modular decomposition, data structures, algorithms, programming languages, database theories, and system architectures are necessary to provide PHInf the knowledge and skills to communicate and collaborate effectively with ITs implementing PHInfSys.\n2. Curricular independence vs. integrations: Siloed, independent courses maximize covering competency breadth, but they also rely on students to integrate across competencies. Courses integrating competencies develop the skills to apply informatics in public health. Examples of integrating competencies include using population health management and surveillance examples in technical courses, such as systems analysis, database, or health information exchange.\n3. Declarative vs. procedural competencies: Declarative competencies are familiarity with analysis tools, such as requirements analysis, business process modeling, use cases, data flow diagrams, and entity relationship diagrams. Procedural competencies are being able to develop requirements, business process models, use cases, data flow diagrams, and entity relationship diagrams. Training in procedural competencies requires experiential practicums in which students apply declarative competencies. Increasing the emphasis on procedural competencies limits the breadth of declarative competencies that can be covered. The PHI design focuses on developing strong procedural competencies in system analysis because it is the core competency used to design and implement effective PHInf.\nIn sum, design challenges include decisions about content, integration (independent\/integrated), and competency type (declarative\/procedural) tradeoffs.\nThe proposed PHI design focuses on public health core competencies and organization\/systems\/information sciences related to analyzing, designing, implementing, and disseminating effective information systems in public health. Integrating competencies across courses maximizes the ability to apply skills in public health settings. For declarative and procedural learning, experiential practicums assure the ability to apply declarative knowledge. Elective credits support students in developing deeper skills in areas they are interested in, such as data management, analytics\/data mining, GIS, visualization\/communication, health information exchange[37], leading community health information exchange, or surveillance systems.[18]\n\nPHI competencies \nA PHI competency is a measurable and demonstrable knowledge related to the role \u201cof developing innovative applications of technology and systems that address public health priorities by analyzing how information is organized and used and evaluating how this work contributes to the scientific field.\u201d[2] The master\u2019s level competencies for PHI professionals build on earlier work .[2][11] Not all competencies listed in earlier work are included for three reasons. First, the selected competencies need to fit into a typical MPH curriculum. While there are many desirable and important competencies, there is limited curriculum time. Second, some competencies reflect work in a practice setting that are not directly measurable (e.g., collaborating with others in program development, offering insights, contributing to decision-making, leading knowledge management). Third, some competencies are more managerial\/executive competencies than professional competencies.\n\nCore competencies \nCore competencies include both public health and PHI. Public health core competencies[6][25] are addressed through the required MPH curriculum courses. This material is extended by integrating competencies related to public health systems, surveillance systems, population health management, community health assessment, eHealth, program monitoring, and evaluation in the PHI courses. For example, using electronic disease surveillance, population health management, community health assessment, and other public health systems as examples in the systems analysis courses educates students in the use of systems analysis to address common public health systems.[16][38]\nPublic health informatics competencies focus on systems analysis and data modeling. These include computer science and health informatics competencies related to nomenclature, standards, platforms, information architecture, interoperability, and systems analysis and data modeling, which require integration of material from organization\/information systems\/systems sciences, public health, and computer sciences. This integration builds the overall competency to design PHInfSys to solve problems in the public health domain. These competencies are noted as central to PHI. For example, \u201cRedesigning Public Health Surveillance in an eHealth World\u201d demonstrates the importance of systems analysis when it points out that \u201c[d]efining requirements is a critical step in developing or acquiring an information system that will effectively support the work of the organization.\u201d[38] Similarly, system selection depends \u201con the clarity and applicability of the use cases you define, and on determining prior to the demonstration how\u2014based on what criteria\u2014the use-case demonstrations will be judged\u201d[38] as well as business process definitions.[38] The description of an informatics savvy health department states that \u201c[i]nformation systems managers and staff require competency in creating formal system requirements.\u201d[7] Requirements, use cases, and business process definitions are produced by systems analysis and require expertise in the system development life cycles (SDLC), national standards, and public policy related to informatics (e.g., meaningful use).[8] Complementary competencies are those that support systems analysis and data modeling[39], such as standards (e.g., HL7), nomenclature (e.g., ICD, CPT)[40], data flow diagrams, relational database theory, and query languages that inform system analysis and are used during implementation.\nAs an example, Table 1 illustrates systems analysis competencies for electronic disease surveillance. The goal of the system is to maximize population health by minimizing transmission of communicable diseases, which is done \u201cthrough early identification, treatment, and resolution of health conditions.\u201d[16] Systems analysis requires integrating competency in systems analysis with expertise in the system\u2019s public health domain (e.g., electronic disease surveillance). The first analysis phase defines the problem, its decision-makers, context, and stakeholders. The second phase does a root cause analysis of gaps in population health related to communicable disease surveillance or system productivity and describes functional requirements, what tasks the communicable disease surveillance system supports, and nonfunctional requirements, criteria that the system has to meet. The third phase describes the logical model, the use cases corresponding to each functional requirement, business processes, data flows, and data model. Design and implementation is the final phase and builds on the analysis in the earlier phases to design and implement the information system that supports communicable disease surveillance staff in their activities.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 1. ystems analysis example for electronic disease surveillance\n\n\n\nDeclarative and procedural competencies \nTables 2 and 3 show the MPH PHI declarative and procedural competencies, which have a strong focus on those competencies also used in systems analysis and data modeling. These competencies reflect the informaticians\u2019 architect role, which focuses on the problem definition and solving skills associated with defining problems, identifying requirements, building logical models, describing core features of information systems (use cases, data flows, data models, business processes), designing implementation alternatives, overseeing implementation, and facilitating user centered design and choice.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 2. Public health informatics (PHI) declarative competencies\n\n\n\n\n\n\n\n\n\n\n\n\n Table 3. Public health informatics (PHI) procedural competencies\n\n\n\nThe declarative and procedural competencies also include a significant focus on research analysis and evaluation methods because PHInfSys are interventions designed to improve population health by improving quality, reducing costs, or both. Given the intervention aspect, a PHInf should have the competencies to be able to evaluate whether the public health information system accomplished its goals and how the information system functions in different contexts (realistic evaluation[32]). Effective evaluation requires enough expertise in public health systems, such as disease surveillance[17], quality improvement[41], public reporting, such as control charts, funnel plots (league tables)[42], and spatial outlier detection[43], to be able to assess how well they are implemented as well as their effectiveness. Competencies in assessing measurement validity and reliability are central to measuring population health.\n\nElectives \nElectives allow students to pursue specializations in related public health and PHI competencies such as:\n\n PHI management: project management, health information management, public health leadership, and leading change for regional and community information systems\n PHI tools: ePublic health, using the web and mobile apps for health promotion, GIS for public health, health care operations research and analytics, public health decision support systems, eHealth, health information exchange, and surveillance systems in depth\n PHI evaluation: decision analysis for policy analysis, cost-effective analysis, program evaluation, usability engineering, and realistic evaluation\n Environmental health systems: Complex systems modeling for population health, infectious diseases, environmental health risk assessment, surveillance of foodborne diseases and food safety hazards\n Analytics and data mining: Advanced statistical computing, statistical learning and data mining, survival analysis, statistics for surveillance and quality improvement\n Computer science: Algorithms and data structures, principles and practice of database systems, data marts and warehouse management, NoSQL and alternatives to relational data models\nOrganizing competencies into courses \nThe design addresses siloing vs. integration in two ways (Table 4). First, given that the application of informatics competencies is conditional on public health context, informatics and public health competencies are strongly integrated. Students need the expertise to apply the informatics concepts in the domain they will practice. Second, a practice-based practicum in the introductory systems analysis course and a practice-based capstone systems analysis course integrate practice and academics.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 4. Public health informatics (PHI) curriculum and competencies\n\n\n\nAs an example, informatics competencies in systems analysis courses (introduction and capstone) are illustrated using PHInfSys. The relational database course (managing electronic health information) uses population health measurement for SQL assignments. Data and information for population health management, which focuses on substantive issues related to population health management uses relational database theory and SQL for assignments in population health management and frames the evaluation of public health interventions using realistic evaluation. In sum, the design emphasizes the integration of systems analysis, database, and evaluation competencies with public health systems and population health management. The objective is to train students in informatics competencies, public health competencies, and the use of informatics skills in public health settings. A separate course addresses competencies that are general and applicable to many areas, such as standards, nomenclature, health information exchange layers, and technical issues.\n\nImplications for policy and practice \nThis paper proposes a master\u2019s level public health curriculum to support workforce development in PHI competencies. The competencies are similar to those described for public health core competencies[6] and in the Advanced Health Informatics Certification[44] with the goal of preparing PHInf who \u201c[p]ractice health informatics with an operational focus on information and knowledge problems that directly impacts the practice of health care, public health, and personal health.\u201d[44] Edward Baker and his colleagues recently made a strong business case for PHI[45] arguing that public health is an information intensive business and that health information systems are transforming the way that information management speeds responses to health threats, promote health, and supports public health decision-making. The foundation of these systems are the essential public health services, a focus on population health, and core informatics competencies, such as system analysis, database theory, and health informatics. The implementation of effective PHInfSys requires individuals with the competencies described in this paper.\nThis program design was based on Carnegie Mellon\u2019s highly successful information systems program which focus on systems analysis and information technology skills to \u201cdesign and implement effective solutions to meet organizational and management needs for information and decision support.\u201d[36] The program was implemented in the School of Public Health at the University of Minnesota in the Fall of 2013. The first year courses have been offered, evaluated, and refined five times, and the second year courses have been offered and refined four times. Thus far, 19 students have earned PHI certificates signifying completion of PHI core courses; 15 will have completed the MPH in PHI by May 2018. The certificate attracted diverse students, including students from health informatics, pharmacy, environmental health, and epidemiology. Applicants have expressed interest in PHI because they feel that data and information is critical to public health research and practice, and they feel that learning informatics and analytics in a public health context will help them to identify opportunities for improving population health. Students in the PHI program have had internships supporting quality improvement and population health management with a safety net clinic network, database administration in health care systems and professional organizations, and supporting standards implementation in electronic disease surveillance systems in state health departments. Graduates of the PHI program have been successful in securing positions across private and public health sectors, and some have pursued further degrees in health informatics. Seven of the 10 MPH-PHI graduates are employed in positions related to PHI. \nTable 5 provides a sample of their job descriptions. The job responsibilities span the spectrum from leading projects in PHI and population health management to offering technical support in data management and research coordination. Future program evaluation should measure the degree to which graduates are engaged in public health related activities, are engaged in PHI related activities, initial placement, and career growth.\n\r\n\n\n\n\n\n\n\n\n\n\n Table 5. Recent master\u2019s in public health public health informatics graduates position examples\n\n\n\nThere are development and sustainability issues that occur because PHI is a new field. First, program development requires collaborating with faculty and stakeholders to communicate what PHI is and a vision of its value.[12] Second, marketing and social media programs that convey PHI\u2019s contribution to new and existing public health students need to be developed. Third, partnerships with stakeholders for course based projects and integrating practice with academics have to be developed. Fourth, PHI program development requires funding commitment from schools of public health until enrollment reaches a break-even point. Finally, PHI faculty who have competency in both public health and informatics need to be developed. Until PHI as a field is more institutionalized, a potential tactic for doing this is recruiting post-doctoral staff from fields within public health (e.g., epidemiology, environmental health) and have them develop PHI competencies while developing PHI related research and\/or recruiting post-doctoral staff from related informatics disciplines[46] and have them develop public health competencies while developing PHI related research and teaching. While these barriers appear substantial, PHI should be sustainable because of the growth in informatics related disciplines and because of skilled PHInf being in high demand as valuable team members in both research and practice.\n\nConclusion \nThe effective and efficient collection of quality public health data and information is intensive. It is the foundation of public health functions, such as surveillance, community health assessment, evaluation, and research. PHI is a profession that focuses on the design of governance, analysis, development, and management of information systems that acquire and integrate data that is used in public health functions, such as disease and environmental surveillance, program evaluation, policy analysis, and research. PHI provides the foundation for organizing and collecting data that is used in performing public health functions. PHI is an emerging, distinct, and needed profession in public health that uses knowledge and skills drawn from computer, information, and organizational sciences and public health to develop critical information systems that provide the data and information that other public health professionals use in their work. PHInf provide the expertise in public health, information, and information technology to take advantage of technology advances to improve the PHInf.\nThis paper proposes a PHI curriculum for schools of public health that can be implemented through MPH and certificate programs. The curriculum provides core competencies in public health and in computer, information, and organizational sciences that are needed by PHInf. The proposed curriculum addresses the challenges of integrating PHI in public health functions, developing declarative knowledge and procedural skills in PHI through a balance of lecture and practicum courses, and allowing the pursuit of specializations. A key focus of the proposed curriculum is its user centered design through systems analysis and systems thinking. This provides PHInf the competencies necessary to lead, design, and implement PHInfSys that take advantage of technology advances.\nThe proposed program is a starting point. Given the rapidity of change in public health practice, public health research, informatics, and information technology, the curriculum will need to evolve to meet changing needs. This evolution should be guided by partnerships with stakeholders in practice, other public health disciplines, and public health researchers to assure that PHInf are effectively designed and developed to meet the needs of those working to improve population health. This development can be guided by the continued evaluation of PHInf to assure they meet user needs and the continued evaluation of PHI programs to assure they are training the workforce that can lead developments in PHInfSys.\nThere is a clear need to develop a PHI workforce. It is imperative that public health institutions, both practice and academic, take action and support the development of a strong PHI workforce. This can be achieved through on-the-job training, fellowships, internships, and academic training. The educational program we propose addresses schools of public health, and provides guidelines for implementing PHI training programs. However, support is also needed from public health practitioners to ensure that PHI training is prioritized to meet the evolving demands of the practice environment. With these tools we can continue to develop an effective and relevant workforce that can navigate the changing trends of public health to improve population health.\n\nAcknowledgements \nWe thank Pat Larkey, Sara Kiesler, Serge Taylor, Bill Hefley, Kathleen Carley, and the Information Systems program at Carnegie Mellon University for foundational ideas. We thank Julie Jacko, Daniel Chan, and Matteo Convertino for their ideas and help in implementing the PHI program at Minnesota. We thank Rochelle Martin, Raul Noriega, Megan Pruente, Cole Schluchter, Nicholas Solberg, Herong Song, and Patrick Williams for information on their current job responsibilities and comments about the PHI program. We thank MPH and certificate PHI students for working with us to develop the program. We thank Ira Moscovice the Health Policy & Management Department, and the School of Public Health at the University of Minnesota for supporting PHI program development.\n\nAuthor contributions \nAll authors contributed to the conceptualization of this paper. DW, RK, and SR led the curriculum conceptualization and development. ML, SR, and DW led the focus on public health informatics (e.g., the informatics savvy health department, immunization systems), CH, SR, and CK led the focus on applications of PHI in electronic disease surveillance system. DW and ML did the first draft. RK, SR, CH, and CK edited and provided components (e.g., description of electronic disease support systems, contents of systems analysis competencies). All read the final version, edited it, and approved it for submission.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Freed, J.; Kyffin, R.; Rashbass, J. et al. (June 2014). \"Knowledge strategy: Harnessing the power of information to improve the public's health\". Public Health England. https:\/\/www.gov.uk\/government\/publications\/knowledge-strategy-harnessing-the-power-of-information-to-improve-the-publics-health .   \n\n\u2191 2.0 2.1 2.2 2.3 2.4 2.5 2.6 Centers for Disease Control and Prevention, University of Washington's Center for Public Health Informatics (September 2009). \"Competencies for Public Health Informaticians\" (PDF). US Department of Health and Human Services, Centers for Disease Control and Prevention. Archived from the original on 13 June 2017. https:\/\/web.archive.org\/web\/20170613082955\/https:\/\/www.cdc.gov\/informaticscompetencies\/pdfs\/phi-competencies.pdf .   \n\n\u2191 Massoudi, B.L.; Chester, K.; Shah, G.H. (2016). \"Public Health Staff Development Needs in Informatics: Findings From a National Survey of Local Health Departments\". Journal of Public Health Management and Practice 22 (Suppl. 6): S58\u2013S62. doi:10.1097\/PHH.0000000000000450. PMC PMC5049962. PMID 27684619. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5049962 .   \n\n\u2191 Beck, A.J.; Leider, J.P.; Coronado, F. et al. (2017). \"State Health Agency and Local Health Department Workforce: Identifying Top Development Needs\". American Journal of Public Health 107 (9): 1418-1424. doi:10.2105\/AJPH.2017.303875. PMID 28727537.   \n\n\u2191 Calhoun, J.G.; Ramiah, K.; Weist, E.M. et al. (2008). \"Development of a core competency model for the master of public health degree\". American Journal of Public Health 98 (9): 1598-607. doi:10.2105\/AJPH.2007.117978. PMC PMC2509588. PMID 18633093. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2509588 .   \n\n\u2191 6.0 6.1 6.2 Council on Education for Public Health (October 2016). \"Accreditation Criteria: Schools of Public Health & Public Health Programs\" (PDF). https:\/\/media.ceph.org\/wp_assets\/2016.Criteria.pdf .   \n\n\u2191 7.0 7.1 LaVenture, M.; Brand, B.; Ross, D.A. et al. (2014). \"Building an informatics-savvy health department: Part I, vision and core strategies\". Journal of Public Health Management and Practice 20 (6): 667\u20139. doi:10.1097\/PHH.0000000000000149. PMID 25250757.   \n\n\u2191 8.0 8.1 LaVenture, M.; Brand, B.; Ross, D.A. et al. (2015). \"Building an informatics-savvy health department II: Operations and tactics\". Journal of Public Health Management and Practice 21 (1): 96\u20139. doi:10.1097\/PHH.0000000000000179. PMID 25414962.   \n\n\u2191 LaVenture, M.; Brand, B.; Baker, E.L. (2017). \"Developing an informatics-savvy health department: From discrete projects to a coordinating program - Part I: Assessment and governance\". Journal of Public Health Management and Practice 23 (3): 325\u20137. doi:10.1097\/PHH.0000000000000551. PMID 28350628.   \n\n\u2191 Brand, B.; LaVenture, M.; Baker, E.L. (2018). \"Developing an informatics-savvy health department: From discrete projects to a coordinating program - Part III: Ensuring well-designed and effectively used information systems\". Journal of Public Health Management and Practice 24 (2): 181\u20134. doi:10.1097\/PHH.0000000000000756. PMID 29360696.   \n\n\u2191 11.0 11.1 11.2 Miner, K.; Alperin, M.; Brogan, C.W. et al. (28 February 2011). \"Applied Public Health Informatics Curriculum - Preface and Modules\". Public Health Informatics Institute. https:\/\/www.phii.org\/resources\/view\/158\/applied-public-health-informatics-curriculum-preface-and-modules .   \n\n\u2191 12.0 12.1 12.2 Fond, M.; Volmert, A.; Kendall-Taylor, N. (September 2015). \"Making Public Health Informatics Visible: Communicating an Emerging Field\" (PDF). FrameWorks Institute. https:\/\/www.frameworksinstitute.org\/assets\/files\/health_care\/phiistrategicmtgfinalseptember2015.pdf .   \n\n\u2191 Institute of Medicine (1988). \"The Future of Public Health\". The National Academies Press. doi:10.17226\/1091. https:\/\/www.nap.edu\/catalog\/1091\/the-future-of-public-health .   \n\n\u2191 Yasnoff, W.A.; Overhage, J.M.; Humphreys, B.L. et al. (2001). \"A national agenda for public health informatics: summarized recommendations from the 2001 AMIA Spring Congress\". JAMIA 8 (6): 535\u201345. PMC PMC130064. PMID 11687561. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130064 .   \n\n\u2191 15.0 15.1 Massoudi, B.L.; Goodman, K.W.; Gotham, I.J. et al. (2012). \"An informatics agenda for public health: Summarized recommendations from the 2011 AMIA PHI Conference\". JAMIA 19 (5): 688-95. doi:10.1136\/amiajnl-2011-000507. PMC PMC3422819. PMID 22395299. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3422819 .   \n\n\u2191 16.0 16.1 16.2 Public Health Informatics Institute (2008). Taking Care of Business: A Collaboration to Define Local Health Department Business Processes (2nd Printing ed.). Public Health Informatics Institute. https:\/\/www.phii.org\/resources\/taking-care-business-second-edition .   \n\n\u2191 17.0 17.1 McNabb, S.J.; Conde, J.M.; Ferland, L. et al. (2016). Transforming Public Health Surveillance: Proactive Measures for Prevention, Detection, and Response. Elsevier. ISBN 9780702063374.   \n\n\u2191 18.0 18.1 18.2 McNabb, S.J.N.; Rayland, P.; Sylvester, J. et al. (2017). \"Informatics enables public health surveillance\". Journal of Health Specialties 5 (2): 55\u201359. doi:10.4103\/jhs.JHS_28_17.   \n\n\u2191 Bachrach, C.A.; Daley, D.M. (2017). \"Shaping a New Field: Three Key Challenges for Population Health Science\". American Journal of Public Health 107 (2): 251\u201352. doi:10.2105\/AJPH.2016.303580. PMC PMC5227949. PMID 28075642. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5227949 .   \n\n\u2191 20.0 20.1 20.2 Kindig, D.A. (2007). \"Understanding population health terminology\". Milbank Q 85 (1): 139\u201361. doi:10.1111\/j.1468-0009.2007.00479.x. PMC PMC2690307. PMID 17319809. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690307 .   \n\n\u2191 Friedman, C.; Rubin, J.; Brown, J. et al. (2015). \"Toward a science of learning systems: A research agenda for the high-functioning Learning Health System\". JAMIA 22 (1): 43\u201350. doi:10.1136\/amiajnl-2014-002977. PMC PMC4433378. PMID 25342177. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4433378 .   \n\n\u2191 22.0 22.1 Yasnoff, W.A.; O'Carroll, P.W.; Koo, D. et al. (2000). \"Public health informatics: improving and transforming public health in the information age\". Journal of Public Health Management and Practics 6 (6): 67\u201375. PMID 18019962.   \n\n\u2191 Public Health Informatics Institute (25 April 2014). \"Workforce Position Classifications and Descriptions\". https:\/\/www.phii.org\/resources\/view\/6423\/workforce-position-classifications-and-descriptions . Retrieved 15 September 2017 .   \n\n\u2191 Public Health Informatics Institute (April 2014). \"Professional Level Public Health Informatician: Sample Position Description and Sample Career Ladder\" (PDF). https:\/\/www.phii.org\/sites\/www.phii.org\/files\/resource\/pdfs\/Professional%20Sample%20Position%20AND%20Career%20Ladder.pdf . Retrieved 05 October 2017 .   \n\n\u2191 25.0 25.1 25.2 Council on Linkages Between Academia and Public Health Practice (26 June 2014). \"Core Competencies for Public Health Professionals\". pp. 24. http:\/\/www.phf.org\/resourcestools\/pages\/core_public_health_competencies.aspx .   \n\n\u2191 Berndt, D.J.; Hevner, A.R.; Studnicki, J. et al. (2003). \"The Catch data warehouse: Support for community health care decision-making\". Decision Support Systems 35 (3): 367\u201384. doi:10.1016\/S0167-9236(02)00114-8.   \n\n\u2191 Kindig, D.; Stoddart, G. (2003). \"What is population health?\". American Journal of Public Health 93 (3): 380-3. PMC PMC1447747. PMID 12604476. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447747 .   \n\n\u2191 28.0 28.1 Simon, H.A. (1981). The Sciences of the Artificial (2nd ed.). MIT Press. ISBN 9780262191937.   \n\n\u2191 29.0 29.1 Dym, C.L.; Agogino, A.M.; Eris, O. et al. (2013). \"Engineering Design Thinking, Teaching, and Learning\". Journal of Engineering Education 94 (1): 103\u201320. doi:10.1002\/j.2168-9830.2005.tb00832.x.   \n\n\u2191 30.0 30.1 Johansson\u2010Sk\u00f6ldberg, U.; Woodilla, J.; \u00c7etinkaya, M. (2013). \"Design Thinking: Past, Present and Possible Futures\". Creativity and Innovation Management 22 (2): 121\u201346. doi:10.1111\/caim.12023.   \n\n\u2191 31.0 31.1 Friedman, C.P. (2012). \"What informatics is and isn't\". JAMIA 20 (2): 224\u201326. doi:10.1136\/amiajnl-2012-001206.   \n\n\u2191 32.0 32.1 32.2 Pawson, R.; Manzano-Santaella, A. (2012). \"A realist diagnostic workshop\". Evaluation 18 (2): 176\u201391. doi:10.1177\/1356389012440912.   \n\n\u2191 Centers for Disease Control and Prevention (2014). \"The Public Health System & the 10 Essential Public Health Services\". National Public Health Performance Standards. https:\/\/www.cdc.gov\/publichealthgateway\/publichealthservices\/essentialhealthservices.html . Retrieved 08 May 2017 .   \n\n\u2191 Harrison, M.I.; Koppel, R.; Bar-Lev, S. (2007). \"Unintended Consequences of Information Technologies in Health Care\u2014An Interactive Sociotechnical Analysis\". JAMIA 14 (5): 542\u201349. doi:10.1197\/jamia.M2384.   \n\n\u2191 Pentland, B.T.; Feldman, M.S. (2008). \"Designing routines: On the folly of designing artifacts, while hoping for patterns of action\". Information and Organization 18 (4): 235\u2013250. doi:10.1016\/j.infoandorg.2008.08.001.   \n\n\u2191 36.0 36.1 \"Information Systems\". Carnegie Mellon University. 2016. https:\/\/www.cmu.edu\/information-systems\/ . Retrieved 29 January 2018 .   \n\n\u2191 Dixon, B., ed. (2016). Health Information Exchange: Navigating and Managing a Network of Health Information Systems. Academic Press. doi:9780128031353.   \n\n\u2191 38.0 38.1 38.2 38.3 Public Health Informatics Institute (June 2012). \"Redesigning Public Health Surveillance in an eHealth World\". Public Health Informatics Institute. https:\/\/www.phii.org\/resources\/view\/1186\/redesigning-public-health-surveillance-ehealth-world .   \n\n\u2191 Dennis, A.; Wixom, B.H.; Roth, R.M. (2015). Systems Analysis and Design (6th ed.). Wiley. ISBN 9781118897843.   \n\n\u2191 Hammond, W.E.; Jaffe, C.; Cimino, J.J. et al. (2014). \"Standards in Biomedical Informatics\". In Shortliffe, E.H.; Cimino, J.J.. Biomedical Informatics (4th ed.). Springer. ISBN 9781447144748.   \n\n\u2191 Woodall, W.H. (2006). \"The Use of Control Charts in Health-Care and Public-Health Surveillance\". Journal of Quality Technology 38 (2): 89\u2013104. doi:10.1080\/00224065.2006.11918593.   \n\n\u2191 Dover, D.C.; Schopflocher, D.P. (2011). \"Using funnel plots in public health surveillance\". Population Health Metrics 9: 58. doi:10.1186\/1478-7954-9-58.   \n\n\u2191 Sherman, R.L.; Henry, K.A.; Tannenbaum, S.L. et al. (2014). \"Applying spatial analysis tools in public health: An example using SaTScan to detect geographic targets for colorectal cancer screening interventions\". Preventing Chronic Disease 11: E41. doi:10.5888\/pcd11.130264. PMC PMC3965324. PMID 24650619. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965324 .   \n\n\u2191 44.0 44.1 Gadd, C.S.; Williamson, J.J.; Steen, E.B. et al. (2016). \"Eligibility requirements for advanced health informatics certification\". JAMIA 23 (4): 851\u20134. doi:10.1093\/jamia\/ocw090. PMID 27358328.   \n\n\u2191 Baker, E.L.; Brand, W.; Davidson, A. et al. (2016). \"Building the Business Case for Public Health Information Systems\". Journal of Public Health Management and Practice 22 (6): 603\u20136. doi:10.1097\/PHH.0000000000000495. PMID 27682729.   \n\n\u2191 Braunstein, M.L.; Laventure, M.; Baker, E.L. (2018). \"Public Health Informatics Incubators: Accelerating Innovation Through Creative Partnerships Between Informatics Experts and Public Health Agencies\". Journal of Public Health Management and Practice 24 (3): 286\u201388. doi:10.1097\/PHH.0000000000000808. PMID 29595587.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. A few grammar and spelling errors were also corrected. In some cases important information was missing from the references, and that information was added. Reference 2 and 11 in the original are unintentionally duplicated; we've removed a duplicate for this version. The same happened with reference 26 and 38 in the original, and it too was not duplicated for this version. The original had reference 46 listed in the references but no corresponding inline citation for 46; it was intentionally omitted for this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\">https:\/\/www.limswiki.org\/index.php\/Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on educationLIMSwiki journal articles on public health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 14 May 2019, at 19:54.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 225 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","97b51c665da19cc225b05dd3ff621e96_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Developing_workforce_capacity_in_public_health_informatics_Core_competencies_and_curriculum_design skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Developing workforce capacity in public health informatics: Core competencies and curriculum design<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>We describe a master\u2019s level <a href=\"https:\/\/www.limswiki.org\/index.php\/Public_health_informatics\" title=\"Public health informatics\" class=\"wiki-link\" data-key=\"f0372a80f101e9f6fd00490dc1ebcedd\">public health informatics<\/a> (PHI) curriculum to support workforce development. Public health decision-making requires intensive <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_management\" title=\"Information management\" class=\"wiki-link\" data-key=\"f8672d270c0750a858ed940158ca0a73\">information management<\/a> to organize responses to health threats and develop effective health education and promotion. PHI competencies prepare the public health workforce to design and implement these information systems. The objective for a master's and certificate in PHI is to prepare public health informaticians with the competencies to work collaboratively with colleagues in public health and other health professions to design and develop information systems that support population health improvement. The PHI competencies are drawn from computer, information, and organizational sciences. A curriculum is proposed to deliver the competencies, and the results of a pilot PHI program are presented. Since the public health workforce needs to use information technology effectively to improve population health, it is essential for public health academic institutions to develop and implement PHI workforce training programs.\n<\/p><p><b>Keywords<\/b>: public health informatics, public health practice, public health workforce, systems analysis, systems design\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>With the increasing use of electronic data collection and storage, there is an increasing focus on information and knowledge management systems.<sup id=\"rdp-ebb-cite_ref-FreedKnow14_1-0\" class=\"reference\"><a href=\"#cite_note-FreedKnow14-1\">[1]<\/a><\/sup> This results in public health workforce needs for public health informatics (PHI) professionals skilled in designing and implementing these systems. PHI professionals are those who \u201cwork in practice, research, or academia and whose primary work function is to use <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">informatics<\/a> to improve the health of populations.\u201d<sup id=\"rdp-ebb-cite_ref-DHHSComp09_2-0\" class=\"reference\"><a href=\"#cite_note-DHHSComp09-2\">[2]<\/a><\/sup> Recent studies highlight the need for informatics training in the public health workforce.<sup id=\"rdp-ebb-cite_ref-MassoudiPublic16_3-0\" class=\"reference\"><a href=\"#cite_note-MassoudiPublic16-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BeckState17_4-0\" class=\"reference\"><a href=\"#cite_note-BeckState17-4\">[4]<\/a><\/sup> Similarly, the Council on Education for Public Health (CEPH) includes informatics as a foundational competency for accreditation of public health programs.<sup id=\"rdp-ebb-cite_ref-CalhounDevelop08_5-0\" class=\"reference\"><a href=\"#cite_note-CalhounDevelop08-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CEPHAccred16_6-0\" class=\"reference\"><a href=\"#cite_note-CEPHAccred16-6\">[6]<\/a><\/sup> We build on the literature for developing informatics' savvy public health departments<sup id=\"rdp-ebb-cite_ref-LaVentureBuild14_7-0\" class=\"reference\"><a href=\"#cite_note-LaVentureBuild14-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LaVentureBuild15_8-0\" class=\"reference\"><a href=\"#cite_note-LaVentureBuild15-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LaVentureDevelop17_9-0\" class=\"reference\"><a href=\"#cite_note-LaVentureDevelop17-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BrandDevelop18_10-0\" class=\"reference\"><a href=\"#cite_note-BrandDevelop18-10\">[10]<\/a><\/sup> and PHI core competency descriptions<sup id=\"rdp-ebb-cite_ref-DHHSComp09_2-1\" class=\"reference\"><a href=\"#cite_note-DHHSComp09-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MinerApplied11_11-0\" class=\"reference\"><a href=\"#cite_note-MinerApplied11-11\">[11]<\/a><\/sup> to design a curriculum for a master's program in public health (MPH) and a certificate in PHI. The proposed curriculum differs from many existing <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health informatics<\/a> programs by emphasizing the public health informatician's architectural role as a systems analyst linking public health users and information technology specialists by guiding the analysis and design of information systems. The focus is on: (a) problem definition for public health information systems (PHInfSys); (b) analyzing, designing, and implementing effective PHInfSys, such as surveillance, community health assessment, and population health management systems;<sup id=\"rdp-ebb-cite_ref-DHHSComp09_2-2\" class=\"reference\"><a href=\"#cite_note-DHHSComp09-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FondMaking15_12-0\" class=\"reference\"><a href=\"#cite_note-FondMaking15-12\">[12]<\/a><\/sup> (c) encoding, collecting, curating, storing, retrieving, and analyzing data to create <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>; (d) assuring system and data governance, management, integration, confidentiality, and security; (e) creating and managing information technologies; and (f) collaborating in and leading inter-disciplinary and cross-cutting teams.\n<\/p><p>The foundation of public health is information, collected across a variety of sources, with the goal of generating and disseminating new knowledge and instituting actions to improve the public\u2019s health. The need for public health education programs to be informed by current public health needs and employment opportunities has been long noted<sup id=\"rdp-ebb-cite_ref-IoMTheFuture88_13-0\" class=\"reference\"><a href=\"#cite_note-IoMTheFuture88-13\">[13]<\/a><\/sup>, as has the importance of establishing academic-practice links with public health agencies and training students in practice situations. In 2001, the need for PHI training was noted in an <a href=\"https:\/\/www.limswiki.org\/index.php\/American_Medical_Informatics_Association\" title=\"American Medical Informatics Association\" class=\"wiki-link\" data-key=\"4d1e3ae25f7b60426902970e99eb18e1\">American Medical Informatics Association<\/a> (AMIA) conference on developing a PHI agenda.<sup id=\"rdp-ebb-cite_ref-YasnoffANat01_14-0\" class=\"reference\"><a href=\"#cite_note-YasnoffANat01-14\">[14]<\/a><\/sup> In 2012, AMIA revisited and updated these recommendations.<sup id=\"rdp-ebb-cite_ref-MassoudiAnInfo12_15-0\" class=\"reference\"><a href=\"#cite_note-MassoudiAnInfo12-15\">[15]<\/a><\/sup> Both reports covered technical topics, such as information architecture and standards; governance and policy, including confidentiality and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information_privacy\" title=\"Information privacy\" class=\"wiki-link\" data-key=\"185f6d9f874e48914b5789317408f782\">privacy<\/a>; and workforce training. These needs are addressed by the PHI training program we describe.\n<\/p><p>The 2012 AMIA report noted that the \u201c[t]he public health value chain is composed of business processes and use cases that describe the flow of data and information. Business processes describe how data, information, and knowledge are used by creating a framework that relates work activities to domains of public health function.\u201d<sup id=\"rdp-ebb-cite_ref-MassoudiAnInfo12_15-1\" class=\"reference\"><a href=\"#cite_note-MassoudiAnInfo12-15\">[15]<\/a><\/sup> PHI is a foundation for modernizing public health value chains, including public health department business processes<sup id=\"rdp-ebb-cite_ref-PHIITaking08_16-0\" class=\"reference\"><a href=\"#cite_note-PHIITaking08-16\">[16]<\/a><\/sup>, public health surveillance<sup id=\"rdp-ebb-cite_ref-McNabbTrans16_17-0\" class=\"reference\"><a href=\"#cite_note-McNabbTrans16-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McNabbInform17_18-0\" class=\"reference\"><a href=\"#cite_note-McNabbInform17-18\">[18]<\/a><\/sup>, public health emergency response, population health science<sup id=\"rdp-ebb-cite_ref-BachrachShaping17_19-0\" class=\"reference\"><a href=\"#cite_note-BachrachShaping17-19\">[19]<\/a><\/sup>, population health management<sup id=\"rdp-ebb-cite_ref-KindigUnder07_20-0\" class=\"reference\"><a href=\"#cite_note-KindigUnder07-20\">[20]<\/a><\/sup>, learning health systems<sup id=\"rdp-ebb-cite_ref-FriedmanToward15_21-0\" class=\"reference\"><a href=\"#cite_note-FriedmanToward15-21\">[21]<\/a><\/sup>, and public health research. PHI is an essential specialization that bridges the digital gap public health agencies face given growing expectations of service and preparedness. PHI integrates knowledge and skills from the information sciences (computer, information, organizational, and systems sciences) with public health expertise and \u201cincludes the conceptualization, design, development, deployment, refinement, maintenance, and evaluation of communication, surveillance, and information systems relevant to public health.\u201d<sup id=\"rdp-ebb-cite_ref-YasnoffPublic00_22-0\" class=\"reference\"><a href=\"#cite_note-YasnoffPublic00-22\">[22]<\/a><\/sup>\n<\/p><p>While PHI shares some core skills with other health informatics domains, such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> or clinical and nursing informatics, it is distinct in necessitating integration of core informatics skills with public health systems, such as surveillance systems, community health assessment, disease\/condition registries, and prevention programs in an overall framework of population health science. Public health informaticians (PHInf) differ from information technologists (ITs) because \u201c[t]he focus of IT is to implement and operate information systems (hardware and software) that meet programmatic needs. In contrast, PHInf have a strategic and systems view of how information systems and technology can impact public health, such as how information systems can support public health decision-making. Unlike IT specialists, PHInf work within the larger context of how information systems function within the political, cultural, economic, and social environment and evaluate their impact within the broad sphere of public health. Thus, PHInf are in the unique position to understand how information systems can improve the practice and science of public health while contributing to the evidence-based practice of public health informatics.\u201d<sup id=\"rdp-ebb-cite_ref-DHHSComp09_2-3\" class=\"reference\"><a href=\"#cite_note-DHHSComp09-2\">[2]<\/a><\/sup>\n<\/p><p>Our paper focuses on training PHInf through MPH and certificate programs to provide them a foundation for a PHI professional career. The professional role is one of four related PHI roles identified by the PHI Institute, the Association of State and Territorial Health Officials, and the National Association of County and City Health Officials (NACCHO): executive; manager; professional; and clinical.<sup id=\"rdp-ebb-cite_ref-PHIIWorkforce14_23-0\" class=\"reference\"><a href=\"#cite_note-PHIIWorkforce14-23\">[23]<\/a><\/sup> The curriculum design highlighted here focuses on the professional role because it is a frontline position that is the foundation for a career path through management to executive leadership.<sup id=\"rdp-ebb-cite_ref-PHIIProf14_24-0\" class=\"reference\"><a href=\"#cite_note-PHIIProf14-24\">[24]<\/a><\/sup>\n<\/p><p>Our paper describes program design objectives and assumptions, design challenges, and core competencies. This overview provides context and prioritization for the program design.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"PHI_program_design_objectives_and_assumptions\">PHI program design objectives and assumptions<\/span><\/h2>\n<p>The objective of PHI training is to educate PHInf with public health and PHI knowledge and skills that enable them to design and implement effective PHInfSys and have a successful PHI career. The design assumes that PHInf require competencies in (a) public health core competencies<sup id=\"rdp-ebb-cite_ref-PHFCore14_25-0\" class=\"reference\"><a href=\"#cite_note-PHFCore14-25\">[25]<\/a><\/sup>, functions, and systems (e.g., disease and environmental surveillance<sup id=\"rdp-ebb-cite_ref-McNabbInform17_18-1\" class=\"reference\"><a href=\"#cite_note-McNabbInform17-18\">[18]<\/a><\/sup>, population health management<sup id=\"rdp-ebb-cite_ref-KindigUnder07_20-1\" class=\"reference\"><a href=\"#cite_note-KindigUnder07-20\">[20]<\/a><\/sup>, and community health assessment<sup id=\"rdp-ebb-cite_ref-KindigUnder07_20-2\" class=\"reference\"><a href=\"#cite_note-KindigUnder07-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BerndtTheCatch03_26-0\" class=\"reference\"><a href=\"#cite_note-BerndtTheCatch03-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KindigWhatIs03_27-0\" class=\"reference\"><a href=\"#cite_note-KindigWhatIs03-27\">[27]<\/a><\/sup>); (b) design thinking and systems analysis<sup id=\"rdp-ebb-cite_ref-SimonTheSci81_28-0\" class=\"reference\"><a href=\"#cite_note-SimonTheSci81-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DymEng13_29-0\" class=\"reference\"><a href=\"#cite_note-DymEng13-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Johansson.E2.80.90Sk.C3.B6ldbergDesign13_30-0\" class=\"reference\"><a href=\"#cite_note-Johansson.E2.80.90Sk.C3.B6ldbergDesign13-30\">[30]<\/a><\/sup>; (c) computer, information, organizational, and systems sciences<sup id=\"rdp-ebb-cite_ref-YasnoffPublic00_22-1\" class=\"reference\"><a href=\"#cite_note-YasnoffPublic00-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FriedmanWhatInfo12_31-0\" class=\"reference\"><a href=\"#cite_note-FriedmanWhatInfo12-31\">[31]<\/a><\/sup>; (d) evaluating information systems effectiveness<sup id=\"rdp-ebb-cite_ref-PawsonARealist12_32-0\" class=\"reference\"><a href=\"#cite_note-PawsonARealist12-32\">[32]<\/a><\/sup>; and (e) teamwork and project management.\n<\/p><p>The core public health functions are assurance, assessment, and advocacy, which are used to maximize population health.<sup id=\"rdp-ebb-cite_ref-CDCPublic17_33-0\" class=\"reference\"><a href=\"#cite_note-CDCPublic17-33\">[33]<\/a><\/sup> The PHI program design focuses on understanding how information systems achieve these functions rather than having a sub-goal focus on technological, computer science, or data mining aspects of PHI. Because PHI focuses on designing PHInfSys it has a foundation in design thinking.<sup id=\"rdp-ebb-cite_ref-SimonTheSci81_28-1\" class=\"reference\"><a href=\"#cite_note-SimonTheSci81-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DymEng13_29-1\" class=\"reference\"><a href=\"#cite_note-DymEng13-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Johansson.E2.80.90Sk.C3.B6ldbergDesign13_30-1\" class=\"reference\"><a href=\"#cite_note-Johansson.E2.80.90Sk.C3.B6ldbergDesign13-30\">[30]<\/a><\/sup> Design thinking focuses on identifying problems as gaps in population health that information systems can help reduce by determining the root causes of those gaps and designing and implementing information systems to reduce population health gaps. Systems analysts are the architects who bridge the gap between users and technologies by understanding and translating user needs to system specifications and collaborating with technologists to implement those systems. The goal of systems analysis is to minimize the frequent disconnects between information system development and the usefulness of the resulting information systems<sup id=\"rdp-ebb-cite_ref-HarrisonUnintended07_34-0\" class=\"reference\"><a href=\"#cite_note-HarrisonUnintended07-34\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PentlandDesign08_35-0\" class=\"reference\"><a href=\"#cite_note-PentlandDesign08-35\">[35]<\/a><\/sup> by avoiding simply automating existing inefficient information pathways.\n<\/p><p>To address these disconnects, effective systems analysis requires that PHInf are \u201cwell grounded in the fundamentals of organization theory, decision-making, teamwork and leadership, and research methods as well as current and emerging information systems technologies.\u201d<sup id=\"rdp-ebb-cite_ref-CMUInfo18_36-0\" class=\"reference\"><a href=\"#cite_note-CMUInfo18-36\">[36]<\/a><\/sup> PHInf also need the ability to design, implement, and evaluate information systems in multiple contexts, using approaches such as realistic evaluation and context-mechanism-outcome configurations (CMOc).<sup id=\"rdp-ebb-cite_ref-PawsonARealist12_32-1\" class=\"reference\"><a href=\"#cite_note-PawsonARealist12-32\">[32]<\/a><\/sup>\n<\/p><p>Finally, PHInf need strong teamwork and project management skills because \u201cPHI is cross-cutting. \u2026 Informaticians see the \u2018big picture\u2019 and \u2018connects the dots\u2019 across all of the other fields related to public health.\u201d<sup id=\"rdp-ebb-cite_ref-FondMaking15_12-1\" class=\"reference\"><a href=\"#cite_note-FondMaking15-12\">[12]<\/a><\/sup> Because information systems span diverse domains, the public health informatician requires the knowledge and skills to collaborate with diverse clients, lead systems analysis and development, and coordinate the diverse specialists implementing information systems, data use and privacy agreements, data and vocabulary standards, evaluation, training, and more.\n<\/p><p>In sum, PHInf require competencies in (a) public health core concepts and systems; (b) systems thinking and systems analysis; (c) computer, information, organizational, and systems sciences; (d) evaluating information systems effectiveness; and (e) the ability to work in and lead teams. These competencies provide the foundation for successfully progressing from being a PHI professional to PHI leadership positions.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"PHI_program_design_challenges\">PHI program design challenges<\/span><\/h2>\n<p>Public health informatics requires many different competencies. This can result in a laundry list of competencies that are not feasible to implement in a credit-constrained curriculum. Implementing a constrained educational program requires competency prioritization and trade-offs. There are three major challenges.\n<\/p><p><b>1. Content<\/b>: PHI requires competencies in distinct domains\u2014public health, health informatics, computer, information, and organizational sciences.<sup id=\"rdp-ebb-cite_ref-DHHSComp09_2-4\" class=\"reference\"><a href=\"#cite_note-DHHSComp09-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MinerApplied11_11-1\" class=\"reference\"><a href=\"#cite_note-MinerApplied11-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FriedmanWhatInfo12_31-1\" class=\"reference\"><a href=\"#cite_note-FriedmanWhatInfo12-31\">[31]<\/a><\/sup> These include required core competencies for public health practice.<sup id=\"rdp-ebb-cite_ref-PHFCore14_25-1\" class=\"reference\"><a href=\"#cite_note-PHFCore14-25\">[25]<\/a><\/sup> Health informatics and computer science focus on technical competencies. Information, systems, and organizational science competencies support solving population health problems because they provide systems analysis and design competencies needed to: (1) understand and translate user needs into system requirements and (2) create designs that can be implemented by ITs. Computer science competencies, such as modular decomposition, data structures, algorithms, programming languages, database theories, and system architectures are necessary to provide PHInf the knowledge and skills to communicate and collaborate effectively with ITs implementing PHInfSys.\n<\/p><p><b>2. Curricular independence vs. integrations<\/b>: Siloed, independent courses maximize covering competency breadth, but they also rely on students to integrate across competencies. Courses integrating competencies develop the skills to apply informatics in public health. Examples of integrating competencies include using population health management and surveillance examples in technical courses, such as systems analysis, database, or health information exchange.\n<\/p><p><b>3. Declarative vs. procedural competencies<\/b>: Declarative competencies are familiarity with analysis tools, such as requirements analysis, business process modeling, use cases, data flow diagrams, and entity relationship diagrams. Procedural competencies are being able to develop requirements, business process models, use cases, data flow diagrams, and entity relationship diagrams. Training in procedural competencies requires experiential practicums in which students apply declarative competencies. Increasing the emphasis on procedural competencies limits the breadth of declarative competencies that can be covered. The PHI design focuses on developing strong procedural competencies in system analysis because it is the core competency used to design and implement effective PHInf.\n<\/p><p>In sum, design challenges include decisions about content, integration (independent\/integrated), and competency type (declarative\/procedural) tradeoffs.\n<\/p><p>The proposed PHI design focuses on public health core competencies and organization\/systems\/information sciences related to analyzing, designing, implementing, and disseminating effective information systems in public health. Integrating competencies across courses maximizes the ability to apply skills in public health settings. For declarative and procedural learning, experiential practicums assure the ability to apply declarative knowledge. Elective credits support students in developing deeper skills in areas they are interested in, such as data management, analytics\/data mining, GIS, visualization\/communication, health information exchange<sup id=\"rdp-ebb-cite_ref-DixonHealth16_37-0\" class=\"reference\"><a href=\"#cite_note-DixonHealth16-37\">[37]<\/a><\/sup>, leading community health information exchange, or surveillance systems.<sup id=\"rdp-ebb-cite_ref-McNabbInform17_18-2\" class=\"reference\"><a href=\"#cite_note-McNabbInform17-18\">[18]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"PHI_competencies\">PHI competencies<\/span><\/h2>\n<p>A PHI competency is a measurable and demonstrable knowledge related to the role \u201cof developing innovative applications of technology and systems that address public health priorities by analyzing how information is organized and used and evaluating how this work contributes to the scientific field.\u201d<sup id=\"rdp-ebb-cite_ref-DHHSComp09_2-5\" class=\"reference\"><a href=\"#cite_note-DHHSComp09-2\">[2]<\/a><\/sup> The master\u2019s level competencies for PHI professionals build on earlier work .<sup id=\"rdp-ebb-cite_ref-DHHSComp09_2-6\" class=\"reference\"><a href=\"#cite_note-DHHSComp09-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MinerApplied11_11-2\" class=\"reference\"><a href=\"#cite_note-MinerApplied11-11\">[11]<\/a><\/sup> Not all competencies listed in earlier work are included for three reasons. First, the selected competencies need to fit into a typical MPH curriculum. While there are many desirable and important competencies, there is limited curriculum time. Second, some competencies reflect work in a practice setting that are not directly measurable (e.g., collaborating with others in program development, offering insights, contributing to decision-making, leading knowledge management). Third, some competencies are more managerial\/executive competencies than professional competencies.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Core_competencies\">Core competencies<\/span><\/h3>\n<p>Core competencies include both public health and PHI. Public health core competencies<sup id=\"rdp-ebb-cite_ref-CEPHAccred16_6-1\" class=\"reference\"><a href=\"#cite_note-CEPHAccred16-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PHFCore14_25-2\" class=\"reference\"><a href=\"#cite_note-PHFCore14-25\">[25]<\/a><\/sup> are addressed through the required MPH curriculum courses. This material is extended by integrating competencies related to public health systems, surveillance systems, population health management, community health assessment, eHealth, program monitoring, and evaluation in the PHI courses. For example, using electronic disease surveillance, population health management, community health assessment, and other public health systems as examples in the systems analysis courses educates students in the use of systems analysis to address common public health systems.<sup id=\"rdp-ebb-cite_ref-PHIITaking08_16-1\" class=\"reference\"><a href=\"#cite_note-PHIITaking08-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PHIIRedes12_38-0\" class=\"reference\"><a href=\"#cite_note-PHIIRedes12-38\">[38]<\/a><\/sup>\n<\/p><p>Public health informatics competencies focus on systems analysis and data modeling. These include computer science and health informatics competencies related to nomenclature, standards, platforms, information architecture, interoperability, and systems analysis and data modeling, which require integration of material from organization\/information systems\/systems sciences, public health, and computer sciences. This integration builds the overall competency to design PHInfSys to solve problems in the public health domain. These competencies are noted as central to PHI. For example, \u201cRedesigning Public Health Surveillance in an eHealth World\u201d demonstrates the importance of systems analysis when it points out that \u201c[d]efining requirements is a critical step in developing or acquiring an information system that will effectively support the work of the organization.\u201d<sup id=\"rdp-ebb-cite_ref-PHIIRedes12_38-1\" class=\"reference\"><a href=\"#cite_note-PHIIRedes12-38\">[38]<\/a><\/sup> Similarly, system selection depends \u201con the clarity and applicability of the use cases you define, and on determining prior to the demonstration how\u2014based on what criteria\u2014the use-case demonstrations will be judged\u201d<sup id=\"rdp-ebb-cite_ref-PHIIRedes12_38-2\" class=\"reference\"><a href=\"#cite_note-PHIIRedes12-38\">[38]<\/a><\/sup> as well as business process definitions.<sup id=\"rdp-ebb-cite_ref-PHIIRedes12_38-3\" class=\"reference\"><a href=\"#cite_note-PHIIRedes12-38\">[38]<\/a><\/sup> The description of an informatics savvy health department states that \u201c[i]nformation systems managers and staff require competency in creating formal system requirements.\u201d<sup id=\"rdp-ebb-cite_ref-LaVentureBuild14_7-1\" class=\"reference\"><a href=\"#cite_note-LaVentureBuild14-7\">[7]<\/a><\/sup> Requirements, use cases, and business process definitions are produced by systems analysis and require expertise in the system development life cycles (SDLC), national standards, and public policy related to informatics (e.g., meaningful use).<sup id=\"rdp-ebb-cite_ref-LaVentureBuild15_8-1\" class=\"reference\"><a href=\"#cite_note-LaVentureBuild15-8\">[8]<\/a><\/sup> Complementary competencies are those that support systems analysis and data modeling<sup id=\"rdp-ebb-cite_ref-DennisSystems15_39-0\" class=\"reference\"><a href=\"#cite_note-DennisSystems15-39\">[39]<\/a><\/sup>, such as standards (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">HL7<\/a>), nomenclature (e.g., ICD, CPT)<sup id=\"rdp-ebb-cite_ref-HammondStand14_40-0\" class=\"reference\"><a href=\"#cite_note-HammondStand14-40\">[40]<\/a><\/sup>, data flow diagrams, relational database theory, and query languages that inform system analysis and are used during implementation.\n<\/p><p>As an example, Table 1 illustrates systems analysis competencies for electronic disease surveillance. The goal of the system is to maximize population health by minimizing transmission of communicable diseases, which is done \u201cthrough early identification, treatment, and resolution of health conditions.\u201d<sup id=\"rdp-ebb-cite_ref-PHIITaking08_16-2\" class=\"reference\"><a href=\"#cite_note-PHIITaking08-16\">[16]<\/a><\/sup> Systems analysis requires integrating competency in systems analysis with expertise in the system\u2019s public health domain (e.g., electronic disease surveillance). The first analysis phase defines the problem, its decision-makers, context, and stakeholders. The second phase does a root cause analysis of gaps in population health related to communicable disease surveillance or system productivity and describes functional requirements, what tasks the communicable disease surveillance system supports, and nonfunctional requirements, criteria that the system has to meet. The third phase describes the logical model, the use cases corresponding to each functional requirement, business processes, data flows, and data model. Design and implementation is the final phase and builds on the analysis in the earlier phases to design and implement the information system that supports communicable disease surveillance staff in their activities.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Wholey_FrontPubHealth2019_6.jpg\" class=\"image wiki-link\" data-key=\"94c5463f96ab6298f2b6f9634be76c50\"><img alt=\"Tab1 Wholey FrontPubHealth2019 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/3e\/Tab1_Wholey_FrontPubHealth2019_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 1.<\/b> ystems analysis example for electronic disease surveillance<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Declarative_and_procedural_competencies\">Declarative and procedural competencies<\/span><\/h3>\n<p>Tables 2 and 3 show the MPH PHI declarative and procedural competencies, which have a strong focus on those competencies also used in systems analysis and data modeling. These competencies reflect the informaticians\u2019 architect role, which focuses on the problem definition and solving skills associated with defining problems, identifying requirements, building logical models, describing core features of information systems (use cases, data flows, data models, business processes), designing implementation alternatives, overseeing implementation, and facilitating user centered design and choice.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab2_Wholey_FrontPubHealth2019_6.jpg\" class=\"image wiki-link\" data-key=\"06419043e236bf23ee9aac576406f447\"><img alt=\"Tab2 Wholey FrontPubHealth2019 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/c1\/Tab2_Wholey_FrontPubHealth2019_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 2.<\/b> Public health informatics (PHI) declarative competencies<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab3_Wholey_FrontPubHealth2019_6.jpg\" class=\"image wiki-link\" data-key=\"02643391cb5d92dae5db646f0f048e74\"><img alt=\"Tab3 Wholey FrontPubHealth2019 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/a\/a3\/Tab3_Wholey_FrontPubHealth2019_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 3.<\/b> Public health informatics (PHI) procedural competencies<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The declarative and procedural competencies also include a significant focus on research analysis and evaluation methods because PHInfSys are interventions designed to improve population health by improving quality, reducing costs, or both. Given the intervention aspect, a PHInf should have the competencies to be able to evaluate whether the public health information system accomplished its goals and how the information system functions in different contexts (realistic evaluation<sup id=\"rdp-ebb-cite_ref-PawsonARealist12_32-2\" class=\"reference\"><a href=\"#cite_note-PawsonARealist12-32\">[32]<\/a><\/sup>). Effective evaluation requires enough expertise in public health systems, such as disease surveillance<sup id=\"rdp-ebb-cite_ref-McNabbTrans16_17-1\" class=\"reference\"><a href=\"#cite_note-McNabbTrans16-17\">[17]<\/a><\/sup>, quality improvement<sup id=\"rdp-ebb-cite_ref-WoodallTheUse18_41-0\" class=\"reference\"><a href=\"#cite_note-WoodallTheUse18-41\">[41]<\/a><\/sup>, public reporting, such as control charts, funnel plots (league tables)<sup id=\"rdp-ebb-cite_ref-DoverUsing11_42-0\" class=\"reference\"><a href=\"#cite_note-DoverUsing11-42\">[42]<\/a><\/sup>, and spatial outlier detection<sup id=\"rdp-ebb-cite_ref-ShermanApplying14_43-0\" class=\"reference\"><a href=\"#cite_note-ShermanApplying14-43\">[43]<\/a><\/sup>, to be able to assess how well they are implemented as well as their effectiveness. Competencies in assessing measurement validity and reliability are central to measuring population health.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Electives\">Electives<\/span><\/h3>\n<p>Electives allow students to pursue specializations in related public health and PHI competencies such as:\n<\/p>\n<ul><li> PHI management: project management, health information management, public health leadership, and leading change for regional and community information systems<\/li>\n<li> PHI tools: ePublic health, using the web and mobile apps for health promotion, GIS for public health, health care operations research and analytics, public health decision support systems, eHealth, health information exchange, and surveillance systems in depth<\/li>\n<li> PHI evaluation: decision analysis for policy analysis, cost-effective analysis, program evaluation, usability engineering, and realistic evaluation<\/li>\n<li> Environmental health systems: Complex systems modeling for population health, infectious diseases, environmental health risk assessment, surveillance of foodborne diseases and food safety hazards<\/li>\n<li> Analytics and data mining: Advanced statistical computing, statistical learning and data mining, survival analysis, statistics for surveillance and quality improvement<\/li>\n<li> Computer science: Algorithms and data structures, principles and practice of database systems, data marts and warehouse management, NoSQL and alternatives to relational data models<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Organizing_competencies_into_courses\">Organizing competencies into courses<\/span><\/h2>\n<p>The design addresses siloing vs. integration in two ways (Table 4). First, given that the application of informatics competencies is conditional on public health context, informatics and public health competencies are strongly integrated. Students need the expertise to apply the informatics concepts in the domain they will practice. Second, a practice-based practicum in the introductory systems analysis course and a practice-based capstone systems analysis course integrate practice and academics.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab4_Wholey_FrontPubHealth2019_6.jpg\" class=\"image wiki-link\" data-key=\"db1c6d6935cbde0e640edea0b5800a17\"><img alt=\"Tab4 Wholey FrontPubHealth2019 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/e\/ed\/Tab4_Wholey_FrontPubHealth2019_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 4.<\/b> Public health informatics (PHI) curriculum and competencies<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As an example, informatics competencies in systems analysis courses (introduction and capstone) are illustrated using PHInfSys. The relational database course (managing electronic health information) uses population health measurement for SQL assignments. Data and information for population health management, which focuses on substantive issues related to population health management uses relational database theory and SQL for assignments in population health management and frames the evaluation of public health interventions using realistic evaluation. In sum, the design emphasizes the integration of systems analysis, database, and evaluation competencies with public health systems and population health management. The objective is to train students in informatics competencies, public health competencies, and the use of informatics skills in public health settings. A separate course addresses competencies that are general and applicable to many areas, such as standards, nomenclature, health information exchange layers, and technical issues.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implications_for_policy_and_practice\">Implications for policy and practice<\/span><\/h2>\n<p>This paper proposes a master\u2019s level public health curriculum to support workforce development in PHI competencies. The competencies are similar to those described for public health core competencies<sup id=\"rdp-ebb-cite_ref-CEPHAccred16_6-2\" class=\"reference\"><a href=\"#cite_note-CEPHAccred16-6\">[6]<\/a><\/sup> and in the Advanced Health Informatics Certification<sup id=\"rdp-ebb-cite_ref-GaddElig16_44-0\" class=\"reference\"><a href=\"#cite_note-GaddElig16-44\">[44]<\/a><\/sup> with the goal of preparing PHInf who \u201c[p]ractice health informatics with an operational focus on information and knowledge problems that directly impacts the practice of health care, public health, and personal health.\u201d<sup id=\"rdp-ebb-cite_ref-GaddElig16_44-1\" class=\"reference\"><a href=\"#cite_note-GaddElig16-44\">[44]<\/a><\/sup> Edward Baker and his colleagues recently made a strong business case for PHI<sup id=\"rdp-ebb-cite_ref-BakerBuild16_45-0\" class=\"reference\"><a href=\"#cite_note-BakerBuild16-45\">[45]<\/a><\/sup> arguing that public health is an information intensive business and that health information systems are transforming the way that information management speeds responses to health threats, promote health, and supports public health decision-making. The foundation of these systems are the essential public health services, a focus on population health, and core informatics competencies, such as system analysis, database theory, and health informatics. The implementation of effective PHInfSys requires individuals with the competencies described in this paper.\n<\/p><p>This program design was based on Carnegie Mellon\u2019s highly successful information systems program which focus on systems analysis and information technology skills to \u201cdesign and implement effective solutions to meet organizational and management needs for information and decision support.\u201d<sup id=\"rdp-ebb-cite_ref-CMUInfo18_36-1\" class=\"reference\"><a href=\"#cite_note-CMUInfo18-36\">[36]<\/a><\/sup> The program was implemented in the School of Public Health at the University of Minnesota in the Fall of 2013. The first year courses have been offered, evaluated, and refined five times, and the second year courses have been offered and refined four times. Thus far, 19 students have earned PHI certificates signifying completion of PHI core courses; 15 will have completed the MPH in PHI by May 2018. The certificate attracted diverse students, including students from health informatics, pharmacy, environmental health, and epidemiology. Applicants have expressed interest in PHI because they feel that data and information is critical to public health research and practice, and they feel that learning informatics and analytics in a public health context will help them to identify opportunities for improving population health. Students in the PHI program have had internships supporting quality improvement and population health management with a safety net clinic network, database administration in health care systems and professional organizations, and supporting standards implementation in electronic disease surveillance systems in state health departments. Graduates of the PHI program have been successful in securing positions across private and public health sectors, and some have pursued further degrees in health informatics. Seven of the 10 MPH-PHI graduates are employed in positions related to PHI. \n<\/p><p>Table 5 provides a sample of their job descriptions. The job responsibilities span the spectrum from leading projects in PHI and population health management to offering technical support in data management and research coordination. Future program evaluation should measure the degree to which graduates are engaged in public health related activities, are engaged in PHI related activities, initial placement, and career growth.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab5_Wholey_FrontPubHealth2019_6.jpg\" class=\"image wiki-link\" data-key=\"dc7932ebcee0138a3f6c12b7a98b08da\"><img alt=\"Tab5 Wholey FrontPubHealth2019 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/be\/Tab5_Wholey_FrontPubHealth2019_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 5.<\/b> Recent master\u2019s in public health public health informatics graduates position examples<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>There are development and sustainability issues that occur because PHI is a new field. First, program development requires collaborating with faculty and stakeholders to communicate what PHI is and a vision of its value.<sup id=\"rdp-ebb-cite_ref-FondMaking15_12-2\" class=\"reference\"><a href=\"#cite_note-FondMaking15-12\">[12]<\/a><\/sup> Second, marketing and social media programs that convey PHI\u2019s contribution to new and existing public health students need to be developed. Third, partnerships with stakeholders for course based projects and integrating practice with academics have to be developed. Fourth, PHI program development requires funding commitment from schools of public health until enrollment reaches a break-even point. Finally, PHI faculty who have competency in both public health and informatics need to be developed. Until PHI as a field is more institutionalized, a potential tactic for doing this is recruiting post-doctoral staff from fields within public health (e.g., epidemiology, environmental health) and have them develop PHI competencies while developing PHI related research and\/or recruiting post-doctoral staff from related informatics disciplines<sup id=\"rdp-ebb-cite_ref-BraunsteinPublic18_46-0\" class=\"reference\"><a href=\"#cite_note-BraunsteinPublic18-46\">[46]<\/a><\/sup> and have them develop public health competencies while developing PHI related research and teaching. While these barriers appear substantial, PHI should be sustainable because of the growth in informatics related disciplines and because of skilled PHInf being in high demand as valuable team members in both research and practice.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The effective and efficient collection of quality public health data and information is intensive. It is the foundation of public health functions, such as surveillance, community health assessment, evaluation, and research. PHI is a profession that focuses on the design of governance, analysis, development, and management of information systems that acquire and integrate data that is used in public health functions, such as disease and environmental surveillance, program evaluation, policy analysis, and research. PHI provides the foundation for organizing and collecting data that is used in performing public health functions. PHI is an emerging, distinct, and needed profession in public health that uses knowledge and skills drawn from computer, information, and organizational sciences and public health to develop critical information systems that provide the data and information that other public health professionals use in their work. PHInf provide the expertise in public health, information, and information technology to take advantage of technology advances to improve the PHInf.\n<\/p><p>This paper proposes a PHI curriculum for schools of public health that can be implemented through MPH and certificate programs. The curriculum provides core competencies in public health and in computer, information, and organizational sciences that are needed by PHInf. The proposed curriculum addresses the challenges of integrating PHI in public health functions, developing declarative knowledge and procedural skills in PHI through a balance of lecture and practicum courses, and allowing the pursuit of specializations. A key focus of the proposed curriculum is its user centered design through systems analysis and systems thinking. This provides PHInf the competencies necessary to lead, design, and implement PHInfSys that take advantage of technology advances.\n<\/p><p>The proposed program is a starting point. Given the rapidity of change in public health practice, public health research, informatics, and information technology, the curriculum will need to evolve to meet changing needs. This evolution should be guided by partnerships with stakeholders in practice, other public health disciplines, and public health researchers to assure that PHInf are effectively designed and developed to meet the needs of those working to improve population health. This development can be guided by the continued evaluation of PHInf to assure they meet user needs and the continued evaluation of PHI programs to assure they are training the workforce that can lead developments in PHInfSys.\n<\/p><p>There is a clear need to develop a PHI workforce. It is imperative that public health institutions, both practice and academic, take action and support the development of a strong PHI workforce. This can be achieved through on-the-job training, fellowships, internships, and academic training. The educational program we propose addresses schools of public health, and provides guidelines for implementing PHI training programs. However, support is also needed from public health practitioners to ensure that PHI training is prioritized to meet the evolving demands of the practice environment. With these tools we can continue to develop an effective and relevant workforce that can navigate the changing trends of public health to improve population health.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We thank Pat Larkey, Sara Kiesler, Serge Taylor, Bill Hefley, Kathleen Carley, and the Information Systems program at Carnegie Mellon University for foundational ideas. We thank Julie Jacko, Daniel Chan, and Matteo Convertino for their ideas and help in implementing the PHI program at Minnesota. We thank Rochelle Martin, Raul Noriega, Megan Pruente, Cole Schluchter, Nicholas Solberg, Herong Song, and Patrick Williams for information on their current job responsibilities and comments about the PHI program. We thank MPH and certificate PHI students for working with us to develop the program. We thank Ira Moscovice the Health Policy & Management Department, and the School of Public Health at the University of Minnesota for supporting PHI program development.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>All authors contributed to the conceptualization of this paper. DW, RK, and SR led the curriculum conceptualization and development. ML, SR, and DW led the focus on public health informatics (e.g., the informatics savvy health department, immunization systems), CH, SR, and CK led the focus on applications of PHI in electronic disease surveillance system. DW and ML did the first draft. RK, SR, CH, and CK edited and provided components (e.g., description of electronic disease support systems, contents of systems analysis competencies). All read the final version, edited it, and approved it for submission.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-FreedKnow14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FreedKnow14_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Freed, J.; Kyffin, R.; Rashbass, J. et al. (June 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gov.uk\/government\/publications\/knowledge-strategy-harnessing-the-power-of-information-to-improve-the-publics-health\" target=\"_blank\">\"Knowledge strategy: Harnessing the power of information to improve the public's health\"<\/a>. Public Health England<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.gov.uk\/government\/publications\/knowledge-strategy-harnessing-the-power-of-information-to-improve-the-publics-health\" target=\"_blank\">https:\/\/www.gov.uk\/government\/publications\/knowledge-strategy-harnessing-the-power-of-information-to-improve-the-publics-health<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Knowledge+strategy%3A+Harnessing+the+power+of+information+to+improve+the+public%27s+health&rft.atitle=&rft.aulast=Freed%2C+J.%3B+Kyffin%2C+R.%3B+Rashbass%2C+J.+et+al.&rft.au=Freed%2C+J.%3B+Kyffin%2C+R.%3B+Rashbass%2C+J.+et+al.&rft.date=June+2014&rft.pub=Public+Health+England&rft_id=https%3A%2F%2Fwww.gov.uk%2Fgovernment%2Fpublications%2Fknowledge-strategy-harnessing-the-power-of-information-to-improve-the-publics-health&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DHHSComp09-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DHHSComp09_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-DHHSComp09_2-1\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-DHHSComp09_2-2\">2.2<\/a><\/sup> <sup><a href=\"#cite_ref-DHHSComp09_2-3\">2.3<\/a><\/sup> <sup><a href=\"#cite_ref-DHHSComp09_2-4\">2.4<\/a><\/sup> <sup><a href=\"#cite_ref-DHHSComp09_2-5\">2.5<\/a><\/sup> <sup><a href=\"#cite_ref-DHHSComp09_2-6\">2.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Centers for Disease Control and Prevention, University of Washington's Center for Public Health Informatics (September 2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20170613082955\/https:\/\/www.cdc.gov\/informaticscompetencies\/pdfs\/phi-competencies.pdf\" target=\"_blank\">\"Competencies for Public Health Informaticians\"<\/a> (PDF). US Department of Health and Human Services, Centers for Disease Control and Prevention. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cdc.gov\/informaticscompetencies\/pdfs\/phi-competencies.pdf\" target=\"_blank\">the original<\/a> on 13 June 2017<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20170613082955\/https:\/\/www.cdc.gov\/informaticscompetencies\/pdfs\/phi-competencies.pdf\" target=\"_blank\">https:\/\/web.archive.org\/web\/20170613082955\/https:\/\/www.cdc.gov\/informaticscompetencies\/pdfs\/phi-competencies.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Competencies+for+Public+Health+Informaticians&rft.atitle=&rft.aulast=Centers+for+Disease+Control+and+Prevention%2C+University+of+Washington%27s+Center+for+Public+Health+Informatics&rft.au=Centers+for+Disease+Control+and+Prevention%2C+University+of+Washington%27s+Center+for+Public+Health+Informatics&rft.date=September+2009&rft.pub=US+Department+of+Health+and+Human+Services%2C+Centers+for+Disease+Control+and+Prevention&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20170613082955%2Fhttps%3A%2F%2Fwww.cdc.gov%2Finformaticscompetencies%2Fpdfs%2Fphi-competencies.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MassoudiPublic16-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MassoudiPublic16_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Massoudi, B.L.; Chester, K.; Shah, G.H. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5049962\" target=\"_blank\">\"Public Health Staff Development Needs in Informatics: Findings From a National Survey of Local Health Departments\"<\/a>. <i>Journal of Public Health Management and Practice<\/i> <b>22<\/b> (Suppl. 6): S58\u2013S62. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPHH.0000000000000450\" target=\"_blank\">10.1097\/PHH.0000000000000450<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5049962\/\" target=\"_blank\">PMC5049962<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27684619\" target=\"_blank\">27684619<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5049962\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5049962<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Public+Health+Staff+Development+Needs+in+Informatics%3A+Findings+From+a+National+Survey+of+Local+Health+Departments&rft.jtitle=Journal+of+Public+Health+Management+and+Practice&rft.aulast=Massoudi%2C+B.L.%3B+Chester%2C+K.%3B+Shah%2C+G.H.&rft.au=Massoudi%2C+B.L.%3B+Chester%2C+K.%3B+Shah%2C+G.H.&rft.date=2016&rft.volume=22&rft.issue=Suppl.+6&rft.pages=S58%E2%80%93S62&rft_id=info:doi\/10.1097%2FPHH.0000000000000450&rft_id=info:pmc\/PMC5049962&rft_id=info:pmid\/27684619&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5049962&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BeckState17-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BeckState17_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Beck, A.J.; Leider, J.P.; Coronado, F. et al. (2017). \"State Health Agency and Local Health Department Workforce: Identifying Top Development Needs\". <i>American Journal of Public Health<\/i> <b>107<\/b> (9): 1418-1424. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2105%2FAJPH.2017.303875\" target=\"_blank\">10.2105\/AJPH.2017.303875<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28727537\" target=\"_blank\">28727537<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=State+Health+Agency+and+Local+Health+Department+Workforce%3A+Identifying+Top+Development+Needs&rft.jtitle=American+Journal+of+Public+Health&rft.aulast=Beck%2C+A.J.%3B+Leider%2C+J.P.%3B+Coronado%2C+F.+et+al.&rft.au=Beck%2C+A.J.%3B+Leider%2C+J.P.%3B+Coronado%2C+F.+et+al.&rft.date=2017&rft.volume=107&rft.issue=9&rft.pages=1418-1424&rft_id=info:doi\/10.2105%2FAJPH.2017.303875&rft_id=info:pmid\/28727537&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CalhounDevelop08-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CalhounDevelop08_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Calhoun, J.G.; Ramiah, K.; Weist, E.M. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2509588\" target=\"_blank\">\"Development of a core competency model for the master of public health degree\"<\/a>. <i>American Journal of Public Health<\/i> <b>98<\/b> (9): 1598-607. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2105%2FAJPH.2007.117978\" target=\"_blank\">10.2105\/AJPH.2007.117978<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2509588\/\" target=\"_blank\">PMC2509588<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18633093\" target=\"_blank\">18633093<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2509588\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2509588<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+of+a+core+competency+model+for+the+master+of+public+health+degree&rft.jtitle=American+Journal+of+Public+Health&rft.aulast=Calhoun%2C+J.G.%3B+Ramiah%2C+K.%3B+Weist%2C+E.M.+et+al.&rft.au=Calhoun%2C+J.G.%3B+Ramiah%2C+K.%3B+Weist%2C+E.M.+et+al.&rft.date=2008&rft.volume=98&rft.issue=9&rft.pages=1598-607&rft_id=info:doi\/10.2105%2FAJPH.2007.117978&rft_id=info:pmc\/PMC2509588&rft_id=info:pmid\/18633093&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2509588&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CEPHAccred16-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CEPHAccred16_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-CEPHAccred16_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-CEPHAccred16_6-2\">6.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Council on Education for Public Health (October 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/media.ceph.org\/wp_assets\/2016.Criteria.pdf\" target=\"_blank\">\"Accreditation Criteria: Schools of Public Health & Public Health Programs\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/media.ceph.org\/wp_assets\/2016.Criteria.pdf\" target=\"_blank\">https:\/\/media.ceph.org\/wp_assets\/2016.Criteria.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Accreditation+Criteria%3A+Schools+of+Public+Health+%26+Public+Health+Programs&rft.atitle=&rft.aulast=Council+on+Education+for+Public+Health&rft.au=Council+on+Education+for+Public+Health&rft.date=October+2016&rft_id=https%3A%2F%2Fmedia.ceph.org%2Fwp_assets%2F2016.Criteria.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LaVentureBuild14-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LaVentureBuild14_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-LaVentureBuild14_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">LaVenture, M.; Brand, B.; Ross, D.A. et al. (2014). \"Building an informatics-savvy health department: Part I, vision and core strategies\". <i>Journal of Public Health Management and Practice<\/i> <b>20<\/b> (6): 667\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPHH.0000000000000149\" target=\"_blank\">10.1097\/PHH.0000000000000149<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25250757\" target=\"_blank\">25250757<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+an+informatics-savvy+health+department%3A+Part+I%2C+vision+and+core+strategies&rft.jtitle=Journal+of+Public+Health+Management+and+Practice&rft.aulast=LaVenture%2C+M.%3B+Brand%2C+B.%3B+Ross%2C+D.A.+et+al.&rft.au=LaVenture%2C+M.%3B+Brand%2C+B.%3B+Ross%2C+D.A.+et+al.&rft.date=2014&rft.volume=20&rft.issue=6&rft.pages=667%E2%80%939&rft_id=info:doi\/10.1097%2FPHH.0000000000000149&rft_id=info:pmid\/25250757&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LaVentureBuild15-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LaVentureBuild15_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-LaVentureBuild15_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">LaVenture, M.; Brand, B.; Ross, D.A. et al. (2015). \"Building an informatics-savvy health department II: Operations and tactics\". <i>Journal of Public Health Management and Practice<\/i> <b>21<\/b> (1): 96\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPHH.0000000000000179\" target=\"_blank\">10.1097\/PHH.0000000000000179<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25414962\" target=\"_blank\">25414962<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+an+informatics-savvy+health+department+II%3A+Operations+and+tactics&rft.jtitle=Journal+of+Public+Health+Management+and+Practice&rft.aulast=LaVenture%2C+M.%3B+Brand%2C+B.%3B+Ross%2C+D.A.+et+al.&rft.au=LaVenture%2C+M.%3B+Brand%2C+B.%3B+Ross%2C+D.A.+et+al.&rft.date=2015&rft.volume=21&rft.issue=1&rft.pages=96%E2%80%939&rft_id=info:doi\/10.1097%2FPHH.0000000000000179&rft_id=info:pmid\/25414962&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LaVentureDevelop17-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LaVentureDevelop17_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">LaVenture, M.; Brand, B.; Baker, E.L. (2017). \"Developing an informatics-savvy health department: From discrete projects to a coordinating program - Part I: Assessment and governance\". <i>Journal of Public Health Management and Practice<\/i> <b>23<\/b> (3): 325\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPHH.0000000000000551\" target=\"_blank\">10.1097\/PHH.0000000000000551<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28350628\" target=\"_blank\">28350628<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Developing+an+informatics-savvy+health+department%3A+From+discrete+projects+to+a+coordinating+program+-+Part+I%3A+Assessment+and+governance&rft.jtitle=Journal+of+Public+Health+Management+and+Practice&rft.aulast=LaVenture%2C+M.%3B+Brand%2C+B.%3B+Baker%2C+E.L.&rft.au=LaVenture%2C+M.%3B+Brand%2C+B.%3B+Baker%2C+E.L.&rft.date=2017&rft.volume=23&rft.issue=3&rft.pages=325%E2%80%937&rft_id=info:doi\/10.1097%2FPHH.0000000000000551&rft_id=info:pmid\/28350628&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrandDevelop18-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrandDevelop18_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brand, B.; LaVenture, M.; Baker, E.L. (2018). \"Developing an informatics-savvy health department: From discrete projects to a coordinating program - Part III: Ensuring well-designed and effectively used information systems\". <i>Journal of Public Health Management and Practice<\/i> <b>24<\/b> (2): 181\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPHH.0000000000000756\" target=\"_blank\">10.1097\/PHH.0000000000000756<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29360696\" target=\"_blank\">29360696<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Developing+an+informatics-savvy+health+department%3A+From+discrete+projects+to+a+coordinating+program+-+Part+III%3A+Ensuring+well-designed+and+effectively+used+information+systems&rft.jtitle=Journal+of+Public+Health+Management+and+Practice&rft.aulast=Brand%2C+B.%3B+LaVenture%2C+M.%3B+Baker%2C+E.L.&rft.au=Brand%2C+B.%3B+LaVenture%2C+M.%3B+Baker%2C+E.L.&rft.date=2018&rft.volume=24&rft.issue=2&rft.pages=181%E2%80%934&rft_id=info:doi\/10.1097%2FPHH.0000000000000756&rft_id=info:pmid\/29360696&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MinerApplied11-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MinerApplied11_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-MinerApplied11_11-1\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-MinerApplied11_11-2\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Miner, K.; Alperin, M.; Brogan, C.W. et al. (28 February 2011). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.phii.org\/resources\/view\/158\/applied-public-health-informatics-curriculum-preface-and-modules\" target=\"_blank\">\"Applied Public Health Informatics Curriculum - Preface and Modules\"<\/a>. Public Health Informatics Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.phii.org\/resources\/view\/158\/applied-public-health-informatics-curriculum-preface-and-modules\" target=\"_blank\">https:\/\/www.phii.org\/resources\/view\/158\/applied-public-health-informatics-curriculum-preface-and-modules<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Applied+Public+Health+Informatics+Curriculum+-+Preface+and+Modules&rft.atitle=&rft.aulast=Miner%2C+K.%3B+Alperin%2C+M.%3B+Brogan%2C+C.W.+et+al.&rft.au=Miner%2C+K.%3B+Alperin%2C+M.%3B+Brogan%2C+C.W.+et+al.&rft.date=28+February+2011&rft.pub=Public+Health+Informatics+Institute&rft_id=https%3A%2F%2Fwww.phii.org%2Fresources%2Fview%2F158%2Fapplied-public-health-informatics-curriculum-preface-and-modules&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FondMaking15-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FondMaking15_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-FondMaking15_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-FondMaking15_12-2\">12.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fond, M.; Volmert, A.; Kendall-Taylor, N. (September 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.frameworksinstitute.org\/assets\/files\/health_care\/phiistrategicmtgfinalseptember2015.pdf\" target=\"_blank\">\"Making Public Health Informatics Visible: Communicating an Emerging Field\"<\/a> (PDF). FrameWorks Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.frameworksinstitute.org\/assets\/files\/health_care\/phiistrategicmtgfinalseptember2015.pdf\" target=\"_blank\">https:\/\/www.frameworksinstitute.org\/assets\/files\/health_care\/phiistrategicmtgfinalseptember2015.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Making+Public+Health+Informatics+Visible%3A+Communicating+an+Emerging+Field&rft.atitle=&rft.aulast=Fond%2C+M.%3B+Volmert%2C+A.%3B+Kendall-Taylor%2C+N.&rft.au=Fond%2C+M.%3B+Volmert%2C+A.%3B+Kendall-Taylor%2C+N.&rft.date=September+2015&rft.pub=FrameWorks+Institute&rft_id=https%3A%2F%2Fwww.frameworksinstitute.org%2Fassets%2Ffiles%2Fhealth_care%2Fphiistrategicmtgfinalseptember2015.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IoMTheFuture88-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IoMTheFuture88_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Institute of Medicine (1988). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nap.edu\/catalog\/1091\/the-future-of-public-health\" target=\"_blank\">\"The Future of Public Health\"<\/a>. The National Academies Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.17226%2F1091\" target=\"_blank\">10.17226\/1091<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nap.edu\/catalog\/1091\/the-future-of-public-health\" target=\"_blank\">https:\/\/www.nap.edu\/catalog\/1091\/the-future-of-public-health<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Future+of+Public+Health&rft.atitle=&rft.aulast=Institute+of+Medicine&rft.au=Institute+of+Medicine&rft.date=1988&rft.pub=The+National+Academies+Press&rft_id=info:doi\/10.17226%2F1091&rft_id=https%3A%2F%2Fwww.nap.edu%2Fcatalog%2F1091%2Fthe-future-of-public-health&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YasnoffANat01-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YasnoffANat01_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yasnoff, W.A.; Overhage, J.M.; Humphreys, B.L. et al. (2001). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130064\" target=\"_blank\">\"A national agenda for public health informatics: summarized recommendations from the 2001 AMIA Spring Congress\"<\/a>. <i>JAMIA<\/i> <b>8<\/b> (6): 535\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC130064\/\" target=\"_blank\">PMC130064<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11687561\" target=\"_blank\">11687561<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130064\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC130064<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+national+agenda+for+public+health+informatics%3A+summarized+recommendations+from+the+2001+AMIA+Spring+Congress&rft.jtitle=JAMIA&rft.aulast=Yasnoff%2C+W.A.%3B+Overhage%2C+J.M.%3B+Humphreys%2C+B.L.+et+al.&rft.au=Yasnoff%2C+W.A.%3B+Overhage%2C+J.M.%3B+Humphreys%2C+B.L.+et+al.&rft.date=2001&rft.volume=8&rft.issue=6&rft.pages=535%E2%80%9345&rft_id=info:pmc\/PMC130064&rft_id=info:pmid\/11687561&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC130064&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MassoudiAnInfo12-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MassoudiAnInfo12_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-MassoudiAnInfo12_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Massoudi, B.L.; Goodman, K.W.; Gotham, I.J. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3422819\" target=\"_blank\">\"An informatics agenda for public health: Summarized recommendations from the 2011 AMIA PHI Conference\"<\/a>. <i>JAMIA<\/i> <b>19<\/b> (5): 688-95. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2011-000507\" target=\"_blank\">10.1136\/amiajnl-2011-000507<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3422819\/\" target=\"_blank\">PMC3422819<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22395299\" target=\"_blank\">22395299<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3422819\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3422819<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+informatics+agenda+for+public+health%3A+Summarized+recommendations+from+the+2011+AMIA+PHI+Conference&rft.jtitle=JAMIA&rft.aulast=Massoudi%2C+B.L.%3B+Goodman%2C+K.W.%3B+Gotham%2C+I.J.+et+al.&rft.au=Massoudi%2C+B.L.%3B+Goodman%2C+K.W.%3B+Gotham%2C+I.J.+et+al.&rft.date=2012&rft.volume=19&rft.issue=5&rft.pages=688-95&rft_id=info:doi\/10.1136%2Famiajnl-2011-000507&rft_id=info:pmc\/PMC3422819&rft_id=info:pmid\/22395299&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3422819&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PHIITaking08-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PHIITaking08_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-PHIITaking08_16-1\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-PHIITaking08_16-2\">16.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Public Health Informatics Institute (2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.phii.org\/resources\/taking-care-business-second-edition\" target=\"_blank\"><i>Taking Care of Business: A Collaboration to Define Local Health Department Business Processes<\/i><\/a> (2nd Printing ed.). Public Health Informatics Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.phii.org\/resources\/taking-care-business-second-edition\" target=\"_blank\">https:\/\/www.phii.org\/resources\/taking-care-business-second-edition<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Taking+Care+of+Business%3A+A+Collaboration+to+Define+Local+Health+Department+Business+Processes&rft.aulast=Public+Health+Informatics+Institute&rft.au=Public+Health+Informatics+Institute&rft.date=2008&rft.edition=2nd+Printing&rft.pub=Public+Health+Informatics+Institute&rft_id=https%3A%2F%2Fwww.phii.org%2Fresources%2Ftaking-care-business-second-edition&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McNabbTrans16-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McNabbTrans16_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-McNabbTrans16_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">McNabb, S.J.; Conde, J.M.; Ferland, L. et al. (2016). <i>Transforming Public Health Surveillance: Proactive Measures for Prevention, Detection, and Response<\/i>. Elsevier. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780702063374.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Transforming+Public+Health+Surveillance%3A+Proactive+Measures+for+Prevention%2C+Detection%2C+and+Response&rft.aulast=McNabb%2C+S.J.%3B+Conde%2C+J.M.%3B+Ferland%2C+L.+et+al.&rft.au=McNabb%2C+S.J.%3B+Conde%2C+J.M.%3B+Ferland%2C+L.+et+al.&rft.date=2016&rft.pub=Elsevier&rft.isbn=9780702063374&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McNabbInform17-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McNabbInform17_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-McNabbInform17_18-1\">18.1<\/a><\/sup> <sup><a href=\"#cite_ref-McNabbInform17_18-2\">18.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McNabb, S.J.N.; Rayland, P.; Sylvester, J. et al. (2017). \"Informatics enables public health surveillance\". <i>Journal of Health Specialties<\/i> <b>5<\/b> (2): 55\u201359. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2Fjhs.JHS_28_17\" target=\"_blank\">10.4103\/jhs.JHS_28_17<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Informatics+enables+public+health+surveillance&rft.jtitle=Journal+of+Health+Specialties&rft.aulast=McNabb%2C+S.J.N.%3B+Rayland%2C+P.%3B+Sylvester%2C+J.+et+al.&rft.au=McNabb%2C+S.J.N.%3B+Rayland%2C+P.%3B+Sylvester%2C+J.+et+al.&rft.date=2017&rft.volume=5&rft.issue=2&rft.pages=55%E2%80%9359&rft_id=info:doi\/10.4103%2Fjhs.JHS_28_17&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BachrachShaping17-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BachrachShaping17_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bachrach, C.A.; Daley, D.M. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5227949\" target=\"_blank\">\"Shaping a New Field: Three Key Challenges for Population Health Science\"<\/a>. <i>American Journal of Public Health<\/i> <b>107<\/b> (2): 251\u201352. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2105%2FAJPH.2016.303580\" target=\"_blank\">10.2105\/AJPH.2016.303580<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5227949\/\" target=\"_blank\">PMC5227949<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28075642\" target=\"_blank\">28075642<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5227949\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5227949<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Shaping+a+New+Field%3A+Three+Key+Challenges+for+Population+Health+Science&rft.jtitle=American+Journal+of+Public+Health&rft.aulast=Bachrach%2C+C.A.%3B+Daley%2C+D.M.&rft.au=Bachrach%2C+C.A.%3B+Daley%2C+D.M.&rft.date=2017&rft.volume=107&rft.issue=2&rft.pages=251%E2%80%9352&rft_id=info:doi\/10.2105%2FAJPH.2016.303580&rft_id=info:pmc\/PMC5227949&rft_id=info:pmid\/28075642&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5227949&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KindigUnder07-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KindigUnder07_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-KindigUnder07_20-1\">20.1<\/a><\/sup> <sup><a href=\"#cite_ref-KindigUnder07_20-2\">20.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kindig, D.A. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690307\" target=\"_blank\">\"Understanding population health terminology\"<\/a>. <i>Milbank Q<\/i> <b>85<\/b> (1): 139\u201361. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1468-0009.2007.00479.x\" target=\"_blank\">10.1111\/j.1468-0009.2007.00479.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2690307\/\" target=\"_blank\">PMC2690307<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17319809\" target=\"_blank\">17319809<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690307\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2690307<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+population+health+terminology&rft.jtitle=Milbank+Q&rft.aulast=Kindig%2C+D.A.&rft.au=Kindig%2C+D.A.&rft.date=2007&rft.volume=85&rft.issue=1&rft.pages=139%E2%80%9361&rft_id=info:doi\/10.1111%2Fj.1468-0009.2007.00479.x&rft_id=info:pmc\/PMC2690307&rft_id=info:pmid\/17319809&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2690307&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FriedmanToward15-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FriedmanToward15_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Friedman, C.; Rubin, J.; Brown, J. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4433378\" target=\"_blank\">\"Toward a science of learning systems: A research agenda for the high-functioning Learning Health System\"<\/a>. <i>JAMIA<\/i> <b>22<\/b> (1): 43\u201350. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2014-002977\" target=\"_blank\">10.1136\/amiajnl-2014-002977<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4433378\/\" target=\"_blank\">PMC4433378<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25342177\" target=\"_blank\">25342177<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4433378\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4433378<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+science+of+learning+systems%3A+A+research+agenda+for+the+high-functioning+Learning+Health+System&rft.jtitle=JAMIA&rft.aulast=Friedman%2C+C.%3B+Rubin%2C+J.%3B+Brown%2C+J.+et+al.&rft.au=Friedman%2C+C.%3B+Rubin%2C+J.%3B+Brown%2C+J.+et+al.&rft.date=2015&rft.volume=22&rft.issue=1&rft.pages=43%E2%80%9350&rft_id=info:doi\/10.1136%2Famiajnl-2014-002977&rft_id=info:pmc\/PMC4433378&rft_id=info:pmid\/25342177&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4433378&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YasnoffPublic00-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-YasnoffPublic00_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-YasnoffPublic00_22-1\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yasnoff, W.A.; O'Carroll, P.W.; Koo, D. et al. (2000). \"Public health informatics: improving and transforming public health in the information age\". <i>Journal of Public Health Management and Practics<\/i> <b>6<\/b> (6): 67\u201375. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18019962\" target=\"_blank\">18019962<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Public+health+informatics%3A+improving+and+transforming+public+health+in+the+information+age&rft.jtitle=Journal+of+Public+Health+Management+and+Practics&rft.aulast=Yasnoff%2C+W.A.%3B+O%27Carroll%2C+P.W.%3B+Koo%2C+D.+et+al.&rft.au=Yasnoff%2C+W.A.%3B+O%27Carroll%2C+P.W.%3B+Koo%2C+D.+et+al.&rft.date=2000&rft.volume=6&rft.issue=6&rft.pages=67%E2%80%9375&rft_id=info:pmid\/18019962&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PHIIWorkforce14-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PHIIWorkforce14_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Public Health Informatics Institute (25 April 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.phii.org\/resources\/view\/6423\/workforce-position-classifications-and-descriptions\" target=\"_blank\">\"Workforce Position Classifications and Descriptions\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.phii.org\/resources\/view\/6423\/workforce-position-classifications-and-descriptions\" target=\"_blank\">https:\/\/www.phii.org\/resources\/view\/6423\/workforce-position-classifications-and-descriptions<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 September 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Workforce+Position+Classifications+and+Descriptions&rft.atitle=&rft.aulast=Public+Health+Informatics+Institute&rft.au=Public+Health+Informatics+Institute&rft.date=25+April+2014&rft_id=https%3A%2F%2Fwww.phii.org%2Fresources%2Fview%2F6423%2Fworkforce-position-classifications-and-descriptions&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PHIIProf14-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PHIIProf14_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Public Health Informatics Institute (April 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.phii.org\/sites\/www.phii.org\/files\/resource\/pdfs\/Professional%20Sample%20Position%20AND%20Career%20Ladder.pdf\" target=\"_blank\">\"Professional Level Public Health Informatician: Sample Position Description and Sample Career Ladder\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.phii.org\/sites\/www.phii.org\/files\/resource\/pdfs\/Professional%20Sample%20Position%20AND%20Career%20Ladder.pdf\" target=\"_blank\">https:\/\/www.phii.org\/sites\/www.phii.org\/files\/resource\/pdfs\/Professional%20Sample%20Position%20AND%20Career%20Ladder.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 October 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Professional+Level+Public+Health+Informatician%3A+Sample+Position+Description+and+Sample+Career+Ladder&rft.atitle=&rft.aulast=Public+Health+Informatics+Institute&rft.au=Public+Health+Informatics+Institute&rft.date=April+2014&rft_id=https%3A%2F%2Fwww.phii.org%2Fsites%2Fwww.phii.org%2Ffiles%2Fresource%2Fpdfs%2FProfessional%2520Sample%2520Position%2520AND%2520Career%2520Ladder.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PHFCore14-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PHFCore14_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-PHFCore14_25-1\">25.1<\/a><\/sup> <sup><a href=\"#cite_ref-PHFCore14_25-2\">25.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Council on Linkages Between Academia and Public Health Practice (26 June 2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.phf.org\/resourcestools\/pages\/core_public_health_competencies.aspx\" target=\"_blank\">\"Core Competencies for Public Health Professionals\"<\/a>. pp. 24<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.phf.org\/resourcestools\/pages\/core_public_health_competencies.aspx\" target=\"_blank\">http:\/\/www.phf.org\/resourcestools\/pages\/core_public_health_competencies.aspx<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Core+Competencies+for+Public+Health+Professionals&rft.atitle=&rft.aulast=Council+on+Linkages+Between+Academia+and+Public+Health+Practice&rft.au=Council+on+Linkages+Between+Academia+and+Public+Health+Practice&rft.date=26+June+2014&rft.pages=pp.+24&rft_id=http%3A%2F%2Fwww.phf.org%2Fresourcestools%2Fpages%2Fcore_public_health_competencies.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BerndtTheCatch03-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BerndtTheCatch03_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berndt, D.J.; Hevner, A.R.; Studnicki, J. et al. (2003). \"The Catch data warehouse: Support for community health care decision-making\". <i>Decision Support Systems<\/i> <b>35<\/b> (3): 367\u201384. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0167-9236%2802%2900114-8\" target=\"_blank\">10.1016\/S0167-9236(02)00114-8<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Catch+data+warehouse%3A+Support+for+community+health+care+decision-making&rft.jtitle=Decision+Support+Systems&rft.aulast=Berndt%2C+D.J.%3B+Hevner%2C+A.R.%3B+Studnicki%2C+J.+et+al.&rft.au=Berndt%2C+D.J.%3B+Hevner%2C+A.R.%3B+Studnicki%2C+J.+et+al.&rft.date=2003&rft.volume=35&rft.issue=3&rft.pages=367%E2%80%9384&rft_id=info:doi\/10.1016%2FS0167-9236%2802%2900114-8&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KindigWhatIs03-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KindigWhatIs03_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kindig, D.; Stoddart, G. (2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447747\" target=\"_blank\">\"What is population health?\"<\/a>. <i>American Journal of Public Health<\/i> <b>93<\/b> (3): 380-3. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1447747\/\" target=\"_blank\">PMC1447747<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12604476\" target=\"_blank\">12604476<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447747\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447747<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+is+population+health%3F&rft.jtitle=American+Journal+of+Public+Health&rft.aulast=Kindig%2C+D.%3B+Stoddart%2C+G.&rft.au=Kindig%2C+D.%3B+Stoddart%2C+G.&rft.date=2003&rft.volume=93&rft.issue=3&rft.pages=380-3&rft_id=info:pmc\/PMC1447747&rft_id=info:pmid\/12604476&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1447747&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SimonTheSci81-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SimonTheSci81_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-SimonTheSci81_28-1\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Simon, H.A. (1981). <i>The Sciences of the Artificial<\/i> (2nd ed.). MIT Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780262191937.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Sciences+of+the+Artificial&rft.aulast=Simon%2C+H.A.&rft.au=Simon%2C+H.A.&rft.date=1981&rft.edition=2nd&rft.pub=MIT+Press&rft.isbn=9780262191937&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DymEng13-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DymEng13_29-0\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-DymEng13_29-1\">29.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dym, C.L.; Agogino, A.M.; Eris, O. et al. (2013). \"Engineering Design Thinking, Teaching, and Learning\". <i>Journal of Engineering Education<\/i> <b>94<\/b> (1): 103\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fj.2168-9830.2005.tb00832.x\" target=\"_blank\">10.1002\/j.2168-9830.2005.tb00832.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Engineering+Design+Thinking%2C+Teaching%2C+and+Learning&rft.jtitle=Journal+of+Engineering+Education&rft.aulast=Dym%2C+C.L.%3B+Agogino%2C+A.M.%3B+Eris%2C+O.+et+al.&rft.au=Dym%2C+C.L.%3B+Agogino%2C+A.M.%3B+Eris%2C+O.+et+al.&rft.date=2013&rft.volume=94&rft.issue=1&rft.pages=103%E2%80%9320&rft_id=info:doi\/10.1002%2Fj.2168-9830.2005.tb00832.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Johansson.E2.80.90Sk.C3.B6ldbergDesign13-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Johansson.E2.80.90Sk.C3.B6ldbergDesign13_30-0\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-Johansson.E2.80.90Sk.C3.B6ldbergDesign13_30-1\">30.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Johansson\u2010Sk\u00f6ldberg, U.; Woodilla, J.; \u00c7etinkaya, M. (2013). \"Design Thinking: Past, Present and Possible Futures\". <i>Creativity and Innovation Management<\/i> <b>22<\/b> (2): 121\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fcaim.12023\" target=\"_blank\">10.1111\/caim.12023<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+Thinking%3A+Past%2C+Present+and+Possible+Futures&rft.jtitle=Creativity+and+Innovation+Management&rft.aulast=Johansson%E2%80%90Sk%C3%B6ldberg%2C+U.%3B+Woodilla%2C+J.%3B+%C3%87etinkaya%2C+M.&rft.au=Johansson%E2%80%90Sk%C3%B6ldberg%2C+U.%3B+Woodilla%2C+J.%3B+%C3%87etinkaya%2C+M.&rft.date=2013&rft.volume=22&rft.issue=2&rft.pages=121%E2%80%9346&rft_id=info:doi\/10.1111%2Fcaim.12023&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FriedmanWhatInfo12-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FriedmanWhatInfo12_31-0\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-FriedmanWhatInfo12_31-1\">31.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Friedman, C.P. (2012). \"What informatics is and isn't\". <i>JAMIA<\/i> <b>20<\/b> (2): 224\u201326. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2012-001206\" target=\"_blank\">10.1136\/amiajnl-2012-001206<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+informatics+is+and+isn%27t&rft.jtitle=JAMIA&rft.aulast=Friedman%2C+C.P.&rft.au=Friedman%2C+C.P.&rft.date=2012&rft.volume=20&rft.issue=2&rft.pages=224%E2%80%9326&rft_id=info:doi\/10.1136%2Famiajnl-2012-001206&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PawsonARealist12-32\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PawsonARealist12_32-0\">32.0<\/a><\/sup> <sup><a href=\"#cite_ref-PawsonARealist12_32-1\">32.1<\/a><\/sup> <sup><a href=\"#cite_ref-PawsonARealist12_32-2\">32.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pawson, R.; Manzano-Santaella, A. (2012). \"A realist diagnostic workshop\". <i>Evaluation<\/i> <b>18<\/b> (2): 176\u201391. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1356389012440912\" target=\"_blank\">10.1177\/1356389012440912<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+realist+diagnostic+workshop&rft.jtitle=Evaluation&rft.aulast=Pawson%2C+R.%3B+Manzano-Santaella%2C+A.&rft.au=Pawson%2C+R.%3B+Manzano-Santaella%2C+A.&rft.date=2012&rft.volume=18&rft.issue=2&rft.pages=176%E2%80%9391&rft_id=info:doi\/10.1177%2F1356389012440912&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CDCPublic17-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CDCPublic17_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Centers for Disease Control and Prevention (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cdc.gov\/publichealthgateway\/publichealthservices\/essentialhealthservices.html\" target=\"_blank\">\"The Public Health System & the 10 Essential Public Health Services\"<\/a>. <i>National Public Health Performance Standards<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cdc.gov\/publichealthgateway\/publichealthservices\/essentialhealthservices.html\" target=\"_blank\">https:\/\/www.cdc.gov\/publichealthgateway\/publichealthservices\/essentialhealthservices.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 May 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Public+Health+System+%26+the+10+Essential+Public+Health+Services&rft.atitle=National+Public+Health+Performance+Standards&rft.aulast=Centers+for+Disease+Control+and+Prevention&rft.au=Centers+for+Disease+Control+and+Prevention&rft.date=2014&rft_id=https%3A%2F%2Fwww.cdc.gov%2Fpublichealthgateway%2Fpublichealthservices%2Fessentialhealthservices.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarrisonUnintended07-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HarrisonUnintended07_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Harrison, M.I.; Koppel, R.; Bar-Lev, S. (2007). \"Unintended Consequences of Information Technologies in Health Care\u2014An Interactive Sociotechnical Analysis\". <i>JAMIA<\/i> <b>14<\/b> (5): 542\u201349. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2384\" target=\"_blank\">10.1197\/jamia.M2384<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Unintended+Consequences+of+Information+Technologies+in+Health+Care%E2%80%94An+Interactive+Sociotechnical+Analysis&rft.jtitle=JAMIA&rft.aulast=Harrison%2C+M.I.%3B+Koppel%2C+R.%3B+Bar-Lev%2C+S.&rft.au=Harrison%2C+M.I.%3B+Koppel%2C+R.%3B+Bar-Lev%2C+S.&rft.date=2007&rft.volume=14&rft.issue=5&rft.pages=542%E2%80%9349&rft_id=info:doi\/10.1197%2Fjamia.M2384&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PentlandDesign08-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PentlandDesign08_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pentland, B.T.; Feldman, M.S. (2008). \"Designing routines: On the folly of designing artifacts, while hoping for patterns of action\". <i>Information and Organization<\/i> <b>18<\/b> (4): 235\u2013250. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.infoandorg.2008.08.001\" target=\"_blank\">10.1016\/j.infoandorg.2008.08.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Designing+routines%3A+On+the+folly+of+designing+artifacts%2C+while+hoping+for+patterns+of+action&rft.jtitle=Information+and+Organization&rft.aulast=Pentland%2C+B.T.%3B+Feldman%2C+M.S.&rft.au=Pentland%2C+B.T.%3B+Feldman%2C+M.S.&rft.date=2008&rft.volume=18&rft.issue=4&rft.pages=235%E2%80%93250&rft_id=info:doi\/10.1016%2Fj.infoandorg.2008.08.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CMUInfo18-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CMUInfo18_36-0\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-CMUInfo18_36-1\">36.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cmu.edu\/information-systems\/\" target=\"_blank\">\"Information Systems\"<\/a>. Carnegie Mellon University. 2016<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cmu.edu\/information-systems\/\" target=\"_blank\">https:\/\/www.cmu.edu\/information-systems\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 29 January 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Information+Systems&rft.atitle=&rft.date=2016&rft.pub=Carnegie+Mellon+University&rft_id=https%3A%2F%2Fwww.cmu.edu%2Finformation-systems%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DixonHealth16-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DixonHealth16_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Dixon, B., ed. (2016). <i>Health Information Exchange: Navigating and Managing a Network of Health Information Systems<\/i>. Academic Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/9780128031353\" target=\"_blank\">9780128031353<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Health+Information+Exchange%3A+Navigating+and+Managing+a+Network+of+Health+Information+Systems&rft.date=2016&rft.pub=Academic+Press&rft_id=info:doi\/9780128031353&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PHIIRedes12-38\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PHIIRedes12_38-0\">38.0<\/a><\/sup> <sup><a href=\"#cite_ref-PHIIRedes12_38-1\">38.1<\/a><\/sup> <sup><a href=\"#cite_ref-PHIIRedes12_38-2\">38.2<\/a><\/sup> <sup><a href=\"#cite_ref-PHIIRedes12_38-3\">38.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Public Health Informatics Institute (June 2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.phii.org\/resources\/view\/1186\/redesigning-public-health-surveillance-ehealth-world\" target=\"_blank\">\"Redesigning Public Health Surveillance in an eHealth World\"<\/a>. Public Health Informatics Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.phii.org\/resources\/view\/1186\/redesigning-public-health-surveillance-ehealth-world\" target=\"_blank\">https:\/\/www.phii.org\/resources\/view\/1186\/redesigning-public-health-surveillance-ehealth-world<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Redesigning+Public+Health+Surveillance+in+an+eHealth+World&rft.atitle=&rft.aulast=Public+Health+Informatics+Institute&rft.au=Public+Health+Informatics+Institute&rft.date=June+2012&rft.pub=Public+Health+Informatics+Institute&rft_id=https%3A%2F%2Fwww.phii.org%2Fresources%2Fview%2F1186%2Fredesigning-public-health-surveillance-ehealth-world&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DennisSystems15-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DennisSystems15_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Dennis, A.; Wixom, B.H.; Roth, R.M. (2015). <i>Systems Analysis and Design<\/i> (6th ed.). Wiley. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781118897843.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Systems+Analysis+and+Design&rft.aulast=Dennis%2C+A.%3B+Wixom%2C+B.H.%3B+Roth%2C+R.M.&rft.au=Dennis%2C+A.%3B+Wixom%2C+B.H.%3B+Roth%2C+R.M.&rft.date=2015&rft.edition=6th&rft.pub=Wiley&rft.isbn=9781118897843&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HammondStand14-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HammondStand14_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hammond, W.E.; Jaffe, C.; Cimino, J.J. et al. (2014). \"Standards in Biomedical Informatics\". In Shortliffe, E.H.; Cimino, J.J.. <i>Biomedical Informatics<\/i> (4th ed.). Springer. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781447144748.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Standards+in+Biomedical+Informatics&rft.atitle=Biomedical+Informatics&rft.aulast=Hammond%2C+W.E.%3B+Jaffe%2C+C.%3B+Cimino%2C+J.J.+et+al.&rft.au=Hammond%2C+W.E.%3B+Jaffe%2C+C.%3B+Cimino%2C+J.J.+et+al.&rft.date=2014&rft.edition=4th&rft.pub=Springer&rft.isbn=9781447144748&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoodallTheUse18-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WoodallTheUse18_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Woodall, W.H. (2006). \"The Use of Control Charts in Health-Care and Public-Health Surveillance\". <i>Journal of Quality Technology<\/i> <b>38<\/b> (2): 89\u2013104. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F00224065.2006.11918593\" target=\"_blank\">10.1080\/00224065.2006.11918593<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Use+of+Control+Charts+in+Health-Care+and+Public-Health+Surveillance&rft.jtitle=Journal+of+Quality+Technology&rft.aulast=Woodall%2C+W.H.&rft.au=Woodall%2C+W.H.&rft.date=2006&rft.volume=38&rft.issue=2&rft.pages=89%E2%80%93104&rft_id=info:doi\/10.1080%2F00224065.2006.11918593&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DoverUsing11-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DoverUsing11_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dover, D.C.; Schopflocher, D.P. (2011). \"Using funnel plots in public health surveillance\". <i>Population Health Metrics<\/i> <b>9<\/b>: 58. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1478-7954-9-58\" target=\"_blank\">10.1186\/1478-7954-9-58<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+funnel+plots+in+public+health+surveillance&rft.jtitle=Population+Health+Metrics&rft.aulast=Dover%2C+D.C.%3B+Schopflocher%2C+D.P.&rft.au=Dover%2C+D.C.%3B+Schopflocher%2C+D.P.&rft.date=2011&rft.volume=9&rft.pages=58&rft_id=info:doi\/10.1186%2F1478-7954-9-58&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShermanApplying14-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShermanApplying14_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sherman, R.L.; Henry, K.A.; Tannenbaum, S.L. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965324\" target=\"_blank\">\"Applying spatial analysis tools in public health: An example using SaTScan to detect geographic targets for colorectal cancer screening interventions\"<\/a>. <i>Preventing Chronic Disease<\/i> <b>11<\/b>: E41. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5888%2Fpcd11.130264\" target=\"_blank\">10.5888\/pcd11.130264<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3965324\/\" target=\"_blank\">PMC3965324<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24650619\" target=\"_blank\">24650619<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965324\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3965324<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applying+spatial+analysis+tools+in+public+health%3A+An+example+using+SaTScan+to+detect+geographic+targets+for+colorectal+cancer+screening+interventions&rft.jtitle=Preventing+Chronic+Disease&rft.aulast=Sherman%2C+R.L.%3B+Henry%2C+K.A.%3B+Tannenbaum%2C+S.L.+et+al.&rft.au=Sherman%2C+R.L.%3B+Henry%2C+K.A.%3B+Tannenbaum%2C+S.L.+et+al.&rft.date=2014&rft.volume=11&rft.pages=E41&rft_id=info:doi\/10.5888%2Fpcd11.130264&rft_id=info:pmc\/PMC3965324&rft_id=info:pmid\/24650619&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3965324&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GaddElig16-44\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GaddElig16_44-0\">44.0<\/a><\/sup> <sup><a href=\"#cite_ref-GaddElig16_44-1\">44.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gadd, C.S.; Williamson, J.J.; Steen, E.B. et al. (2016). \"Eligibility requirements for advanced health informatics certification\". <i>JAMIA<\/i> <b>23<\/b> (4): 851\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fjamia%2Focw090\" target=\"_blank\">10.1093\/jamia\/ocw090<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27358328\" target=\"_blank\">27358328<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Eligibility+requirements+for+advanced+health+informatics+certification&rft.jtitle=JAMIA&rft.aulast=Gadd%2C+C.S.%3B+Williamson%2C+J.J.%3B+Steen%2C+E.B.+et+al.&rft.au=Gadd%2C+C.S.%3B+Williamson%2C+J.J.%3B+Steen%2C+E.B.+et+al.&rft.date=2016&rft.volume=23&rft.issue=4&rft.pages=851%E2%80%934&rft_id=info:doi\/10.1093%2Fjamia%2Focw090&rft_id=info:pmid\/27358328&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BakerBuild16-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BakerBuild16_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Baker, E.L.; Brand, W.; Davidson, A. et al. (2016). \"Building the Business Case for Public Health Information Systems\". <i>Journal of Public Health Management and Practice<\/i> <b>22<\/b> (6): 603\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPHH.0000000000000495\" target=\"_blank\">10.1097\/PHH.0000000000000495<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27682729\" target=\"_blank\">27682729<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+the+Business+Case+for+Public+Health+Information+Systems&rft.jtitle=Journal+of+Public+Health+Management+and+Practice&rft.aulast=Baker%2C+E.L.%3B+Brand%2C+W.%3B+Davidson%2C+A.+et+al.&rft.au=Baker%2C+E.L.%3B+Brand%2C+W.%3B+Davidson%2C+A.+et+al.&rft.date=2016&rft.volume=22&rft.issue=6&rft.pages=603%E2%80%936&rft_id=info:doi\/10.1097%2FPHH.0000000000000495&rft_id=info:pmid\/27682729&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BraunsteinPublic18-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BraunsteinPublic18_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Braunstein, M.L.; Laventure, M.; Baker, E.L. (2018). \"Public Health Informatics Incubators: Accelerating Innovation Through Creative Partnerships Between Informatics Experts and Public Health Agencies\". <i>Journal of Public Health Management and Practice<\/i> <b>24<\/b> (3): 286\u201388. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FPHH.0000000000000808\" target=\"_blank\">10.1097\/PHH.0000000000000808<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29595587\" target=\"_blank\">29595587<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Public+Health+Informatics+Incubators%3A+Accelerating+Innovation+Through+Creative+Partnerships+Between+Informatics+Experts+and+Public+Health+Agencies&rft.jtitle=Journal+of+Public+Health+Management+and+Practice&rft.aulast=Braunstein%2C+M.L.%3B+Laventure%2C+M.%3B+Baker%2C+E.L.&rft.au=Braunstein%2C+M.L.%3B+Laventure%2C+M.%3B+Baker%2C+E.L.&rft.date=2018&rft.volume=24&rft.issue=3&rft.pages=286%E2%80%9388&rft_id=info:doi\/10.1097%2FPHH.0000000000000808&rft_id=info:pmid\/29595587&rfr_id=info:sid\/en.wikipedia.org:Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. A few grammar and spelling errors were also corrected. In some cases important information was missing from the references, and that information was added. Reference 2 and 11 in the original are unintentionally duplicated; we've removed a duplicate for this version. The same happened with reference 26 and 38 in the original, and it too was not duplicated for this version. The original had reference 46 listed in the references but no corresponding inline citation for 46; it was intentionally omitted for this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165458\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.139 seconds\nReal time usage: 1.197 seconds\nPreprocessor visited node count: 34814\/1000000\nPreprocessor generated node count: 41424\/1000000\nPost\u2010expand include size: 271523\/2097152 bytes\nTemplate argument size: 92276\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 1133.608 1 - -total\n 87.23% 988.828 1 - Template:Reflist\n 77.15% 874.606 46 - Template:Citation\/core\n 51.82% 587.442 28 - Template:Cite_journal\n 20.05% 227.338 12 - Template:Cite_web\n 9.56% 108.427 6 - Template:Cite_book\n 7.17% 81.277 58 - Template:Citation\/identifier\n 5.94% 67.356 1 - Template:Infobox_journal_article\n 5.68% 64.390 1 - Template:Infobox\n 4.44% 50.313 49 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11047-0!*!0!!en!5!* and timestamp 20190701165457 and revision id 35615\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design\">https:\/\/www.limswiki.org\/index.php\/Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","97b51c665da19cc225b05dd3ff621e96_images":["https:\/\/www.limswiki.org\/images\/3\/3e\/Tab1_Wholey_FrontPubHealth2019_6.jpg","https:\/\/www.limswiki.org\/images\/c\/c1\/Tab2_Wholey_FrontPubHealth2019_6.jpg","https:\/\/www.limswiki.org\/images\/a\/a3\/Tab3_Wholey_FrontPubHealth2019_6.jpg","https:\/\/www.limswiki.org\/images\/e\/ed\/Tab4_Wholey_FrontPubHealth2019_6.jpg","https:\/\/www.limswiki.org\/images\/b\/be\/Tab5_Wholey_FrontPubHealth2019_6.jpg"],"97b51c665da19cc225b05dd3ff621e96_timestamp":1562000097,"36d4da5bfaa11917913bb8842c095eca_type":"article","36d4da5bfaa11917913bb8842c095eca_title":"Determining the hospital information system (HIS) success rate: Development of a new instrument and case study (Ebnehoseini et al. 2019)","36d4da5bfaa11917913bb8842c095eca_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study","36d4da5bfaa11917913bb8842c095eca_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Determining the hospital information system (HIS) success rate: Development of a new instrument and case study\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDetermining the hospital information system (HIS) success rate: Development of a new instrument and case studyJournal\n \nOpen Access Macedonian Journal of Medical SciencesAuthor(s)\n \nEbnehoseini, Zahara; Tabesh, Hamed; Deldar, Kolsoum; Mostafavi, Sayyed M.; Tara, MahmoodAuthor affiliation(s)\n \nMashhad University of Medical Sciences, Shahroud University of Medical SciencesPrimary contact\n \nEmail: TaraM at mums dot ac dot irYear published\n \n2019Volume and issue\n \n7(9)Page(s)\n \n1407\u201314DOI\n \n10.3889\/oamjms.2019.294ISSN\n \n1857-9655Distribution license\n \nCreative Commons Attribution-NonCommercial 4.0 InternationalWebsite\n \nhttps:\/\/www.id-press.eu\/mjms\/article\/view\/2647Download\n \nhttps:\/\/www.id-press.eu\/mjms\/article\/view\/oamjms.2019.294\/3327 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methodology \n\n3.1 Instrument \n3.2 Study settings \n3.3 Participants \n3.4 Statistics \n\n\n4 Results \n\n4.1 Participants \n4.2 Reliability and validity \n4.3 HIS success rate \n\n4.3.1 HIS success rate in system quality \n4.3.2 HIS success rate in information quality \n4.3.3 HIS success rate in service quality \n4.3.4 HIS success rate in system use \n4.3.5 HIS success rate in satisfaction, usefulness, and net benefits \n4.3.6 Total HIS success rate \n\n\n\n\n5 Discussion \n6 Funding \n7 Competing interests \n8 References \n9 Notes \n\n\n\nAbstract \nBackground: A hospital information system (HIS) is a type of health information system which is widely used in clinical settings. Determining the success rate of a HIS is an ongoing area of research since its implications are of interest for researchers, physicians, and managers. \nAim: In the present study, we develop a novel instrument to measure HIS success rate based on users\u2019 viewpoints in a teaching hospital.\nMethods: The study was conducted in Ibn-e Sina and Dr. Hejazi Psychiatry Hospital and education center in Mashhad, Iran. The instrument for data collection was a self-administered structured questionnaire based on the information systems success model (ISSM), covering seven dimensions, which includes system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefits. The verification of content validity was carried out by an expert panel. The internal consistency of dimensions was measured by Cronbach\u2019s alpha. Pearson\u2019s correlation coefficient was calculated to evaluate the significance of associations between dimensions. The HIS success rate on users\u2019 viewpoints was determined.\nResults: A total of 125 users participated in the study. The instrument was validated by an expert panel with the content validity index (CVI) at 0.85 and content validity ratio (CVR) at 0.86. The overall Cronbach\u2019s alpha value of the instrument was 0.93. The Pearson correlation coefficient showed significant positive relationships among the investigated dimensions. On average, HIS success rate in the hospital under study was 65% (CI: 64%, 67%). The dimensions of \u201cusefulness,\u201d \u201csystem quality,\u201d and \u201cnet benefits\u201d showed the highest rates of success, respectively. \nConclusion: The instrument used in this study can be adopted for HIS evaluation in future studies. In the current study, a method was developed to determine the HIS success rate based on users\u2019 viewpoints. This method allows for the comparison of HIS success rates in various hospitals. As well, our findings underscore the viewpoints of HIS users in a developing country.\nKeywords: electronic health record, evaluation, hospital information systems, information system success model, ISSM\n\nIntroduction \nHospitals have implemented health information systems to provide timely and accurate information, thus fulfilling managerial needs and improving healthcare effectiveness and efficiency at a reasonable cost.[1]\nThe hospital information system (HIS) is one of the health information systems which has been extensively utilized. There are major concerns regarding the quality of these systems; therefore, it is vital to continuously evaluate the HIS. As well, HIS implementation requires a great deal of investment. For example, the Health Information Technology for Economic and Clinical Health Act (HITECH) of 2009 mandates about $10 million for each hospital in the United States.[2] Therefore, post-implementation evaluation of HIS success is of critical importance. Determining HIS success rate is an ongoing area of research since its implications are of interest not only for researchers but also to physicians and managers. The results of the evaluation can reveal the value of a HIS and provide essential information for subsequent decisions.[3]\nThere are many existing frameworks to evaluate health information systems. The information systems success model (ISSM) is one of the most well-known and valid frameworks to assess these systems.[4] ISSM was developed by DeLone and Mclean in 1990[5] and was subsequently updated in 2003.[6] ISSM is comprised of seven dimensions, including system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefits.[6] ISSM focuses on the assessment of technological aspects of information systems based on users\u2019 viewpoints and explains the influence of technology on system use as well as users\u2019 satisfaction. The success of information systems depends on the interaction of different ISSM dimensions.[7] Some of the studies which have deployed this framework to assess HIS include works by Tilahun and Fritz[4], Sicotte et al.[8], Otieno et al.[9], Ojo[3], Messeri et al.[10], and Aggelidis et al.[11]\nIn 2014, the Iranian Ministry of Health and Medical Education mandated that public hospitals transfer patients\u2019 medical records to SEPAS (Iran\u2019s electronic health record [EHR] project). This led to a rapid growth in the implementation of a HIS in hospitals across the country. Following the implementation of EHRs and HISs in Saudi Arabia[12], Korea[2], and Taiwan[13][14], the success of these systems was evaluated. In the case of Iran, a developing country, to the best of our knowledge there have been very few studies which have evaluated the success of a HIS. The studies by Tavakoli et al.[15], Abdekhoda et al.[16], and Saghaeiannejad-Isfahani et al.[17] offer some examples to this end. Nevertheless, these studies did not collectively provide conclusive evidence on the success of HISs. As such, there is a need for additional studies to gather further evidence on the effectiveness and efficiency of the HIS in Iran.\nIn the present study, we report on the development of an instrument to evaluate the rate of HIS success based on ISSM, which can be used to compare HIS success rates in different hospitals and clinical settings. Furthermore, in a case study, we investigate the success rate of HIS based on users\u2019 viewpoints in a teaching hospital in Iran.\n\nMethodology \nThis was a descriptive cross-sectional study.\n\nInstrument \nA self-administrated structured questionnaire was developed based on ISSM to collect data. The questionnaire comprised of seven dimensions, including system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefits. Each dimension encompasses several evaluation measures. The questions were designed based on the ISSM evaluation measures and dimensions.\nVerification of the content validity was carried out by an expert panel. The panel consisted of nine medical informatics experts. Expert validity was measured using a content validity index (CVI) and content validity ratio (CVR). The internal consistency of each dimension was measured by Cronbach\u2019s alpha. Pearson\u2019s correlation was calculated to evaluate the significance of associations between dimensions. \nThe final draft of the instrument consisted of two sections. The first section includes users\u2019 characteristics such as age, gender, education level, working hours per month, ICDL certification, work experience, computer experience, and HIS experience (n=8 questions). The second section includes 60 questions on a five-point Likert scale ranging from 1 (completely disagree) to 5 (completely agree), which covers 16 evaluation measures in seven dimensions (Table 1).\n\n\n\n\n\n\n\nTable 1. Dimensions and evaluation measures\n\n\nDimensions\n\nEvaluation Measure\r\n(number of questions)\n\nDefinition\n\n\nSystem quality\n\nAdaptability (n=1)\n\n\u201cData from different sources can be consolidated or compared without inconsistencies.\u201d[18]\n\n\nReliability (n=1)\n\n\u201cThe dependability and consistency of access and uptime of systems\u201d[18][19]\n\n\nAvailability (n=6)\n\nAccess to the computer resource and infrastructure where and when it is needed\n\n\nUsability (n=6)\n\n\u201cThe capability in human functional terms to be used easily and effectively by the specified range of users, given specified training and user support, to fulfill the specified range of tasks, within the specified range of environmental scenarios\u201d[20]\n\n\nInformation quality\n\nSecurity (n=3)\n\n\u201cThe safeguarding of data from misappropriation or unauthorized alteration or loss\u201d[21]\n\n\nEase of understanding (n=3)\n\n\u201cEase of determining what a data element on a report or file means, or what is excluded or included in calculating it\u201d[18]\n\n\nCompleteness (n=2)\n\n\u201cThe comprehensive of the output information content\u201d[21]\n\n\nPersonalization (n=1)\n\nAbility to personalize HIS for users\n\n\nRelevance (n=1)\n\n\u201cThe degree of congruence between what the user wants or requires and what is provided by the information products and services\u201d[21]\n\n\nService quality\n\nResponsiveness (n=4)\n\n\"Willingness to help customers and provide prompt service\u201d[22]\n\n\nAssurance (n=4)\n\n\u201cKnowledge and courtesy of employees and their ability to inspire trust and confidence\u201d[22]\n\n\nEmpathy (n=4)\n\n\"Caring individualized attention the service provider gives\u201d[22]\n\n\nSatisfaction\n\nEvaluation (Measure was dimension) (n=4)\n\n\u201cThe level of overall user\u2019s satisfaction with HIS\u201d[19]\n\n\nSystem use\n\nEvaluation (Measure was equal to dimension) (n=1)\n\nThe number of hours that users use the HIS\n\n\nSystem usefulness\n\nEvaluation (Measure was equal to dimension) (n=4)\n\n\"The degree to which user believes that using the new system would improve his\/her job performance\u201d[23]\n\n\nNet benefits\n\nEvaluation (Measure was equal to dimension) (n=18)\n\n\"As the 'impacts' of HIS have evolved beyond the immediate user, researchers have suggested additional IS impact measures, such as workgroup impacts, inter-organizational and industry impacts, consumer impacts, and societal impacts.\u201d[6]\n\n\n\nStudy settings \nThe study was conducted in Ibn-e Sina and Dr. Hejazi Psychiatry Hospital and education center. The case hospital was a 900-bed teaching hospital, which has been in operation for more than 40 years. The hospital is the largest psychiatric hospital in Northeastern Iran. Patients from five neighboring provinces (with an approximate population of eleven million) are referred to this case hospital for professional psychological services. The hospital consists of 16 specialized and sub-specialized departments such as psychiatric emergency, pediatric psychiatry, geriatric psychiatry, drug abuse treatment, and adult psychiatry. Average bed occupancy and bed turnover ratio is 1.01 and 75%, respectively. The length of stay in the hospital is nine days on average.\nIn 2002, Mashhad University of Medical Sciences implemented a customized HIS named IHIS (Iranian Hospital Information System). Currently, IHIS is implemented in all hospitals affiliated with the university. The IHIS installation of the case hospital covers information systems (IS) for different services, including admission, discharge and transfer, inpatient, outpatient, pharmacy, laboratory, radiology, accounting, and insurance.\n\nParticipants \nTarget respondents were users in four departments, including psychiatric departments, administrative and financial departments, nursing management departments, and para-clinics departments. Given the high number of nurses in the psychiatric departments (n=190), a sample size of 64 participants for nurses was calculated. As well, seven secretaries of the psychiatric departments also participated in the study, making a total of 71 participants from the psychiatric department. All users in other departments (administrative and financial departments, nursing management department, and para-clinics departments) were asked to participate in the current study, of whom a total of 61 participants agreed to participate. Thus, the total number of participants for the study was 125 individuals. Empirical data were collected targeting users with over six months of HIS working experience.\nThe study was approved by the ethical committee of Mashhad University of Medical Sciences before the instrument was officially distributed to all users to protect the rights and privacy of the participants. \nThe researchers met all users in person and invited them to participate in the study. Questionnaires were provided to users who agreed to participate in the study.\n\nStatistics \nSummary statistics for characteristics of participants were calculated as frequencies and proportions. The negative responses were reversed. HIS success rate in the case hospital was calculated in two steps. First, the rate of HIS success by each user in evaluation measures, dimensions, and total dimensions of the instrument was determined. This was performed using the following formula, keeping in mind that five was the maximum score for each question on a 1-5 point Likert scale:\n\n Maximum HIS success rate of evaluation measures, dimensions, and total dimensions by each user = number of questions * 5\n Acquired HIS success rate of evaluation measures, dimensions, and total dimensions by each user = sum of the acquired score for each question on a 1-5 point Likert scale by each user\n HIS success rate = (Acquired HIS success rate\/Maximum HIS success rate) * 100\nIn the following paragraphs, a sample of conducted calculations will be explained. The \u201cAvailability\u201d evaluation measure belonged to the system quality dimension. This evaluation measure has six related questions, and users responded to each question on a 1-5 point Likert scale. The maximum possible score for \u201cAvailability\u201d by each user was equal to 1 (one user) * 6 (the number of questions) * 5 (the maximum score for each question on a 1-5 point Likert scale) = 30. The HIS success rate for \u201cAvailability\u201d by \u201cuser one\u201d was 27. Therefore, the HIS success rate for \u201cAvailability\u201d by \u201cuser one\u201d = (27\/30)*100 = 90%.\nThe dimension of \u201cSystem quality\u201d consisted of 14 questions. The maximum possible score for system quality by each user was equal to 1 (one user) * 14 (the number of questions) * 5 (the maximum score for each question) = 70. The HIS success rate of \"System quality\" by \u201cuser one\u201d was 51. Therefore, the HIS success rate for system quality by \u201cuser one\u201d = (51\/70)*100 = 72.9%.\nCombining all dimensions of the instrument, we find a total of 61 questions. Accordingly, the maximum possible score of all dimensions by each user was equal to 1 (one user) * 60 (the number of questions) * 5 (the maximum score for each question) = 300. The HIS success rate for total dimensions of the instrument by \u201cuser one\u201d was 205. Therefore, the HIS success rate for total dimensions of the instrument by \u201cuser one\u201d = (205\/300)*100 = 68.3%.\nIn the second step, the mean and confidence interval of the HIS success rate were determined for evaluation measures, dimensions, and total dimensions of the instrument by four groups of users. These groups include para-clinics departments, administrative and financial departments, nursing management department, and psychiatric departments. An assessment of the normality of data is a necessity for many statistical tests. Hence, to compare HIS success rate among four user groups, the normality of data in each ISSM dimension was assessed. Some researchers suggested the Shapiro-Wilk test as the best option for testing the normality of data.[24] In our study, the Shapiro-Wilk test was used to assess the normality of data. In the current study, comparisons of the HIS success rate were made among four user groups using ANOVA for normal variables and the Kruskal-Wallis test for non-normal variables. A p-value of less than 0.05 was considered statistically significant.\nIn this study, two post-hoc tests were chosen to identify significant inter-group differences at p \u2265 0.05. The Tukey honestly significant difference (HSD) test for normal variables and the Mann\u2013Whitney test for non-normal variables was conducted. Data analysis was performed using SPSS, version 11.5 statistical software.\n\nResults \nParticipants \nIn this study, the researchers met with 191 target users, among which 125 individuals agreed to participate in the study. Table 2 demonstrates the characteristics of the participants. Over two-thirds of the participants were women. The age range of users was 22 to 55 years, and most of the participants were aged 20-30. Only 2% of participants were without a university education. More than 77% of the participants had a two- or four-year university degree. 61% of participants had 5-15 years of work experience. 68% of the users had an ICDL certificate. 80% of the users had over three years of HIS experience, and 68% of them had more than six years of computer experience.\n\n\n\n\n\n\n\nTable 2. Participants' characteristics\n\n\nUser's characteristics\n\nSubgroups\n\nFrequency (%)\n\n\nSex\n\nMale\n\n48 (38.4)\n\n\nFemale\n\n77 (61.6)\n\n\nAge (years)\n\n20\u201329\n\n20 (16.0)\n\n\n30\u201339\n\n73 (58.4)\n\n\n40\u201349\n\n30 (24.0)\n\n\n\u2265 50\n\n2 (1.6)\n\n\nEducation level\n\nHigh school diploma\n\n2 (1.6)\n\n\nAssociate's\n\n2 (1.6)\n\n\nBachelor's\n\n94 (75.2)\n\n\nMaster's\n\n27 (21.6)\n\n\nWorking hours per month\n\n< Regular + 20 overtime hours\n\n41 (20.8)\n\n\nRegular + 20\u201360 overtime hours\n\n44 (35.2)\n\n\nRegular + 60\u2013120 overtime hours\n\n30 (24.0)\n\n\n> Regular + 120 overtime hours\n\n10 (8.0)\n\n\nWork experience (years)\n\n< 5\n\n23 (18.4)\n\n\n5\u20139\n\n35 (28.0)\n\n\n10\u201314\n\n41 (32.8)\n\n\n15\u201319\n\n13 (10.4)\n\n\n20\u201324\n\n9 (7.2)\n\n\n\u2265 25\n\n4 (3.2)\n\n\nICDL certification\n\nYes\n\n97 (77.6)\n\n\nNo\n\n28 (22.4)\n\n\nComputer experience (years)\n\n< 1\n\n15 (12.0)\n\n\n1\u20133\n\n9 (7.2)\n\n\n> 3\n\n101 (80.8)\n\n\nHIS experience (years)\n\n< 1\n\n3 (2.4)\n\n\n1\u20133\n\n10 (8.0)\n\n\n4\u20136\n\n14 (11.2)\n\n\n> 6\n\n98 (78.4)\n\n\n\nReliability and validity \nThe instrument was validated by an expert panel with CVI at 0.85 and CVR at 0.86. The overall Cronbach\u2019s alpha value of the instrument was determined as 0.93, representing high reliability. This value was between 0.476 and 0.943 for different instrument dimensions, as shown in Table 3. The value of Cronbach\u2019s alpha was satisfactory among the five dimensions of system quality, information quality, service quality, usefulness, and net benefits. The value of Cronbach\u2019s alpha for the satisfaction dimension was low, and its value was 47.0. Because the dimension of \u201csystem use\u201d had just one question, its Cronbach\u2019s alpha was not calculated. The Pearson correlation coefficient showed significant positive relationships among the investigated dimensions (p-value: 0.01). Pearson correlation coefficient was between 0.197 and 0.707 for different dimensions (Table 3).\n\n\n\n\n\n\n\nTable 3. Correlations among all dimensions (n = 125)\r\n* Correlation is significant at the 0.05 significance level (2-tailed)\r\n** Correlation is significant at the 0.01 significance level (2-tailed)\n\n\nDimensions (Cronbach\u2019s alpha)\n\nSystem quality\n\nInformation quality\n\nService quality\n\nSystem use\n\nSatisfaction\n\nUsefulness\n\nNet benefits\n\n\nSystem quality (0.719)\n\n1\n\n0.401**\n\n0.255**\n\n0.450**\n\n0.319**\n\n0.270**\n\n0.348**\n\n\nInformation quality (0.611)\n\n\n\n1\n\n0.454**\n\n0.315**\n\n0.471**\n\n0.495**\n\n0.454**\n\n\nService quality (0.785)\n\n\n\n\n\n1\n\n0.197*\n\n0.329**\n\n0.285**\n\n0.352**\n\n\nSatisfaction (0.476)\n\n\n\n\n\n\n\n1\n\n0.423**\n\n0.444**\n\n0.411**\n\n\nSystem use (not calculated)\n\n\n\n\n\n\n\n\n\n1\n\n0.660**\n\n0.632**\n\n\nUsefulness (0.926)\n\n\n\n\n\n\n\n\n\n\n\n1\n\n0.707**\n\n\nNet benefits (0.943)\n\n\n\n\n\n\n\n\n\n\n\n\n\n1\n\n\n\nHIS success rate \nThe dimensions of \u201cUsefulness,\u201d \u201cSystem quality,\u201d and \u201cNet benefits\u201d obtained the highest mean rates of success, respectively. The \u201cService quality,\u201d \u201cInformation quality,\u201d and \"System use\" dimensions ranked fifth through seventh, respectively. Table 4 demonstrates the mean success rates for all HIS evaluation dimensions. These results will be discussed further in the following paragraphs.\n\n\n\n\n\n\n\nTable 4. Mean success rate of evaluation dimensions\r\nThe significant results within each group of users are indicated by letters a and b; values not sharing a common letter differ significantly (P < 0.05).\n\n\nUsers\/Dimensions\n\nMean (95% confidence interval)\n\n\nFinancial and administrative departments\n\nPsychiatric departments\n\nNursing management department\n\nPara-clinical department\n\nAll users\n\n\nSystem quality\n\n0.74 (0.71, 0.77)\n\n0.65 (0.63, 0.67)\n\n0.71 (0.65, 0.78)b\n\n0.64 (0.59, 0.70)b\n\n0.67 (0.65, 0.69)\n\n\nInformation quality\n\n0.65 (0.61, 0.69)\n\n0.62 (0.60, 0.64)\n\n0.62 (0.56, 0.68)\n\n0.62 (0.58, 0.66)\n\n0.63 (0.61, 0.64)\n\n\nService quality\n\n0.62 (0.56, 0.67)\n\n0.63 (0.61, 0.66)\n\n0.62 (0.56, 0.69)\n\n0.65 (0.62, 0.69)\n\n0.63 (0.61, 0.65)\n\n\nSystem use\n\n0.63 (0.47, 0.79)a\n\n0.30 (0.25, 0.35)b\n\n0.27 (0.16, 0.37)b\n\n0.41 (0.29, 0.54)b\n\n0.37 (0.32, 0.41)\n\n\nSatisfaction\n\n0.68 (0.63, 0.73)\n\n0.63 (0.60, 0.66)\n\n0.68 (0.61, 0.75)\n\n0.66 (0.62, 0.71)\n\n0.65 (0.63, 0.67)\n\n\nUsefulness\n\n0.78 (0.71, 0.85)\n\n0.71 (0.67, 0.75)\n\n0.77 (0.67, 0.86)\n\n0.76 (0.72, 0.80)\n\n0.74 (0.71, 0.76)\n\n\nNet benefits\n\n0.68 (0.62, 0.74)\n\n0.64 (0.61, 0.68)\n\n0.70 (0.60, 0.79)\n\n0.66 (0.61, 0.71)\n\n0.66 (0.63, 0.68)\n\n\nTotal HIS success\n\n0.69 (0.65, 0.72)\n\n0.64 (0.62, 0.66)\n\n0.67 (0.61, 0.74)\n\n0.65 (0.62, 0.69)\n\n0.65 (0.64, 0.67)\n\n\n\nHIS success rate in system quality \nThe dimension of \u201cSystem quality\u201d included four evaluation measures, with a general mean success rate of 0.67 (95% CI: 65%, 69%) from the users\u2019 perspective. Figure 1 shows the mean success rate of evaluation measures in the \u201cSystem quality\u201d dimension. Evaluation measures of \u201cAdaptability\u201d and \u201cReliability\u201d acquired a mean success rate of smaller than 60%. Moreover, evaluation measures of \"Availability\"and \"Usability\u201d acquired a mean success rate of greater than 60%.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1: The mean success rate on the dimension of \"System quality\"\n\n\n\nThe findings of this study showed that there is a significant difference in the mean success rate in the \u201cAvailability\u201d evaluation measure between para-clinical and financial and administrative departments (p < 0.001). A significant difference was observed in the psychiatric departments (p < 0.001). The mean \u201cAvailability\u201d in the financial and administrative departments, nursing management department, para-clinical, and psychiatric departments were 88%, 77%, 72%, and 68%, respectively.\nThe results of the study illustrated that ICDL certificate and the \u201cAvailability\u201d evaluation measure had a significant difference (p-value: 0.037). The \u201cUsability\u201d evaluation measure was significantly associated with education level. There was a significant difference in the mean success rate for the \u201cUsability\u201d evaluation measure between the users with a diploma and bachelor\u2019s degrees (p-value: 0.033). This difference was observed between the users with the bachelor\u2019s and master\u2019s degrees (p-value: 0.033). The mean success rate in this evaluation measure for the users with a diploma, bachelor\u2019s, and master\u2019s degrees were 78%, 66%, and 64%, respectively.\nThe significant results within each group of users are indicated by letters a and b; values not sharing a common letter differ significantly (P < 0.05).\n\nHIS success rate in information quality \nThe \u201cInformation quality\u201d dimension includes five measures, with a total of 0.63 (95% CI: 61%, 64%) mean success rate on users\u2019 viewpoints. The range of mean success rate on the evaluation measures in this dimension was 0.42 to 0.73 (Figure 2). The minimum and maximum success rates belong to \u201cHIS privatization\u201d and \u201cHIS\u2019s information relevancy,\u201d respectively. There was no significant difference in the mean success rate between the evaluation measures of this dimension and users\u2019 characteristics.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2: The mean success rate on the dimension of \"Information quality\"\n\n\n\nHIS success rate in service quality \nThe \u201cService quality\u201d dimension includes three evaluation measures. In general, this dimension acquired 0.63 (95% CI: 61%, 65%) of the mean success rate from users (Figure 3). A significant difference in the mean success rate between evaluation measures of this dimension and either users\u2019 characteristics or groups of users\u2019 was not observed.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3: The mean success rate on the dimension of \"Service quality\"\n\n\n\nHIS success rate in system use \nCompared to other dimensions, \u201cSystem use\u201d was found to have the lowest mean success rate at 0.37 (95% CI: 32%, 41%) according to users\u2019 viewpoints. A significant difference between \u201cSystem use\u201d and users\u2019 categories was observed (p < 0.001). The \u201cSystem use\u201d in the financial and administrative departments was higher than in other groups. The mean rate of system use among users of financial and administrative departments, para-clinical departments, psychiatric departments, and the nursing management department were 0.68, 0.41, 0.30, and 0.27, respectively. There was a significant difference between the ICDL certificate and system use (p < 0.001).\n\nHIS success rate in satisfaction, usefulness, and net benefits \nThe mean success rate of the \u201cSatisfaction\u201d dimension was 0.65 (95% CI: 63%, 67%). The results of the current study revealed that there is a significant difference between the mean success rate of this dimension and the number of working hours per month (p-value: 0.041). The mean success rate of \u201cSatisfaction\u201d for users with higher working hours per month was larger in comparison with other users.\nThe \u201cUsefulness\u201d dimension attained the highest mean success rate at 0.74 (95% CI: 71%, 76%) according to users\u2019 viewpoints. The \u201cNet benefits\u201d dimension acquired 0.66 (95% CI: 63%, 68%) of the mean success rate on users\u2019 viewpoints. A significant difference between the mean success rate of these dimensions and users\u2019 characteristics was not observed.\n\nTotal HIS success rate \nOn average, \u201cTotal HIS success rate\u201d in the case hospital was 0.65 (95% CI: 64%, 67%) (Figure 4). Mean total HIS success was not significantly associated with users\u2019 groups. There was no significant difference between the mean total HIS success and users\u2019 characteristics, including age, gender, education, work experience, working hours per month, ICDL certificate, computer use experience, and HIS user experience.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4: The mean total HIS success rate\n\n\n\nDiscussion \nIn the current study, an ISSM-based instrument was constructed to evaluate a HIS. The instrument was found to have a high rate of validity and reliability. This can be applied to evaluate other HISs in future studies. As well, a method was developed to determine the HIS success rate based on users\u2019 viewpoints. This method enables the users to compare HIS success rate in various hospitals. The association between the mean success rate between evaluation dimensions and user characteristics such as age, gender and users\u2019 categories was investigated. The results of the present study highlight the characteristics and opinions of HIS users in a developing country. The most important findings of the study will be discussed in the following paragraphs.\nThe ISSM has been modified and validated in healthcare settings in a variety of empirical studies.[3][4][9][12][25][26][27] The present study validates a questionnaire based on the ISSM framework in the context of hospital information systems in a teaching hospital of a developing country. The results of the current study demonstrated that there is a significant correlation between the seven dimensions of ISSM, which includes system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefit.\nThe results of the study revealed that users hold positive views on the HIS\u2019s success in the case hospital. Overall, the HIS success rate was acceptable in the evaluation measures such as usability, information security, completeness, and job relevancy. Most users viewed dimensions of system quality, system use, and satisfaction. Top et al.[28] illustrated that EHRs are useful and capable of enhancing the quality of information and the quality of workflow. They also found that nurses can easily enter, access, and read data using EHRs and can conduct their work faster. According to a study by Lambooij et al.[27], users believed that an EHR is easy to use and aligned with their work. They perceived that the quality of patient data is better when EHRs are easier to use and better aligned with their daily routine. The findings of the study by Bossen et al.[26] on EHR evaluation also reported similar results. In total, users had positive experiences with the EHR and its operational reliability, response time, login, and support. Moreover, EHR performance was acceptable. We observed a satisfactory average success rate in the usability evaluation measure. However, a significant difference between usability and education level was observed in our study. Users with a diploma, bachelor\u2019s and master\u2019s degrees had different perceptions of usability. The results of the studies by Calisir et al.[29] and Brown[30] confirm our results. The results of their study revealed that education level has a significant effect on perceived ease-of-use and usability. In 2018, a study was conducted on IHIS\u2019s usability and identified usability problems. It seems that these usability problems in IHIS influenced users\u2019 perceptions of usability with different levels of education.[31]\nIn this study, we noticed that there is a significant difference among users\u2019 groups and the availability of computer resources. Computer resources are more easily available in the financial and administrative departments compared with the other three groups. Meanwhile, a significant difference between system use and users\u2019 groups was observed. Users of financial and administrative departments use HIS more often compared with other users. Rate of HIS use in users with more abundant computer resources was higher than other users. The result of the study by Lu et al. demonstrated that the availability of computer resources is a key factor in HIS use.[14]\nThe findings of the study by Asadi et al.[32] revealed that information systems in Iran are mainly focused on financial objectives. Our findings revealed that mean system use among users of the nursing management department and psychiatric departments is generally low. The results of the study by Top et al.[28] are congruent with these findings. There might be a significant difference between the rate of system use, availability of computer resources, and users groups because managers and policymakers are focused on a HIS\u2019s financial objectives. The findings of the current study also demonstrated that nursing managers had access to more computer resources compared with users of psychiatric departments. However, no significant difference was observed on HIS use rate between these two groups. Besides, findings of a study by Tubaishat[33] showed that the professional rank of nurses does not influence the rate of HIS use.\nWe faced a couple of limitations in this study. In the present study, system use, which is a key measure in HIS evaluation, was evaluated by users\u2019 self-report. This could lead to subjective evaluation of this measure. A second limitation was that the number of computers was insufficient in the psychiatric departments, mostly running at low speed. As a result, it was often the case that one user logged into the HIS and other users used that HIS under the username of the first user. Therefore, we were unable to accurately report the true amount of system use based on HIS log records.\nIn conclusion, in the current study, an instrument based on the ISSM framework and method was developed to determine the HIS success rate based on users\u2019 viewpoints. This instrument covers seven dimensions, including system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefits. The proposed method enabled the researchers to determine the HIS success rate in accordance with these dimensions. The value of HIS success rate in each dimension is a quantitative measure. Hence, for future studies researchers are equipped with an objective measure to compare HIS success rates across a wide range of hospital settings.\nAs well, our findings underscore the characteristics and opinions of HIS users in a developing country. Our results showed that HIS in the case hospital acquired an acceptable success rate (69%) based on users\u2019 viewpoints. Of note, some dimensions such as usability require modifications and improvements.\n\nFunding \nThis research did not receive any financial support.\n\nCompeting interests \nThe authors have declared that no competing interests exist.\n\nReferences \n\n\n\u2191 Chen, R.F.; Hsiao, J.L. (2012). \"An investigation on physicians' acceptance of hospital information systems: A case study\". International Journal of Medical Informatics 81 (12): 810\u201320. doi:10.1016\/j.ijmedinf.2012.05.003. PMID 22652011.   \n\n\u2191 2.0 2.1 Iqbal, U.; Ho, C.H.; Li, Y.C. et al. (2013). \"The relationship between usage intention and adoption of electronic health records at primary care clinics\". Computer Methods and Programs in Biomedicine 112 (3): 731\u20137. doi:10.1016\/j.cmpb.2013.09.001. PMID 24091088.   \n\n\u2191 3.0 3.1 3.2 Ojo, A.I. (2017). \"Validation of the DeLone and McLean Information Systems Success Model\". Healthcare Informatics Research 23 (1): 60\u201366. doi:10.4258\/hir.2017.23.1.60. PMC PMC5334133. PMID 28261532. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5334133 .   \n\n\u2191 4.0 4.1 4.2 Tilahun, B.; Fritz, F. (2015). \"Modeling antecedents of electronic medical record system implementation success in low-resource setting hospitals\". BMC Medical Informatics and Decision Making 15: 61. doi:10.1186\/s12911-015-0192-0. PMC PMC4522063. PMID 26231051. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4522063 .   \n\n\u2191 DeLone, W.H.; McLean, E.R. (1992). \"Information Systems Success: The Quest for the Dependent Variable\". Information Systems Research 3 (1): 1\u201395. doi:10.1287\/isre.3.1.60.   \n\n\u2191 6.0 6.1 6.2 DeLone, W.H.; McLean, E.R. (2003). \"The DeLone and McLean Model of Information Systems Success: A Ten-Year Update\". Journal of Management Information Systems 19 (4): 9\u201330. doi:10.1080\/07421222.2003.11045748.   \n\n\u2191 Otieno, G.O.; Hinako, T.; Motohiro, A. et al. (2008). \"Measuring effectiveness of electronic medical records systems: Towards building a composite index for benchmarking hospitals\". International Journal of Medical Informatics 77 (10): 657\u201369. doi:10.1016\/j.ijmedinf.2008.01.002. PMID 18313352.   \n\n\u2191 Sicotte, C.; Par\u00e9, G.; Bini, K.K. et al. (2010). \"Virtual organization of hospital medical imaging: A user satisfaction survey\". Journal of Digital Imaging 23 (6): 689\u2013700. doi:10.1007\/s10278-009-9220-x. PMC PMC3046687. PMID 19588196. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687 .   \n\n\u2191 9.0 9.1 Otieno, O.G.; Toyama, H.; Asonuma, M. et al. (2007). \"Nurses' views on the use, quality and user satisfaction with electronic medical records: Questionnaire development\". Journal of Advanced Nursing 60 (2): 209\u201319. doi:10.1111\/j.1365-2648.2007.04384.x. PMID 17877568.   \n\n\u2191 Messeri, P.; Khan, S.; Millery, M. et al. (2013). \"An information systems model of the determinants of electronic health record use\". Applied Clinical Informatics 4 (2): 185\u2013200. doi:10.4338\/ACI-2013-01-RA-0005. PMC PMC3716425. PMID 23874357. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3716425 .   \n\n\u2191 Aggelidis, V.P.; Chatzoglou, P.D. (2012). \"Hospital information systems: Measuring end user computing satisfaction (EUCS)\". Journal of Biomedical Informatics 45 (3): 566\u201379. doi:10.1016\/j.jbi.2012.02.009. PMID 22426283.   \n\n\u2191 12.0 12.1 Alharthi, H.; Youssef, A.; Radwan, S. et al. (2014). \"Physician satisfaction with electronic medical records in a major Saudi Government hospital\". Journal of Taibah University Medical Sciences 9 (3): 213\u201318. doi:10.1016\/j.jtumed.2014.01.004.   \n\n\u2191 Wang, J.Y.; Ho, H.Y.; Chen, J.D. et al. (2015). \"Attitudes toward inter-hospital electronic patient record exchange: Discrepancies among physicians, medical record staff, and patients\". BMC Health Services Research 15: 264. doi:10.1186\/s12913-015-0896-y. PMC PMC4499194. PMID 26162748. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4499194 .   \n\n\u2191 14.0 14.1 Lu, C.H.; Hsiao, J.L.; Chen, R.F. (2012). \"Factors determining nurse acceptance of hospital information systems\". Computers, Informatics, Nursing 30 (5): 257\u201364. doi:10.1097\/NCN.0b013e318224b4cf. PMID 22228251.   \n\n\u2191 Tavakoli, N.; Jahanbakhsh, M.; Shahin, A. et al. (2013). \"Electronic medical record in central polyclinic of isfahan oil industry: a case study based on technology acceptance model\". Acta Informatica Medica 21 (1): 23\u20135. doi:10.5455\/AIM.2012.21.23-25. PMC PMC3610586. PMID 23572857. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3610586 .   \n\n\u2191 Abdekhoda, M.; Ahmadi, M.; Gohari, M. et al. (2015). \"The effects of organizational contextual factors on physicians' attitude toward adoption of Electronic Medical Records\". Journal of Biomedical Informatics 53: 174\u20139. doi:10.1016\/j.jbi.2014.10.008. PMID 25445481.   \n\n\u2191 Saghaeiannejad-Isfahani, S.; Jahanbakhsh, M.; Habibi, M. et al. (2014). \"A Survey on the Users' Satisfaction with the Hospital Information Systems (HISs) based on DeLone and McLean's Model in the Medical-Teaching Hospitals in Isfahan City\". Acta Informatica Medica 22 (3): 179-82. doi:10.5455\/aim.2014.22.179-182. PMC PMC4130683. PMID 25132711. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4130683 .   \n\n\u2191 18.0 18.1 18.2 Goodhue, D.L.; Thompson, R.L. (1995). \"Task-Technology Fit and Individual Performance\". MIS Quarterly 19 (2): 213\u201336. doi:10.2307\/249689.   \n\n\u2191 19.0 19.1 Garcia-Smith, D.; Effken, J.A. (2013). \"Development and initial evaluation of the Clinical Information Systems Success Model (CISSM)\". International Journal of Medical Informatics 82 (6): 539\u201352. doi:10.1016\/j.ijmedinf.2013.01.011. PMID 23497819.   \n\n\u2191 Shackel, B. (1991). \"Usability\u2014Context, framework, definition, design and evaluation\". Human Factors for Informatics Usability. Cambridge University Press. pp. 21\u201337. ISBN 0521365708.   \n\n\u2191 21.0 21.1 21.2 Bailey, J.E.; Pearson, S.W. (1983). \"Development of a Tool for Measuring and Analyzing Computer User Satisfaction\". Management Science 29 (5): 519\u2013637. doi:10.1287\/mnsc.29.5.530.   \n\n\u2191 22.0 22.1 22.2 Pitt, L.F.; Watson, R.T.; Kavan, C.B. (1995). \"Service Quality: A Measure of Information Systems Effectiveness\". MIS Quarterly 19 (2): 173\u201387. doi:10.2307\/249687.   \n\n\u2191 Venkatesh, V.; Bala, H. (2008). \"Technology Acceptance Model 3 and a Research Agenda on Interventions\". Decision Sciences 39 (2): 273\u2013315. doi:10.1111\/j.1540-5915.2008.00192.x.   \n\n\u2191 Goodhue, D.L. (1995). \"Understanding User Evaluations of Information Systems\". Management Science 41 (12): 1827\u20131977. doi:10.1287\/mnsc.41.12.1827.   \n\n\u2191 Top, M.; Gider, O. (2012). \"Nurses' views on electronic medical records (EMR) in Turkey: an analysis according to use, quality and user satisfaction\". Journal of Medical Systems 36 (3): 1979\u201388. doi:10.1007\/s10916-011-9657-6. PMID 21302133.   \n\n\u2191 26.0 26.1 Bossen, C.; Jensen, L.G.; Udsen, F.W. (2013). \"Evaluation of a comprehensive EHR based on the DeLone and McLean model for IS success: Approach, results, and success factors\". International Journal of Medical Informatics 82 (10): 940\u201353. doi:10.1016\/j.ijmedinf.2013.05.010. PMID 23827768.   \n\n\u2191 27.0 27.1 Lambooij, M.S.; Drewes, H.W.; Koster, F. (2017). \"Use of electronic medical records and quality of patient data: Different reaction patterns of doctors and nurses to the hospital organization\". BMC Medical Informatics and Decision Making 17 (1): 17. doi:10.1186\/s12911-017-0412-x. PMC PMC5303309. PMID 28187729. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5303309 .   \n\n\u2191 28.0 28.1 Top, M.; Yilmaz, A.; Karabulut, E. et al. (2015). \"Validation of a nurses' views on electronic medical record systems (EMR) questionnaire in Turkish health system\". Journal of Medical Systems 39 (6): 67. doi:10.1007\/s10916-015-0250-2. PMID 25957164.   \n\n\u2191 Calisir, F.; Altin Gumussoy, C.; Bayram, A. (2009). \"Predicting the behavioral intention to use enterprise resource planning systems: An exploratory extension of the technology acceptance model\". Management Research News 32 (7): 597\u2013613. doi:10.1108\/01409170910965215.   \n\n\u2191 Brown, I.T.J. (2002). \"Individual and Technological Factors Affecting Perceived Ease of Use of Web\u2010based Learning Technologies in a Developing Country\". EJISDC 9 (1): 1\u201315. doi:10.1002\/j.1681-4835.2002.tb00055.x.   \n\n\u2191 Ebnehoseini, Z.; Tara, M.; Meraji, M. et al. (2018). \"Usability Evaluation of an Admission, Discharge, and Transfer Information System: A Heuristic Evaluation\". Open Access Macedonian Journal of Medical Sciences 6 (11): 1941\u201345. doi:10.3889\/oamjms.2018.392. PMC PMC6290422. PMID 30559840. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6290422 .   \n\n\u2191 Asadi, F.; Moghaddasi, H.; Rabiei, R. et al. (2015). \"The Evaluation of SEPAS National Project Based on Electronic Health Record System (EHRS) Coordinates in Iran\". Acta Informatica Medica 23 (6): 369-73. doi:10.5455\/aim.2015.23.369-373. PMC PMC4720822. PMID 26862248. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720822 .   \n\n\u2191 Tubaishat, A. (2017). \"Evaluation of Electronic Health Record Implementation in Hospitals\". Computers, Informatics, Nursing 35 (7): 364\u201372. doi:10.1097\/CIN.0000000000000328. PMID 28221186.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. The original age ranges, work experience ranges, and HIS experience ranges in Table 2 overlapped inappropriately; they have been updated here to what presumably the authors intended. The original erroneously refers to Table 3 as Table 4 twice; that is updated in this version. The authors had the incorrect mean success rate in \"HIS success rate in service quality\"; it was updated to the correct number for this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\">https:\/\/www.limswiki.org\/index.php\/Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on public health informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 11 June 2019, at 23:01.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 233 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","36d4da5bfaa11917913bb8842c095eca_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Determining_the_hospital_information_system_HIS_success_rate_Development_of_a_new_instrument_and_case_study skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Determining the hospital information system (HIS) success rate: Development of a new instrument and case study<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: A <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital_information_system\" title=\"Hospital information system\" class=\"wiki-link\" data-key=\"d8385de7b1f39a39d793f8ce349b448d\">hospital information system<\/a> (HIS) is a type of health information system which is widely used in clinical settings. Determining the success rate of a HIS is an ongoing area of research since its implications are of interest for researchers, physicians, and managers. \n<\/p><p><b>Aim<\/b>: In the present study, we develop a novel instrument to measure HIS success rate based on users\u2019 viewpoints in a teaching <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a>.\n<\/p><p><b>Methods<\/b>: The study was conducted in Ibn-e Sina and Dr. Hejazi Psychiatry Hospital and education center in Mashhad, Iran. The instrument for data collection was a self-administered structured questionnaire based on the information systems success model (ISSM), covering seven dimensions, which includes system quality, <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> quality, service quality, system use, usefulness, satisfaction, and net benefits. The verification of content validity was carried out by an expert panel. The internal consistency of dimensions was measured by Cronbach\u2019s alpha. Pearson\u2019s correlation coefficient was calculated to evaluate the significance of associations between dimensions. The HIS success rate on users\u2019 viewpoints was determined.\n<\/p><p><b>Results<\/b>: A total of 125 users participated in the study. The instrument was validated by an expert panel with the content validity index (CVI) at 0.85 and content validity ratio (CVR) at 0.86. The overall Cronbach\u2019s alpha value of the instrument was 0.93. The Pearson correlation coefficient showed significant positive relationships among the investigated dimensions. On average, HIS success rate in the hospital under study was 65% (CI: 64%, 67%). The dimensions of \u201cusefulness,\u201d \u201csystem quality,\u201d and \u201cnet benefits\u201d showed the highest rates of success, respectively. \n<\/p><p><b>Conclusion<\/b>: The instrument used in this study can be adopted for HIS evaluation in future studies. In the current study, a method was developed to determine the HIS success rate based on users\u2019 viewpoints. This method allows for the comparison of HIS success rates in various hospitals. As well, our findings underscore the viewpoints of HIS users in a developing country.\n<\/p><p><b>Keywords<\/b>: electronic health record, evaluation, hospital information systems, information system success model, ISSM\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">Hospitals<\/a> have implemented health information systems to provide timely and accurate <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, thus fulfilling managerial needs and improving healthcare effectiveness and efficiency at a reasonable cost.<sup id=\"rdp-ebb-cite_ref-ChenAnInvest12_1-0\" class=\"reference\"><a href=\"#cite_note-ChenAnInvest12-1\">[1]<\/a><\/sup>\n<\/p><p>The <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital_information_system\" title=\"Hospital information system\" class=\"wiki-link\" data-key=\"d8385de7b1f39a39d793f8ce349b448d\">hospital information system<\/a> (HIS) is one of the health information systems which has been extensively utilized. There are major concerns regarding the quality of these systems; therefore, it is vital to continuously evaluate the HIS. As well, HIS implementation requires a great deal of investment. For example, the <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Information_Technology_for_Economic_and_Clinical_Health_Act\" title=\"Health Information Technology for Economic and Clinical Health Act\" class=\"wiki-link\" data-key=\"89c9e20984bbda2e628b6d1d28ec3ad5\">Health Information Technology for Economic and Clinical Health Act<\/a> (HITECH) of 2009 mandates about $10 million for each hospital in the United States.<sup id=\"rdp-ebb-cite_ref-IqbalTheRelat13_2-0\" class=\"reference\"><a href=\"#cite_note-IqbalTheRelat13-2\">[2]<\/a><\/sup> Therefore, post-implementation evaluation of HIS success is of critical importance. Determining HIS success rate is an ongoing area of research since its implications are of interest not only for researchers but also to physicians and managers. The results of the evaluation can reveal the value of a HIS and provide essential information for subsequent decisions.<sup id=\"rdp-ebb-cite_ref-OjoValid17_3-0\" class=\"reference\"><a href=\"#cite_note-OjoValid17-3\">[3]<\/a><\/sup>\n<\/p><p>There are many existing frameworks to evaluate health information systems. The information systems success model (ISSM) is one of the most well-known and valid frameworks to assess these systems.<sup id=\"rdp-ebb-cite_ref-TilahunModel15_4-0\" class=\"reference\"><a href=\"#cite_note-TilahunModel15-4\">[4]<\/a><\/sup> ISSM was developed by DeLone and Mclean in 1990<sup id=\"rdp-ebb-cite_ref-DeLoneInfo92_5-0\" class=\"reference\"><a href=\"#cite_note-DeLoneInfo92-5\">[5]<\/a><\/sup> and was subsequently updated in 2003.<sup id=\"rdp-ebb-cite_ref-DeLoneTheDeLone03_6-0\" class=\"reference\"><a href=\"#cite_note-DeLoneTheDeLone03-6\">[6]<\/a><\/sup> ISSM is comprised of seven dimensions, including system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefits.<sup id=\"rdp-ebb-cite_ref-DeLoneTheDeLone03_6-1\" class=\"reference\"><a href=\"#cite_note-DeLoneTheDeLone03-6\">[6]<\/a><\/sup> ISSM focuses on the assessment of technological aspects of information systems based on users\u2019 viewpoints and explains the influence of technology on system use as well as users\u2019 satisfaction. The success of information systems depends on the interaction of different ISSM dimensions.<sup id=\"rdp-ebb-cite_ref-OtienoMeasur08_7-0\" class=\"reference\"><a href=\"#cite_note-OtienoMeasur08-7\">[7]<\/a><\/sup> Some of the studies which have deployed this framework to assess HIS include works by Tilahun and Fritz<sup id=\"rdp-ebb-cite_ref-TilahunModel15_4-1\" class=\"reference\"><a href=\"#cite_note-TilahunModel15-4\">[4]<\/a><\/sup>, Sicotte <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-SicotteVirtual10_8-0\" class=\"reference\"><a href=\"#cite_note-SicotteVirtual10-8\">[8]<\/a><\/sup>, Otieno <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-OtienoNurses07_9-0\" class=\"reference\"><a href=\"#cite_note-OtienoNurses07-9\">[9]<\/a><\/sup>, Ojo<sup id=\"rdp-ebb-cite_ref-OjoValid17_3-1\" class=\"reference\"><a href=\"#cite_note-OjoValid17-3\">[3]<\/a><\/sup>, Messeri <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-MesseriAnInfo13_10-0\" class=\"reference\"><a href=\"#cite_note-MesseriAnInfo13-10\">[10]<\/a><\/sup>, and Aggelidis <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-AggelidisHosp12_11-0\" class=\"reference\"><a href=\"#cite_note-AggelidisHosp12-11\">[11]<\/a><\/sup>\n<\/p><p>In 2014, the Iranian Ministry of Health and Medical Education mandated that public hospitals transfer patients\u2019 medical records to SEPAS (Iran\u2019s <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a> [EHR] project). This led to a rapid growth in the implementation of a HIS in hospitals across the country. Following the implementation of EHRs and HISs in Saudi Arabia<sup id=\"rdp-ebb-cite_ref-AlharthiPhys14_12-0\" class=\"reference\"><a href=\"#cite_note-AlharthiPhys14-12\">[12]<\/a><\/sup>, Korea<sup id=\"rdp-ebb-cite_ref-IqbalTheRelat13_2-1\" class=\"reference\"><a href=\"#cite_note-IqbalTheRelat13-2\">[2]<\/a><\/sup>, and Taiwan<sup id=\"rdp-ebb-cite_ref-WangAttitudes15_13-0\" class=\"reference\"><a href=\"#cite_note-WangAttitudes15-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LuFactors12_14-0\" class=\"reference\"><a href=\"#cite_note-LuFactors12-14\">[14]<\/a><\/sup>, the success of these systems was evaluated. In the case of Iran, a developing country, to the best of our knowledge there have been very few studies which have evaluated the success of a HIS. The studies by Tavakoli <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TavakoliElectro13_15-0\" class=\"reference\"><a href=\"#cite_note-TavakoliElectro13-15\">[15]<\/a><\/sup>, Abdekhoda <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-AbdekhodaTheEff15_16-0\" class=\"reference\"><a href=\"#cite_note-AbdekhodaTheEff15-16\">[16]<\/a><\/sup>, and Saghaeiannejad-Isfahani <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-Saghaeiannejad-IsfahaniASurv14_17-0\" class=\"reference\"><a href=\"#cite_note-Saghaeiannejad-IsfahaniASurv14-17\">[17]<\/a><\/sup> offer some examples to this end. Nevertheless, these studies did not collectively provide conclusive evidence on the success of HISs. As such, there is a need for additional studies to gather further evidence on the effectiveness and efficiency of the HIS in Iran.\n<\/p><p>In the present study, we report on the development of an instrument to evaluate the rate of HIS success based on ISSM, which can be used to compare HIS success rates in different hospitals and clinical settings. Furthermore, in a case study, we investigate the success rate of HIS based on users\u2019 viewpoints in a teaching hospital in Iran.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methodology\">Methodology<\/span><\/h2>\n<p>This was a descriptive cross-sectional study.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Instrument\">Instrument<\/span><\/h3>\n<p>A self-administrated structured questionnaire was developed based on ISSM to collect data. The questionnaire comprised of seven dimensions, including system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefits. Each dimension encompasses several evaluation measures. The questions were designed based on the ISSM evaluation measures and dimensions.\n<\/p><p>Verification of the content validity was carried out by an expert panel. The panel consisted of nine medical informatics experts. Expert validity was measured using a content validity index (CVI) and content validity ratio (CVR). The internal consistency of each dimension was measured by Cronbach\u2019s alpha. Pearson\u2019s correlation was calculated to evaluate the significance of associations between dimensions. \n<\/p><p>The final draft of the instrument consisted of two sections. The first section includes users\u2019 characteristics such as age, gender, education level, working hours per month, ICDL certification, work experience, computer experience, and HIS experience (n=8 questions). The second section includes 60 questions on a five-point Likert scale ranging from 1 (completely disagree) to 5 (completely agree), which covers 16 evaluation measures in seven dimensions (Table 1).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 1.<\/b> Dimensions and evaluation measures\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Dimensions\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Evaluation Measure<br \/>(number of questions)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Definition\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">System quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Adaptability (n=1)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cData from different sources can be consolidated or compared without inconsistencies.\u201d<sup id=\"rdp-ebb-cite_ref-GoodhueTask95_18-0\" class=\"reference\"><a href=\"#cite_note-GoodhueTask95-18\">[18]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reliability (n=1)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThe dependability and consistency of access and uptime of systems\u201d<sup id=\"rdp-ebb-cite_ref-GoodhueTask95_18-1\" class=\"reference\"><a href=\"#cite_note-GoodhueTask95-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Garcia-SmithDevelop13_19-0\" class=\"reference\"><a href=\"#cite_note-Garcia-SmithDevelop13-19\">[19]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Availability (n=6)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Access to the computer resource and infrastructure where and when it is needed\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Usability (n=6)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThe capability in human functional terms to be used easily and effectively by the specified range of users, given specified training and user support, to fulfill the specified range of tasks, within the specified range of environmental scenarios\u201d<sup id=\"rdp-ebb-cite_ref-ShackelHuman91_20-0\" class=\"reference\"><a href=\"#cite_note-ShackelHuman91-20\">[20]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"5\">Information quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security (n=3)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThe safeguarding of data from misappropriation or unauthorized alteration or loss\u201d<sup id=\"rdp-ebb-cite_ref-BaileyDevelop83_21-0\" class=\"reference\"><a href=\"#cite_note-BaileyDevelop83-21\">[21]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ease of understanding (n=3)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cEase of determining what a data element on a report or file means, or what is excluded or included in calculating it\u201d<sup id=\"rdp-ebb-cite_ref-GoodhueTask95_18-2\" class=\"reference\"><a href=\"#cite_note-GoodhueTask95-18\">[18]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Completeness (n=2)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThe comprehensive of the output information content\u201d<sup id=\"rdp-ebb-cite_ref-BaileyDevelop83_21-1\" class=\"reference\"><a href=\"#cite_note-BaileyDevelop83-21\">[21]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Personalization (n=1)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ability to personalize HIS for users\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Relevance (n=1)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThe degree of congruence between what the user wants or requires and what is provided by the information products and services\u201d<sup id=\"rdp-ebb-cite_ref-BaileyDevelop83_21-2\" class=\"reference\"><a href=\"#cite_note-BaileyDevelop83-21\">[21]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">Service quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Responsiveness (n=4)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Willingness to help customers and provide prompt service\u201d<sup id=\"rdp-ebb-cite_ref-PittService95_22-0\" class=\"reference\"><a href=\"#cite_note-PittService95-22\">[22]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Assurance (n=4)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cKnowledge and courtesy of employees and their ability to inspire trust and confidence\u201d<sup id=\"rdp-ebb-cite_ref-PittService95_22-1\" class=\"reference\"><a href=\"#cite_note-PittService95-22\">[22]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Empathy (n=4)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"Caring individualized attention the service provider gives\u201d<sup id=\"rdp-ebb-cite_ref-PittService95_22-2\" class=\"reference\"><a href=\"#cite_note-PittService95-22\">[22]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Satisfaction\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluation (Measure was dimension) (n=4)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cThe level of overall user\u2019s satisfaction with HIS\u201d<sup id=\"rdp-ebb-cite_ref-Garcia-SmithDevelop13_19-1\" class=\"reference\"><a href=\"#cite_note-Garcia-SmithDevelop13-19\">[19]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluation (Measure was equal to dimension) (n=1)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The number of hours that users use the HIS\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System usefulness\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluation (Measure was equal to dimension) (n=4)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"The degree to which user believes that using the new system would improve his\/her job performance\u201d<sup id=\"rdp-ebb-cite_ref-VenkateshTech08_23-0\" class=\"reference\"><a href=\"#cite_note-VenkateshTech08-23\">[23]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Net benefits\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluation (Measure was equal to dimension) (n=18)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"As the 'impacts' of HIS have evolved beyond the immediate user, researchers have suggested additional IS impact measures, such as workgroup impacts, inter-organizational and industry impacts, consumer impacts, and societal impacts.\u201d<sup id=\"rdp-ebb-cite_ref-DeLoneTheDeLone03_6-2\" class=\"reference\"><a href=\"#cite_note-DeLoneTheDeLone03-6\">[6]<\/a><\/sup>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Study_settings\">Study settings<\/span><\/h3>\n<p>The study was conducted in Ibn-e Sina and Dr. Hejazi Psychiatry Hospital and education center. The case hospital was a 900-bed teaching hospital, which has been in operation for more than 40 years. The hospital is the largest psychiatric hospital in Northeastern Iran. Patients from five neighboring provinces (with an approximate population of eleven million) are referred to this case hospital for professional psychological services. The hospital consists of 16 specialized and sub-specialized departments such as psychiatric emergency, pediatric psychiatry, geriatric psychiatry, drug abuse treatment, and adult psychiatry. Average bed occupancy and bed turnover ratio is 1.01 and 75%, respectively. The length of stay in the hospital is nine days on average.\n<\/p><p>In 2002, Mashhad University of Medical Sciences implemented a customized HIS named IHIS (Iranian Hospital Information System). Currently, IHIS is implemented in all hospitals affiliated with the university. The IHIS installation of the case hospital covers information systems (IS) for different services, including admission, discharge and transfer, inpatient, outpatient, pharmacy, <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a>, radiology, accounting, and insurance.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Participants\">Participants<\/span><\/h3>\n<p>Target respondents were users in four departments, including psychiatric departments, administrative and financial departments, nursing management departments, and para-clinics departments. Given the high number of nurses in the psychiatric departments (n=190), a sample size of 64 participants for nurses was calculated. As well, seven secretaries of the psychiatric departments also participated in the study, making a total of 71 participants from the psychiatric department. All users in other departments (administrative and financial departments, nursing management department, and para-clinics departments) were asked to participate in the current study, of whom a total of 61 participants agreed to participate. Thus, the total number of participants for the study was 125 individuals. Empirical data were collected targeting users with over six months of HIS working experience.\n<\/p><p>The study was approved by the ethical committee of Mashhad University of Medical Sciences before the instrument was officially distributed to all users to protect the rights and privacy of the participants. \n<\/p><p>The researchers met all users in person and invited them to participate in the study. Questionnaires were provided to users who agreed to participate in the study.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Statistics\">Statistics<\/span><\/h3>\n<p>Summary statistics for characteristics of participants were calculated as frequencies and proportions. The negative responses were reversed. HIS success rate in the case hospital was calculated in two steps. First, the rate of HIS success by each user in evaluation measures, dimensions, and total dimensions of the instrument was determined. This was performed using the following formula, keeping in mind that five was the maximum score for each question on a 1-5 point Likert scale:\n<\/p>\n<dl><dd> <b>Maximum HIS success rate of evaluation measures, dimensions, and total dimensions by each user<\/b> = number of questions * 5<\/dd><\/dl>\n<dl><dd> <b>Acquired HIS success rate of evaluation measures, dimensions, and total dimensions by each user<\/b> = sum of the acquired score for each question on a 1-5 point Likert scale by each user<\/dd><\/dl>\n<dl><dd> <b>HIS success rate<\/b> = (Acquired HIS success rate\/Maximum HIS success rate) * 100<\/dd><\/dl>\n<p>In the following paragraphs, a sample of conducted calculations will be explained. The \u201cAvailability\u201d evaluation measure belonged to the system quality dimension. This evaluation measure has six related questions, and users responded to each question on a 1-5 point Likert scale. The maximum possible score for \u201cAvailability\u201d by each user was equal to 1 (one user) * 6 (the number of questions) * 5 (the maximum score for each question on a 1-5 point Likert scale) = 30. The HIS success rate for \u201cAvailability\u201d by \u201cuser one\u201d was 27. Therefore, the HIS success rate for \u201cAvailability\u201d by \u201cuser one\u201d = (27\/30)*100 = 90%.\n<\/p><p>The dimension of \u201cSystem quality\u201d consisted of 14 questions. The maximum possible score for system quality by each user was equal to 1 (one user) * 14 (the number of questions) * 5 (the maximum score for each question) = 70. The HIS success rate of \"System quality\" by \u201cuser one\u201d was 51. Therefore, the HIS success rate for system quality by \u201cuser one\u201d = (51\/70)*100 = 72.9%.\n<\/p><p>Combining all dimensions of the instrument, we find a total of 61 questions. Accordingly, the maximum possible score of all dimensions by each user was equal to 1 (one user) * 60 (the number of questions) * 5 (the maximum score for each question) = 300. The HIS success rate for total dimensions of the instrument by \u201cuser one\u201d was 205. Therefore, the HIS success rate for total dimensions of the instrument by \u201cuser one\u201d = (205\/300)*100 = 68.3%.\n<\/p><p>In the second step, the mean and confidence interval of the HIS success rate were determined for evaluation measures, dimensions, and total dimensions of the instrument by four groups of users. These groups include para-clinics departments, administrative and financial departments, nursing management department, and psychiatric departments. An assessment of the normality of data is a necessity for many statistical tests. Hence, to compare HIS success rate among four user groups, the normality of data in each ISSM dimension was assessed. Some researchers suggested the Shapiro-Wilk test as the best option for testing the normality of data.<sup id=\"rdp-ebb-cite_ref-GoodhueUnder95_24-0\" class=\"reference\"><a href=\"#cite_note-GoodhueUnder95-24\">[24]<\/a><\/sup> In our study, the Shapiro-Wilk test was used to assess the normality of data. In the current study, comparisons of the HIS success rate were made among four user groups using ANOVA for normal variables and the Kruskal-Wallis test for non-normal variables. A p-value of less than 0.05 was considered statistically significant.\n<\/p><p>In this study, two post-hoc tests were chosen to identify significant inter-group differences at p \u2265 0.05. The Tukey honestly significant difference (HSD) test for normal variables and the Mann\u2013Whitney test for non-normal variables was conducted. Data analysis was performed using SPSS, version 11.5 statistical software.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Participants_2\">Participants<\/span><\/h3>\n<p>In this study, the researchers met with 191 target users, among which 125 individuals agreed to participate in the study. Table 2 demonstrates the characteristics of the participants. Over two-thirds of the participants were women. The age range of users was 22 to 55 years, and most of the participants were aged 20-30. Only 2% of participants were without a university education. More than 77% of the participants had a two- or four-year university degree. 61% of participants had 5-15 years of work experience. 68% of the users had an ICDL certificate. 80% of the users had over three years of HIS experience, and 68% of them had more than six years of computer experience.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 2.<\/b> Participants' characteristics\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">User's characteristics\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Subgroups\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Frequency (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">Sex\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Male\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">48 (38.4)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Female\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">77 (61.6)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">Age (years)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\u201329\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20 (16.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30\u201339\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">73 (58.4)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">40\u201349\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30 (24.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2265 50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (1.6)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">Education level\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">High school diploma\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (1.6)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Associate's\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (1.6)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bachelor's\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">94 (75.2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Master's\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">27 (21.6)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">Working hours per month\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< Regular + 20 overtime hours\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">41 (20.8)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Regular + 20\u201360 overtime hours\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">44 (35.2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Regular + 60\u2013120 overtime hours\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">30 (24.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">> Regular + 120 overtime hours\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10 (8.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"6\">Work experience (years)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23 (18.4)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\u20139\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">35 (28.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\u201314\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">41 (32.8)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\u201319\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13 (10.4)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\u201324\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9 (7.2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2265 25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4 (3.2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"2\">ICDL certification\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97 (77.6)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28 (22.4)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"3\">Computer experience (years)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15 (12.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\u20133\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9 (7.2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">> 3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">101 (80.8)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"4\">HIS experience (years)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">< 1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 (2.4)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\u20133\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10 (8.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\u20136\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14 (11.2)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">> 6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98 (78.4)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Reliability_and_validity\">Reliability and validity<\/span><\/h3>\n<p>The instrument was validated by an expert panel with CVI at 0.85 and CVR at 0.86. The overall Cronbach\u2019s alpha value of the instrument was determined as 0.93, representing high reliability. This value was between 0.476 and 0.943 for different instrument dimensions, as shown in Table 3. The value of Cronbach\u2019s alpha was satisfactory among the five dimensions of system quality, information quality, service quality, usefulness, and net benefits. The value of Cronbach\u2019s alpha for the satisfaction dimension was low, and its value was 47.0. Because the dimension of \u201csystem use\u201d had just one question, its Cronbach\u2019s alpha was not calculated. The Pearson correlation coefficient showed significant positive relationships among the investigated dimensions (p-value: 0.01). Pearson correlation coefficient was between 0.197 and 0.707 for different dimensions (Table 3).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 3.<\/b> Correlations among all dimensions (n = 125)<br \/>* Correlation is significant at the 0.05 significance level (2-tailed)<br \/>** Correlation is significant at the 0.01 significance level (2-tailed)\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Dimensions (Cronbach\u2019s alpha)\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">System quality\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Information quality\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Service quality\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">System use\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Satisfaction\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Usefulness\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Net benefits\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System quality (0.719)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.401**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.255**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.450**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.319**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.270**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.348**\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Information quality (0.611)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.454**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.315**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.471**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.495**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.454**\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Service quality (0.785)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.197*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.329**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.285**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.352**\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Satisfaction (0.476)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.423**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.444**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.411**\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System use (not calculated)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.660**\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.632**\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Usefulness (0.926)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.707**\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Net benefits (0.943)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"HIS_success_rate\">HIS success rate<\/span><\/h3>\n<p>The dimensions of \u201cUsefulness,\u201d \u201cSystem quality,\u201d and \u201cNet benefits\u201d obtained the highest mean rates of success, respectively. The \u201cService quality,\u201d \u201cInformation quality,\u201d and \"System use\" dimensions ranked fifth through seventh, respectively. Table 4 demonstrates the mean success rates for all HIS evaluation dimensions. These results will be discussed further in the following paragraphs.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 4.<\/b> Mean success rate of evaluation dimensions<br \/>The significant results within each group of users are indicated by letters <sup>a<\/sup> and <sup>b<\/sup>; values not sharing a common letter differ significantly (P < 0.05).\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\" rowspan=\"2\">Users\/Dimensions\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\" colspan=\"5\">Mean (95% confidence interval)\n<\/th><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Financial and administrative departments\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Psychiatric departments\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Nursing management department\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Para-clinical department\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">All users\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.74 (0.71, 0.77)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65 (0.63, 0.67)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.71 (0.65, 0.78)<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64 (0.59, 0.70)<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.67 (0.65, 0.69)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Information quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65 (0.61, 0.69)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62 (0.60, 0.64)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62 (0.56, 0.68)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62 (0.58, 0.66)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.63 (0.61, 0.64)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Service quality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62 (0.56, 0.67)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.63 (0.61, 0.66)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.62 (0.56, 0.69)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65 (0.62, 0.69)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.63 (0.61, 0.65)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.63 (0.47, 0.79)<sup>a<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.30 (0.25, 0.35)<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.27 (0.16, 0.37)<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.41 (0.29, 0.54)<sup>b<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.37 (0.32, 0.41)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Satisfaction\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.68 (0.63, 0.73)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.63 (0.60, 0.66)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.68 (0.61, 0.75)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.66 (0.62, 0.71)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65 (0.63, 0.67)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Usefulness\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.78 (0.71, 0.85)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.71 (0.67, 0.75)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.77 (0.67, 0.86)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.76 (0.72, 0.80)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.74 (0.71, 0.76)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Net benefits\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.68 (0.62, 0.74)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64 (0.61, 0.68)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.70 (0.60, 0.79)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.66 (0.61, 0.71)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.66 (0.63, 0.68)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Total HIS success<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.69 (0.65, 0.72)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.64 (0.62, 0.66)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.67 (0.61, 0.74)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65 (0.62, 0.69)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.65 (0.64, 0.67)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"HIS_success_rate_in_system_quality\">HIS success rate in system quality<\/span><\/h4>\n<p>The dimension of \u201cSystem quality\u201d included four evaluation measures, with a general mean success rate of 0.67 (95% CI: 65%, 69%) from the users\u2019 perspective. Figure 1 shows the mean success rate of evaluation measures in the \u201cSystem quality\u201d dimension. Evaluation measures of \u201cAdaptability\u201d and \u201cReliability\u201d acquired a mean success rate of smaller than 60%. Moreover, evaluation measures of \"Availability\"and \"Usability\u201d acquired a mean success rate of greater than 60%.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" class=\"image wiki-link\" data-key=\"fcf5374b1360ec73fec24e5a045b62bd\"><img alt=\"Fig1 Ebnehoseini OAccessMacJofMedSci2019 7-9.png\" src=\"https:\/\/www.limswiki.org\/images\/f\/fe\/Fig1_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1:<\/b> The mean success rate on the dimension of \"System quality\"<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The findings of this study showed that there is a significant difference in the mean success rate in the \u201cAvailability\u201d evaluation measure between para-clinical and financial and administrative departments (p < 0.001). A significant difference was observed in the psychiatric departments (p < 0.001). The mean \u201cAvailability\u201d in the financial and administrative departments, nursing management department, para-clinical, and psychiatric departments were 88%, 77%, 72%, and 68%, respectively.\n<\/p><p>The results of the study illustrated that ICDL certificate and the \u201cAvailability\u201d evaluation measure had a significant difference (p-value: 0.037). The \u201cUsability\u201d evaluation measure was significantly associated with education level. There was a significant difference in the mean success rate for the \u201cUsability\u201d evaluation measure between the users with a diploma and bachelor\u2019s degrees (p-value: 0.033). This difference was observed between the users with the bachelor\u2019s and master\u2019s degrees (p-value: 0.033). The mean success rate in this evaluation measure for the users with a diploma, bachelor\u2019s, and master\u2019s degrees were 78%, 66%, and 64%, respectively.\n<\/p><p>The significant results within each group of users are indicated by letters <sup>a<\/sup> and <sup>b<\/sup>; values not sharing a common letter differ significantly (P < 0.05).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"HIS_success_rate_in_information_quality\">HIS success rate in information quality<\/span><\/h4>\n<p>The \u201cInformation quality\u201d dimension includes five measures, with a total of 0.63 (95% CI: 61%, 64%) mean success rate on users\u2019 viewpoints. The range of mean success rate on the evaluation measures in this dimension was 0.42 to 0.73 (Figure 2). The minimum and maximum success rates belong to \u201cHIS privatization\u201d and \u201cHIS\u2019s information relevancy,\u201d respectively. There was no significant difference in the mean success rate between the evaluation measures of this dimension and users\u2019 characteristics.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" class=\"image wiki-link\" data-key=\"c8154816c3e041e8339024500bba3715\"><img alt=\"Fig2 Ebnehoseini OAccessMacJofMedSci2019 7-9.png\" src=\"https:\/\/www.limswiki.org\/images\/1\/1d\/Fig2_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2:<\/b> The mean success rate on the dimension of \"Information quality\"<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"HIS_success_rate_in_service_quality\">HIS success rate in service quality<\/span><\/h4>\n<p>The \u201cService quality\u201d dimension includes three evaluation measures. In general, this dimension acquired 0.63 (95% CI: 61%, 65%) of the mean success rate from users (Figure 3). A significant difference in the mean success rate between evaluation measures of this dimension and either users\u2019 characteristics or groups of users\u2019 was not observed.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" class=\"image wiki-link\" data-key=\"950617472837a8dda44fbb1a1c969cae\"><img alt=\"Fig3 Ebnehoseini OAccessMacJofMedSci2019 7-9.png\" src=\"https:\/\/www.limswiki.org\/images\/b\/bf\/Fig3_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3:<\/b> The mean success rate on the dimension of \"Service quality\"<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"HIS_success_rate_in_system_use\">HIS success rate in system use<\/span><\/h4>\n<p>Compared to other dimensions, \u201cSystem use\u201d was found to have the lowest mean success rate at 0.37 (95% CI: 32%, 41%) according to users\u2019 viewpoints. A significant difference between \u201cSystem use\u201d and users\u2019 categories was observed (p < 0.001). The \u201cSystem use\u201d in the financial and administrative departments was higher than in other groups. The mean rate of system use among users of financial and administrative departments, para-clinical departments, psychiatric departments, and the nursing management department were 0.68, 0.41, 0.30, and 0.27, respectively. There was a significant difference between the ICDL certificate and system use (p < 0.001).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"HIS_success_rate_in_satisfaction.2C_usefulness.2C_and_net_benefits\">HIS success rate in satisfaction, usefulness, and net benefits<\/span><\/h4>\n<p>The mean success rate of the \u201cSatisfaction\u201d dimension was 0.65 (95% CI: 63%, 67%). The results of the current study revealed that there is a significant difference between the mean success rate of this dimension and the number of working hours per month (p-value: 0.041). The mean success rate of \u201cSatisfaction\u201d for users with higher working hours per month was larger in comparison with other users.\n<\/p><p>The \u201cUsefulness\u201d dimension attained the highest mean success rate at 0.74 (95% CI: 71%, 76%) according to users\u2019 viewpoints. The \u201cNet benefits\u201d dimension acquired 0.66 (95% CI: 63%, 68%) of the mean success rate on users\u2019 viewpoints. A significant difference between the mean success rate of these dimensions and users\u2019 characteristics was not observed.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Total_HIS_success_rate\">Total HIS success rate<\/span><\/h4>\n<p>On average, \u201cTotal HIS success rate\u201d in the case hospital was 0.65 (95% CI: 64%, 67%) (Figure 4). Mean total HIS success was not significantly associated with users\u2019 groups. There was no significant difference between the mean total HIS success and users\u2019 characteristics, including age, gender, education, work experience, working hours per month, ICDL certificate, computer use experience, and HIS user experience.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" class=\"image wiki-link\" data-key=\"58582aa544c6122328e64634a8c2f997\"><img alt=\"Fig4 Ebnehoseini OAccessMacJofMedSci2019 7-9.png\" src=\"https:\/\/www.limswiki.org\/images\/3\/34\/Fig4_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4:<\/b> The mean total HIS success rate<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>In the current study, an ISSM-based instrument was constructed to evaluate a HIS. The instrument was found to have a high rate of validity and reliability. This can be applied to evaluate other HISs in future studies. As well, a method was developed to determine the HIS success rate based on users\u2019 viewpoints. This method enables the users to compare HIS success rate in various hospitals. The association between the mean success rate between evaluation dimensions and user characteristics such as age, gender and users\u2019 categories was investigated. The results of the present study highlight the characteristics and opinions of HIS users in a developing country. The most important findings of the study will be discussed in the following paragraphs.\n<\/p><p>The ISSM has been modified and validated in healthcare settings in a variety of empirical studies.<sup id=\"rdp-ebb-cite_ref-OjoValid17_3-2\" class=\"reference\"><a href=\"#cite_note-OjoValid17-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TilahunModel15_4-2\" class=\"reference\"><a href=\"#cite_note-TilahunModel15-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OtienoNurses07_9-1\" class=\"reference\"><a href=\"#cite_note-OtienoNurses07-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlharthiPhys14_12-1\" class=\"reference\"><a href=\"#cite_note-AlharthiPhys14-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TopNurses12_25-0\" class=\"reference\"><a href=\"#cite_note-TopNurses12-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BossenEval13_26-0\" class=\"reference\"><a href=\"#cite_note-BossenEval13-26\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LambooijUseOf17_27-0\" class=\"reference\"><a href=\"#cite_note-LambooijUseOf17-27\">[27]<\/a><\/sup> The present study validates a questionnaire based on the ISSM framework in the context of hospital information systems in a teaching hospital of a developing country. The results of the current study demonstrated that there is a significant correlation between the seven dimensions of ISSM, which includes system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefit.\n<\/p><p>The results of the study revealed that users hold positive views on the HIS\u2019s success in the case hospital. Overall, the HIS success rate was acceptable in the evaluation measures such as usability, information security, completeness, and job relevancy. Most users viewed dimensions of system quality, system use, and satisfaction. Top <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TopValid15_28-0\" class=\"reference\"><a href=\"#cite_note-TopValid15-28\">[28]<\/a><\/sup> illustrated that EHRs are useful and capable of enhancing the quality of information and the quality of workflow. They also found that nurses can easily enter, access, and read data using EHRs and can conduct their work faster. According to a study by Lambooij <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-LambooijUseOf17_27-1\" class=\"reference\"><a href=\"#cite_note-LambooijUseOf17-27\">[27]<\/a><\/sup>, users believed that an EHR is easy to use and aligned with their work. They perceived that the quality of patient data is better when EHRs are easier to use and better aligned with their daily routine. The findings of the study by Bossen <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-BossenEval13_26-1\" class=\"reference\"><a href=\"#cite_note-BossenEval13-26\">[26]<\/a><\/sup> on EHR evaluation also reported similar results. In total, users had positive experiences with the EHR and its operational reliability, response time, login, and support. Moreover, EHR performance was acceptable. We observed a satisfactory average success rate in the usability evaluation measure. However, a significant difference between usability and education level was observed in our study. Users with a diploma, bachelor\u2019s and master\u2019s degrees had different perceptions of usability. The results of the studies by Calisir <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CalisirPredict09_29-0\" class=\"reference\"><a href=\"#cite_note-CalisirPredict09-29\">[29]<\/a><\/sup> and Brown<sup id=\"rdp-ebb-cite_ref-BrownIndivid17_30-0\" class=\"reference\"><a href=\"#cite_note-BrownIndivid17-30\">[30]<\/a><\/sup> confirm our results. The results of their study revealed that education level has a significant effect on perceived ease-of-use and usability. In 2018, a study was conducted on IHIS\u2019s usability and identified usability problems. It seems that these usability problems in IHIS influenced users\u2019 perceptions of usability with different levels of education.<sup id=\"rdp-ebb-cite_ref-EbnehoseiniUsability18_31-0\" class=\"reference\"><a href=\"#cite_note-EbnehoseiniUsability18-31\">[31]<\/a><\/sup>\n<\/p><p>In this study, we noticed that there is a significant difference among users\u2019 groups and the availability of computer resources. Computer resources are more easily available in the financial and administrative departments compared with the other three groups. Meanwhile, a significant difference between system use and users\u2019 groups was observed. Users of financial and administrative departments use HIS more often compared with other users. Rate of HIS use in users with more abundant computer resources was higher than other users. The result of the study by Lu <i>et al.<\/i> demonstrated that the availability of computer resources is a key factor in HIS use.<sup id=\"rdp-ebb-cite_ref-LuFactors12_14-1\" class=\"reference\"><a href=\"#cite_note-LuFactors12-14\">[14]<\/a><\/sup>\n<\/p><p>The findings of the study by Asadi <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-AsadiTheEval15_32-0\" class=\"reference\"><a href=\"#cite_note-AsadiTheEval15-32\">[32]<\/a><\/sup> revealed that information systems in Iran are mainly focused on financial objectives. Our findings revealed that mean system use among users of the nursing management department and psychiatric departments is generally low. The results of the study by Top <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-TopValid15_28-1\" class=\"reference\"><a href=\"#cite_note-TopValid15-28\">[28]<\/a><\/sup> are congruent with these findings. There might be a significant difference between the rate of system use, availability of computer resources, and users groups because managers and policymakers are focused on a HIS\u2019s financial objectives. The findings of the current study also demonstrated that nursing managers had access to more computer resources compared with users of psychiatric departments. However, no significant difference was observed on HIS use rate between these two groups. Besides, findings of a study by Tubaishat<sup id=\"rdp-ebb-cite_ref-TubaishatEval17_33-0\" class=\"reference\"><a href=\"#cite_note-TubaishatEval17-33\">[33]<\/a><\/sup> showed that the professional rank of nurses does not influence the rate of HIS use.\n<\/p><p>We faced a couple of limitations in this study. In the present study, system use, which is a key measure in HIS evaluation, was evaluated by users\u2019 self-report. This could lead to subjective evaluation of this measure. A second limitation was that the number of computers was insufficient in the psychiatric departments, mostly running at low speed. As a result, it was often the case that one user logged into the HIS and other users used that HIS under the username of the first user. Therefore, we were unable to accurately report the true amount of system use based on HIS log records.\n<\/p><p>In conclusion, in the current study, an instrument based on the ISSM framework and method was developed to determine the HIS success rate based on users\u2019 viewpoints. This instrument covers seven dimensions, including system quality, information quality, service quality, system use, usefulness, satisfaction, and net benefits. The proposed method enabled the researchers to determine the HIS success rate in accordance with these dimensions. The value of HIS success rate in each dimension is a quantitative measure. Hence, for future studies researchers are equipped with an objective measure to compare HIS success rates across a wide range of hospital settings.\n<\/p><p>As well, our findings underscore the characteristics and opinions of HIS users in a developing country. Our results showed that HIS in the case hospital acquired an acceptable success rate (69%) based on users\u2019 viewpoints. Of note, some dimensions such as usability require modifications and improvements.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h2>\n<p>This research did not receive any financial support.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors have declared that no competing interests exist.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-ChenAnInvest12-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChenAnInvest12_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chen, R.F.; Hsiao, J.L. (2012). \"An investigation on physicians' acceptance of hospital information systems: A case study\". <i>International Journal of Medical Informatics<\/i> <b>81<\/b> (12): 810\u201320. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2012.05.003\" target=\"_blank\">10.1016\/j.ijmedinf.2012.05.003<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22652011\" target=\"_blank\">22652011<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+investigation+on+physicians%27+acceptance+of+hospital+information+systems%3A+A+case+study&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Chen%2C+R.F.%3B+Hsiao%2C+J.L.&rft.au=Chen%2C+R.F.%3B+Hsiao%2C+J.L.&rft.date=2012&rft.volume=81&rft.issue=12&rft.pages=810%E2%80%9320&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2012.05.003&rft_id=info:pmid\/22652011&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IqbalTheRelat13-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-IqbalTheRelat13_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-IqbalTheRelat13_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Iqbal, U.; Ho, C.H.; Li, Y.C. et al. (2013). \"The relationship between usage intention and adoption of electronic health records at primary care clinics\". <i>Computer Methods and Programs in Biomedicine<\/i> <b>112<\/b> (3): 731\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cmpb.2013.09.001\" target=\"_blank\">10.1016\/j.cmpb.2013.09.001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24091088\" target=\"_blank\">24091088<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+relationship+between+usage+intention+and+adoption+of+electronic+health+records+at+primary+care+clinics&rft.jtitle=Computer+Methods+and+Programs+in+Biomedicine&rft.aulast=Iqbal%2C+U.%3B+Ho%2C+C.H.%3B+Li%2C+Y.C.+et+al.&rft.au=Iqbal%2C+U.%3B+Ho%2C+C.H.%3B+Li%2C+Y.C.+et+al.&rft.date=2013&rft.volume=112&rft.issue=3&rft.pages=731%E2%80%937&rft_id=info:doi\/10.1016%2Fj.cmpb.2013.09.001&rft_id=info:pmid\/24091088&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OjoValid17-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OjoValid17_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-OjoValid17_3-1\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-OjoValid17_3-2\">3.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ojo, A.I. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5334133\" target=\"_blank\">\"Validation of the DeLone and McLean Information Systems Success Model\"<\/a>. <i>Healthcare Informatics Research<\/i> <b>23<\/b> (1): 60\u201366. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4258%2Fhir.2017.23.1.60\" target=\"_blank\">10.4258\/hir.2017.23.1.60<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5334133\/\" target=\"_blank\">PMC5334133<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28261532\" target=\"_blank\">28261532<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5334133\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5334133<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Validation+of+the+DeLone+and+McLean+Information+Systems+Success+Model&rft.jtitle=Healthcare+Informatics+Research&rft.aulast=Ojo%2C+A.I.&rft.au=Ojo%2C+A.I.&rft.date=2017&rft.volume=23&rft.issue=1&rft.pages=60%E2%80%9366&rft_id=info:doi\/10.4258%2Fhir.2017.23.1.60&rft_id=info:pmc\/PMC5334133&rft_id=info:pmid\/28261532&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5334133&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TilahunModel15-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TilahunModel15_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-TilahunModel15_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-TilahunModel15_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tilahun, B.; Fritz, F. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4522063\" target=\"_blank\">\"Modeling antecedents of electronic medical record system implementation success in low-resource setting hospitals\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>15<\/b>: 61. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12911-015-0192-0\" target=\"_blank\">10.1186\/s12911-015-0192-0<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4522063\/\" target=\"_blank\">PMC4522063<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26231051\" target=\"_blank\">26231051<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4522063\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4522063<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Modeling+antecedents+of+electronic+medical+record+system+implementation+success+in+low-resource+setting+hospitals&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Tilahun%2C+B.%3B+Fritz%2C+F.&rft.au=Tilahun%2C+B.%3B+Fritz%2C+F.&rft.date=2015&rft.volume=15&rft.pages=61&rft_id=info:doi\/10.1186%2Fs12911-015-0192-0&rft_id=info:pmc\/PMC4522063&rft_id=info:pmid\/26231051&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4522063&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeLoneInfo92-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeLoneInfo92_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DeLone, W.H.; McLean, E.R. (1992). \"Information Systems Success: The Quest for the Dependent Variable\". <i>Information Systems Research<\/i> <b>3<\/b> (1): 1\u201395. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Fisre.3.1.60\" target=\"_blank\">10.1287\/isre.3.1.60<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+Systems+Success%3A+The+Quest+for+the+Dependent+Variable&rft.jtitle=Information+Systems+Research&rft.aulast=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.au=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.date=1992&rft.volume=3&rft.issue=1&rft.pages=1%E2%80%9395&rft_id=info:doi\/10.1287%2Fisre.3.1.60&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeLoneTheDeLone03-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DeLoneTheDeLone03_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-DeLoneTheDeLone03_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-DeLoneTheDeLone03_6-2\">6.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DeLone, W.H.; McLean, E.R. (2003). \"The DeLone and McLean Model of Information Systems Success: A Ten-Year Update\". <i>Journal of Management Information Systems<\/i> <b>19<\/b> (4): 9\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F07421222.2003.11045748\" target=\"_blank\">10.1080\/07421222.2003.11045748<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+DeLone+and+McLean+Model+of+Information+Systems+Success%3A+A+Ten-Year+Update&rft.jtitle=Journal+of+Management+Information+Systems&rft.aulast=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.au=DeLone%2C+W.H.%3B+McLean%2C+E.R.&rft.date=2003&rft.volume=19&rft.issue=4&rft.pages=9%E2%80%9330&rft_id=info:doi\/10.1080%2F07421222.2003.11045748&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OtienoMeasur08-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OtienoMeasur08_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Otieno, G.O.; Hinako, T.; Motohiro, A. et al. (2008). \"Measuring effectiveness of electronic medical records systems: Towards building a composite index for benchmarking hospitals\". <i>International Journal of Medical Informatics<\/i> <b>77<\/b> (10): 657\u201369. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2008.01.002\" target=\"_blank\">10.1016\/j.ijmedinf.2008.01.002<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18313352\" target=\"_blank\">18313352<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Measuring+effectiveness+of+electronic+medical+records+systems%3A+Towards+building+a+composite+index+for+benchmarking+hospitals&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Otieno%2C+G.O.%3B+Hinako%2C+T.%3B+Motohiro%2C+A.+et+al.&rft.au=Otieno%2C+G.O.%3B+Hinako%2C+T.%3B+Motohiro%2C+A.+et+al.&rft.date=2008&rft.volume=77&rft.issue=10&rft.pages=657%E2%80%9369&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2008.01.002&rft_id=info:pmid\/18313352&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SicotteVirtual10-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SicotteVirtual10_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sicotte, C.; Par\u00e9, G.; Bini, K.K. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687\" target=\"_blank\">\"Virtual organization of hospital medical imaging: A user satisfaction survey\"<\/a>. <i>Journal of Digital Imaging<\/i> <b>23<\/b> (6): 689\u2013700. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10278-009-9220-x\" target=\"_blank\">10.1007\/s10278-009-9220-x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3046687\/\" target=\"_blank\">PMC3046687<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19588196\" target=\"_blank\">19588196<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3046687<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Virtual+organization+of+hospital+medical+imaging%3A+A+user+satisfaction+survey&rft.jtitle=Journal+of+Digital+Imaging&rft.aulast=Sicotte%2C+C.%3B+Par%C3%A9%2C+G.%3B+Bini%2C+K.K.+et+al.&rft.au=Sicotte%2C+C.%3B+Par%C3%A9%2C+G.%3B+Bini%2C+K.K.+et+al.&rft.date=2010&rft.volume=23&rft.issue=6&rft.pages=689%E2%80%93700&rft_id=info:doi\/10.1007%2Fs10278-009-9220-x&rft_id=info:pmc\/PMC3046687&rft_id=info:pmid\/19588196&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3046687&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OtienoNurses07-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OtienoNurses07_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-OtienoNurses07_9-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Otieno, O.G.; Toyama, H.; Asonuma, M. et al. (2007). \"Nurses' views on the use, quality and user satisfaction with electronic medical records: Questionnaire development\". <i>Journal of Advanced Nursing<\/i> <b>60<\/b> (2): 209\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1365-2648.2007.04384.x\" target=\"_blank\">10.1111\/j.1365-2648.2007.04384.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17877568\" target=\"_blank\">17877568<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nurses%27+views+on+the+use%2C+quality+and+user+satisfaction+with+electronic+medical+records%3A+Questionnaire+development&rft.jtitle=Journal+of+Advanced+Nursing&rft.aulast=Otieno%2C+O.G.%3B+Toyama%2C+H.%3B+Asonuma%2C+M.+et+al.&rft.au=Otieno%2C+O.G.%3B+Toyama%2C+H.%3B+Asonuma%2C+M.+et+al.&rft.date=2007&rft.volume=60&rft.issue=2&rft.pages=209%E2%80%9319&rft_id=info:doi\/10.1111%2Fj.1365-2648.2007.04384.x&rft_id=info:pmid\/17877568&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MesseriAnInfo13-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MesseriAnInfo13_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Messeri, P.; Khan, S.; Millery, M. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3716425\" target=\"_blank\">\"An information systems model of the determinants of electronic health record use\"<\/a>. <i>Applied Clinical Informatics<\/i> <b>4<\/b> (2): 185\u2013200. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4338%2FACI-2013-01-RA-0005\" target=\"_blank\">10.4338\/ACI-2013-01-RA-0005<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3716425\/\" target=\"_blank\">PMC3716425<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23874357\" target=\"_blank\">23874357<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3716425\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3716425<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+information+systems+model+of+the+determinants+of+electronic+health+record+use&rft.jtitle=Applied+Clinical+Informatics&rft.aulast=Messeri%2C+P.%3B+Khan%2C+S.%3B+Millery%2C+M.+et+al.&rft.au=Messeri%2C+P.%3B+Khan%2C+S.%3B+Millery%2C+M.+et+al.&rft.date=2013&rft.volume=4&rft.issue=2&rft.pages=185%E2%80%93200&rft_id=info:doi\/10.4338%2FACI-2013-01-RA-0005&rft_id=info:pmc\/PMC3716425&rft_id=info:pmid\/23874357&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3716425&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AggelidisHosp12-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AggelidisHosp12_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aggelidis, V.P.; Chatzoglou, P.D. (2012). \"Hospital information systems: Measuring end user computing satisfaction (EUCS)\". <i>Journal of Biomedical Informatics<\/i> <b>45<\/b> (3): 566\u201379. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2012.02.009\" target=\"_blank\">10.1016\/j.jbi.2012.02.009<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22426283\" target=\"_blank\">22426283<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hospital+information+systems%3A+Measuring+end+user+computing+satisfaction+%28EUCS%29&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Aggelidis%2C+V.P.%3B+Chatzoglou%2C+P.D.&rft.au=Aggelidis%2C+V.P.%3B+Chatzoglou%2C+P.D.&rft.date=2012&rft.volume=45&rft.issue=3&rft.pages=566%E2%80%9379&rft_id=info:doi\/10.1016%2Fj.jbi.2012.02.009&rft_id=info:pmid\/22426283&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlharthiPhys14-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlharthiPhys14_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlharthiPhys14_12-1\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alharthi, H.; Youssef, A.; Radwan, S. et al. (2014). \"Physician satisfaction with electronic medical records in a major Saudi Government hospital\". <i>Journal of Taibah University Medical Sciences<\/i> <b>9<\/b> (3): 213\u201318. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jtumed.2014.01.004\" target=\"_blank\">10.1016\/j.jtumed.2014.01.004<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Physician+satisfaction+with+electronic+medical+records+in+a+major+Saudi+Government+hospital&rft.jtitle=Journal+of+Taibah+University+Medical+Sciences&rft.aulast=Alharthi%2C+H.%3B+Youssef%2C+A.%3B+Radwan%2C+S.+et+al.&rft.au=Alharthi%2C+H.%3B+Youssef%2C+A.%3B+Radwan%2C+S.+et+al.&rft.date=2014&rft.volume=9&rft.issue=3&rft.pages=213%E2%80%9318&rft_id=info:doi\/10.1016%2Fj.jtumed.2014.01.004&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangAttitudes15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangAttitudes15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, J.Y.; Ho, H.Y.; Chen, J.D. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4499194\" target=\"_blank\">\"Attitudes toward inter-hospital electronic patient record exchange: Discrepancies among physicians, medical record staff, and patients\"<\/a>. <i>BMC Health Services Research<\/i> <b>15<\/b>: 264. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12913-015-0896-y\" target=\"_blank\">10.1186\/s12913-015-0896-y<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4499194\/\" target=\"_blank\">PMC4499194<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26162748\" target=\"_blank\">26162748<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4499194\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4499194<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Attitudes+toward+inter-hospital+electronic+patient+record+exchange%3A+Discrepancies+among+physicians%2C+medical+record+staff%2C+and+patients&rft.jtitle=BMC+Health+Services+Research&rft.aulast=Wang%2C+J.Y.%3B+Ho%2C+H.Y.%3B+Chen%2C+J.D.+et+al.&rft.au=Wang%2C+J.Y.%3B+Ho%2C+H.Y.%3B+Chen%2C+J.D.+et+al.&rft.date=2015&rft.volume=15&rft.pages=264&rft_id=info:doi\/10.1186%2Fs12913-015-0896-y&rft_id=info:pmc\/PMC4499194&rft_id=info:pmid\/26162748&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4499194&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LuFactors12-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LuFactors12_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-LuFactors12_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lu, C.H.; Hsiao, J.L.; Chen, R.F. (2012). \"Factors determining nurse acceptance of hospital information systems\". <i>Computers, Informatics, Nursing<\/i> <b>30<\/b> (5): 257\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FNCN.0b013e318224b4cf\" target=\"_blank\">10.1097\/NCN.0b013e318224b4cf<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22228251\" target=\"_blank\">22228251<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Factors+determining+nurse+acceptance+of+hospital+information+systems&rft.jtitle=Computers%2C+Informatics%2C+Nursing&rft.aulast=Lu%2C+C.H.%3B+Hsiao%2C+J.L.%3B+Chen%2C+R.F.&rft.au=Lu%2C+C.H.%3B+Hsiao%2C+J.L.%3B+Chen%2C+R.F.&rft.date=2012&rft.volume=30&rft.issue=5&rft.pages=257%E2%80%9364&rft_id=info:doi\/10.1097%2FNCN.0b013e318224b4cf&rft_id=info:pmid\/22228251&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TavakoliElectro13-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TavakoliElectro13_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tavakoli, N.; Jahanbakhsh, M.; Shahin, A. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3610586\" target=\"_blank\">\"Electronic medical record in central polyclinic of isfahan oil industry: a case study based on technology acceptance model\"<\/a>. <i>Acta Informatica Medica<\/i> <b>21<\/b> (1): 23\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5455%2FAIM.2012.21.23-25\" target=\"_blank\">10.5455\/AIM.2012.21.23-25<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3610586\/\" target=\"_blank\">PMC3610586<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23572857\" target=\"_blank\">23572857<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3610586\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3610586<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+medical+record+in+central+polyclinic+of+isfahan+oil+industry%3A+a+case+study+based+on+technology+acceptance+model&rft.jtitle=Acta+Informatica+Medica&rft.aulast=Tavakoli%2C+N.%3B+Jahanbakhsh%2C+M.%3B+Shahin%2C+A.+et+al.&rft.au=Tavakoli%2C+N.%3B+Jahanbakhsh%2C+M.%3B+Shahin%2C+A.+et+al.&rft.date=2013&rft.volume=21&rft.issue=1&rft.pages=23%E2%80%935&rft_id=info:doi\/10.5455%2FAIM.2012.21.23-25&rft_id=info:pmc\/PMC3610586&rft_id=info:pmid\/23572857&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3610586&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbdekhodaTheEff15-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbdekhodaTheEff15_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abdekhoda, M.; Ahmadi, M.; Gohari, M. et al. (2015). \"The effects of organizational contextual factors on physicians' attitude toward adoption of Electronic Medical Records\". <i>Journal of Biomedical Informatics<\/i> <b>53<\/b>: 174\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2014.10.008\" target=\"_blank\">10.1016\/j.jbi.2014.10.008<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25445481\" target=\"_blank\">25445481<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+effects+of+organizational+contextual+factors+on+physicians%27+attitude+toward+adoption+of+Electronic+Medical+Records&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Abdekhoda%2C+M.%3B+Ahmadi%2C+M.%3B+Gohari%2C+M.+et+al.&rft.au=Abdekhoda%2C+M.%3B+Ahmadi%2C+M.%3B+Gohari%2C+M.+et+al.&rft.date=2015&rft.volume=53&rft.pages=174%E2%80%939&rft_id=info:doi\/10.1016%2Fj.jbi.2014.10.008&rft_id=info:pmid\/25445481&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Saghaeiannejad-IsfahaniASurv14-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Saghaeiannejad-IsfahaniASurv14_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saghaeiannejad-Isfahani, S.; Jahanbakhsh, M.; Habibi, M. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4130683\" target=\"_blank\">\"A Survey on the Users' Satisfaction with the Hospital Information Systems (HISs) based on DeLone and McLean's Model in the Medical-Teaching Hospitals in Isfahan City\"<\/a>. <i>Acta Informatica Medica<\/i> <b>22<\/b> (3): 179-82. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5455%2Faim.2014.22.179-182\" target=\"_blank\">10.5455\/aim.2014.22.179-182<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4130683\/\" target=\"_blank\">PMC4130683<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25132711\" target=\"_blank\">25132711<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4130683\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4130683<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Survey+on+the+Users%27+Satisfaction+with+the+Hospital+Information+Systems+%28HISs%29+based+on+DeLone+and+McLean%27s+Model+in+the+Medical-Teaching+Hospitals+in+Isfahan+City&rft.jtitle=Acta+Informatica+Medica&rft.aulast=Saghaeiannejad-Isfahani%2C+S.%3B+Jahanbakhsh%2C+M.%3B+Habibi%2C+M.+et+al.&rft.au=Saghaeiannejad-Isfahani%2C+S.%3B+Jahanbakhsh%2C+M.%3B+Habibi%2C+M.+et+al.&rft.date=2014&rft.volume=22&rft.issue=3&rft.pages=179-82&rft_id=info:doi\/10.5455%2Faim.2014.22.179-182&rft_id=info:pmc\/PMC4130683&rft_id=info:pmid\/25132711&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4130683&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoodhueTask95-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GoodhueTask95_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-GoodhueTask95_18-1\">18.1<\/a><\/sup> <sup><a href=\"#cite_ref-GoodhueTask95_18-2\">18.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goodhue, D.L.; Thompson, R.L. (1995). \"Task-Technology Fit and Individual Performance\". <i>MIS Quarterly<\/i> <b>19<\/b> (2): 213\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2307%2F249689\" target=\"_blank\">10.2307\/249689<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Task-Technology+Fit+and+Individual+Performance&rft.jtitle=MIS+Quarterly&rft.aulast=Goodhue%2C+D.L.%3B+Thompson%2C+R.L.&rft.au=Goodhue%2C+D.L.%3B+Thompson%2C+R.L.&rft.date=1995&rft.volume=19&rft.issue=2&rft.pages=213%E2%80%9336&rft_id=info:doi\/10.2307%2F249689&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Garcia-SmithDevelop13-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Garcia-SmithDevelop13_19-0\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-Garcia-SmithDevelop13_19-1\">19.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Garcia-Smith, D.; Effken, J.A. (2013). \"Development and initial evaluation of the Clinical Information Systems Success Model (CISSM)\". <i>International Journal of Medical Informatics<\/i> <b>82<\/b> (6): 539\u201352. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2013.01.011\" target=\"_blank\">10.1016\/j.ijmedinf.2013.01.011<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23497819\" target=\"_blank\">23497819<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+and+initial+evaluation+of+the+Clinical+Information+Systems+Success+Model+%28CISSM%29&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Garcia-Smith%2C+D.%3B+Effken%2C+J.A.&rft.au=Garcia-Smith%2C+D.%3B+Effken%2C+J.A.&rft.date=2013&rft.volume=82&rft.issue=6&rft.pages=539%E2%80%9352&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2013.01.011&rft_id=info:pmid\/23497819&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShackelHuman91-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShackelHuman91_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Shackel, B. (1991). \"Usability\u2014Context, framework, definition, design and evaluation\". <i>Human Factors for Informatics Usability<\/i>. Cambridge University Press. pp. 21\u201337. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 0521365708.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Usability%E2%80%94Context%2C+framework%2C+definition%2C+design+and+evaluation&rft.atitle=Human+Factors+for+Informatics+Usability&rft.aulast=Shackel%2C+B.&rft.au=Shackel%2C+B.&rft.date=1991&rft.pages=pp.%26nbsp%3B21%E2%80%9337&rft.pub=Cambridge+University+Press&rft.isbn=0521365708&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaileyDevelop83-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BaileyDevelop83_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-BaileyDevelop83_21-1\">21.1<\/a><\/sup> <sup><a href=\"#cite_ref-BaileyDevelop83_21-2\">21.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bailey, J.E.; Pearson, S.W. (1983). \"Development of a Tool for Measuring and Analyzing Computer User Satisfaction\". <i>Management Science<\/i> <b>29<\/b> (5): 519\u2013637. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Fmnsc.29.5.530\" target=\"_blank\">10.1287\/mnsc.29.5.530<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+of+a+Tool+for+Measuring+and+Analyzing+Computer+User+Satisfaction&rft.jtitle=Management+Science&rft.aulast=Bailey%2C+J.E.%3B+Pearson%2C+S.W.&rft.au=Bailey%2C+J.E.%3B+Pearson%2C+S.W.&rft.date=1983&rft.volume=29&rft.issue=5&rft.pages=519%E2%80%93637&rft_id=info:doi\/10.1287%2Fmnsc.29.5.530&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PittService95-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PittService95_22-0\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-PittService95_22-1\">22.1<\/a><\/sup> <sup><a href=\"#cite_ref-PittService95_22-2\">22.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pitt, L.F.; Watson, R.T.; Kavan, C.B. (1995). \"Service Quality: A Measure of Information Systems Effectiveness\". <i>MIS Quarterly<\/i> <b>19<\/b> (2): 173\u201387. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2307%2F249687\" target=\"_blank\">10.2307\/249687<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Service+Quality%3A+A+Measure+of+Information+Systems+Effectiveness&rft.jtitle=MIS+Quarterly&rft.aulast=Pitt%2C+L.F.%3B+Watson%2C+R.T.%3B+Kavan%2C+C.B.&rft.au=Pitt%2C+L.F.%3B+Watson%2C+R.T.%3B+Kavan%2C+C.B.&rft.date=1995&rft.volume=19&rft.issue=2&rft.pages=173%E2%80%9387&rft_id=info:doi\/10.2307%2F249687&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VenkateshTech08-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VenkateshTech08_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Venkatesh, V.; Bala, H. (2008). \"Technology Acceptance Model 3 and a Research Agenda on Interventions\". <i>Decision Sciences<\/i> <b>39<\/b> (2): 273\u2013315. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1540-5915.2008.00192.x\" target=\"_blank\">10.1111\/j.1540-5915.2008.00192.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Technology+Acceptance+Model+3+and+a+Research+Agenda+on+Interventions&rft.jtitle=Decision+Sciences&rft.aulast=Venkatesh%2C+V.%3B+Bala%2C+H.&rft.au=Venkatesh%2C+V.%3B+Bala%2C+H.&rft.date=2008&rft.volume=39&rft.issue=2&rft.pages=273%E2%80%93315&rft_id=info:doi\/10.1111%2Fj.1540-5915.2008.00192.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoodhueUnder95-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoodhueUnder95_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goodhue, D.L. (1995). \"Understanding User Evaluations of Information Systems\". <i>Management Science<\/i> <b>41<\/b> (12): 1827\u20131977. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Fmnsc.41.12.1827\" target=\"_blank\">10.1287\/mnsc.41.12.1827<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+User+Evaluations+of+Information+Systems&rft.jtitle=Management+Science&rft.aulast=Goodhue%2C+D.L.&rft.au=Goodhue%2C+D.L.&rft.date=1995&rft.volume=41&rft.issue=12&rft.pages=1827%E2%80%931977&rft_id=info:doi\/10.1287%2Fmnsc.41.12.1827&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TopNurses12-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TopNurses12_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Top, M.; Gider, O. (2012). \"Nurses' views on electronic medical records (EMR) in Turkey: an analysis according to use, quality and user satisfaction\". <i>Journal of Medical Systems<\/i> <b>36<\/b> (3): 1979\u201388. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10916-011-9657-6\" target=\"_blank\">10.1007\/s10916-011-9657-6<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21302133\" target=\"_blank\">21302133<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nurses%27+views+on+electronic+medical+records+%28EMR%29+in+Turkey%3A+an+analysis+according+to+use%2C+quality+and+user+satisfaction&rft.jtitle=Journal+of+Medical+Systems&rft.aulast=Top%2C+M.%3B+Gider%2C+O.&rft.au=Top%2C+M.%3B+Gider%2C+O.&rft.date=2012&rft.volume=36&rft.issue=3&rft.pages=1979%E2%80%9388&rft_id=info:doi\/10.1007%2Fs10916-011-9657-6&rft_id=info:pmid\/21302133&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BossenEval13-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BossenEval13_26-0\">26.0<\/a><\/sup> <sup><a href=\"#cite_ref-BossenEval13_26-1\">26.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bossen, C.; Jensen, L.G.; Udsen, F.W. (2013). \"Evaluation of a comprehensive EHR based on the DeLone and McLean model for IS success: Approach, results, and success factors\". <i>International Journal of Medical Informatics<\/i> <b>82<\/b> (10): 940\u201353. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2013.05.010\" target=\"_blank\">10.1016\/j.ijmedinf.2013.05.010<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23827768\" target=\"_blank\">23827768<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+a+comprehensive+EHR+based+on+the+DeLone+and+McLean+model+for+IS+success%3A+Approach%2C+results%2C+and+success+factors&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Bossen%2C+C.%3B+Jensen%2C+L.G.%3B+Udsen%2C+F.W.&rft.au=Bossen%2C+C.%3B+Jensen%2C+L.G.%3B+Udsen%2C+F.W.&rft.date=2013&rft.volume=82&rft.issue=10&rft.pages=940%E2%80%9353&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2013.05.010&rft_id=info:pmid\/23827768&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LambooijUseOf17-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LambooijUseOf17_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-LambooijUseOf17_27-1\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lambooij, M.S.; Drewes, H.W.; Koster, F. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5303309\" target=\"_blank\">\"Use of electronic medical records and quality of patient data: Different reaction patterns of doctors and nurses to the hospital organization\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>17<\/b> (1): 17. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12911-017-0412-x\" target=\"_blank\">10.1186\/s12911-017-0412-x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5303309\/\" target=\"_blank\">PMC5303309<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28187729\" target=\"_blank\">28187729<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5303309\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5303309<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+electronic+medical+records+and+quality+of+patient+data%3A+Different+reaction+patterns+of+doctors+and+nurses+to+the+hospital+organization&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Lambooij%2C+M.S.%3B+Drewes%2C+H.W.%3B+Koster%2C+F.&rft.au=Lambooij%2C+M.S.%3B+Drewes%2C+H.W.%3B+Koster%2C+F.&rft.date=2017&rft.volume=17&rft.issue=1&rft.pages=17&rft_id=info:doi\/10.1186%2Fs12911-017-0412-x&rft_id=info:pmc\/PMC5303309&rft_id=info:pmid\/28187729&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5303309&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TopValid15-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TopValid15_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-TopValid15_28-1\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Top, M.; Yilmaz, A.; Karabulut, E. et al. (2015). \"Validation of a nurses' views on electronic medical record systems (EMR) questionnaire in Turkish health system\". <i>Journal of Medical Systems<\/i> <b>39<\/b> (6): 67. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10916-015-0250-2\" target=\"_blank\">10.1007\/s10916-015-0250-2<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25957164\" target=\"_blank\">25957164<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Validation+of+a+nurses%27+views+on+electronic+medical+record+systems+%28EMR%29+questionnaire+in+Turkish+health+system&rft.jtitle=Journal+of+Medical+Systems&rft.aulast=Top%2C+M.%3B+Yilmaz%2C+A.%3B+Karabulut%2C+E.+et+al.&rft.au=Top%2C+M.%3B+Yilmaz%2C+A.%3B+Karabulut%2C+E.+et+al.&rft.date=2015&rft.volume=39&rft.issue=6&rft.pages=67&rft_id=info:doi\/10.1007%2Fs10916-015-0250-2&rft_id=info:pmid\/25957164&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CalisirPredict09-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CalisirPredict09_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Calisir, F.; Altin Gumussoy, C.; Bayram, A. (2009). \"Predicting the behavioral intention to use enterprise resource planning systems: An exploratory extension of the technology acceptance model\". <i>Management Research News<\/i> <b>32<\/b> (7): 597\u2013613. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1108%2F01409170910965215\" target=\"_blank\">10.1108\/01409170910965215<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Predicting+the+behavioral+intention+to+use+enterprise+resource+planning+systems%3A+An+exploratory+extension+of+the+technology+acceptance+model&rft.jtitle=Management+Research+News&rft.aulast=Calisir%2C+F.%3B+Altin+Gumussoy%2C+C.%3B+Bayram%2C+A.&rft.au=Calisir%2C+F.%3B+Altin+Gumussoy%2C+C.%3B+Bayram%2C+A.&rft.date=2009&rft.volume=32&rft.issue=7&rft.pages=597%E2%80%93613&rft_id=info:doi\/10.1108%2F01409170910965215&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrownIndivid17-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrownIndivid17_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brown, I.T.J. (2002). \"Individual and Technological Factors Affecting Perceived Ease of Use of Web\u2010based Learning Technologies in a Developing Country\". <i>EJISDC<\/i> <b>9<\/b> (1): 1\u201315. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fj.1681-4835.2002.tb00055.x\" target=\"_blank\">10.1002\/j.1681-4835.2002.tb00055.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Individual+and+Technological+Factors+Affecting+Perceived+Ease+of+Use+of+Web%E2%80%90based+Learning+Technologies+in+a+Developing+Country&rft.jtitle=EJISDC&rft.aulast=Brown%2C+I.T.J.&rft.au=Brown%2C+I.T.J.&rft.date=2002&rft.volume=9&rft.issue=1&rft.pages=1%E2%80%9315&rft_id=info:doi\/10.1002%2Fj.1681-4835.2002.tb00055.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EbnehoseiniUsability18-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EbnehoseiniUsability18_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ebnehoseini, Z.; Tara, M.; Meraji, M. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6290422\" target=\"_blank\">\"Usability Evaluation of an Admission, Discharge, and Transfer Information System: A Heuristic Evaluation\"<\/a>. <i>Open Access Macedonian Journal of Medical Sciences<\/i> <b>6<\/b> (11): 1941\u201345. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3889%2Foamjms.2018.392\" target=\"_blank\">10.3889\/oamjms.2018.392<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6290422\/\" target=\"_blank\">PMC6290422<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30559840\" target=\"_blank\">30559840<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6290422\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6290422<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Usability+Evaluation+of+an+Admission%2C+Discharge%2C+and+Transfer+Information+System%3A+A+Heuristic+Evaluation&rft.jtitle=Open+Access+Macedonian+Journal+of+Medical+Sciences&rft.aulast=Ebnehoseini%2C+Z.%3B+Tara%2C+M.%3B+Meraji%2C+M.+et+al.&rft.au=Ebnehoseini%2C+Z.%3B+Tara%2C+M.%3B+Meraji%2C+M.+et+al.&rft.date=2018&rft.volume=6&rft.issue=11&rft.pages=1941%E2%80%9345&rft_id=info:doi\/10.3889%2Foamjms.2018.392&rft_id=info:pmc\/PMC6290422&rft_id=info:pmid\/30559840&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6290422&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AsadiTheEval15-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AsadiTheEval15_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Asadi, F.; Moghaddasi, H.; Rabiei, R. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720822\" target=\"_blank\">\"The Evaluation of SEPAS National Project Based on Electronic Health Record System (EHRS) Coordinates in Iran\"<\/a>. <i>Acta Informatica Medica<\/i> <b>23<\/b> (6): 369-73. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5455%2Faim.2015.23.369-373\" target=\"_blank\">10.5455\/aim.2015.23.369-373<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4720822\/\" target=\"_blank\">PMC4720822<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26862248\" target=\"_blank\">26862248<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720822\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4720822<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Evaluation+of+SEPAS+National+Project+Based+on+Electronic+Health+Record+System+%28EHRS%29+Coordinates+in+Iran&rft.jtitle=Acta+Informatica+Medica&rft.aulast=Asadi%2C+F.%3B+Moghaddasi%2C+H.%3B+Rabiei%2C+R.+et+al.&rft.au=Asadi%2C+F.%3B+Moghaddasi%2C+H.%3B+Rabiei%2C+R.+et+al.&rft.date=2015&rft.volume=23&rft.issue=6&rft.pages=369-73&rft_id=info:doi\/10.5455%2Faim.2015.23.369-373&rft_id=info:pmc\/PMC4720822&rft_id=info:pmid\/26862248&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4720822&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TubaishatEval17-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TubaishatEval17_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tubaishat, A. (2017). \"Evaluation of Electronic Health Record Implementation in Hospitals\". <i>Computers, Informatics, Nursing<\/i> <b>35<\/b> (7): 364\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FCIN.0000000000000328\" target=\"_blank\">10.1097\/CIN.0000000000000328<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28221186\" target=\"_blank\">28221186<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+Electronic+Health+Record+Implementation+in+Hospitals&rft.jtitle=Computers%2C+Informatics%2C+Nursing&rft.aulast=Tubaishat%2C+A.&rft.au=Tubaishat%2C+A.&rft.date=2017&rft.volume=35&rft.issue=7&rft.pages=364%E2%80%9372&rft_id=info:doi\/10.1097%2FCIN.0000000000000328&rft_id=info:pmid\/28221186&rfr_id=info:sid\/en.wikipedia.org:Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference. The original age ranges, work experience ranges, and HIS experience ranges in Table 2 overlapped inappropriately; they have been updated here to what presumably the authors intended. The original erroneously refers to Table 3 as Table 4 twice; that is updated in this version. The authors had the incorrect mean success rate in \"HIS success rate in service quality\"; it was updated to the correct number for this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165457\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.873 seconds\nReal time usage: 0.910 seconds\nPreprocessor visited node count: 27312\/1000000\nPreprocessor generated node count: 36343\/1000000\nPost\u2010expand include size: 227712\/2097152 bytes\nTemplate argument size: 72557\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 853.720 1 - -total\n 86.17% 735.635 1 - Template:Reflist\n 77.16% 658.771 32 - Template:Cite_journal\n 76.08% 649.472 33 - Template:Citation\/core\n 10.52% 89.807 65 - Template:Citation\/identifier\n 7.79% 66.492 1 - Template:Infobox_journal_article\n 7.46% 63.653 1 - Template:Infobox\n 4.47% 38.182 80 - Template:Infobox\/row\n 4.06% 34.694 34 - Template:Citation\/make_link\n 3.93% 33.578 140 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11069-0!*!0!!en!5!* and timestamp 20190701165456 and revision id 35876\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study\">https:\/\/www.limswiki.org\/index.php\/Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","36d4da5bfaa11917913bb8842c095eca_images":["https:\/\/www.limswiki.org\/images\/f\/fe\/Fig1_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png","https:\/\/www.limswiki.org\/images\/1\/1d\/Fig2_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png","https:\/\/www.limswiki.org\/images\/b\/bf\/Fig3_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png","https:\/\/www.limswiki.org\/images\/3\/34\/Fig4_Ebnehoseini_OAccessMacJofMedSci2019_7-9.png"],"36d4da5bfaa11917913bb8842c095eca_timestamp":1562000096,"c2dd7f033df4bcb56d74373a750e4a88_type":"article","c2dd7f033df4bcb56d74373a750e4a88_title":"Laboratory information management software for engineered mini-protein therapeutic workflow (Brusniak et al. 2019)","c2dd7f033df4bcb56d74373a750e4a88_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow","c2dd7f033df4bcb56d74373a750e4a88_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Laboratory information management software for engineered mini-protein therapeutic workflow\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nLaboratory information management software for engineered mini-protein therapeutic workflowJournal\n \nBMC BioinformaticsAuthor(s)\n \nBrusniak, Mi-Youn; Ramos, Hector; Lee, Bernard; Olson, James M.Author affiliation(s)\n \nFred Hutchinson Cancer Research CenterPrimary contact\n \nEmail: mbrusnia at fredhutch dot orgYear published\n \n2019Volume and issue\n \n20Page(s)\n \n343DOI\n \n10.1186\/s12859-019-2935-xISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-019-2935-xDownload\n \nhttps:\/\/bmcbioinformatics.biomedcentral.com\/track\/pdf\/10.1186\/s12859-019-2935-x (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Implementation \n\n3.1 Protein engineering compound lineage tracking and customizable assay views for candidate therapeutics prioritization \n3.2 Generalized assay collection pipeline and input transformation codes \n3.3 External pipeline for enhanced flexibility and faster data transformation \n3.4 Project portfolio management and relevant data reports \n3.5 External software LIMS integration \n3.6 Additional automation with scheduled tasks in windows operating system \n3.7 Liquid chromatography peak classifier module \n\n\n4 Results \n5 Conclusion \n6 Abbreviations \n7 Declarations \n\n7.1 Acknowledgements \n7.2 Funding \n7.3 Authors\u2019 contributions \n7.4 Competing interests \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nBackground: Protein-based therapeutics are one of the fastest growing classes of novel medical interventions in areas such as cancer, infectious disease, and inflammation. Protein engineering plays an important role in the optimization of desired therapeutic properties such as reducing immunogenicity, increasing stability for storage, increasing target specificity, etc. One category of protein therapeutics is nature-inspired bioengineered cystine-dense peptides (CDPs) for various biological targets. These engineered proteins are often further modified by synthetic chemistry. For example, candidate mini-proteins can be conjugated into active small molecule drugs. We refer to modified mini-proteins as \"optides\" (optimized peptides). To efficiently serve the multidisciplinary lab scientists with varied therapeutic portfolio research goals in a non-commercial setting, a cost-effective, extendable laboratory information management system (LIMS) is\/was needed.\nResults: We have developed a LIMS named Optide-Hunter for a generalized engineered protein compounds workflow that tracks entities and assays from creation to preclinical experiments. The implementation and custom modules are built using LabKey Server, which is an open-source platform for scientific data integration and analysis. Optide-Hunter contains a compound registry, in-silico assays, high-throughput production, large-scale production, in vivo assays, and data extraction from a specimen-tracking database. It is used to store, extract, and view data for various therapeutics projects. Optide-Hunter also includes stand-alone external processing software (HPLCPeakClassifierApp) for automated chromatogram classification. The HPLCPeakClassifierApp is used for pre-processing of high-performance liquid chromatography (HPLC) data prior to loading to Optide-Hunter. The custom implementation is done using data transformation modules in R, SQL, JavaScript, and Java, while being open-source to assist new users in customizing it for their unique workflows. (Instructions for exploring a deployed version of Optide-Hunter can be found on the LabKey website.)\nConclusion: The Optide-Hunter LIMS system is designed and built to track the processes of engineering, producing, and prioritizing protein therapeutic candidates. It can be easily adapted and extended for use in small or large research laboratories where multidisciplinary scientists are collaborating to engineer compounds for potential therapeutic or protein science applications. Exploration of open-source Optide-Hunter can help any bioinformatics scientist adapt, extend, and deploy an equivalent system tailored to each laboratory\u2019s workflow.\nKeywords: laboratory information management system, therapeutic protein, HPLC\/UPLC peak classification, protein engineering, LabKey software\n\nBackground \nFollowing significant advancements in biologics and biopharmaceuticals, protein-based therapeutics has surpassed 10 percent of the entire pharmaceutical market and is expected to be an even larger proportion of the market in the future.[1] Peptide and protein drugs target a wide variety of therapeutic areas such as cancer, inflammation, endocrine disorders, infectious diseases, and more.[2] In the development of peptide and protein therapeutics, protein engineering is an essential part of achieving the desired therapeutic properties in terms of target specificity, stability, pharmacokinetics, pharmacodynamics, etc. Protein engineering is not limited to amino acid sequence alteration. Conjugation with small molecule (dye or drug) can be used to produce antibody drug conjugates (ADCs) or peptide-drug conjugates (PDCs).[3]\nTracking conjugations and other modification steps while manufacturing and producing therapeutic proteins is challenging because it involves many more processing steps than the small-molecule, high-throughput design equivalent. These steps can be complex, and it is crucial that the process steps are captured for repeatability, whether the engineered protein production is being performed in a good manufacturing practice (GMP) or good laboratory practice (GLP) research lab, pre-clinical lab, or in an academic lab. It is also important to keep track of protein generation lineage for retrieval of data with related sequences, especially in high-throughput engineering processes. Frequently, it is beneficial to search previously engineered proteins that possess sequence similarity. As an example, our laboratory investigates nature-inspired cystine-dense peptides (CDPs) that originated from spider, snake, grasshopper, and other species. We have made more than a thousand CDPs and characterized them based on their expressability in our mammalian cell expression system.[4] We usually start from natural amino acid sequences (homologues) that are then modified to improve binding, serum half-life, and many other pharmacodynamic or pharmacokinetic properties (e.g., 14C-labelled peptides for autoradiography-based biodistribution or alanine scanning for structure activity relationship). During the sequence engineering, it is desirable to maintain the evolutionary lineage of the candidate CDPs from the natural homologue sequences so as to better inform further mutation or modification strategies.\nThere are several commercially available laboratory information management systems (LIMS) that can be used to address lineage and process tracking. Many can be configured as needed and some can be customized through software development. Our lab simultaneously faced the building of a LIMS system while creating an experiment pipeline, as is common among academic research groups. Therefore, it was difficult to prepare reasonable software requirement specifications to establish ready-made\/turn-key solutions up front. We found that the open-source LabKey platform provided a budget-friendly and easily extendable and adaptable LIMS solution. LabKey is a well-documented open-source platform for scientific data integration and analysis in a broad array of experimental settings.[5] This manuscript describes the customization of LabKey Server into Optide-Hunter for application to our engineered peptide therapeutic candidates\u2019 workflow. The customization includes our custom code for multiple open-source modules under an Apache 2.0 license. The hope is that Optide-Hunter can assist other academic labs or small biotechnology organizations to jump-start their protein engineering-based therapeutics workflows and easily adapt the provided code and example server for their unique needs. With the exception of the FreezerPro integration connector, the modules introduced in this publication are free of charge to set up. LabKey provides purchasable add-on special instrument connection packages and annual support if a user desires guidance from LabKey personnel rather than its user community.\n\nImplementation \nProtein engineering compound lineage tracking and customizable assay views for candidate therapeutics prioritization \nFigure 1 illustrates the various engineering pathways for therapeutic proteins. Compound registration starts with bioinformatics research and data mining for candidate peptides. Some of our compounds have a Uniprot number because they are native proteins produced by plants, animals, microbes, or other organisms. However, other compounds are de novo protein designs generated through the use of computational modeling software. From the parent sequences, variant sequences are registered. The variant sequence proteins can be chemically synthesized or be expressed by recombinant expression vector systems using bioengineering techniques. When the bioengineering platform is used to generate proteins, the construct sequences with prefix and suffix are added (e.g., enzyme cleavage site, polyhistidine-tag, etc). The construct can be used in either large scale (up to 10\u2009mg\/L in 2\u2009L cell culture) or high-throughput scale (up to 20\u2009\u03bcg in 1\u2009mL scale 96 well plate culture). The proteins are screened by in vitro, ex vivo, or in vivo assays without further molecular structure modification. However, sometimes, the proteins are chemically modified (e.g., PDC) prior to biologic assays. Several properties (e.g., purity, expressability by recombinant protein expression systems, synthesizability, etc.) are considered prior to progressing further along the drug discovery pipeline. Thus, based on predefined criteria, some sequences return to previous steps for redesign, which are denoted by red arrows in Fig. 1.\n\n\n\n\n\n\n\n\n\n Figure 1: Protein Engineering Workflow. The bioinformatics data\/literature mining with or without therapeutic targets is the starting point of root protein sequences. The software allows in-silico designed protein sequences as starting points as well as those with Uniprot designations. The majority of proteins we have explored are from sequences harvested from publicly available genomes. Thus, they have species and Uniprot numbers in the Homologue sample set database fields. Black arrows show the typical engineering paths. The dotted line from high throughput production is rare due to the amount of protein produced at this scale and current lack of efficient purification protocols. Red arrows indicate going back up the hierarchy to redesign proteins based on failures or other criteria (purity, express-ability by recombinant protein expression system, synthesizability etc.)\n\n\n\nOptide-Hunter utilizes the LabKey \u201cSample Set\u201d data container and \u201cParent Column\u201d lookup field as a database foreign key constraint. This ensures that all sequences must have a valid parent ID to be accepted for registration. Thus, all sequences that are derived from registered parents can be reviewed as illustrated in Fig. 2.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 2: Retrieving Lineage. Both parent and child samples of a given engineered protein can be easily retrieved and displayed. This figure illustrates that the HMG0001351 has its three additional variance sequence children (VAR0001396, VAR0001397 and VAR0001464). Derivation is listed in the \u201cRuns using this material or a derived material\u201d table. All the blue words are hyperlinks for getting additional information.\n\n\n\nOptide-Hunter also implemented a custom module called \u201cAssayReport\u201d and added it as an optional module to the list of other existing LabKey modules, as shown in Fig. 3A. This module provides the \u201cMolecular Properties Assay Report\u201d view that enables a user to filter child compound property values for comparison or prioritization in the therapeutics discovery pipeline, as in Fig. 3B. The module also enables a user to filter through the graphical utility, shown in Fig. 3C. The report can be easily customized through the \u201cEdit Report\u201d function by an administrator or system developer, and the source code can be updated to add or replace sample and assay data, as shown in Fig. 3D.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 3: Engineered Protein Selection Module. (a) A new module called \u201cAssayReport\u201d is available as an optional module to the currently distributed modules in the compound registry page. (b) Upon parent compound selection, all variants from the parent are shown and also filtered by user specified property values or (c) filtered by selection of a specified region in the scatter plot for further selection of engineered proteins. (d) Administrators, developers or bioinformaticians can customize through the \u201cEdit Report\u201d function and adding or replacing JSON code as needed.\n\n\n\nGeneralized assay collection pipeline and input transformation codes \nOptide-Hunter server deployment has an in silico function that calculates molecular properties of registered proteins. As shown in Fig. 1, our therapeutics discovery pipeline is not linear. However, proteins are generally designed and interrogated using in silico assays, then manufactured by high-throughput protein expression systems. Based on the high-throughput assay data, large scale production is initiated and the peptide is then further modified chemically. Those intermediate and final products are stored in a third-party specimen tracking LIMS called FreezerPro. Some of the compounds are further studied in vivo. In Optide-Hunter, the assay workflows can be identified using navigation menus configured in the LabKey headers referred to as \u201cInSilicoAssay,\u201d \u201cHTProduction,\u201d \u201cOTDProduction,\u201d \u201cChemProduction,\u201d \u201cFreezerPro,\u201d and \u201cVIVOAssay,\u201d shown in Fig. 4. To capture the assay data in a useful way, we used the LabKey data transform functions on data input from Microsoft Excel files. This was done with R code installed in Optide-Hunter (shown in the \u201cFiles\u201d panel in Fig. 4). \n\r\n\n\n\n\n\n\n\n\n\n\n Figure 4: Data Transformation R codes. All code mentioned in the paper are embedded in the deployed LabKey customization interfaces such as folder schema. Our custom transformation R code resides in the top-level Optide project folder and is downloadable for further customization.\n\n\n\nFor example, InSilicoAssay_onInsert.R parses tabular Excel files that are uploaded by a user and checks for duplicated sequences in order to avoid duplicate DNA synthesis orders. Next, molecular properties such as average mass, monoisotopic mass, net charge at pH\u20097.4, and hydrophobicity are calculated and inserted into the database. An R developer can customize the calculations by adapting the InSilicoAssay_onInsert.R source code. We customized the mass calculation for our CDP compounds because the disulfide bond formations of every cysteine site are important. The mass calculation accounts for the pair of hydrogen bonds lost at cysteine sites when disulfide bonds are formed. Similarly, WBAStandardTransformation.R parses whole body autoradiography (WBA) data with radioactive concentration standards and uses the linear fit values to transform the raw data. The R transform script then uses the linear fit to calculate decay per minute (dpm) of a compound\u2019s radioactivity in various tissues (brain, Xenograft tumor, blood, etc.). These two examples are among several scripts that can be easily adapted for use with any R package or other major scripting language. The RLabKey package is essential for data retrieval and insertion into the underlying LabKey database.\n\nExternal pipeline for enhanced flexibility and faster data transformation \nWhen data processing requires additional functions, heavy computational demand, or user input parameters, invoking a LabKey external pipeline module provided flexibility that a transformation script (described in the previous section) cannot. The external pipelines, as opposed to the built-in processing pipelines, were built and deployed via Apache Tomcat, which is a necessary component of LabKey Server. The external module then runs as desired, reporting its status to the user: complete, running, or in error. Optide-Hunter currently deploys four such customized external pipelines. One loads 96 well plate HPLC data to our HTProduction Assay. A user selects the desired chromatography files and executes the corresponding \u201cImport Data\u201d option \u201cUpdate HPLC Assay into database,\u201d shown in Fig. 5A. These custom modules use standard LabKey module structure and are provided. The tasks and pipelines can be edited in the folder to easily create a new external pipeline. The insert_jpegs.R component file contains an algorithm to parse the chromatogram jpeg file names to find matched compounds in the database. It then populates corresponding property values in the HTProduction Assay database table. Optide-Hunter also uses an external pipeline module called \u201cGenerate HT Plates from an HT Delivery form\u201d that creates HT barcodes automatically with delivered DNA information and inserts them into the database (Fig. 5B). In this case, the module accepts several user input parameters prior to processing (Fig. 5C).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 5: Data Transformation R codes. All code mentioned in the paper are embedded in the deployed LabKey customization interfaces such as folder schema. Our custom transformation R code resides in the top-level Optide project folder and is downloadable for further customization.\n\n\n\nProject portfolio management and relevant data reports \nWhen a research lab manages several project portfolios for different therapeutic targets, querying for specific sample information is important. While a LIMS typically provides data querying services, interrogating similar or the same compound across multiple therapeutic target evaluations is quite difficult. Thus, a custom reporting module that gathers all relevant data from various assays (HTProduction, OTDProduction, CHEMProduction etc) becomes an indispensable tool for investigators. For example, a lab scientist generates data for one assay but needs to simultaneously view another type of data. Similarly, project managers may only want to investigate promising subsets of samples. In our platform, a user submits a set of \u201cConstruct\u201d sample IDs (Construct IDs) as keys to query the entire set of assays contained in our system. The resulting page\u2019s URL contains the queried IDs so users can bookmark the entire URL containing the target therapeutic program compounds as shown in Fig. 6A. When new assay or production data associated with the queried construct IDs become available, it automatically populates. This feature is found under the \u201cPrograms\u201d menu in the top banner, and the source code is visible to administrators and developers by clicking \u201cEdit Source,\u201d shown at the bottom in Fig. 6B.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 6: Custom Report to Retrieve Data Associated with Investigating Compounds. (a) Compound IDs retrieve all assay data associated with each ID along with parent compound information. (b) Administrators or developers can click the edit button (pencil icon) in the top box to see the source code schema.\n\n\n\nExternal software LIMS integration \nThe compounds that we have made are deposited in -20F or -80F freezers. Specimen tracking is done through FreezerPro, which is commercially licensed on a per-user basis. Therefore, few lab scientists are authorized to access the specimens for accessioning or releasing. When compounds are produced, they are aliquoted to several vials for future use. Most of our lab users and program managers need limited information about each specimen to devise experiments, such as the total amount of each compound. They do not need to know non-research related information like the location of each specimen. We have implemented LabKey\u2019s integration with FreezerPro to provide filtered FreezerPro data for users. (Fig. 7).\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 7: External LIMS Interface and Custom code. XML is used to configure filtering or mapping between Optide-Hunter and third party databases such as FreezerPro.\n\n\n\nAdditional automation with scheduled tasks in windows operating system \nSome assay tables contain data that need to be processed after other steps are taken and new related data become available. For example, one scientist may analyze a protein using gel electrophoresis and report their findings in the assay table in the LIMS. Then another lab scientist may run the same protein in liquid chromatography\u2013mass spectrometry (LC-MS) and report their findings in the same assay table. After the relevant data are populated in the database, the scheduled tasks are invoked to perform calculations and insert the results into a designated column in the assay table. For example, mass spectrometry m\/z data is reported by a lab scientist. The scheduled task code is run overnight on the associated compound(s) and assigns a \u201ctrue or false\u201d validation status for the compound. More specifically, we calculate monoisotopic mass with full disulfide bond formation and compare it with various charge states. If the measured m\/z is one of the available values, the scheduled script ascribes a status of \u201ctrue\u201d to the compound. The frequency (minutes, hours, or days) of the scheduled job is easily adjusted through Microsoft Window Operating System task scheduler. The scripts utilize the RLabKey package in R and can be easily customized for the needs of another lab.\n\nLiquid chromatography peak classifier module \nThis workflow also contains stand-alone modules by a custom application intended to automatically provide quality scores for liquid chromatography (LC) measurements. Bioengineered expressed proteins are run through either Agilent high-performance liquid chromatography (HPLC) or by Waters ultra performance liquid chromatography (UPLC) for trace characteristics. In addition to a general trace assessment of protein production, CDPs require an additional disulfide bond formation assessment. More specifically, CDPs are a promising molecular class due to their particular arrangement of disulfide bonds in their core that provide structural stability. This signature disulfide bond formation can be a critical attribute, for example, in the development of an orally delivered therapeutic compound. Thus, for each produced protein, two LC traces are obtained: one trace is from intact purified protein and the other trace is the dithiothreitol (DTT) treated protein. With an overlaying of the two traces, the protein is classified as (1) perfect, (2) perfect-partial reduction, (3) simple, and (4) complex (Fig. 8). In order to classify compounds\u2019 biophysical properties in a few defined categories in a consistent manner, we developed a method and stand-alone software (HPLCPeakClassifierApp). Our method involves a blank sample being run every three sets of pairs of protein samples (DTT treated and DTT not-treated pair). All sample UV absorption trace values are normalized by subtracting the low noise values of the preceding blank sample.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 8: HPLCPeakClassifierApp Classification. The blue trace is from intact protein without DTT treatment and the red trace is from the same protein with DTT treatment. The number of peaks is identified after blank sample normalization provided that the trace is greater than user defined signal-to-noise ratio. (a) Each protein is classified as \u201cPerfect\u201d when there is one blue trace and one shifted red trace that indicates high protein purity with disulfide bond formation. (b) Protein is classified as \u201cPerfect-Partial\u201d when there is a single peak in the blue trace and the red trace has two peaks of which one overlaps with the blue trace, indicating that disulfide bond formation is partially reduced. This type of protein can be of particular therapeutic interest since it shows higher resistance to DTT reduction, which implies that the peptide may remain intact in the typical reductive intracellular environment. (c) Protein is classified as \u201cSimple\u201d when either the blue or red trace has two peaks including shoulder peaks. (d) Protein is classified as \u201cComplex\u201d when either the blue or red trace has more than two peaks.\n\n\n\nThe HPLCPeakClassifierApp has various input parameters for each lab to classify the traces that align with their research objectives. For example, the signal to noise ratio (\u2212SN) value controls how many peaks are found within user-defined retention time (RT) ranges (\u2212MinRTForPeak, \u2212MaxRTForPeak). By providing RT range as an input parameter, labs can change the values to accommodate different solvent gradients. The classification referred to as \u201cSimple\u201d is a more subjective classification that heavily depends on the screen objectives and can be stringent or permissive. Thus, a parameter called \u201c-Classification\u201d is provided to define number of acceptable peaks to be classified as \u201csimple.\u201d\n\nResults \nThe compilation of integration and automation efforts has produced Optide-Hunter software, which is composed of pluggable R, SQL, HTML, JSON and Java code. It is a web server-based LIMS and stand-alone LC processing toolset for engineered protein therapeutics discovery, including in silico design, bioengineered proteins, synthetic chemical modification, high-throughput plate-based in vitro and animal model-based in vivo systems. Optide-Hunter is an actively deployed system that continuously collects data for repeatability and collaboration among our scientists who are in different physical locations. We provide here access to a deployed web version of our system with fake sample data for the community to evaluate. We provide open-source versions of our software for those who desire it, to adapt them for their workflow by modifying or extending the code and database table design. The instructions for evaluation can be found at the LabKey website, and all of the Optide code is embedded in the LabKey custom platform. We also packaged the source code in one place for easy retrieval by bioinformaticians or software engineers.\n\nConclusion \nTherapeutic development based on engineered protein platforms has been gaining ground in many disease indication fields. However, academic labs or start-up companies face two challenges in obtaining a useful LIMS. Often the workflow platform itself is under construction, and it is hard to generate solid software requirement specifications up front. Furthermore, the cost of commercial LIMS can be prohibitive. This paper addresses the unmet need for those labs that require cost-effective and flexible LIMS for early-stage experimental pipeline development for engineered protein therapeutics development.\n\nAbbreviations \nADC: antibody drug conjugates\nHPLC: high-performance liquid chromatrography\nLC-MS: liquid chromatography\u2013mass spectrometry\nLIMS: laboratory information management system\nPDC: peptide-drug conjugates\nUPLC: ultra performance liquid chromatography\nWBA: whole body autoradiography\n\nDeclarations \nAcknowledgements \nThe authors are grateful to LabKey teams for providing technical support. We are also grateful to the Fred Hutchinson Cancer Research Center Molecular Design and Therapeutics team for adapting the platform.\n\nFunding \nThis work was funded by philanthropic support from Project Violet (https:\/\/www.fredhutch.org\/en\/labs\/clinical\/projects\/project-violet.html).\nThe funding body played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.\n\nAuthors\u2019 contributions \nMB conceived the original Optide-Hunter framework, and MB subsequently oversaw Optide-Hunter framework design and their implementation. MB, HR implemented codes. BL provided support, LabKey data modelling guidance, and coordinated development support at Labkey to address issues associated with custom code. BL, MB, HR and JO contributed manuscript preparation and approved the final manuscript.\n\nCompeting interests \nNone declared.\n\nReferences \n\n\n\u2191 Usmani, S.S.; Bedi, G.; Samuel, J.S. et al. (2017). \"THPdb: Database of FDA-approved peptide and protein therapeutics\". PLoS One 12 (7): e0181748. doi:10.1371\/journal.pone.0181748. PMC PMC5536290. PMID 28759605. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5536290 .   \n\n\u2191 Lagass\u00e9, H.A.; Alexaki, A.; Simhadri, V.L. et al. (2017). \"Recent advances in (therapeutic protein) drug development\". F1000Research 6: 113. doi:10.12688\/f1000research.9970.1. PMC PMC5302153. PMID 28232867. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302153 .   \n\n\u2191 Ma, L.; Wang, C.; He, Z. et al. (2017). \"Peptide-Drug Conjugate: A Novel Drug Design Approach\". Current Medicinal Chemistry 24 (31): 3373-3396. doi:10.2174\/0929867324666170404142840. PMID 28393694.   \n\n\u2191 Correnti, C.E.; Gewe, M.M.; Mehlin, C. et al. (2018). Screening, large-scale production and structure-based classification of cystine-dense peptides. 25. pp. 170\u201378. doi:10.1038\/s41594-018-0033-9. PMC PMC5840021. PMID 29483648. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5840021 .   \n\n\u2191 Nelson, E.K.; Britt P.; Josh E. et al. (2011). \"LabKey Server: An open source platform for scientific data integration, analysis and collaboration\". BMC Bioinformatics 12 (71). doi:10.1186\/1471-2105-12-71. PMC PMC3062597. PMID 21385461. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow\">https:\/\/www.limswiki.org\/index.php\/Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on laboratory informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 24 June 2019, at 22:53.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 96 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","c2dd7f033df4bcb56d74373a750e4a88_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Laboratory information management software for engineered mini-protein therapeutic workflow<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background<\/b>: Protein-based therapeutics are one of the fastest growing classes of novel medical interventions in areas such as cancer, infectious disease, and inflammation. Protein engineering plays an important role in the optimization of desired therapeutic properties such as reducing immunogenicity, increasing stability for storage, increasing target specificity, etc. One category of protein therapeutics is nature-inspired bioengineered cystine-dense peptides (CDPs) for various biological targets. These engineered proteins are often further modified by synthetic chemistry. For example, candidate mini-proteins can be conjugated into active small molecule drugs. We refer to modified mini-proteins as \"optides\" (optimized peptides). To efficiently serve the multidisciplinary lab scientists with varied therapeutic portfolio research goals in a non-commercial setting, a cost-effective, extendable <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS) is\/was needed.\n<\/p><p><b>Results<\/b>: We have developed a LIMS named Optide-Hunter for a generalized engineered protein compounds <a href=\"https:\/\/www.limswiki.org\/index.php\/Workflow\" title=\"Workflow\" class=\"wiki-link\" data-key=\"92bd8748272e20d891008dcb8243e8a8\">workflow<\/a> that tracks entities and assays from creation to preclinical experiments. The implementation and custom modules are built using <a href=\"https:\/\/www.limswiki.org\/index.php\/LabKey_Server\" title=\"LabKey Server\" class=\"wiki-link\" data-key=\"856c6fe009646beaaed7ecd21dda4932\">LabKey Server<\/a>, which is an open-source platform for scientific data integration and analysis. Optide-Hunter contains a compound registry, in-silico assays, high-throughput production, large-scale production, <i>in vivo<\/i> assays, and data extraction from a specimen-tracking database. It is used to store, extract, and view data for various therapeutics projects. Optide-Hunter also includes stand-alone external processing software (HPLCPeakClassifierApp) for automated <a href=\"https:\/\/www.limswiki.org\/index.php\/Chromatography\" title=\"Chromatography\" class=\"wiki-link\" data-key=\"2615535d1f14c6cffdfad7285999ad9d\">chromatogram<\/a> classification. The HPLCPeakClassifierApp is used for pre-processing of <a href=\"https:\/\/www.limswiki.org\/index.php\/High-performance_liquid_chromatography\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"dc274e995eb18113903eebaef69c0cf9\">high-performance liquid chromatography<\/a> (HPLC) data prior to loading to Optide-Hunter. The custom implementation is done using data transformation modules in <a href=\"https:\/\/www.limswiki.org\/index.php\/R_(programming_language)\" title=\"R (programming language)\" class=\"wiki-link\" data-key=\"1b0aa598f071aca4c5b4ee08d8bb2bde\">R<\/a>, SQL, JavaScript, and Java, while being open-source to assist new users in customizing it for their unique workflows. (Instructions for exploring a deployed version of Optide-Hunter can be found on the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labkey.com\/case%20study\/optide-hunter\/\" target=\"_blank\">LabKey website<\/a>.)\n<\/p><p><b>Conclusion<\/b>: The Optide-Hunter LIMS system is designed and built to track the processes of engineering, producing, and prioritizing protein therapeutic candidates. It can be easily adapted and extended for use in small or large research <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a> where multidisciplinary scientists are collaborating to engineer compounds for potential therapeutic or protein science applications. Exploration of open-source Optide-Hunter can help any <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> scientist adapt, extend, and deploy an equivalent system tailored to each laboratory\u2019s workflow.\n<\/p><p><b>Keywords<\/b>: laboratory information management system, therapeutic protein, HPLC\/UPLC peak classification, protein engineering, LabKey software\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Following significant advancements in biologics and biopharmaceuticals, protein-based therapeutics has surpassed 10 percent of the entire pharmaceutical market and is expected to be an even larger proportion of the market in the future.<sup id=\"rdp-ebb-cite_ref-UsmaniTHPdb17_1-0\" class=\"reference\"><a href=\"#cite_note-UsmaniTHPdb17-1\">[1]<\/a><\/sup> Peptide and protein drugs target a wide variety of therapeutic areas such as cancer, inflammation, endocrine disorders, infectious diseases, and more.<sup id=\"rdp-ebb-cite_ref-Lagass.C3.A9Research17_2-0\" class=\"reference\"><a href=\"#cite_note-Lagass.C3.A9Research17-2\">[2]<\/a><\/sup> In the development of peptide and protein therapeutics, protein engineering is an essential part of achieving the desired therapeutic properties in terms of target specificity, stability, pharmacokinetics, pharmacodynamics, etc. Protein engineering is not limited to <a href=\"https:\/\/www.limswiki.org\/index.php\/Protein_sequencing\" title=\"Protein sequencing\" class=\"wiki-link\" data-key=\"efdac3647b70690761eee6ade33c0892\">amino acid sequence<\/a> alteration. Conjugation with small molecule (dye or drug) can be used to produce antibody drug conjugates (ADCs) or peptide-drug conjugates (PDCs).<sup id=\"rdp-ebb-cite_ref-MaPeptide17_3-0\" class=\"reference\"><a href=\"#cite_note-MaPeptide17-3\">[3]<\/a><\/sup>\n<\/p><p>Tracking conjugations and other modification steps while manufacturing and producing therapeutic proteins is challenging because it involves many more processing steps than the small-molecule, high-throughput design equivalent. These steps can be complex, and it is crucial that the process steps are captured for repeatability, whether the engineered protein production is being performed in a <a href=\"https:\/\/www.limswiki.org\/index.php\/Good_manufacturing_practice\" title=\"Good manufacturing practice\" class=\"wiki-link\" data-key=\"50b520c825b1702be310362c66e0343f\">good manufacturing practice<\/a> (GMP) or good laboratory practice (GLP) research lab, pre-clinical lab, or in an academic lab. It is also important to keep track of protein generation lineage for retrieval of data with related sequences, especially in high-throughput engineering processes. Frequently, it is beneficial to search previously engineered proteins that possess sequence similarity. As an example, our laboratory investigates nature-inspired cystine-dense peptides (CDPs) that originated from spider, snake, grasshopper, and other species. We have made more than a thousand CDPs and characterized them based on their expressability in our mammalian cell expression system.<sup id=\"rdp-ebb-cite_ref-CorrentiScreen18_4-0\" class=\"reference\"><a href=\"#cite_note-CorrentiScreen18-4\">[4]<\/a><\/sup> We usually start from natural amino acid sequences (homologues) that are then modified to improve binding, serum half-life, and many other pharmacodynamic or pharmacokinetic properties (e.g., <sup>14<\/sup>C-labelled peptides for autoradiography-based biodistribution or alanine scanning for structure activity relationship). During the sequence engineering, it is desirable to maintain the evolutionary lineage of the candidate CDPs from the natural homologue sequences so as to better inform further mutation or modification strategies.\n<\/p><p>There are several commercially available <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS) that can be used to address lineage and process tracking. Many can be configured as needed and some can be customized through software development. Our lab simultaneously faced the building of a LIMS system while creating an experiment pipeline, as is common among academic research groups. Therefore, it was difficult to prepare reasonable software requirement specifications to establish ready-made\/turn-key solutions up front. We found that the open-source <a href=\"https:\/\/www.limswiki.org\/index.php\/LabKey_Server\" title=\"LabKey Server\" class=\"wiki-link\" data-key=\"856c6fe009646beaaed7ecd21dda4932\">LabKey<\/a> platform provided a budget-friendly and easily extendable and adaptable LIMS solution. LabKey is a well-documented open-source platform for scientific data integration and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analysis<\/a> in a broad array of experimental settings.<sup id=\"rdp-ebb-cite_ref-NelsonLabKey11_5-0\" class=\"reference\"><a href=\"#cite_note-NelsonLabKey11-5\">[5]<\/a><\/sup> This manuscript describes the customization of LabKey Server into Optide-Hunter for application to our engineered peptide therapeutic candidates\u2019 workflow. The customization includes our custom code for multiple open-source modules under an Apache 2.0 license. The hope is that Optide-Hunter can assist other academic labs or small biotechnology organizations to jump-start their protein engineering-based therapeutics workflows and easily adapt the provided code and example server for their unique needs. With the exception of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Brooks_Automation,_Inc.\" title=\"Brooks Automation, Inc.\" class=\"wiki-link\" data-key=\"7b55775625992e844044c5db215b4a91\">FreezerPro<\/a> integration connector, the modules introduced in this publication are free of charge to set up. LabKey provides purchasable add-on special instrument connection packages and annual support if a user desires guidance from LabKey personnel rather than its user community.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Protein_engineering_compound_lineage_tracking_and_customizable_assay_views_for_candidate_therapeutics_prioritization\">Protein engineering compound lineage tracking and customizable assay views for candidate therapeutics prioritization<\/span><\/h3>\n<p>Figure 1 illustrates the various engineering pathways for therapeutic proteins. Compound registration starts with <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> research and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_mining\" title=\"Data mining\" class=\"wiki-link\" data-key=\"be09d3680fe1608addedf6f62692ee47\">data mining<\/a> for candidate peptides. Some of our compounds have a Uniprot number because they are native proteins produced by plants, animals, microbes, or other organisms. However, other compounds are <i>de novo<\/i> protein designs generated through the use of computational modeling software. From the parent sequences, variant sequences are registered. The variant sequence proteins can be chemically synthesized or be expressed by recombinant expression vector systems using bioengineering techniques. When the bioengineering platform is used to generate proteins, the construct sequences with prefix and suffix are added (e.g., enzyme cleavage site, polyhistidine-tag, etc). The construct can be used in either large scale (up to 10\u2009mg\/L in 2\u2009L cell culture) or high-throughput scale (up to 20\u2009\u03bcg in 1\u2009mL scale 96 well plate culture). The proteins are screened by <i>in vitro<\/i>, <i>ex vivo<\/i>, or <i>in vivo<\/i> assays without further molecular structure modification. However, sometimes, the proteins are chemically modified (e.g., PDC) prior to biologic assays. Several properties (e.g., purity, expressability by recombinant protein expression systems, synthesizability, etc.) are considered prior to progressing further along the drug discovery pipeline. Thus, based on predefined criteria, some sequences return to previous steps for redesign, which are denoted by red arrows in Fig. 1.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"b95f0007c4c650481e3b70b6325e63ee\"><img alt=\"Fig1 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/6\/63\/Fig1_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1:<\/b> Protein Engineering Workflow. The bioinformatics data\/literature mining with or without therapeutic targets is the starting point of root protein sequences. The software allows in-silico designed protein sequences as starting points as well as those with Uniprot designations. The majority of proteins we have explored are from sequences harvested from publicly available genomes. Thus, they have species and Uniprot numbers in the Homologue sample set database fields. Black arrows show the typical engineering paths. The dotted line from high throughput production is rare due to the amount of protein produced at this scale and current lack of efficient purification protocols. Red arrows indicate going back up the hierarchy to redesign proteins based on failures or other criteria (purity, express-ability by recombinant protein expression system, synthesizability etc.)<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Optide-Hunter utilizes the LabKey \u201cSample Set\u201d data container and \u201cParent Column\u201d lookup field as a database foreign key constraint. This ensures that all sequences must have a valid parent ID to be accepted for registration. Thus, all sequences that are derived from registered parents can be reviewed as illustrated in Fig. 2.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"6cd58f7ce7963c9399a1b79bfbf06e21\"><img alt=\"Fig2 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/9\/9f\/Fig2_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2:<\/b> Retrieving Lineage. Both parent and child samples of a given engineered protein can be easily retrieved and displayed. This figure illustrates that the HMG0001351 has its three additional variance sequence children (VAR0001396, VAR0001397 and VAR0001464). Derivation is listed in the \u201cRuns using this material or a derived material\u201d table. All the blue words are hyperlinks for getting additional information.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Optide-Hunter also implemented a custom module called \u201cAssayReport\u201d and added it as an optional module to the list of other existing LabKey modules, as shown in Fig. 3A. This module provides the \u201cMolecular Properties Assay Report\u201d view that enables a user to filter child compound property values for comparison or prioritization in the therapeutics discovery pipeline, as in Fig. 3B. The module also enables a user to filter through the graphical utility, shown in Fig. 3C. The report can be easily customized through the \u201cEdit Report\u201d function by an administrator or system developer, and the source code can be updated to add or replace sample and assay data, as shown in Fig. 3D.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"d380d9ec30a881c5689e35366611dd65\"><img alt=\"Fig3 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/3\/3e\/Fig3_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3:<\/b> Engineered Protein Selection Module. (<b>a<\/b>) A new module called \u201cAssayReport\u201d is available as an optional module to the currently distributed modules in the compound registry page. (<b>b<\/b>) Upon parent compound selection, all variants from the parent are shown and also filtered by user specified property values or (<b>c<\/b>) filtered by selection of a specified region in the scatter plot for further selection of engineered proteins. (<b>d<\/b>) Administrators, developers or bioinformaticians can customize through the \u201cEdit Report\u201d function and adding or replacing JSON code as needed.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Generalized_assay_collection_pipeline_and_input_transformation_codes\">Generalized assay collection pipeline and input transformation codes<\/span><\/h3>\n<p>Optide-Hunter server deployment has an <i>in silico<\/i> function that calculates molecular properties of registered proteins. As shown in Fig. 1, our therapeutics discovery pipeline is not linear. However, proteins are generally designed and interrogated using <i>in silico<\/i> assays, then manufactured by high-throughput protein expression systems. Based on the high-throughput assay data, large scale production is initiated and the peptide is then further modified chemically. Those intermediate and final products are stored in a third-party specimen tracking LIMS called FreezerPro. Some of the compounds are further studied <i>in vivo<\/i>. In Optide-Hunter, the assay workflows can be identified using navigation menus configured in the LabKey headers referred to as \u201cInSilicoAssay,\u201d \u201cHTProduction,\u201d \u201cOTDProduction,\u201d \u201cChemProduction,\u201d \u201cFreezerPro,\u201d and \u201cVIVOAssay,\u201d shown in Fig. 4. To capture the assay data in a useful way, we used the LabKey data transform functions on data input from Microsoft Excel files. This was done with R code installed in Optide-Hunter (shown in the \u201cFiles\u201d panel in Fig. 4). \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"eb16039bec6f5f24c1cea54d5001c572\"><img alt=\"Fig4 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/5\/53\/Fig4_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4:<\/b> Data Transformation R codes. All code mentioned in the paper are embedded in the deployed LabKey customization interfaces such as folder schema. Our custom transformation R code resides in the top-level Optide project folder and is downloadable for further customization.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For example, InSilicoAssay_onInsert.R parses tabular Excel files that are uploaded by a user and checks for duplicated sequences in order to avoid duplicate DNA synthesis orders. Next, molecular properties such as average mass, monoisotopic mass, net charge at pH\u20097.4, and hydrophobicity are calculated and inserted into the database. An R developer can customize the calculations by adapting the InSilicoAssay_onInsert.R source code. We customized the mass calculation for our CDP compounds because the disulfide bond formations of every cysteine site are important. The mass calculation accounts for the pair of hydrogen bonds lost at cysteine sites when disulfide bonds are formed. Similarly, WBAStandardTransformation.R parses whole body autoradiography (WBA) data with radioactive concentration standards and uses the linear fit values to transform the raw data. The R transform script then uses the linear fit to calculate decay per minute (dpm) of a compound\u2019s radioactivity in various tissues (brain, Xenograft tumor, blood, etc.). These two examples are among several scripts that can be easily adapted for use with any R package or other major scripting language. The RLabKey package is essential for data retrieval and insertion into the underlying LabKey database.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"External_pipeline_for_enhanced_flexibility_and_faster_data_transformation\">External pipeline for enhanced flexibility and faster data transformation<\/span><\/h3>\n<p>When data processing requires additional functions, heavy computational demand, or user input parameters, invoking a LabKey external pipeline module provided flexibility that a transformation script (described in the previous section) cannot. The external pipelines, as opposed to the built-in processing pipelines, were built and deployed via <a href=\"https:\/\/www.limswiki.org\/index.php\/Apache_Tomcat\" title=\"Apache Tomcat\" class=\"wiki-link\" data-key=\"6fd6693ebebef576e0e80cf1c328d360\">Apache Tomcat<\/a>, which is a necessary component of LabKey Server. The external module then runs as desired, reporting its status to the user: complete, running, or in error. Optide-Hunter currently deploys four such customized external pipelines. One loads 96 well plate HPLC data to our HTProduction Assay. A user selects the desired <a href=\"https:\/\/www.limswiki.org\/index.php\/Chromatography\" title=\"Chromatography\" class=\"wiki-link\" data-key=\"2615535d1f14c6cffdfad7285999ad9d\">chromatography<\/a> files and executes the corresponding \u201cImport Data\u201d option \u201cUpdate HPLC Assay into database,\u201d shown in Fig. 5A. These custom modules use standard LabKey module structure and are provided. The tasks and pipelines can be edited in the folder to easily create a new external pipeline. The insert_jpegs.R component file contains an algorithm to parse the chromatogram jpeg file names to find matched compounds in the database. It then populates corresponding property values in the HTProduction Assay database table. Optide-Hunter also uses an external pipeline module called \u201cGenerate HT Plates from an HT Delivery form\u201d that creates HT barcodes automatically with delivered DNA information and inserts them into the database (Fig. 5B). In this case, the module accepts several user input parameters prior to processing (Fig. 5C).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"fc19d7d6ece510716f989f607d01c59f\"><img alt=\"Fig5 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/e\/ee\/Fig5_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5:<\/b> Data Transformation R codes. All code mentioned in the paper are embedded in the deployed LabKey customization interfaces such as folder schema. Our custom transformation R code resides in the top-level Optide project folder and is downloadable for further customization.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Project_portfolio_management_and_relevant_data_reports\">Project portfolio management and relevant data reports<\/span><\/h3>\n<p>When a research lab manages several project portfolios for different therapeutic targets, querying for specific sample <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> is important. While a LIMS typically provides data querying services, interrogating similar or the same compound across multiple therapeutic target evaluations is quite difficult. Thus, a custom reporting module that gathers all relevant data from various assays (HTProduction, OTDProduction, CHEMProduction etc) becomes an indispensable tool for investigators. For example, a lab scientist generates data for one assay but needs to simultaneously view another type of data. Similarly, project managers may only want to investigate promising subsets of samples. In our platform, a user submits a set of \u201cConstruct\u201d sample IDs (Construct IDs) as keys to query the entire set of assays contained in our system. The resulting page\u2019s URL contains the queried IDs so users can bookmark the entire URL containing the target therapeutic program compounds as shown in Fig. 6A. When new assay or production data associated with the queried construct IDs become available, it automatically populates. This feature is found under the \u201cPrograms\u201d menu in the top banner, and the source code is visible to administrators and developers by clicking \u201cEdit Source,\u201d shown at the bottom in Fig. 6B.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"7f1fd357272ea3c74368ea674a5b31e4\"><img alt=\"Fig6 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/3\/39\/Fig6_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 6:<\/b> Custom Report to Retrieve Data Associated with Investigating Compounds. (<b>a<\/b>) Compound IDs retrieve all assay data associated with each ID along with parent compound information. (<b>b<\/b>) Administrators or developers can click the edit button (pencil icon) in the top box to see the source code schema.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"External_software_LIMS_integration\">External software LIMS integration<\/span><\/h3>\n<p>The compounds that we have made are deposited in -20F or -80F freezers. Specimen tracking is done through FreezerPro, which is commercially licensed on a per-user basis. Therefore, few lab scientists are authorized to access the specimens for accessioning or releasing. When compounds are produced, they are aliquoted to several vials for future use. Most of our lab users and program managers need limited information about each specimen to devise experiments, such as the total amount of each compound. They do not need to know non-research related information like the location of each specimen. We have implemented LabKey\u2019s integration with FreezerPro to provide filtered FreezerPro data for users. (Fig. 7).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"d10d1bbb010ff9ab6f80a71ba8840ed0\"><img alt=\"Fig7 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/2\/2c\/Fig7_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 7:<\/b> External LIMS Interface and Custom code. XML is used to configure filtering or mapping between Optide-Hunter and third party databases such as FreezerPro.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Additional_automation_with_scheduled_tasks_in_windows_operating_system\">Additional automation with scheduled tasks in windows operating system<\/span><\/h3>\n<p>Some assay tables contain data that need to be processed after other steps are taken and new related data become available. For example, one scientist may analyze a protein using gel electrophoresis and report their findings in the assay table in the LIMS. Then another lab scientist may run the same protein in liquid chromatography\u2013mass spectrometry (LC-MS) and report their findings in the same assay table. After the relevant data are populated in the database, the scheduled tasks are invoked to perform calculations and insert the results into a designated column in the assay table. For example, <a href=\"https:\/\/www.limswiki.org\/index.php\/Mass_spectrometry\" title=\"Mass spectrometry\" class=\"wiki-link\" data-key=\"fb548eafe2596c35d7ea741849aa83d4\">mass spectrometry<\/a> m\/z data is reported by a lab scientist. The scheduled task code is run overnight on the associated compound(s) and assigns a \u201ctrue or false\u201d validation status for the compound. More specifically, we calculate monoisotopic mass with full disulfide bond formation and compare it with various charge states. If the measured m\/z is one of the available values, the scheduled script ascribes a status of \u201ctrue\u201d to the compound. The frequency (minutes, hours, or days) of the scheduled job is easily adjusted through Microsoft Window Operating System task scheduler. The scripts utilize the RLabKey package in R and can be easily customized for the needs of another lab.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Liquid_chromatography_peak_classifier_module\">Liquid chromatography peak classifier module<\/span><\/h3>\n<p>This workflow also contains stand-alone modules by a custom application intended to automatically provide quality scores for liquid chromatography (LC) measurements. Bioengineered expressed proteins are run through either <a href=\"https:\/\/www.limswiki.org\/index.php\/Agilent_Technologies,_Inc.\" title=\"Agilent Technologies, Inc.\" class=\"wiki-link\" data-key=\"dcea1a676a012bcbe3af9562dd17f8a0\">Agilent<\/a> <a href=\"https:\/\/www.limswiki.org\/index.php\/High-performance_liquid_chromatography\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"dc274e995eb18113903eebaef69c0cf9\">high-performance liquid chromatography<\/a> (HPLC) or by <a href=\"https:\/\/www.limswiki.org\/index.php\/Waters_Corporation\" title=\"Waters Corporation\" class=\"wiki-link\" data-key=\"d17153eab93be07066660e3cff4b84c5\">Waters<\/a> ultra performance liquid chromatography (UPLC) for trace characteristics. In addition to a general trace assessment of protein production, CDPs require an additional disulfide bond formation assessment. More specifically, CDPs are a promising molecular class due to their particular arrangement of disulfide bonds in their core that provide structural stability. This signature disulfide bond formation can be a critical attribute, for example, in the development of an orally delivered therapeutic compound. Thus, for each produced protein, two LC traces are obtained: one trace is from intact purified protein and the other trace is the dithiothreitol (DTT) treated protein. With an overlaying of the two traces, the protein is classified as (1) perfect, (2) perfect-partial reduction, (3) simple, and (4) complex (Fig. 8). In order to classify compounds\u2019 biophysical properties in a few defined categories in a consistent manner, we developed a method and stand-alone software (HPLCPeakClassifierApp). Our method involves a blank sample being run every three sets of pairs of protein samples (DTT treated and DTT not-treated pair). All sample UV absorption trace values are normalized by subtracting the low noise values of the preceding blank sample.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Brusniak_BMCBioinformatics2019_20.png\" class=\"image wiki-link\" data-key=\"a8c497bdf6e1708a3709d74d945be859\"><img alt=\"Fig8 Brusniak BMCBioinformatics2019 20.png\" src=\"https:\/\/www.limswiki.org\/images\/3\/33\/Fig8_Brusniak_BMCBioinformatics2019_20.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 8:<\/b> HPLCPeakClassifierApp Classification. The blue trace is from intact protein without DTT treatment and the red trace is from the same protein with DTT treatment. The number of peaks is identified after blank sample normalization provided that the trace is greater than user defined signal-to-noise ratio. (<b>a<\/b>) Each protein is classified as \u201cPerfect\u201d when there is one blue trace and one shifted red trace that indicates high protein purity with disulfide bond formation. (<b>b<\/b>) Protein is classified as \u201cPerfect-Partial\u201d when there is a single peak in the blue trace and the red trace has two peaks of which one overlaps with the blue trace, indicating that disulfide bond formation is partially reduced. This type of protein can be of particular therapeutic interest since it shows higher resistance to DTT reduction, which implies that the peptide may remain intact in the typical reductive intracellular environment. (<b>c<\/b>) Protein is classified as \u201cSimple\u201d when either the blue or red trace has two peaks including shoulder peaks. (<b>d<\/b>) Protein is classified as \u201cComplex\u201d when either the blue or red trace has more than two peaks.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The HPLCPeakClassifierApp has various input parameters for each lab to classify the traces that align with their research objectives. For example, the signal to noise ratio (\u2212SN) value controls how many peaks are found within user-defined retention time (RT) ranges (\u2212MinRTForPeak, \u2212MaxRTForPeak). By providing RT range as an input parameter, labs can change the values to accommodate different solvent gradients. The classification referred to as \u201cSimple\u201d is a more subjective classification that heavily depends on the screen objectives and can be stringent or permissive. Thus, a parameter called \u201c-Classification\u201d is provided to define number of acceptable peaks to be classified as \u201csimple.\u201d\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>The compilation of integration and automation efforts has produced Optide-Hunter software, which is composed of pluggable R, SQL, HTML, JSON and Java code. It is a web server-based LIMS and stand-alone LC processing toolset for engineered protein therapeutics discovery, including <i>in silico<\/i> design, bioengineered proteins, synthetic chemical modification, high-throughput plate-based <i>in vitro<\/i> and animal model-based <i>in vivo<\/i> systems. Optide-Hunter is an actively deployed system that continuously collects data for repeatability and collaboration among our scientists who are in different physical locations. We provide here access to a deployed web version of our system with fake sample data for the community to evaluate. We provide open-source versions of our software for those who desire it, to adapt them for their workflow by modifying or extending the code and database table design. The instructions for evaluation can be found at the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.labkey.com\/case%20study\/optide-hunter\" target=\"_blank\">LabKey website<\/a>, and all of the Optide code is embedded in the LabKey custom platform. We also packaged the source code in one place for easy retrieval by bioinformaticians or software engineers.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Therapeutic development based on engineered protein platforms has been gaining ground in many disease indication fields. However, academic labs or start-up companies face two challenges in obtaining a useful LIMS. Often the workflow platform itself is under construction, and it is hard to generate solid software requirement specifications up front. Furthermore, the cost of commercial LIMS can be prohibitive. This paper addresses the unmet need for those labs that require cost-effective and flexible LIMS for early-stage experimental pipeline development for engineered protein therapeutics development.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>ADC<\/b>: antibody drug conjugates\n<\/p><p><b>HPLC<\/b>: high-performance liquid chromatrography\n<\/p><p><b>LC-MS<\/b>: liquid chromatography\u2013mass spectrometry\n<\/p><p><b>LIMS<\/b>: laboratory information management system\n<\/p><p><b>PDC<\/b>: peptide-drug conjugates\n<\/p><p><b>UPLC<\/b>: ultra performance liquid chromatography\n<\/p><p><b>WBA<\/b>: whole body autoradiography\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h3>\n<p>The authors are grateful to LabKey teams for providing technical support. We are also grateful to the Fred Hutchinson Cancer Research Center Molecular Design and Therapeutics team for adapting the platform.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was funded by philanthropic support from Project Violet (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.fredhutch.org\/en\/labs\/clinical\/projects\/project-violet.html\" target=\"_blank\">https:\/\/www.fredhutch.org\/en\/labs\/clinical\/projects\/project-violet.html<\/a>).\n<\/p><p>The funding body played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h3>\n<p>MB conceived the original Optide-Hunter framework, and MB subsequently oversaw Optide-Hunter framework design and their implementation. MB, HR implemented codes. BL provided support, LabKey data modelling guidance, and coordinated development support at Labkey to address issues associated with custom code. BL, MB, HR and JO contributed manuscript preparation and approved the final manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>None declared.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-UsmaniTHPdb17-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UsmaniTHPdb17_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Usmani, S.S.; Bedi, G.; Samuel, J.S. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5536290\" target=\"_blank\">\"THPdb: Database of FDA-approved peptide and protein therapeutics\"<\/a>. <i>PLoS One<\/i> <b>12<\/b> (7): e0181748. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0181748\" target=\"_blank\">10.1371\/journal.pone.0181748<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5536290\/\" target=\"_blank\">PMC5536290<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28759605\" target=\"_blank\">28759605<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5536290\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5536290<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=THPdb%3A+Database+of+FDA-approved+peptide+and+protein+therapeutics&rft.jtitle=PLoS+One&rft.aulast=Usmani%2C+S.S.%3B+Bedi%2C+G.%3B+Samuel%2C+J.S.+et+al.&rft.au=Usmani%2C+S.S.%3B+Bedi%2C+G.%3B+Samuel%2C+J.S.+et+al.&rft.date=2017&rft.volume=12&rft.issue=7&rft.pages=e0181748&rft_id=info:doi\/10.1371%2Fjournal.pone.0181748&rft_id=info:pmc\/PMC5536290&rft_id=info:pmid\/28759605&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5536290&rfr_id=info:sid\/en.wikipedia.org:Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Lagass.C3.A9Research17-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Lagass.C3.A9Research17_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lagass\u00e9, H.A.; Alexaki, A.; Simhadri, V.L. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302153\" target=\"_blank\">\"Recent advances in (therapeutic protein) drug development\"<\/a>. <i>F1000Research<\/i> <b>6<\/b>: 113. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12688%2Ff1000research.9970.1\" target=\"_blank\">10.12688\/f1000research.9970.1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5302153\/\" target=\"_blank\">PMC5302153<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28232867\" target=\"_blank\">28232867<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302153\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5302153<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Recent+advances+in+%28therapeutic+protein%29+drug+development&rft.jtitle=F1000Research&rft.aulast=Lagass%C3%A9%2C+H.A.%3B+Alexaki%2C+A.%3B+Simhadri%2C+V.L.+et+al.&rft.au=Lagass%C3%A9%2C+H.A.%3B+Alexaki%2C+A.%3B+Simhadri%2C+V.L.+et+al.&rft.date=2017&rft.volume=6&rft.pages=113&rft_id=info:doi\/10.12688%2Ff1000research.9970.1&rft_id=info:pmc\/PMC5302153&rft_id=info:pmid\/28232867&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5302153&rfr_id=info:sid\/en.wikipedia.org:Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaPeptide17-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaPeptide17_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ma, L.; Wang, C.; He, Z. et al. (2017). \"Peptide-Drug Conjugate: A Novel Drug Design Approach\". <i>Current Medicinal Chemistry<\/i> <b>24<\/b> (31): 3373-3396. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2174%2F0929867324666170404142840\" target=\"_blank\">10.2174\/0929867324666170404142840<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28393694\" target=\"_blank\">28393694<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Peptide-Drug+Conjugate%3A+A+Novel+Drug+Design+Approach&rft.jtitle=Current+Medicinal+Chemistry&rft.aulast=Ma%2C+L.%3B+Wang%2C+C.%3B+He%2C+Z.+et+al.&rft.au=Ma%2C+L.%3B+Wang%2C+C.%3B+He%2C+Z.+et+al.&rft.date=2017&rft.volume=24&rft.issue=31&rft.pages=3373-3396&rft_id=info:doi\/10.2174%2F0929867324666170404142840&rft_id=info:pmid\/28393694&rfr_id=info:sid\/en.wikipedia.org:Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CorrentiScreen18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CorrentiScreen18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Correnti, C.E.; Gewe, M.M.; Mehlin, C. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5840021\" target=\"_blank\"><i>Screening, large-scale production and structure-based classification of cystine-dense peptides<\/i><\/a>. <b>25<\/b>. pp. 170\u201378. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs41594-018-0033-9\" target=\"_blank\">10.1038\/s41594-018-0033-9<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5840021\/\" target=\"_blank\">PMC5840021<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29483648\" target=\"_blank\">29483648<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5840021\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5840021<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Screening%2C+large-scale+production+and+structure-based+classification+of+cystine-dense+peptides&rft.aulast=Correnti%2C+C.E.%3B+Gewe%2C+M.M.%3B+Mehlin%2C+C.+et+al.&rft.au=Correnti%2C+C.E.%3B+Gewe%2C+M.M.%3B+Mehlin%2C+C.+et+al.&rft.date=2018&rft.volume=25&rft.issue=3&rft.pages=pp.+170%E2%80%9378&rft_id=info:doi\/10.1038%2Fs41594-018-0033-9&rft_id=info:pmc\/PMC5840021&rft_id=info:pmid\/29483648&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5840021&rfr_id=info:sid\/en.wikipedia.org:Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NelsonLabKey11-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NelsonLabKey11_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nelson, E.K.; Britt P.; Josh E. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597\" target=\"_blank\">\"LabKey Server: An open source platform for scientific data integration, analysis and collaboration\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b> (71). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-71\" target=\"_blank\">10.1186\/1471-2105-12-71<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3062597\/\" target=\"_blank\">PMC3062597<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21385461\" target=\"_blank\">21385461<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LabKey+Server%3A+An+open+source+platform+for+scientific+data+integration%2C+analysis+and+collaboration&rft.jtitle=BMC+Bioinformatics&rft.aulast=Nelson%2C+E.K.%3B+Britt+P.%3B+Josh+E.+et+al.&rft.au=Nelson%2C+E.K.%3B+Britt+P.%3B+Josh+E.+et+al.&rft.date=2011&rft.volume=12&rft.issue=71&rft_id=info:doi\/10.1186%2F1471-2105-12-71&rft_id=info:pmc\/PMC3062597&rft_id=info:pmid\/21385461&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3062597&rfr_id=info:sid\/en.wikipedia.org:Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, spelling, and grammar. We also added PMCID and DOI when they were missing from the original reference.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165456\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.266 seconds\nReal time usage: 0.309 seconds\nPreprocessor visited node count: 6244\/1000000\nPreprocessor generated node count: 26996\/1000000\nPost\u2010expand include size: 53098\/2097152 bytes\nTemplate argument size: 16355\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 251.615 1 - -total\n 59.34% 149.319 1 - Template:Reflist\n 53.69% 135.093 5 - Template:Cite_journal\n 50.34% 126.675 5 - Template:Citation\/core\n 34.77% 87.476 1 - Template:Infobox_journal_article\n 33.25% 83.659 1 - Template:Infobox\n 17.98% 45.250 80 - Template:Infobox\/row\n 9.54% 23.995 14 - Template:Citation\/identifier\n 4.71% 11.844 5 - Template:Citation\/make_link\n 3.49% 8.769 32 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11077-0!*!0!!en!5!* and timestamp 20190701165456 and revision id 35939\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow\">https:\/\/www.limswiki.org\/index.php\/Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","c2dd7f033df4bcb56d74373a750e4a88_images":["https:\/\/www.limswiki.org\/images\/6\/63\/Fig1_Brusniak_BMCBioinformatics2019_20.png","https:\/\/www.limswiki.org\/images\/9\/9f\/Fig2_Brusniak_BMCBioinformatics2019_20.png","https:\/\/www.limswiki.org\/images\/3\/3e\/Fig3_Brusniak_BMCBioinformatics2019_20.png","https:\/\/www.limswiki.org\/images\/5\/53\/Fig4_Brusniak_BMCBioinformatics2019_20.png","https:\/\/www.limswiki.org\/images\/e\/ee\/Fig5_Brusniak_BMCBioinformatics2019_20.png","https:\/\/www.limswiki.org\/images\/3\/39\/Fig6_Brusniak_BMCBioinformatics2019_20.png","https:\/\/www.limswiki.org\/images\/2\/2c\/Fig7_Brusniak_BMCBioinformatics2019_20.png","https:\/\/www.limswiki.org\/images\/3\/33\/Fig8_Brusniak_BMCBioinformatics2019_20.png"],"c2dd7f033df4bcb56d74373a750e4a88_timestamp":1562000096,"7685e865ec42d1c3178d9cb2493521d4_type":"article","7685e865ec42d1c3178d9cb2493521d4_title":"Assessing cyberbiosecurity vulnerabilities and infrastructure resilience (Schabacker et al. 2019)","7685e865ec42d1c3178d9cb2493521d4_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience","7685e865ec42d1c3178d9cb2493521d4_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Assessing cyberbiosecurity vulnerabilities and infrastructure resilience\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nAssessing cyberbiosecurity vulnerabilities and infrastructure resilienceJournal\n \nFrontiers in Bioengineering and BiotechnologyAuthor(s)\n \nSchabacker, Daniel S.; Levy, Leslie-Anne; Evans, Nate J.; Fowler, Jennifer M.; Dickey, Ellen A.Author affiliation(s)\n \nArgonne National LaboratoryPrimary contact\n \nEmail: dschabacker at anl dot govEditors\n \nDiEuliis, DianeYear published\n \n2019Volume and issue\n \n7Page(s)\n \n61DOI\n \n10.3389\/fbioe.2019.00061ISSN\n \n2296-4185Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00061\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00061\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Risk mitigation in the era of converging technologies \n4 Cyberbiosecurity \n5 Understanding key terms \n6 Considering dependencies and interdependencies in cyberbiosecurity \n7 Focusing on vulnerability \n8 Considering the human factor in cyberbiosecurity \n9 Roadmap for a cyberbiosecurity assessment framework \n10 Acknowledgements \n\n10.1 Author contributions \n10.2 Funding \n10.3 Conflict of interest \n\n\n11 References \n12 Notes \n\n\n\nAbstract \nThe convergence of advances in biotechnology with laboratory automation, access to data, and computational biology has democratized biotechnology and accelerated the development of new therapeutics. However, increased access to biotechnology in the digital age has also introduced additional security concerns and ultimately spawned the new discipline of cyberbiosecurity, which encompasses cybersecurity, cyber-physical security, and biosecurity considerations. With the emergence of this new discipline comes the need for a logical, repeatable, and shared approach for evaluating facility and system vulnerabilities to cyberbiosecurity threats. In this paper, we outline the foundation of an assessment framework for cyberbiosecurity, accounting for both security and resilience factors in the physical and cyber domains. This is a unique problem set, yet despite the complexity of the cyberbiosecurity field in terms of operations and governance, previous experience developing and implementing physical and cyber assessments applicable to a wide spectrum of critical infrastructure sectors provides a validated point of departure for a cyberbiosecurity assessment framework. This approach proposes to integrate existing capabilities and proven methodologies from the infrastructure assessment realm (e.g., decision science, physical security, infrastructure resilience, cybersecurity) with new expertise and requirements in the cyberbiosecurity space (e.g., biotechnology, biomanufacturing, genomics) in order to forge a flexible and defensible approach to identifying and mitigating vulnerabilities. Determining where vulnerabilities reside within cyberbiosecurity business processes can help public and private sector partners create an assessment framework to identify mitigation options for consideration that are both economically and practically viable and, ultimately, allow them to manage risk more effectively.\nKeywords: cyberbiosecurity, vulnerability, resilience, risk, convergence, emerging, converging, technology\n\nIntroduction \nAn important initial step in effectively managing risk is developing a comprehensive understanding of vulnerabilities. Stakeholders can then identify economical and practical options to mitigate vulnerabilities. Risk in the biological sciences has been managed through the implementation of standard biosecurity practices, through which vulnerabilities are (a) identified and (b) mitigated through regularly updated training, policies, and enhanced physical security. To prevent unauthorized access to high-consequence biological agents, the U.S. Government (USG) stood up the Federal Select Agent Program (FSAP), which added extensive requirements (e.g., background checks, registration by institutions, increased oversight) for those seeking access to \"biological select agents and toxins\" (BSATs). The BSAT list is based on taxonomic classifications and includes 67 high-consequence biological agents and toxins. Advances in genetic engineering tools (e.g., CRISPR Cas 9 systems) along with the convergence of lab automation, computational biology, and access to publicly available genomic databases will dramatically impact the effectiveness of the FSAP as well as other biosecurity policies and practices. It will no longer be necessary to obtain physical samples to exploit a biological agent; access to publicly available genomic databases, biofoundries, lab automation, and computational biology enables the design and production of high-consequence biological agents and toxins. These biological agents may be entirely new to nature and unconstrained by taxonomic classification such as the BSAT list.[1] This new digital environment in which biological research increasingly takes place must be systematically assessed for vulnerabilities in order to effectively manage evolving risks. The new discipline of cyberbiosecurity\u2014which includes biosecurity, cyber-physical security, and cybersecurity\u2014directly addresses the unique risks associated with biotechnology in an increasingly digital environment.[2][3]\nIn this paper, we outline the foundation of an assessment framework for cyberbiosecurity, accounting for both security and resilience factors in the physical and cyber domains. When implemented, the assessment framework will help partners identify and prioritize vulnerabilities. Importantly, the prioritization of vulnerabilities will result from a defensible, transparent, and reproducible assessment. In conjunction with an understanding of the consequences of disruption, risk mitigation strategies can be developed and considered in return-on-investment (ROI) analyses. ROIs will allow stakeholders to make informed decisions on how best to allocate limited resources for maximum impact.\nWhile biosecurity is one of the three disciplines comprising cyberbiosecurity (e.g., biosecurity, cyber-physical security, and cybersecurity), it is well-established and will not be discussed due to space limitations.\n\nRisk mitigation in the era of converging technologies \nEmerging and converging technologies present new risks to security that require new methodologies for risk prioritization and mitigation.\nThe accelerated pace of technological advancements across nearly all scientific disciplines has been driven largely by the convergence of advancements in scientific disciplines associated with computation, networking, automation, and access to data. Convergence occurs where scientific disciplines or key enabling technologies combine with other disciplines or enabling technologies and promise new or improved capabilities. Convergence is more than the simple combination of different disciplines or technologies. It leads to synergies, adding more value through convergence.[4]\nWhile converging technologies lead to fast and far-reaching improvements, they also create new security challenges and risks. We often try to address new risks with methods that were successful in the past; however, they may not be appropriate for the systemic risks posed by the increasing interconnectivity and complexity associated with converging technologies.[4] Additionally, with highly interconnected systems, the risk from dependencies and interdependencies must be considered. Therefore, we must take a more systemic approach to assessing and mitigating risks resulting from converging technologies.\nEmerging and converging technologies have significantly increased the number of vulnerabilities to national security to levels that are untenable for the government and private sector to address in their entirety. They simply do not have the resources required to implement mitigation strategies to address risks with a low probability of occurrence and\/or low consequence. Current conversations do not prioritize potential courses of action based on defensible integrated risk assessments that consider both probability and consequence in the context of converging technologies.\n\nCyberbiosecurity \nThe exploration of life sciences has become increasingly dependent upon internet-connected machinery and devices. Internet-dependent infrastructure is critical to computation and discovery of new avenues of research. The subsequent dependence upon technology and internet-connected devices begs the need to secure this infrastructure. For example, attackers could exploit unsecured networks and remotely manipulate biological material, creating new threats with devastating potential.[3] Cyberbiosecurity aims to understand and reduce the risks associated with conducting research using advanced technologies in the bioscience field. Science exploration depends increasingly upon cloud services, cyber-physical devices, internet-connected machines, remote databases, and many other cyber-vulnerable technologies. This convergence of science and cybersecurity opens the field to a new threat landscape.\nBelow are two examples of vulnerabilities that may not be individually identifiable in either a biosecurity or a cybersecurity context but are only apparent when both disciplines are considered.\nBringing together advances in synthetic biology and genetic engineering with machine learning, advanced modeling, metabolic engineering, and access to publicly available databases containing complete genome sequences of pathogens, including virulence factors, will enable the design of novel high consequence biological agents completely in silico. Minimal laboratory infrastructure and equipment would be required. Moreover, the vast array of publicly available open-source tools enable execution of these processes by less experienced personnel.\nAdvances in laboratory automation have enabled tacit knowledge (e.g., hands-on know-how), traditionally requiring years of professional laboratory training, to be codified into executable code controlling automated laboratory equipment. The ability of automated laboratory equipment to reproducibly perform tasks\u2014once limited to well-trained laboratorians\u2014has been monetized in the form of commercial biological production facilities (e.g., biofoundries). These biofoundries may unwittingly produce components of high-consequence biological agents solely from digital information provided by the customer. To request synthesis services, the customer simply goes to the website of the biofoundry and uploads the required biological data (e.g., DNA sequences, amino acid sequences, etc.). To obscure the identity and\/or functional properties of the final product, several biofoundries can be used, each synthesizing seemingly innocuous products representing only a portion of the final product.\nFurthermore, contributions to the exploration of science are built upon the open and sharing nature of samples and knowledge. This inherent openness and trust that exist in the scientific community is ripe for exploitation.[2] In order to thwart attackers and keep data secure, it is paramount that the CIA triad of confidentiality, integrity, and availability of scientific data is upheld in this digital era. Compromising any of the pillars within the CIA triad could lead to unwanted consequences. For example, attackers could:\n\n exploit vulnerable infrastructure and steal proprietary sequences from a biotechnology firm, ruining the confidentiality of the stolen intellectual property;\n manipulate DNA sequences for malicious intent, thereby destroying the integrity of a given sample or changing a sample to be something other then what is intended; or\n degrade systems, compromising the availability of cyber-physical devices that are used to perform needed functions.\nEnsuring the confidentiality, integrity, and availability of both the physical material and the associated digital information is essential to ensuring the safety and security of scientific advances in bioscience.\n\nUnderstanding key terms \nDefining the key elements of the emerging field of cyberbiosecurity is important to ensuring a common understanding of the relevant technical issues that arise from this new hybrid discipline. It is equally important to define key terms related to risk, particularly for audiences that may not already be familiar with the core concepts relevant to biosecurity, cyber-physical security, and cybersecurity assessments, policies, and practices. An important foundational document in this regard is the DHS Risk Lexicon, published in 2010 by the U.S. Department of Homeland Security to level-set terminology across the homeland security enterprise.[5]\nAs framed in the DHS Risk Lexicon, risk is the potential for an unwanted outcome resulting from an incident, event, or occurrence, as determined by its likelihood and the associated consequences. Evaluating the probability of adversarial attacks is challenging due in part to the lack of historical data in which to ground quantitative estimates, an inability to project that future deliberate threats will resemble those of the past, and the inherent challenges in evaluating the intent and capability of entities seeking to exploit weaknesses. Thus, risk in the Homeland Security space has been framed as a function of three elements: the threats to which an asset or system is susceptible, the vulnerabilities of the asset or system to the threat, and the potential consequences arising from the degradation of the asset or system. Each of these elements is defined below[5]:\n\nThreat\nnatural or human-caused occurrence, individual, entity, or action that has or indicates the potential to harm life, information, operations, the environment, and\/or property\nVulnerability\nphysical feature or operational attribute that renders an entity open to exploitation or susceptible to a given hazard\nConsequence\nthe effect of an event, incident, or occurrence; consequence is commonly deconstructed and measured in four categories: human, economic, mission, and psychological\nWhen talking about risk, it is also important to define what a hazard is due to its direct correlation and impact on vulnerabilities, threats, and consequences of an asset. A hazard is a natural or man-made source or cause of harm or difficulty. Threats are typically directed at an entity, asset, system, network, or geographic area, while a hazard is a natural or accidental phenomenon that is not driven consciously by an adversary.\nAlthough not typically identified as one of the three core factors driving risk, resilience is an additional consideration that impacts assessments of risk and ensuing strategies for managing it. As a result, it is relevant to understanding ways to evaluate cyberbiosecurity. Resilience is the ability to resist, absorb, recover from, or successfully adapt to adversity or a change in conditions.[5] Resilience features play a role in both the vulnerability and consequence variables in risk. Resilience measures can reduce vulnerability to various threats and hazards through protective measures that improve an organization's ability to resist an event or absorb its effects with minimal impact. Similarly, on the consequence side, resilience measures can enhance an entity's ability to quickly adapt and respond to an incident, as well as to recover and return to normal operations more quickly.[5][6]\nTaking into consideration all of these inputs, organizations can institute defensible, repeatable, and actionable processes to analyze risk and ultimately, to make decisions on how to manage it. Risk management is the process of identifying, analyzing, and communicating risk and then accepting, avoiding, transferring, or controlling it to an acceptable level and at an acceptable cost.[5] Risk management involves knowing the threats and hazards that could potentially impact a given organization, the vulnerabilities that render it susceptible to particular hazards, and the various consequences that might result. Figure 1 illustrates how these various components combine to drive risk-based decision-making.[7]\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 Risk management. By understanding the likelihood of various threats and hazards, associated vulnerabilities, potential consequences, and resilience characteristics, stakeholders can make informed decisions on ways to manage risk (i.e., accept, transfer, avoid, or mitigate).\n\n\n\nCyberbiosecurity is a new field that brings together different disciplines in new ways, triggering a pressing need for new thinking in terms of relevant threats, vulnerabilities, and consequences. Existing approaches used in biosecurity, cyber-physical security, and cybersecurity communities provide important foundational concepts and organizing principles, but they do not adequately capture emergent features related to biological and biomedical systems. Biosecurity, cyber-physical security, and cybersecurity are defined below.\n\nBiosecurity\ndescribes the protection, control, and accountability of biological materials in order to prevent their unauthorized access, loss, theft, misuse, diversion, or intentional release\nCyber-physical security\naddresses the potentially high-consequence dependency between physical systems and the special-purpose computers that control and monitor them\nCybersecurity\naddresses the risks of computer and network systems used for managing processes and sharing and protecting information\nConsidering dependencies and interdependencies in cyberbiosecurity \nIn addition to the concepts defined in the previous section, another concept that is relevant to understanding risk\u2014including but not limited to the cyberbiosecurity domain\u2014is the notion of how dependencies and interdependencies among and between complex systems impact overall risk. Dependencies and interdependencies are key to how the public and private sector understand, analyze, and manage risk within and across critical infrastructure sectors and other complex systems.\nA dependency is a unidirectional relationship between two assets, in which the operations of one asset affect the operations of the other. For example, a water treatment plant may depend on an external data source to process its water for potability.\nAn interdependency is a bidirectional relationship between two assets, in which the operations of both assets affect each other. For example, the water treatment plant requires communications for its supervisory control and data acquisition (SCADA) system, and, in turn, provides water used by the communications system to cool its equipment. An interdependency is effectively a combination of two dependencies; therefore, understanding an interdependency requires analyses of the one-way dependencies that comprise it.[8]\nEffective analysis of dependencies and interdependencies (whether for critical infrastructure, cyberbiosecurity, or other fields of study) requires some basic frameworks for defining, categorizing, and characterizing key features. For example, since infrastructure systems are constantly interacting with their environment and using inputs to generate outputs, it is important to identify where a dependency or interdependency exists within this activity chain. Upstream dependencies are the products or services provided to one system by an external source that are necessary to support its operations and functions. Internal dependencies involve interactions among internal operations, functions, and missions of the system. Downstream dependencies speak to the consumers or recipients who rely on the system's output and are affected by service disruptions or resource degradation.[8]\nDependencies and independencies are effectively risk multipliers: they can amplify vulnerabilities and consequences that arise from different threats and hazards. For example, loss of a service such as electric power can potentially affect other infrastructure systems that require power to operate, exacerbating the effects of the original power outage and possibly triggering other unanticipated downstream impacts. The presence of dependencies and interdependencies within the cyberbiosecurity domain make the already complex task of understanding risk that much more complicated, requiring analysts not only to evaluate threat, vulnerability, and consequence factors, but also to characterize relevant dependencies and interdependencies that can render complex systems more susceptible to disruption or exploitation.\n\nFocusing on vulnerability \nWhile the field of cyberbiosecurity is new, community members can leverage extensive knowledge and applications from other fields in order to begin stitching together an overarching framework for understanding relevant threats, vulnerabilities, and consequences from a cyberbiosecurity perspective, whether at a facility, system, or organizational level.\nBiolabs need an assessment toolkit that: (1) apply to a wide range of assets and systems across different sectors; (2) produce repeatable, defensible, and actionable results; (3) balance the need for efficiency with the need for detailed data; and (4) build on sound scientific principles, industry standards, and recognized best practices. The approaches above have been used to build and deliver multiple infrastructure assessment tools focused on vulnerability (e.g., Infrastructure Survey Tool [IST], Cyber Infrastructure Survey Tool [Cyber IST], Modified Infrastructure Survey Tool) and are based on the principles of decision analysis, an approach that can be used to manage risk under conditions of uncertainty.[9][10] When combined with additional analyses that evaluate potential threats and consequences of disruptions or loss, these processes can help biosecurity partners understand their broader risk environment and potential courses of action to mitigate risk.\nOne example application that could be helpful to the biosecurity community is the IST, which DHS field personnel use to evaluate security and resilience at critical infrastructure facilities nationwide in partnership with infrastructure owners and operators. The IST includes an index\u2014the Protective Measures Index (PMI)\u2014that characterizes the protective measures posture of individual facilities based on their most vulnerable aspects.[11][12] The PMI aggregates data collected through a structured onsite assessment process into four levels of information (or subcomponents) across five major categories. For each subcomponent, an index corresponding to the weighted sum of its subcomponents is calculated. This process results in an overall PMI that ranges from 0 (low protection) to 100 (high protection) for the critical infrastructure analyzed, as well as index values for various subcomponents.[6]\nThe decision analysis methodology used to define the PMI was specifically developed to integrate the major elements that are relevant to protecting critical infrastructure. The methodology integrates physical elements that are traditionally part of protection analysis (e.g., fencing, gates, entry controls, intrusion detection systems) as well as operational elements (e.g., security management, security planning, information-sharing mechanisms). The process for identifying specific security characteristics that contribute to protection at a facility and then establishing relative weights required a series of structured elicitation sessions with subject matter experts from public and private sectors.[7]\nUltimately, organizing PMI components into different levels and ranking their relative importance allows for the creation of reproducible results and visually compelling outputs that help owners and operators of critical infrastructure make tradeoff decisions on potential courses of action. Furthermore, the use of a consistent index and the consistent deployment of the toolset for a decade has allowed users to compare their results with other assets in the same sector.\nAnother example that could be helpful to analysis is the Cyber IST, which focuses on critical cyber services. A cyber service is any combination of equipment and devices (hardware), applications and platforms (software), communications, and data that have been integrated to provide specific business services. In this case that would classify as lab systems whose loss would result in physical destruction, safety, and health effects (e.g., a chemical release or loss of environment controls); theft of sensitive information that can be exploited; business interruption (e.g., denial of service); or other economic loss to the organization or its customers\/users. The Cyber IST generates a Cyber Protection and Resilience Index (CPRI) as its mechanism for organizations to use in comparative analysis.\nIn cybersecurity, identified threats, vulnerabilities, and consequences are often categorized into how these risks affect the confidentiality, integrity, and availability of a critical cyber service. These factors are considered the three most significant elements of reliable cybersecurity. Confidentiality limits who has access to information. Integrity governs how and when information is modified. Availability is the assurance that people who are authorized to access the information are able to do so. The question set for the Cyber IST was developed by subject matter experts based on the CIA triad, to assess how businesses help uphold the confidentiality, integrity, and availability of their critical cyber services.[13] This same question set provides the basis for assessing confidentiality, integrity, and availability of critical cyber services or assets within the context of cyberbiosecurity.\n\nConsidering the human factor in cyberbiosecurity \nInsiders pose substantial threats to cyberbiosecurity because they already have authorized access to critical systems. Most security measures are designed to protect the organization from external attacks and are often more difficult to implement to protect from internal attacks. The potential consequences of threats from insiders vary by the amount of trust and authority given to them.[14]\nInsiders include not only employees of the organization but also employees of trusted business partners, if those partners have access to the organization's systems, equipment, or data. The threats posed by insiders include both unintentional and intentional, both of which should be accounted for in cyberbiosecurity assessment frameworks. Unintentional incidents often result from negligence or misjudgment. Intentional incidents include insiders who commit fraud for financial gain or seek to sabotage the organization.\nBoth unintentional and intentional insider incidents can result from actions taken by external actors. For example, unintentional insider incidents may involve insider personnel responding to phishing or social engineering attacks from outside parties, while intentional incidents could involve personnel colluding with external actors, either voluntarily or under pressure. Insiders could willingly participate based on involvement in a cause or support to foreign government or organization, or they may fall victim to recruitment by a criminal enterprise either because of financial or personal troubles.[15]\n\nRoadmap for a cyberbiosecurity assessment framework \nMoving forward, the diverse community of researchers and practitioners in the cyberbiosecurity domain should collaborate to establish a common vulnerability assessment framework that is grounded in decision science, apply lessons learned from parallel efforts in related fields, and reflect the complex multidisciplinary cyberbiosecurity environment. Key steps in this process should include:\n\n engaging subject matter experts in decision science, biotechnology, biosecurity, cyber-physical security, cybersecurity, and physical security in a collaborative assessment development process;\n defining functional requirements of assessment processes to ensure common understanding of goals, objectives, and constraints;\n characterizing the biotechnology ecosystem based on facility type (e.g., universities, biofoundries, pharmaceutical companies) and supporting systems (e.g., bioprocess, supply chain, supporting information systems, facility infrastructure) to identify likely assessment candidates and pathways;\n identifying relevant industry standards, legal frameworks, and regulatory regimes that apply to cyberbiosecurity;\n establishing a comprehensive taxonomy of characteristics in physical assets and cyber systems in the biotechnology community that influence security posture (e.g., access control, security management, personnel, response protocols, dependencies);\n conducting an iterative elicitation process to establish subject matter expert consensus on relative importance of security characteristics and their subcomponents in order to facilitate data aggregation, comparison with like entities, and alternatives analysis; and\n exploring potential approaches for collecting assessment data and visualizing assessment results.\nAcknowledgements \nAuthor contributions \nDS conceived the manuscript and all authors have jointly contributed to the manuscript, with particular contribution of L-AL to the text on the cyberbiosecurity assessment framework and dependencies. Contribution of NE to the text on cybersecurity and the human factor. All authors have read and approved the manuscript for publication.\n\nFunding \nWriting of the manuscript was funded by Argonne National Laboratory. Argonne National Laboratory's work was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Wintle, B.C.; Boehm, C.R.; Rhodes, C. et al. (2017). \"A transatlantic perspective on 20 emerging issues in biological engineering\". eLife 6: e30247. doi:10.7554\/eLife.30247. PMC PMC5685469. PMID 29132504. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469 .   \n\n\u2191 2.0 2.1 Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". Trends in Biotechnology 36 (1): 4\u20137. doi:10.1016\/j.tibtech.2017.10.012. PMID 29224719.   \n\n\u2191 3.0 3.1 Murch, R.S.; Wo, W.K.; Buchholz, W.G. et al. (2018). \"Cyberbiosecurity: An Emerging New Discipline to Help Safeguard the Bioeconomy\". Frontiers in Bioengineering and Biotechnology 6: 39. doi:10.3389\/fbioe.2018.00039. PMC PMC5895716. PMID 29675411. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5895716 .   \n\n\u2191 4.0 4.1 Dengg, A., ed. (2018) (PDF). Tomorrow's Technology: A Double-Edged Sword. Federal Ministry of Defence, Republic of Austria. ISBN 9783903121317. http:\/\/www.bundesheer.at\/pdf_pool\/publikationen\/buch_dengg_tomorrows_technology_web.pdf .   \n\n\u2191 5.0 5.1 5.2 5.3 5.4 Risk Steering Committee (September 2010). \"DHS Risk Lexicon, 2010 Edition\" (PDF). Department of Homeland Security. https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/dhs-risk-lexicon-2010_0.pdf .   \n\n\u2191 6.0 6.1 Petit, F.D.P.; Bassett, G.W.; Black, R. et al. (15 July 2013). \"Resilience Measurement Index: An Indicator of Critical Infrastructure Resilience\". Argonne National Laboratory. doi:10.2172\/1087819. https:\/\/www.osti.gov\/biblio\/1087819-resilience-measurement-index-indicator-critical-infrastructure-resilience .   \n\n\u2191 7.0 7.1 Petit, F.D.P.; Bassett, G.W.; Buehring, W.A. et al. (21 November 2013). \"Protective Measures Index and Vulnerability Index: Indicators of Critical Infrastructure Protection and Vulnerability\". Argonne National Laboratory. doi:10.2172\/1108161. https:\/\/www.osti.gov\/biblio\/1108161 .   \n\n\u2191 8.0 8.1 Petit, F.D.P.; Verner, D.; Brannegan, D. et al. (01 June 2015). \"Analysis of Critical Infrastructure Dependencies and Interdependencies\". Argonne National Laboratory. doi:10.2172\/1184636. https:\/\/www.osti.gov\/biblio\/1184636-analysis-critical-infrastructure-dependencies-interdependencies .   \n\n\u2191 Keeney, R.L.; Raiffa, H. (1976). Decisions with multiple objectives: Preferences and value tradeoffs. Wiley. pp. 569. ISBN 9780471465102.   \n\n\u2191 Keeney, R.L. (1992). Value\u2010focused thinking: A path to creative decision making. Harvard University Press. ISBN 9780674931971.   \n\n\u2191 Fisher, R.E.; Buehring, W.A.; Whitfield, R.G. et al. (14 October 2009). \"Constructing vulnerabilty and protective measures indices for the enhanced critical infrastructure protection program\". Argonne National Laboratory. doi:10.2172\/966343. https:\/\/www.osti.gov\/biblio\/966343-constructing-vulnerabilty-protective-measures-indices-enhanced-critical-infrastructure-protection-program .   \n\n\u2191 Collins, M.; Petit, F.; Buehring, W. et al. (2011). \"Protective measures and vulnerability indices for the Enhanced Critical Infrastructure Protection Programme\". International Journal of Critical Infrastructures 7 (3): 200\u201319. doi:10.1504\/IJCIS.2011.042976.   \n\n\u2191 Joyce, A.L.; Petit, F.D.; Phillips, J.A. et al. (01 October 2017). \"Cyber Protection and Resilience Index: An Indicator of an Organization's Cyber Protection and Resilience Program\". Argonne National Laboratory. doi:10.2172\/1433503. https:\/\/www.osti.gov\/biblio\/1433503-cyber-protection-resilience-index-indicator-organization-cyber-protection-resilience-program .   \n\n\u2191 Evans, N.J. (2009). \"Information technology social engineering: An academic definition and study of social engineering - Analyzing the human firewall\". Graduate Theses and Dissertations. doi:10.31274\/etd-180810-436. https:\/\/lib.dr.iastate.edu\/etd\/10709\/ .   \n\n\u2191 Perkins, D.; Fabregas, E. (2017). Mitigating Insider Threats through Strengthening Organizations' Culture of Biosafety, Biosecurity, and Responsible Conduct [White paper]. Submitted to the Committee on a Decadal Survey of Social and Behavioral Sciences and Applications to National Security, National Academies of Sciences, Engineering, and Medicine. pp. 1\u20136. http:\/\/sites.nationalacademies.org\/DBASSE\/BBCSS\/DBASSE_175673 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\">https:\/\/www.limswiki.org\/index.php\/Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on cybersecurity\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 7 May 2019, at 16:07.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 234 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","7685e865ec42d1c3178d9cb2493521d4_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Assessing cyberbiosecurity vulnerabilities and infrastructure resilience<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The convergence of advances in <a href=\"https:\/\/www.limswiki.org\/index.php\/Biotechnology\" title=\"Biotechnology\" class=\"wiki-link\" data-key=\"115005039d4cf0b4ef55ec14dc6d66da\">biotechnology<\/a> with <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">laboratory automation<\/a>, access to data, and computational biology has democratized biotechnology and accelerated the development of new therapeutics. However, increased access to biotechnology in the digital age has also introduced additional security concerns and ultimately spawned the new discipline of cyberbiosecurity, which encompasses <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a>, cyber-physical security, and biosecurity considerations. With the emergence of this new discipline comes the need for a logical, repeatable, and shared approach for evaluating facility and system vulnerabilities to cyberbiosecurity threats. In this paper, we outline the foundation of an assessment framework for cyberbiosecurity, accounting for both security and resilience factors in the physical and cyber domains. This is a unique problem set, yet despite the complexity of the cyberbiosecurity field in terms of operations and governance, previous experience developing and implementing physical and cyber assessments applicable to a wide spectrum of critical infrastructure sectors provides a validated point of departure for a cyberbiosecurity assessment framework. This approach proposes to integrate existing capabilities and proven methodologies from the infrastructure assessment realm (e.g., decision science, physical security, infrastructure resilience, cybersecurity) with new expertise and requirements in the cyberbiosecurity space (e.g., biotechnology, biomanufacturing, <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomics<\/a>) in order to forge a flexible and defensible approach to identifying and mitigating vulnerabilities. Determining where vulnerabilities reside within cyberbiosecurity business processes can help public and private sector partners create an assessment framework to identify mitigation options for consideration that are both economically and practically viable and, ultimately, allow them to manage risk more effectively.\n<\/p><p><b>Keywords<\/b>: cyberbiosecurity, vulnerability, resilience, risk, convergence, emerging, converging, technology\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>An important initial step in effectively managing risk is developing a comprehensive understanding of vulnerabilities. Stakeholders can then identify economical and practical options to mitigate vulnerabilities. Risk in the biological sciences has been managed through the implementation of standard biosecurity practices, through which vulnerabilities are (a) identified and (b) mitigated through regularly updated training, policies, and enhanced physical security. To prevent unauthorized access to high-consequence biological agents, the U.S. Government (USG) stood up the Federal Select Agent Program (FSAP), which added extensive requirements (e.g., background checks, registration by institutions, increased oversight) for those seeking access to \"biological select agents and toxins\" (BSATs). The BSAT list is based on taxonomic classifications and includes 67 high-consequence biological agents and toxins. Advances in genetic engineering tools (e.g., CRISPR Cas 9 systems) along with the convergence of lab automation, computational biology, and access to publicly available genomic databases will dramatically impact the effectiveness of the FSAP as well as other biosecurity policies and practices. It will no longer be necessary to obtain physical samples to exploit a biological agent; access to publicly available genomic databases, biofoundries, lab automation, and computational biology enables the design and production of high-consequence biological agents and toxins. These biological agents may be entirely new to nature and unconstrained by taxonomic classification such as the BSAT list.<sup id=\"rdp-ebb-cite_ref-WintleATrans17_1-0\" class=\"reference\"><a href=\"#cite_note-WintleATrans17-1\">[1]<\/a><\/sup> This new digital environment in which biological research increasingly takes place must be systematically assessed for vulnerabilities in order to effectively manage evolving risks. The new discipline of cyberbiosecurity\u2014which includes biosecurity, cyber-physical security, and cybersecurity\u2014directly addresses the unique risks associated with biotechnology in an increasingly digital environment.<sup id=\"rdp-ebb-cite_ref-PeccoudCyber18_2-0\" class=\"reference\"><a href=\"#cite_note-PeccoudCyber18-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MurchCyber18_3-0\" class=\"reference\"><a href=\"#cite_note-MurchCyber18-3\">[3]<\/a><\/sup>\n<\/p><p>In this paper, we outline the foundation of an assessment framework for cyberbiosecurity, accounting for both security and resilience factors in the physical and cyber domains. When implemented, the assessment framework will help partners identify and prioritize vulnerabilities. Importantly, the prioritization of vulnerabilities will result from a defensible, transparent, and reproducible assessment. In conjunction with an understanding of the consequences of disruption, risk mitigation strategies can be developed and considered in return-on-investment (ROI) analyses. ROIs will allow stakeholders to make informed decisions on how best to allocate limited resources for maximum impact.\n<\/p><p>While biosecurity is one of the three disciplines comprising cyberbiosecurity (e.g., biosecurity, cyber-physical security, and cybersecurity), it is well-established and will not be discussed due to space limitations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Risk_mitigation_in_the_era_of_converging_technologies\">Risk mitigation in the era of converging technologies<\/span><\/h2>\n<p>Emerging and converging technologies present new risks to security that require new methodologies for risk prioritization and mitigation.\n<\/p><p>The accelerated pace of technological advancements across nearly all scientific disciplines has been driven largely by the convergence of advancements in scientific disciplines associated with computation, networking, automation, and access to data. Convergence occurs where scientific disciplines or key enabling technologies combine with other disciplines or enabling technologies and promise new or improved capabilities. Convergence is more than the simple combination of different disciplines or technologies. It leads to synergies, adding more value through convergence.<sup id=\"rdp-ebb-cite_ref-DenggTomorrow18_4-0\" class=\"reference\"><a href=\"#cite_note-DenggTomorrow18-4\">[4]<\/a><\/sup>\n<\/p><p>While converging technologies lead to fast and far-reaching improvements, they also create new security challenges and risks. We often try to address new risks with methods that were successful in the past; however, they may not be appropriate for the systemic risks posed by the increasing interconnectivity and complexity associated with converging technologies.<sup id=\"rdp-ebb-cite_ref-DenggTomorrow18_4-1\" class=\"reference\"><a href=\"#cite_note-DenggTomorrow18-4\">[4]<\/a><\/sup> Additionally, with highly interconnected systems, the risk from dependencies and interdependencies must be considered. Therefore, we must take a more systemic approach to assessing and mitigating risks resulting from converging technologies.\n<\/p><p>Emerging and converging technologies have significantly increased the number of vulnerabilities to national security to levels that are untenable for the government and private sector to address in their entirety. They simply do not have the resources required to implement mitigation strategies to address risks with a low probability of occurrence and\/or low consequence. Current conversations do not prioritize potential courses of action based on defensible integrated risk assessments that consider both probability and consequence in the context of converging technologies.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Cyberbiosecurity\">Cyberbiosecurity<\/span><\/h2>\n<p>The exploration of life sciences has become increasingly dependent upon internet-connected machinery and devices. Internet-dependent infrastructure is critical to computation and discovery of new avenues of research. The subsequent dependence upon technology and internet-connected devices begs the need to secure this infrastructure. For example, attackers could exploit unsecured networks and remotely manipulate biological material, creating new threats with devastating potential.<sup id=\"rdp-ebb-cite_ref-MurchCyber18_3-1\" class=\"reference\"><a href=\"#cite_note-MurchCyber18-3\">[3]<\/a><\/sup> Cyberbiosecurity aims to understand and reduce the risks associated with conducting research using advanced technologies in the bioscience field. Science exploration depends increasingly upon <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud services<\/a>, cyber-physical devices, internet-connected machines, remote databases, and many other cyber-vulnerable technologies. This convergence of science and cybersecurity opens the field to a new threat landscape.\n<\/p><p>Below are two examples of vulnerabilities that may not be individually identifiable in either a biosecurity or a cybersecurity context but are only apparent when both disciplines are considered.\n<\/p><p>Bringing together advances in synthetic biology and genetic engineering with machine learning, advanced modeling, metabolic engineering, and access to publicly available databases containing complete genome sequences of pathogens, including virulence factors, will enable the design of novel high consequence biological agents completely <i>in silico<\/i>. Minimal laboratory infrastructure and equipment would be required. Moreover, the vast array of publicly available open-source tools enable execution of these processes by less experienced personnel.\n<\/p><p>Advances in laboratory automation have enabled tacit knowledge (e.g., hands-on know-how), traditionally requiring years of professional laboratory training, to be codified into executable code controlling automated laboratory equipment. The ability of automated laboratory equipment to reproducibly perform tasks\u2014once limited to well-trained laboratorians\u2014has been monetized in the form of commercial biological production facilities (e.g., biofoundries). These biofoundries may unwittingly produce components of high-consequence biological agents solely from digital information provided by the customer. To request synthesis services, the customer simply goes to the website of the biofoundry and uploads the required biological data (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/DNA_sequencing\" title=\"DNA sequencing\" class=\"wiki-link\" data-key=\"7ff86b38049c37e30858efd13bd00925\">DNA sequences<\/a>, amino acid sequences, etc.). To obscure the identity and\/or functional properties of the final product, several biofoundries can be used, each synthesizing seemingly innocuous products representing only a portion of the final product.\n<\/p><p>Furthermore, contributions to the exploration of science are built upon the open and sharing nature of samples and knowledge. This inherent openness and trust that exist in the scientific community is ripe for exploitation.<sup id=\"rdp-ebb-cite_ref-PeccoudCyber18_2-1\" class=\"reference\"><a href=\"#cite_note-PeccoudCyber18-2\">[2]<\/a><\/sup> In order to thwart attackers and keep data secure, it is paramount that the CIA triad of confidentiality, integrity, and availability of scientific data is upheld in this digital era. Compromising any of the pillars within the CIA triad could lead to unwanted consequences. For example, attackers could:\n<\/p>\n<ul><li> exploit vulnerable infrastructure and steal proprietary sequences from a biotechnology firm, ruining the <i>confidentiality<\/i> of the stolen intellectual property;<\/li><\/ul>\n<ul><li> manipulate DNA sequences for malicious intent, thereby destroying the <i>integrity<\/i> of a given sample or changing a sample to be something other then what is intended; or<\/li><\/ul>\n<ul><li> degrade systems, compromising the <i>availability<\/i> of cyber-physical devices that are used to perform needed functions.<\/li><\/ul>\n<p>Ensuring the confidentiality, integrity, and availability of both the physical material and the associated digital information is essential to ensuring the safety and security of scientific advances in bioscience.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Understanding_key_terms\">Understanding key terms<\/span><\/h2>\n<p>Defining the key elements of the emerging field of cyberbiosecurity is important to ensuring a common understanding of the relevant technical issues that arise from this new hybrid discipline. It is equally important to define key terms related to risk, particularly for audiences that may not already be familiar with the core concepts relevant to biosecurity, cyber-physical security, and cybersecurity assessments, policies, and practices. An important foundational document in this regard is the <i>DHS Risk Lexicon<\/i>, published in 2010 by the U.S. Department of Homeland Security to level-set terminology across the homeland security enterprise.<sup id=\"rdp-ebb-cite_ref-DHSRisk10_5-0\" class=\"reference\"><a href=\"#cite_note-DHSRisk10-5\">[5]<\/a><\/sup>\n<\/p><p>As framed in the <i>DHS Risk Lexicon<\/i>, risk is the potential for an unwanted outcome resulting from an incident, event, or occurrence, as determined by its likelihood and the associated consequences. Evaluating the probability of adversarial attacks is challenging due in part to the lack of historical data in which to ground quantitative estimates, an inability to project that future deliberate threats will resemble those of the past, and the inherent challenges in evaluating the intent and capability of entities seeking to exploit weaknesses. Thus, risk in the Homeland Security space has been framed as a function of three elements: the threats to which an asset or system is susceptible, the vulnerabilities of the asset or system to the threat, and the potential consequences arising from the degradation of the asset or system. Each of these elements is defined below<sup id=\"rdp-ebb-cite_ref-DHSRisk10_5-1\" class=\"reference\"><a href=\"#cite_note-DHSRisk10-5\">[5]<\/a><\/sup>:\n<\/p>\n<dl><dt>Threat<\/dt>\n<dd>natural or human-caused occurrence, individual, entity, or action that has or indicates the potential to harm life, information, operations, the environment, and\/or property<\/dd><\/dl>\n<dl><dt>Vulnerability<\/dt>\n<dd>physical feature or operational attribute that renders an entity open to exploitation or susceptible to a given hazard<\/dd><\/dl>\n<dl><dt>Consequence<\/dt>\n<dd>the effect of an event, incident, or occurrence; consequence is commonly deconstructed and measured in four categories: human, economic, mission, and psychological<\/dd><\/dl>\n<p>When talking about risk, it is also important to define what a hazard is due to its direct correlation and impact on vulnerabilities, threats, and consequences of an asset. A hazard is a natural or man-made source or cause of harm or difficulty. Threats are typically directed at an entity, asset, system, network, or geographic area, while a hazard is a natural or accidental phenomenon that is not driven consciously by an adversary.\n<\/p><p>Although not typically identified as one of the three core factors driving risk, resilience is an additional consideration that impacts assessments of risk and ensuing strategies for managing it. As a result, it is relevant to understanding ways to evaluate cyberbiosecurity. Resilience is the ability to resist, absorb, recover from, or successfully adapt to adversity or a change in conditions.<sup id=\"rdp-ebb-cite_ref-DHSRisk10_5-2\" class=\"reference\"><a href=\"#cite_note-DHSRisk10-5\">[5]<\/a><\/sup> Resilience features play a role in both the vulnerability and consequence variables in risk. Resilience measures can reduce vulnerability to various threats and hazards through protective measures that improve an organization's ability to resist an event or absorb its effects with minimal impact. Similarly, on the consequence side, resilience measures can enhance an entity's ability to quickly adapt and respond to an incident, as well as to recover and return to normal operations more quickly.<sup id=\"rdp-ebb-cite_ref-DHSRisk10_5-3\" class=\"reference\"><a href=\"#cite_note-DHSRisk10-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PetitResil13_6-0\" class=\"reference\"><a href=\"#cite_note-PetitResil13-6\">[6]<\/a><\/sup>\n<\/p><p>Taking into consideration all of these inputs, organizations can institute defensible, repeatable, and actionable processes to analyze risk and ultimately, to make decisions on how to manage it. Risk management is the process of identifying, analyzing, and communicating risk and then accepting, avoiding, transferring, or controlling it to an acceptable level and at an acceptable cost.<sup id=\"rdp-ebb-cite_ref-DHSRisk10_5-4\" class=\"reference\"><a href=\"#cite_note-DHSRisk10-5\">[5]<\/a><\/sup> Risk management involves knowing the threats and hazards that could potentially impact a given organization, the vulnerabilities that render it susceptible to particular hazards, and the various consequences that might result. Figure 1 illustrates how these various components combine to drive risk-based decision-making.<sup id=\"rdp-ebb-cite_ref-PetitProtect13_7-0\" class=\"reference\"><a href=\"#cite_note-PetitProtect13-7\">[7]<\/a><\/sup>\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Schabacker_FrontBioengBiotech2019_7.jpg\" class=\"image wiki-link\" data-key=\"5f4a5bd916cd4565e5b922b55843954b\"><img alt=\"Fig1 Schabacker FrontBioengBiotech2019 7.jpg\" src=\"https:\/\/www.limswiki.org\/images\/6\/69\/Fig1_Schabacker_FrontBioengBiotech2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> Risk management. By understanding the likelihood of various threats and hazards, associated vulnerabilities, potential consequences, and resilience characteristics, stakeholders can make informed decisions on ways to manage risk (i.e., accept, transfer, avoid, or mitigate).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Cyberbiosecurity is a new field that brings together different disciplines in new ways, triggering a pressing need for new thinking in terms of relevant threats, vulnerabilities, and consequences. Existing approaches used in biosecurity, cyber-physical security, and cybersecurity communities provide important foundational concepts and organizing principles, but they do not adequately capture emergent features related to biological and biomedical systems. Biosecurity, cyber-physical security, and cybersecurity are defined below.\n<\/p>\n<dl><dt>Biosecurity<\/dt>\n<dd>describes the protection, control, and accountability of biological materials in order to prevent their unauthorized access, loss, theft, misuse, diversion, or intentional release<\/dd><\/dl>\n<dl><dt>Cyber-physical security<\/dt>\n<dd>addresses the potentially high-consequence dependency between physical systems and the special-purpose computers that control and monitor them<\/dd><\/dl>\n<dl><dt>Cybersecurity<\/dt>\n<dd>addresses the risks of computer and network systems used for managing processes and sharing and protecting information<\/dd><\/dl>\n<h2><span class=\"mw-headline\" id=\"Considering_dependencies_and_interdependencies_in_cyberbiosecurity\">Considering dependencies and interdependencies in cyberbiosecurity<\/span><\/h2>\n<p>In addition to the concepts defined in the previous section, another concept that is relevant to understanding risk\u2014including but not limited to the cyberbiosecurity domain\u2014is the notion of how dependencies and interdependencies among and between complex systems impact overall risk. Dependencies and interdependencies are key to how the public and private sector understand, analyze, and manage risk within and across critical infrastructure sectors and other complex systems.\n<\/p><p>A dependency is a unidirectional relationship between two assets, in which the operations of one asset affect the operations of the other. For example, a water treatment plant may depend on an external data source to process its water for potability.\n<\/p><p>An interdependency is a bidirectional relationship between two assets, in which the operations of both assets affect each other. For example, the water treatment plant requires communications for its <a href=\"https:\/\/www.limswiki.org\/index.php\/Supervisory_control_and_data_acquisition\" title=\"Supervisory control and data acquisition\" class=\"mw-redirect wiki-link\" data-key=\"15a9cf66d1585180cc0c2afeb1a0f817\">supervisory control and data acquisition<\/a> (SCADA) system, and, in turn, provides water used by the communications system to cool its equipment. An interdependency is effectively a combination of two dependencies; therefore, understanding an interdependency requires analyses of the one-way dependencies that comprise it.<sup id=\"rdp-ebb-cite_ref-PetitAnal15_8-0\" class=\"reference\"><a href=\"#cite_note-PetitAnal15-8\">[8]<\/a><\/sup>\n<\/p><p>Effective analysis of dependencies and interdependencies (whether for critical infrastructure, cyberbiosecurity, or other fields of study) requires some basic frameworks for defining, categorizing, and characterizing key features. For example, since infrastructure systems are constantly interacting with their environment and using inputs to generate outputs, it is important to identify where a dependency or interdependency exists within this activity chain. Upstream dependencies are the products or services provided to one system by an external source that are necessary to support its operations and functions. Internal dependencies involve interactions among internal operations, functions, and missions of the system. Downstream dependencies speak to the consumers or recipients who rely on the system's output and are affected by service disruptions or resource degradation.<sup id=\"rdp-ebb-cite_ref-PetitAnal15_8-1\" class=\"reference\"><a href=\"#cite_note-PetitAnal15-8\">[8]<\/a><\/sup>\n<\/p><p>Dependencies and independencies are effectively risk multipliers: they can amplify vulnerabilities and consequences that arise from different threats and hazards. For example, loss of a service such as electric power can potentially affect other infrastructure systems that require power to operate, exacerbating the effects of the original power outage and possibly triggering other unanticipated downstream impacts. The presence of dependencies and interdependencies within the cyberbiosecurity domain make the already complex task of understanding risk that much more complicated, requiring analysts not only to evaluate threat, vulnerability, and consequence factors, but also to characterize relevant dependencies and interdependencies that can render complex systems more susceptible to disruption or exploitation.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Focusing_on_vulnerability\">Focusing on vulnerability<\/span><\/h2>\n<p>While the field of cyberbiosecurity is new, community members can leverage extensive knowledge and applications from other fields in order to begin stitching together an overarching framework for understanding relevant threats, vulnerabilities, and consequences from a cyberbiosecurity perspective, whether at a facility, system, or organizational level.\n<\/p><p>Biolabs need an assessment toolkit that: (1) apply to a wide range of assets and systems across different sectors; (2) produce repeatable, defensible, and actionable results; (3) balance the need for efficiency with the need for detailed data; and (4) build on sound scientific principles, industry standards, and recognized best practices. The approaches above have been used to build and deliver multiple infrastructure assessment tools focused on vulnerability (e.g., Infrastructure Survey Tool [IST], Cyber Infrastructure Survey Tool [Cyber IST], Modified Infrastructure Survey Tool) and are based on the principles of decision analysis, an approach that can be used to manage risk under conditions of uncertainty.<sup id=\"rdp-ebb-cite_ref-KeeneyDecisions76_9-0\" class=\"reference\"><a href=\"#cite_note-KeeneyDecisions76-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KeeneyValue92_10-0\" class=\"reference\"><a href=\"#cite_note-KeeneyValue92-10\">[10]<\/a><\/sup> When combined with additional analyses that evaluate potential threats and consequences of disruptions or loss, these processes can help biosecurity partners understand their broader risk environment and potential courses of action to mitigate risk.\n<\/p><p>One example application that could be helpful to the biosecurity community is the IST, which DHS field personnel use to evaluate security and resilience at critical infrastructure facilities nationwide in partnership with infrastructure owners and operators. The IST includes an index\u2014the Protective Measures Index (PMI)\u2014that characterizes the protective measures posture of individual facilities based on their most vulnerable aspects.<sup id=\"rdp-ebb-cite_ref-FisherConstruct09_11-0\" class=\"reference\"><a href=\"#cite_note-FisherConstruct09-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CollinsProtect11_12-0\" class=\"reference\"><a href=\"#cite_note-CollinsProtect11-12\">[12]<\/a><\/sup> The PMI aggregates data collected through a structured onsite assessment process into four levels of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> (or subcomponents) across five major categories. For each subcomponent, an index corresponding to the weighted sum of its subcomponents is calculated. This process results in an overall PMI that ranges from 0 (low protection) to 100 (high protection) for the critical infrastructure analyzed, as well as index values for various subcomponents.<sup id=\"rdp-ebb-cite_ref-PetitResil13_6-1\" class=\"reference\"><a href=\"#cite_note-PetitResil13-6\">[6]<\/a><\/sup>\n<\/p><p>The decision analysis methodology used to define the PMI was specifically developed to integrate the major elements that are relevant to protecting critical infrastructure. The methodology integrates physical elements that are traditionally part of protection analysis (e.g., fencing, gates, entry controls, intrusion detection systems) as well as operational elements (e.g., security management, security planning, information-sharing mechanisms). The process for identifying specific security characteristics that contribute to protection at a facility and then establishing relative weights required a series of structured elicitation sessions with subject matter experts from public and private sectors.<sup id=\"rdp-ebb-cite_ref-PetitProtect13_7-1\" class=\"reference\"><a href=\"#cite_note-PetitProtect13-7\">[7]<\/a><\/sup>\n<\/p><p>Ultimately, organizing PMI components into different levels and ranking their relative importance allows for the creation of reproducible results and visually compelling outputs that help owners and operators of critical infrastructure make tradeoff decisions on potential courses of action. Furthermore, the use of a consistent index and the consistent deployment of the toolset for a decade has allowed users to compare their results with other assets in the same sector.\n<\/p><p>Another example that could be helpful to analysis is the Cyber IST, which focuses on critical cyber services. A cyber service is any combination of equipment and devices (hardware), applications and platforms (software), communications, and data that have been integrated to provide specific business services. In this case that would classify as lab systems whose loss would result in physical destruction, safety, and health effects (e.g., a chemical release or loss of environment controls); theft of sensitive information that can be exploited; business interruption (e.g., denial of service); or other economic loss to the organization or its customers\/users. The Cyber IST generates a Cyber Protection and Resilience Index (CPRI) as its mechanism for organizations to use in comparative analysis.\n<\/p><p>In cybersecurity, identified threats, vulnerabilities, and consequences are often categorized into how these risks affect the confidentiality, integrity, and availability of a critical cyber service. These factors are considered the three most significant elements of reliable cybersecurity. Confidentiality limits who has access to information. Integrity governs how and when information is modified. Availability is the assurance that people who are authorized to access the information are able to do so. The question set for the Cyber IST was developed by subject matter experts based on the CIA triad, to assess how businesses help uphold the confidentiality, integrity, and availability of their critical cyber services.<sup id=\"rdp-ebb-cite_ref-JoyceCyber17_13-0\" class=\"reference\"><a href=\"#cite_note-JoyceCyber17-13\">[13]<\/a><\/sup> This same question set provides the basis for assessing confidentiality, integrity, and availability of critical cyber services or assets within the context of cyberbiosecurity.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Considering_the_human_factor_in_cyberbiosecurity\">Considering the human factor in cyberbiosecurity<\/span><\/h2>\n<p>Insiders pose substantial threats to cyberbiosecurity because they already have authorized access to critical systems. Most security measures are designed to protect the organization from external attacks and are often more difficult to implement to protect from internal attacks. The potential consequences of threats from insiders vary by the amount of trust and authority given to them.<sup id=\"rdp-ebb-cite_ref-EvansInfo09_14-0\" class=\"reference\"><a href=\"#cite_note-EvansInfo09-14\">[14]<\/a><\/sup>\n<\/p><p>Insiders include not only employees of the organization but also employees of trusted business partners, if those partners have access to the organization's systems, equipment, or data. The threats posed by insiders include both unintentional and intentional, both of which should be accounted for in cyberbiosecurity assessment frameworks. Unintentional incidents often result from negligence or misjudgment. Intentional incidents include insiders who commit fraud for financial gain or seek to sabotage the organization.\n<\/p><p>Both unintentional and intentional insider incidents can result from actions taken by external actors. For example, unintentional insider incidents may involve insider personnel responding to phishing or social engineering attacks from outside parties, while intentional incidents could involve personnel colluding with external actors, either voluntarily or under pressure. Insiders could willingly participate based on involvement in a cause or support to foreign government or organization, or they may fall victim to recruitment by a criminal enterprise either because of financial or personal troubles.<sup id=\"rdp-ebb-cite_ref-PerkinsMit18_15-0\" class=\"reference\"><a href=\"#cite_note-PerkinsMit18-15\">[15]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Roadmap_for_a_cyberbiosecurity_assessment_framework\">Roadmap for a cyberbiosecurity assessment framework<\/span><\/h2>\n<p>Moving forward, the diverse community of researchers and practitioners in the cyberbiosecurity domain should collaborate to establish a common vulnerability assessment framework that is grounded in decision science, apply lessons learned from parallel efforts in related fields, and reflect the complex multidisciplinary cyberbiosecurity environment. Key steps in this process should include:\n<\/p>\n<ul><li> engaging subject matter experts in decision science, biotechnology, biosecurity, cyber-physical security, cybersecurity, and physical security in a collaborative assessment development process;<\/li><\/ul>\n<ul><li> defining functional requirements of assessment processes to ensure common understanding of goals, objectives, and constraints;<\/li><\/ul>\n<ul><li> characterizing the biotechnology ecosystem based on facility type (e.g., universities, biofoundries, pharmaceutical companies) and supporting systems (e.g., bioprocess, supply chain, supporting information systems, facility infrastructure) to identify likely assessment candidates and pathways;<\/li><\/ul>\n<ul><li> identifying relevant industry standards, legal frameworks, and regulatory regimes that apply to cyberbiosecurity;<\/li><\/ul>\n<ul><li> establishing a comprehensive taxonomy of characteristics in physical assets and cyber systems in the biotechnology community that influence security posture (e.g., access control, security management, personnel, response protocols, dependencies);<\/li><\/ul>\n<ul><li> conducting an iterative elicitation process to establish subject matter expert consensus on relative importance of security characteristics and their subcomponents in order to facilitate data aggregation, comparison with like entities, and alternatives analysis; and<\/li><\/ul>\n<ul><li> exploring potential approaches for collecting assessment data and visualizing assessment results.<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>DS conceived the manuscript and all authors have jointly contributed to the manuscript, with particular contribution of L-AL to the text on the cyberbiosecurity assessment framework and dependencies. Contribution of NE to the text on cybersecurity and the human factor. All authors have read and approved the manuscript for publication.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>Writing of the manuscript was funded by Argonne National Laboratory. Argonne National Laboratory's work was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-WintleATrans17-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WintleATrans17_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wintle, B.C.; Boehm, C.R.; Rhodes, C. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469\" target=\"_blank\">\"A transatlantic perspective on 20 emerging issues in biological engineering\"<\/a>. <i>eLife<\/i> <b>6<\/b>: e30247. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7554%2FeLife.30247\" target=\"_blank\">10.7554\/eLife.30247<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5685469\/\" target=\"_blank\">PMC5685469<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29132504\" target=\"_blank\">29132504<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+transatlantic+perspective+on+20+emerging+issues+in+biological+engineering&rft.jtitle=eLife&rft.aulast=Wintle%2C+B.C.%3B+Boehm%2C+C.R.%3B+Rhodes%2C+C.+et+al.&rft.au=Wintle%2C+B.C.%3B+Boehm%2C+C.R.%3B+Rhodes%2C+C.+et+al.&rft.date=2017&rft.volume=6&rft.pages=e30247&rft_id=info:doi\/10.7554%2FeLife.30247&rft_id=info:pmc\/PMC5685469&rft_id=info:pmid\/29132504&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5685469&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PeccoudCyber18-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PeccoudCyber18_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-PeccoudCyber18_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". <i>Trends in Biotechnology<\/i> <b>36<\/b> (1): 4\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.tibtech.2017.10.012\" target=\"_blank\">10.1016\/j.tibtech.2017.10.012<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29224719\" target=\"_blank\">29224719<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyberbiosecurity%3A+From+Naive+Trust+to+Risk+Awareness&rft.jtitle=Trends+in+Biotechnology&rft.aulast=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.au=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.date=2018&rft.volume=36&rft.issue=1&rft.pages=4%E2%80%937&rft_id=info:doi\/10.1016%2Fj.tibtech.2017.10.012&rft_id=info:pmid\/29224719&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MurchCyber18-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MurchCyber18_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-MurchCyber18_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Murch, R.S.; Wo, W.K.; Buchholz, W.G. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5895716\" target=\"_blank\">\"Cyberbiosecurity: An Emerging New Discipline to Help Safeguard the Bioeconomy\"<\/a>. <i>Frontiers in Bioengineering and Biotechnology<\/i> <b>6<\/b>: 39. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffbioe.2018.00039\" target=\"_blank\">10.3389\/fbioe.2018.00039<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5895716\/\" target=\"_blank\">PMC5895716<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29675411\" target=\"_blank\">29675411<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5895716\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5895716<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyberbiosecurity%3A+An+Emerging+New+Discipline+to+Help+Safeguard+the+Bioeconomy&rft.jtitle=Frontiers+in+Bioengineering+and+Biotechnology&rft.aulast=Murch%2C+R.S.%3B+Wo%2C+W.K.%3B+Buchholz%2C+W.G.+et+al.&rft.au=Murch%2C+R.S.%3B+Wo%2C+W.K.%3B+Buchholz%2C+W.G.+et+al.&rft.date=2018&rft.volume=6&rft.pages=39&rft_id=info:doi\/10.3389%2Ffbioe.2018.00039&rft_id=info:pmc\/PMC5895716&rft_id=info:pmid\/29675411&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5895716&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DenggTomorrow18-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DenggTomorrow18_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-DenggTomorrow18_4-1\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Dengg, A., ed. (2018) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.bundesheer.at\/pdf_pool\/publikationen\/buch_dengg_tomorrows_technology_web.pdf\" target=\"_blank\"><i>Tomorrow's Technology: A Double-Edged Sword<\/i><\/a>. Federal Ministry of Defence, Republic of Austria. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783903121317<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.bundesheer.at\/pdf_pool\/publikationen\/buch_dengg_tomorrows_technology_web.pdf\" target=\"_blank\">http:\/\/www.bundesheer.at\/pdf_pool\/publikationen\/buch_dengg_tomorrows_technology_web.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Tomorrow%27s+Technology%3A+A+Double-Edged+Sword&rft.date=2018&rft.pub=Federal+Ministry+of+Defence%2C+Republic+of+Austria&rft.isbn=9783903121317&rft_id=http%3A%2F%2Fwww.bundesheer.at%2Fpdf_pool%2Fpublikationen%2Fbuch_dengg_tomorrows_technology_web.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DHSRisk10-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DHSRisk10_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-DHSRisk10_5-1\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-DHSRisk10_5-2\">5.2<\/a><\/sup> <sup><a href=\"#cite_ref-DHSRisk10_5-3\">5.3<\/a><\/sup> <sup><a href=\"#cite_ref-DHSRisk10_5-4\">5.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Risk Steering Committee (September 2010). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/dhs-risk-lexicon-2010_0.pdf\" target=\"_blank\">\"DHS Risk Lexicon, 2010 Edition\"<\/a> (PDF). Department of Homeland Security<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/dhs-risk-lexicon-2010_0.pdf\" target=\"_blank\">https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/dhs-risk-lexicon-2010_0.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=DHS+Risk+Lexicon%2C+2010+Edition&rft.atitle=&rft.aulast=Risk+Steering+Committee&rft.au=Risk+Steering+Committee&rft.date=September+2010&rft.pub=Department+of+Homeland+Security&rft_id=https%3A%2F%2Fwww.dhs.gov%2Fsites%2Fdefault%2Ffiles%2Fpublications%2Fdhs-risk-lexicon-2010_0.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PetitResil13-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PetitResil13_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-PetitResil13_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Petit, F.D.P.; Bassett, G.W.; Black, R. et al. (15 July 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.osti.gov\/biblio\/1087819-resilience-measurement-index-indicator-critical-infrastructure-resilience\" target=\"_blank\">\"Resilience Measurement Index: An Indicator of Critical Infrastructure Resilience\"<\/a>. Argonne National Laboratory. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2172%2F1087819\" target=\"_blank\">10.2172\/1087819<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.osti.gov\/biblio\/1087819-resilience-measurement-index-indicator-critical-infrastructure-resilience\" target=\"_blank\">https:\/\/www.osti.gov\/biblio\/1087819-resilience-measurement-index-indicator-critical-infrastructure-resilience<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Resilience+Measurement+Index%3A+An+Indicator+of+Critical+Infrastructure+Resilience&rft.atitle=&rft.aulast=Petit%2C+F.D.P.%3B+Bassett%2C+G.W.%3B+Black%2C+R.+et+al.&rft.au=Petit%2C+F.D.P.%3B+Bassett%2C+G.W.%3B+Black%2C+R.+et+al.&rft.date=15+July+2013&rft.pub=Argonne+National+Laboratory&rft_id=info:doi\/10.2172%2F1087819&rft_id=https%3A%2F%2Fwww.osti.gov%2Fbiblio%2F1087819-resilience-measurement-index-indicator-critical-infrastructure-resilience&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PetitProtect13-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PetitProtect13_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-PetitProtect13_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Petit, F.D.P.; Bassett, G.W.; Buehring, W.A. et al. (21 November 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.osti.gov\/biblio\/1108161\" target=\"_blank\">\"Protective Measures Index and Vulnerability Index: Indicators of Critical Infrastructure Protection and Vulnerability\"<\/a>. Argonne National Laboratory. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2172%2F1108161\" target=\"_blank\">10.2172\/1108161<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.osti.gov\/biblio\/1108161\" target=\"_blank\">https:\/\/www.osti.gov\/biblio\/1108161<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Protective+Measures+Index+and+Vulnerability+Index%3A+Indicators+of+Critical+Infrastructure+Protection+and+Vulnerability&rft.atitle=&rft.aulast=Petit%2C+F.D.P.%3B+Bassett%2C+G.W.%3B+Buehring%2C+W.A.+et+al.&rft.au=Petit%2C+F.D.P.%3B+Bassett%2C+G.W.%3B+Buehring%2C+W.A.+et+al.&rft.date=21+November+2013&rft.pub=Argonne+National+Laboratory&rft_id=info:doi\/10.2172%2F1108161&rft_id=https%3A%2F%2Fwww.osti.gov%2Fbiblio%2F1108161&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PetitAnal15-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PetitAnal15_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-PetitAnal15_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Petit, F.D.P.; Verner, D.; Brannegan, D. et al. (01 June 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.osti.gov\/biblio\/1184636-analysis-critical-infrastructure-dependencies-interdependencies\" target=\"_blank\">\"Analysis of Critical Infrastructure Dependencies and Interdependencies\"<\/a>. Argonne National Laboratory. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2172%2F1184636\" target=\"_blank\">10.2172\/1184636<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.osti.gov\/biblio\/1184636-analysis-critical-infrastructure-dependencies-interdependencies\" target=\"_blank\">https:\/\/www.osti.gov\/biblio\/1184636-analysis-critical-infrastructure-dependencies-interdependencies<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Analysis+of+Critical+Infrastructure+Dependencies+and+Interdependencies&rft.atitle=&rft.aulast=Petit%2C+F.D.P.%3B+Verner%2C+D.%3B+Brannegan%2C+D.+et+al.&rft.au=Petit%2C+F.D.P.%3B+Verner%2C+D.%3B+Brannegan%2C+D.+et+al.&rft.date=01+June+2015&rft.pub=Argonne+National+Laboratory&rft_id=info:doi\/10.2172%2F1184636&rft_id=https%3A%2F%2Fwww.osti.gov%2Fbiblio%2F1184636-analysis-critical-infrastructure-dependencies-interdependencies&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KeeneyDecisions76-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KeeneyDecisions76_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Keeney, R.L.; Raiffa, H. (1976). <i>Decisions with multiple objectives: Preferences and value tradeoffs<\/i>. Wiley. pp. 569. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780471465102.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Decisions+with+multiple+objectives%3A+Preferences+and+value+tradeoffs&rft.aulast=Keeney%2C+R.L.%3B+Raiffa%2C+H.&rft.au=Keeney%2C+R.L.%3B+Raiffa%2C+H.&rft.date=1976&rft.pages=pp.%26nbsp%3B569&rft.pub=Wiley&rft.isbn=9780471465102&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KeeneyValue92-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KeeneyValue92_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Keeney, R.L. (1992). <i>Value\u2010focused thinking: A path to creative decision making<\/i>. Harvard University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780674931971.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Value%E2%80%90focused+thinking%3A+A+path+to+creative+decision+making&rft.aulast=Keeney%2C+R.L.&rft.au=Keeney%2C+R.L.&rft.date=1992&rft.pub=Harvard+University+Press&rft.isbn=9780674931971&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FisherConstruct09-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FisherConstruct09_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fisher, R.E.; Buehring, W.A.; Whitfield, R.G. et al. (14 October 2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.osti.gov\/biblio\/966343-constructing-vulnerabilty-protective-measures-indices-enhanced-critical-infrastructure-protection-program\" target=\"_blank\">\"Constructing vulnerabilty and protective measures indices for the enhanced critical infrastructure protection program\"<\/a>. Argonne National Laboratory. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2172%2F966343\" target=\"_blank\">10.2172\/966343<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.osti.gov\/biblio\/966343-constructing-vulnerabilty-protective-measures-indices-enhanced-critical-infrastructure-protection-program\" target=\"_blank\">https:\/\/www.osti.gov\/biblio\/966343-constructing-vulnerabilty-protective-measures-indices-enhanced-critical-infrastructure-protection-program<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Constructing+vulnerabilty+and+protective+measures+indices+for+the+enhanced+critical+infrastructure+protection+program&rft.atitle=&rft.aulast=Fisher%2C+R.E.%3B+Buehring%2C+W.A.%3B+Whitfield%2C+R.G.+et+al.&rft.au=Fisher%2C+R.E.%3B+Buehring%2C+W.A.%3B+Whitfield%2C+R.G.+et+al.&rft.date=14+October+2009&rft.pub=Argonne+National+Laboratory&rft_id=info:doi\/10.2172%2F966343&rft_id=https%3A%2F%2Fwww.osti.gov%2Fbiblio%2F966343-constructing-vulnerabilty-protective-measures-indices-enhanced-critical-infrastructure-protection-program&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CollinsProtect11-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CollinsProtect11_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Collins, M.; Petit, F.; Buehring, W. et al. (2011). \"Protective measures and vulnerability indices for the Enhanced Critical Infrastructure Protection Programme\". <i>International Journal of Critical Infrastructures<\/i> <b>7<\/b> (3): 200\u201319. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1504%2FIJCIS.2011.042976\" target=\"_blank\">10.1504\/IJCIS.2011.042976<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Protective+measures+and+vulnerability+indices+for+the+Enhanced+Critical+Infrastructure+Protection+Programme&rft.jtitle=International+Journal+of+Critical+Infrastructures&rft.aulast=Collins%2C+M.%3B+Petit%2C+F.%3B+Buehring%2C+W.+et+al.&rft.au=Collins%2C+M.%3B+Petit%2C+F.%3B+Buehring%2C+W.+et+al.&rft.date=2011&rft.volume=7&rft.issue=3&rft.pages=200%E2%80%9319&rft_id=info:doi\/10.1504%2FIJCIS.2011.042976&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JoyceCyber17-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JoyceCyber17_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Joyce, A.L.; Petit, F.D.; Phillips, J.A. et al. (01 October 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.osti.gov\/biblio\/1433503-cyber-protection-resilience-index-indicator-organization-cyber-protection-resilience-program\" target=\"_blank\">\"Cyber Protection and Resilience Index: An Indicator of an Organization's Cyber Protection and Resilience Program\"<\/a>. Argonne National Laboratory. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2172%2F1433503\" target=\"_blank\">10.2172\/1433503<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.osti.gov\/biblio\/1433503-cyber-protection-resilience-index-indicator-organization-cyber-protection-resilience-program\" target=\"_blank\">https:\/\/www.osti.gov\/biblio\/1433503-cyber-protection-resilience-index-indicator-organization-cyber-protection-resilience-program<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cyber+Protection+and+Resilience+Index%3A+An+Indicator+of+an+Organization%27s+Cyber+Protection+and+Resilience+Program&rft.atitle=&rft.aulast=Joyce%2C+A.L.%3B+Petit%2C+F.D.%3B+Phillips%2C+J.A.+et+al.&rft.au=Joyce%2C+A.L.%3B+Petit%2C+F.D.%3B+Phillips%2C+J.A.+et+al.&rft.date=01+October+2017&rft.pub=Argonne+National+Laboratory&rft_id=info:doi\/10.2172%2F1433503&rft_id=https%3A%2F%2Fwww.osti.gov%2Fbiblio%2F1433503-cyber-protection-resilience-index-indicator-organization-cyber-protection-resilience-program&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EvansInfo09-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EvansInfo09_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Evans, N.J. (2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/lib.dr.iastate.edu\/etd\/10709\/\" target=\"_blank\">\"Information technology social engineering: An academic definition and study of social engineering - Analyzing the human firewall\"<\/a>. <i>Graduate Theses and Dissertations<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.31274%2Fetd-180810-436\" target=\"_blank\">10.31274\/etd-180810-436<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/lib.dr.iastate.edu\/etd\/10709\/\" target=\"_blank\">https:\/\/lib.dr.iastate.edu\/etd\/10709\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Information+technology+social+engineering%3A+An+academic+definition+and+study+of+social+engineering+-+Analyzing+the+human+firewall&rft.atitle=Graduate+Theses+and+Dissertations&rft.aulast=Evans%2C+N.J.&rft.au=Evans%2C+N.J.&rft.date=2009&rft_id=info:doi\/10.31274%2Fetd-180810-436&rft_id=https%3A%2F%2Flib.dr.iastate.edu%2Fetd%2F10709%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PerkinsMit18-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PerkinsMit18_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Perkins, D.; Fabregas, E. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/sites.nationalacademies.org\/DBASSE\/BBCSS\/DBASSE_175673\" target=\"_blank\"><i>Mitigating Insider Threats through Strengthening Organizations' Culture of Biosafety, Biosecurity, and Responsible Conduct [White paper<\/i><\/a>]. Submitted to the Committee on a Decadal Survey of Social and Behavioral Sciences and Applications to National Security, National Academies of Sciences, Engineering, and Medicine. pp. 1\u20136<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/sites.nationalacademies.org\/DBASSE\/BBCSS\/DBASSE_175673\" target=\"_blank\">http:\/\/sites.nationalacademies.org\/DBASSE\/BBCSS\/DBASSE_175673<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Mitigating+Insider+Threats+through+Strengthening+Organizations%27+Culture+of+Biosafety%2C+Biosecurity%2C+and+Responsible+Conduct+%5BWhite+paper%5D&rft.aulast=Perkins%2C+D.%3B+Fabregas%2C+E.&rft.au=Perkins%2C+D.%3B+Fabregas%2C+E.&rft.date=2017&rft.series=Submitted+to+the+Committee+on+a+Decadal+Survey+of+Social+and+Behavioral+Sciences+and+Applications+to+National+Security%2C+National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rft.pages=pp.%26nbsp%3B1%E2%80%936&rft_id=http%3A%2F%2Fsites.nationalacademies.org%2FDBASSE%2FBBCSS%2FDBASSE_175673&rfr_id=info:sid\/en.wikipedia.org:Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165455\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.423 seconds\nReal time usage: 0.452 seconds\nPreprocessor visited node count: 12486\/1000000\nPreprocessor generated node count: 34200\/1000000\nPost\u2010expand include size: 100785\/2097152 bytes\nTemplate argument size: 36792\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 424.471 1 - -total\n 78.22% 332.011 1 - Template:Reflist\n 65.26% 277.019 15 - Template:Citation\/core\n 28.33% 120.266 7 - Template:Cite_web\n 25.75% 109.305 4 - Template:Cite_journal\n 16.93% 71.873 4 - Template:Cite_book\n 15.35% 65.142 1 - Template:Infobox_journal_article\n 14.70% 62.385 1 - Template:Infobox\n 8.87% 37.668 80 - Template:Infobox\/row\n 6.45% 27.381 18 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11044-0!*!0!!en!5!* and timestamp 20190701165455 and revision id 35566\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience\">https:\/\/www.limswiki.org\/index.php\/Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","7685e865ec42d1c3178d9cb2493521d4_images":["https:\/\/www.limswiki.org\/images\/6\/69\/Fig1_Schabacker_FrontBioengBiotech2019_7.jpg"],"7685e865ec42d1c3178d9cb2493521d4_timestamp":1562000095,"d53838ebf5172963d6c6496807f9617e_type":"article","d53838ebf5172963d6c6496807f9617e_title":"Cyberbiosecurity: An emerging new discipline to help safeguard the bioeconomy (Murch et al. 2018)","d53838ebf5172963d6c6496807f9617e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy","d53838ebf5172963d6c6496807f9617e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Cyberbiosecurity: An emerging new discipline to help safeguard the bioeconomy\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nCyberbiosecurity: An emerging new discipline to help safeguard the bioeconomyJournal\n \nFrontiers in Bioengineering and BiotechnologyAuthor(s)\n \nMurch, Randall S.; So, William K.; Buchholz, Wallace G.; Raman, Sanjay; Peccoud, JeanAuthor affiliation(s)\n \nVirginia Tech, Federal Bureau of Investigation, University of Nebraska, Colorado State UniversityPrimary contact\n \nEmail: rmurch at vt dot eduEditors\n \nBerns, Kenneth I.Year published\n \n2018Volume and issue\n \n6Page(s)\n \n39DOI\n \n10.3389\/fbioe.2018.00039ISSN\n \n2296-4185Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2018.00039\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2018.00039\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Background \n4 Adding another dimension: Cyberbiosecurity systems analysis of a biomanufacturing facility \n5 Moving cyberbiosecurity forward \n6 Acknowledgements \n\n6.1 Author contributions \n6.2 Funding \n6.3 Conflict of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nCyberbiosecurity is being proposed as a formal new enterprise which encompasses cybersecurity, cyber-physical security, and biosecurity as applied to biological and biomedical-based systems. In recent years, an array of important meetings and public discussions, commentaries, and publications have occurred that highlight numerous vulnerabilities. While necessary first steps, they do not provide a systematized structure for effectively promoting communication, education and training, elucidation, and prioritization for analysis, research, development, testing and evaluation, and implementation of scientific and technological standards of practice, policy, or regulatory or legal considerations for protecting the bioeconomy. Further, experts in biosecurity and cybersecurity are generally not aware of each other's domains, expertise, perspectives, priorities, or where mutually supported opportunities exist for which positive outcomes could result. Creating, promoting, and advancing a new discipline can assist with formal, beneficial, and continuing engagements. Recent key activities and publications that inform the creation of cyberbiosecurity are briefly reviewed, as is the expansion of cyberbiosecurity to include biomanufacturing, which is supported by a rigorous analysis of a biomanufacturing facility. Recommendations are provided to initialize cyberbiosecurity and place it on a trajectory to establish a structured and sustainable discipline, forum, and enterprise.\nKeywords: cyberbiosecurity, bioeconomy, biosecurity, biomanufacturing, cybersecurity, cyber-physical security, supply chain\n\nIntroduction \nWe propose \u201ccyberbiosecurity\u201d as an emerging hybridized discipline at the interface of cybersecurity, cyber-physical security, and biosecurity. Initially, we define this term as \u201cunderstanding the vulnerabilities to unwanted surveillance, intrusions, and malicious and harmful activities which can occur within or at the interfaces of commingled life and medical sciences, cyber, cyber-physical, supply chain, and infrastructure systems, and developing and instituting measures to prevent, protect against, mitigate, investigate, and attribute such threats as it pertains to security, competitiveness, and resilience.\u201d We emphasize this is an initial definition; we fully expect that the definition and the landscape will rapidly evolve, requiring the definition to be revised. We also contend that, because of its diversity and extent, cyberbiosecurity needs its own systematics, so that it can be better communicated, organized, explored, advanced, and implemented. Here, we also posit that cyberbiosecurity contributes to a larger strategic objective of \u201csafeguarding the bioeconomy,\u201d[1] a concept advanced in the U.S. which seeks to increase security and resilience of the bioeconomy to protect its rapidly changing cyber-life science topology.\nThus far, what we are proposing to call cyberbiosecurity has primarily been initiated out of two principal sets of activities. The first set of activities involved a study[2] and three workshops[1][3][4], which were primarily focused on security issues with respect to \u201cbig data\u201d and the relationship with the \u201cbioeconomy.\u201d The second set was a first-ever systems analysis of a biomanufacturing facility, which expands the view to include a different \u201ctarget set\u201d and approach to understanding vulnerabilities with sharp acuity. This tasked study was conducted to comprehensively understand the vulnerabilities with respect to a wide range of unwanted intrusions and nefarious activities in the life science, cyber, cyber-physical, infrastructure, and supply chain aspects, and determine what measures could be taken or developed and implemented to anticipate, detect, identify, prevent, mitigate, respond to, and attribute such potential exploitation. The first published paper on cyberbiosecurity primarily focuses on the security of the biotechnology interface with cyberspace.[5] In addition to the system analysis as part of the second set, a small workshop was held in the U.S. that sought to scope and stimulate interest in the government, academic, corporate, and non-profit sectors, create a core constituency, understand what topics and themes could constitute cyberbiosecurity, identify priorities, and begin to develop a campaign and timeline. The workshop was highly successful. These endeavors, together with additional recent activities and publications[6][7][8][9], have added to scoping the future of cyberbiosecurity yet to come.\n\nBackground \nSimply stated, since its inception, biosecurity has been primarily focused on reducing the risks associated with the misuse of science which could cause harm to humans, animals, plants, and the environment through the creation, production, and deliberate or accidental release of infectious disease agents or their byproducts (e.g., toxins). Cybersecurity has been a separate field which has been primarily focused on the security of information technology systems, from personal computers and communications devices to large infrastructures and networks. Up until just the past few years, the \u201ccyber\u201d overlaps with biosecurity have not been realized or fleshed out. The important interrelationship between biosecurity and cybersecurity is gaining increasing attention. We posit that the two must work collaboratively and will not be effective working separately. Cyberbiosecurity actually started with thinking about a particular set of problems being confronted by the life sciences. As a result of our recent work, described below, other dimensions are being added. Establishing a unifying discipline, crafting its systematics, and identifying an evolutionary path forward are within reach.\nThe economic strength and growth of the United States have been due to a culture and environment that foster innovation. Those developments could not be possible without significant contributions by science and engineering. The intersection among economic growth and the biological sciences contributions\u2014the bioeconomy\u2014has recently been recognized as an important component of national security. For the U.S., the bioeconomy accounts for an estimated $4 trillion annually, nearly 25% of the GDP. That contribution ranges from pharmaceuticals to renewable energy, from environmental remediation to public health resilience, and from agriculture to emerging disease response. As part of the U.S. national security architecture, \u201csafeguarding the sciences\u201d is a priority. In doing so, the U.S. Federal Bureau of Investigation (FBI) and other federal agencies also fulfill the U.S. obligation to the Biological Toxins and Weapons Convention (BTWC) and compliance to the United Nations Security Council Resolution (UNSCR) 1540, preventing the misuse of biological material, technology, and expertise, and encouraging the enforcement of the related statutes. The FBI also sponsors and actively engages the International Genetically Engineered Machine (iGEM) competition to inculcate a culture of security among international students, who will become leaders of research, industry, and policymaking. At the same time, the FBI works with U.S. policymakers[10] to redefine the scope of the biosecurity spectrum for the twenty-first century, a century with an unprecedented pace of biological research and innovation, and the use of diverse and large datasets (big data) to assist global scientific and societal priorities and opportunities. Concomitant to both realized and future benefits and growth, the life sciences are becoming increasingly digitized\u2014while at the same time intellectual property protection, cyber intrusion, and the protection of personal medical and genomic information becoming more important\u2014and the impacts on science, trade, and commerce loom large. Engagements with the science media[6] and testimonies[10] have raised these issues to advance both U.S. competitiveness and national security.\nIn 2014, the American Association for the Advancement of Science (AAAS), FBI, and the United Nations Interregional Crime and Justice Research Institute (UNICRI) published a report entitled \u201cNational and Transnational Security Implications of Big Data in the Life Sciences.[2] Briefly, this report starts by helping to understand \u201cbig data\u201d; massive, diverse data sets that are created, reside, are analyzed in, and move in information ecosystems. For the life sciences, big data refers to datasets including \u201craw data, combined data, or published data from the health-care system, pharmaceutical industry, genomics and other \u2013omics fields, clinical research, environment, agriculture, and microbiome efforts.\u201d Further, they state that big data also includes analytic technologies and outputs, such as from \u201cdata integration, data mining, data fusion, image and speech recognition, natural language processing, machine learning, social media analysis, and Bayesian analysis.\u201d A number of areas that have drawn and need attention are pointed out, such as the security of the cyber infrastructure and data repositories, and the privacy and confidentiality of individuals. In our view, their focus on the security risks of big data in the life sciences falls into just two major categories, i.e., inappropriate access to data and analytic technologies through vulnerabilities in the data and cyber infrastructure. As such, the use of big data technologies to integrate current data and enable the design of a harmful biological agent should be revisited and refined. Thanks to this team's efforts, not only do we have a useful topology of big data, the beginnings of a structure for thinking about security implications at the bio-cyber interface (technical, legal, institutional, and individual) and a set of high-level recommendations for a path forward.\nFrom 2014 to 2016, three workshops were organized by the U.S. National Academies on behalf of the FBI under the theme of \u201cSafeguarding the Bioeconomy.\u201d The first[1] laid the foundation for the next two. Presentations and discussions focused on the security implications of the convergence in the life and chemical sciences with physical, mathematical, computational, engineering, and social and behavioral sciences. In addition to broader contexts, two specific technologies received focus: neuromorphic computing and 3-D bioprinting. The second workshop[3] introduced a range of new threats to and vulnerabilities of the bioeconomy, which at the time had not received focused consideration with respect to U.S. \u201ccompetiveness, security, economic growth, and global leadership in research and innovation.\u201d This workshop was built on three major themes: the role of informatics in the bioeconomy, criminal threats and vulnerabilities in the existing and near-future bioeconomy, and securing and flourishing the bioeconomy for the future. Rapid growth of this sector creates increasing security risks to proprietary materials and informatics, brings about an increase in frequency in industrial espionage and data hacks, and decreases the effectiveness of traditional security measures. Still, alternative and adaptive security measures could be implemented even with the inherent openness of emerging technologies upon which the bioeconomy is dependent. Workshop participants not only provided more detail on the threats and vulnerabilities but also both comprehensive categories and specific approaches that could be taken to address the problems and concerns identified. The third workshop[4] principally focused on data generation and access with respect to the bioeconomy within several categories of both clinical and non-clinical data, from the perspectives of biosecurity, data policy and regulation, future implications, technology advances, data sovereignty and sharing, cybersecurity, and international implications. Taken together, these events significantly expanded the view of what the emerging discipline of cyberbiosecurity could encompass.\nPauwels and her co-authors also raise important concerns and recommendations for the security of biotechnology in cyberspace. In the first, she and Vidyarthi[7] raise concerns over data breaches of health care information and what it means for the biotechnology industry. Protecting digital DNA and personal medical information is highlighted, as well as the fact that a then recent U.S. Presidential cybersecurity initiative put significant resources into shoring up cyberinfrastructure. Unfortunately, the need for improvements to protecting the bioeconomy, which is heavily dependent on information systems and infrastructure, was not recognized. The report outlined the implications of not protecting the bioeconomy dimension. Their recommendations were primarily focused on protecting genomic data. In the second report, Pauwels and Dunlap[9] go into more depth framing potential cyber-vulnerabilities for specific types of biotechnologies: genome-editing; DNA assembly, synthesis and printing; portable genomic sequencers; artificial intelligence for understanding biological complexity; autonomous systems and robotics in cloud labs; and lab-on-a-chip and microfluidic technologies, all of which have cyber-physical interfaces. These authors also suggest governance systems and policy recommendations which might be harnessed to address the lab-focused concerns they raise.\nOther recent publications also highlight the complexity of the enterprise we are terming \u201ccyberbiosecurity\u201d and concerns over security, robustness, and resiliency. These include:\n\n security of personal genomic data when foreign companies purchase all or part of a U.S. company or are contracted for genomic or health care data services, which provides access to sensitive personal information[8];\n the continuing vulnerability of electronic health records[11] and health care systems[12][13][14];\n imposing control over DNA sequencing through DNA-encoded malware[15];\n synthetic biology supply chain vulnerabilities[16];\n cybersecurity compromise of large industrial biopharma companies[17][18]; and\n high-level studies which are systematically examining U.S. biodefense programs and capabilities.[19][20]\nThe darkweb\/darknet[21][22][23] could be included as it interfaces with dual use life science endeavors and biopharma research, development, intellectual property, and products, compromising the integrity of critical life science and health cyber-supported technologies and infrastructures. Because of the reliance on bioinformatics, the security of synthetic DNA could also be included, as well.[24] Clearly, this rapidly expanding galaxy does needs a universally accepted definition, common terms of reference, and defined boundaries and structure for best value, ordered evolution, and impact.\n\nAdding another dimension: Cyberbiosecurity systems analysis of a biomanufacturing facility \nNow we add another dimension to cyberbiosecurity and take an approach that we posit that should be incorporated with other aspects discussed earlier. The biopharma industry itself has its own substantial equities and investments in the research, development, production, and sale of vaccines, therapeutics, and prophylactics for the global market. The U.S. Government has substantial investments in the development and production of critical vaccines and biotherapeutics for both civilian and military purposes. Concomitantly, experts are increasingly recognizing that biomanufacturing itself is potentially vulnerable to unwanted or illicit activities which could result in damaging outcomes. These could include the theft of intellectual property, disruption of the supply chain, manipulation of the bioprocess development and bioproduction, cyberattacks on key information technology components and cyberphysical interfaces, the corruption of critical data, and manipulation of security systems and infrastructure upon which secure and safe facility operations are dependent. Our sponsor was not interested in generalizations or esoteric approximations about the security vulnerabilities of a biomanufacturing facility but wanted a comprehensive, detailed, actionable analysis.\nThus, we undertook an in-depth, multidimensional analysis of an existing biomanufacturing facility to identify and project security gaps and vulnerabilities, make recommendations with respect to addressing those identified and projected, and set the stage for more specific and comprehensive measures to be undertaken, whether they exist or have to be developed and validated. The systems analysis approach used was designed to assess the state of security at present, determine what an acceptable state of security would be, and provide guidance and recommendations to take the facility from its current state to the desired state.\nThe bioprocess development\/bioproduction facility used as the \u201ctest bed\u201d for this analysis designs, develops, and produces clinical trial quantities of protein-based biotherapeutics and the associated documentation for commercial and government clients. If the outputs from this \u201ctest bed\u201d meet client expectations and the client receives government approval, the client scales up production and the product is marketed. This facility was studied as a system, consisting of four key, interrelated subsystems: end-to-end bioprocess development\/biomanufacturing; the supply chain; the supporting information systems infrastructure and cyber-physical interfaces with bioprocess development and biomanufacturing; and facility infrastructure, including its relationship to the facility's host infrastructure. The systems analysis was a phased process with project management methods applied. The facility or any of its components or operations were not compromised, corrupted, or altered during this project in any manner or form. Rather, it was studied thoroughly yet benignly. The analysis included human factors and \u201cdownstream\u201d considerations, as well.\nThe systems aspects of a biomanufacturing facility which are potentially vulnerable to security threats and the solutions required are summarized in Figure 1.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 A systems view of protecting a biomanufacturing facility. For each defensive set identified, multiple threats and impacts were identified and potentially more than one pathway or technique could be used by an adversary to achieve their objectives. GMP, Good Manufacturing Practice; IT, Information Technology; IS, Information Systems; VPN, Virtual Private Network; IP, Intellectual Property.\n\n\n\nDue to space limitations, we provide only a top-level view of the analysis. Key overarching findings include:\n\n Vulnerabilities can exist across the entire system, from bioprocess development and GMP to supply chain, to cyber-physical and infrastructure; there are potentially more than one might anticipate a priori.\n Successful exploitation of vulnerabilities can occur through passive and active means for passive and active purposes, depending upon adversaries' intentions, objectives, accesses, knowledge and resources, and outcomes sought.\n Exploitation of some vulnerabilities require direct access to facilities or components; personnel and physical security aspects should not be overlooked.\n Adversaries can use combinations and sequences of methods and targeting, both subtle and not, to attempt to and achieve their objectives.\n The operational capabilities of adversaries, not just technical, must be considered and accounted for in planning for and implementing security measures.\nWe emphasize that, while there are general principles that apply and observations that will derive from such analyses, the analysis design and execution and the resulting solution set, should be tailored on a facility-by-facility basis. We note that the defensive areas noted may not be singular, but rather require combinations of defensive approaches and techniques to be identified and implemented to ensure optimal security robustness.\nWhat is considerably important from this analysis is that a rigorous study of a facility such as this can result in the identification and characterization of discrete vulnerabilities, gaps, shortfalls, and opportunities for which readily-available solutions can be implemented, or otherwise can be developed, tested, and implemented. We did not conduct detailed studies regarding how genomics can be compromised as it relates to biomanufacturing because we were directed not to, but we are well aware of plausible scenarios and what the effects could be.\nOur analysis demonstrates that biomanufacturing facilities can benefit from comprehensive, multidisciplinary analyses to identify security vulnerabilities, leading to solutions to mitigate or address them. This, in turn, raises the prospect of the development and validation of a set of methods or protocols would be in order which could be used by facility staff or external service providers to shore up individual facilities, from do-it-yourself to large biopharma. Walking this out, guidelines or standards could be developed, established, and accepted to ensure consistency and quality of the analyses conducted, the credentials of the personnel doing so, and the quality and effectiveness of measures undertaken. While sophisticated adversaries could design and execute sophisticated attacks, it is likely in many instances that relatively straightforward methods and practices could raise the bar considerably to reduce risk. Lastly, combining analyses of this sort could be used as a basis for informed investments in research, development, testing, and evaluation for solutions to the most worrisome current and future threats.\n\nMoving cyberbiosecurity forward \nMany other critical cyber-enabled life science and biomedical technologies, systems, and applications naturally lend themselves to inclusion within cyberbiosecurity. These include, but are not limited to, personalized genomics, medical and fitness technologies, 3-D printing of critical personalized medical devices, and medical laboratory and surgical robotics. A more comprehensive system is warranted. Cyberbiosecurity could be expanded to include cyber-bio systems within agriculture and farm-to-table food production, processing and distribution systems, and within natural resource and environmental management. Direct and ordered engagements of the pertinent sectors of the life sciences, biosecurity, and cyber-cybersecurity communities should occur. Academia, industry, government, and non-profits (including policy, regulatory, and legal experts) need to begin to learn to communicate with and educate each other, harmoniously identify and develop priorities and opportunities, and specify \u201cnext steps.\u201d A major opportunity exists right now to propose a unified structure and common vernacular. Lastly, while definition and assemblage of cyberbiosecurity is occurring, national or international strategies should be pursued to harmonize the emerging enterprise and foster measurable value, success, and sustainability.\n\nAcknowledgements \nAuthor contributions \nRM: Co-originator of cyberbiosecurity concept; Lead author, responsible for structure, content and figure; responsible for considering, incorporating co-author contributions and suggested modifications; responsible for final version. WS: Co-originator of Safeguarding the Bioeconomy concept and campaign; contributed or reviewed and edited paragraphs related to the AAAS-FBI-UNICRI Big Data report, the US National Academies workshops and the relevant FBI programs and initiatives. SR: Co-originator of cyberbiosecurity concept; overall quality assurance and readability reviews and modifications; contributions to section on cyberbiosecurity applied to biomanufacturing. WB: Contributions to section on cyberbiosecurity applied to biomanufacturing; reviews to ensure content quality and readability of paper. JP: Co-originator of cyberbiosecurity concept; review and critique of manuscript to ensure complementarity and alignment with first paper published on cyberbiosecurity (biotechnology focus) in another journal (he is the lead author; RM, SR, and WB are co-authors).\n\nFunding \nThe funding for the systems analysis of the Test Bed Facility was provided through a contract award (Contract FA4600-12-D-9000, Task Order 0065) from the U. S. Department of Defense, United States Strategic Command to the National Strategic Research Institute, University of Nebraska.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 1.0 1.1 1.2 Board on Chemical Sciences and Technology; Board on Life Sciences (2014). Meeting Recap: Workshop - Convergence: Safeguarding Technology in the Bioeconomy. The National Academies of Sciences, Engineering, and Medicine.   \n\n\u2191 2.0 2.1 FBI WMD Directorate, American Association for the Advancement of Science, United Nations Interregional Crime and Justice Research Institute (2014). \"National and Transnational Security Implications of Big Data in the Life Sciences\" (PDF). AAAS. http:\/\/www.aaas.org\/sites\/default\/files\/AAAS-FBI-UNICRI_Big_Data_Report_111014.pdf .   \n\n\u2191 3.0 3.1 Board on Chemical Sciences and Technology (2015) (PDF). Meeting Recap: Safeguarding the Bioeconomy: Applications and Implications of Emerging Science. The National Academies of Sciences, Engineering, and Medicine. https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf .   \n\n\u2191 4.0 4.1 Board on Life Sciences and Board on Chemical Sciences and Technology (2016) (PDF). Meeting Recap: Safeguarding the Bioeconomy III: Securing Life Sciences Data. The National Academies of Sciences, Engineering, and Medicine. https:\/\/www.ibpforum.org\/sites\/default\/files\/Safeguarding_the_Bioeconomy_III_Recap.pdf .   \n\n\u2191 Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". Trends in Biotechnology 36 (1): 4\u20137. doi:10.1016\/j.tibtech.2017.10.012. PMID 29224719.   \n\n\u2191 6.0 6.1 Kozminski, K.G.; Drubin, D.G. (2015). \"Biosecurity in the age of Big Data: A conversation with the FBI\". Molecular Biology of the Cell 26 (22): 3894\u201397. doi:10.1091\/mbc.E14-01-0027. PMC PMC4710219. PMID 26543195. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4710219 .   \n\n\u2191 7.0 7.1 Pauwels, E.; Vidyarthi, A. (29 March 2016). \"How Our Unhealthy Cybersecurity Infrastructure Is Hurting Biotechnology\". Wilson Briefs. Wilson Center. https:\/\/www.wilsoncenter.org\/publication\/how-our-unhealthy-cybersecurity-infrastructure-hurting-biotechnology .   \n\n\u2191 8.0 8.1 Pauwels, E.; Vidyarthi, A. (19 November 2017). \"Who Will Own The Secrets In Our Genes? A U.S. \u2013 China Race in Artificial Intelligence and Genomics\". Wilson Briefs. Wilson Center. https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics .   \n\n\u2191 9.0 9.1 Pauwels, E.; Dunlap, E. (07 September 2017). \"The Intelligent and Connected Bio-Labs of the Future: Promise and Peril in the Fourth Industrial Revolution\". Wilson Briefs. Wilson Center. https:\/\/www.wilsoncenter.org\/publication\/the-intelligent-and-connected-bio-labs-the-future-promise-and-peril-the-fourth .   \n\n\u2191 10.0 10.1 You, E.H. (16 March 2017). \"Safeguarding the Bioeconomy: U.S. Opportunities and Challenges - Testimony for the U.S.-China Economic and Security Review Commission\" (PDF). https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf .   \n\n\u2191 Weise, E. (05 February 2015). \"Millions of Anthem customers alerted to hack\". USA Today. https:\/\/www.usatoday.com\/story\/tech\/2015\/02\/05\/anthem-health-care-computer-security-breach\/22917635\/ .   \n\n\u2191 Hackett, R. (17 July 2015). \"UCLA Health System data breach may affect millions\". Fortune. http:\/\/fortune.com\/2015\/07\/17\/ucla-health-system-data-breach\/ .   \n\n\u2191 Winton, R. (18 February 2016). \"Hollywood hospital pays $17,000 in bitcoin to hackers; FBI investigating\". Los Angeles Times. https:\/\/www.latimes.com\/business\/technology\/la-me-ln-hollywood-hospital-bitcoin-20160217-story.html .   \n\n\u2191 Griffin, A. (12 May 2017). \"NHS hack: Cyber attack takes 16 hospitals offline as patients are turned away\". Independent. https:\/\/www.independent.co.uk\/news\/uk\/home-news\/nhs-cyber-attack-hack-hospitals-16-patients-turned-away-wanna-decryptor-a7733196.html .   \n\n\u2191 Greenberg, A. (10 August 2017). \"Biohackers encoded malware in a strand of DNA\". Wired. https:\/\/www.wired.com\/story\/malware-dna-hack\/ .   \n\n\u2191 Frazar, S.L.; Hund, G.E.; Bonheyo, G.T. et al. (2017). \"Defining the synthetic biology supply chain\". Health Security 15 (4): 392-400. doi:10.1089\/hs.2016.0083. PMID 28767286.   \n\n\u2191 Collier, K. (30 June 2017). \"Merck IT systems still crippled in Petya's aftermath\". CyberScoop. https:\/\/www.cyberscoop.com\/merck-petya-ransomware-ukraine\/ .   \n\n\u2191 Shaban, H.; Nakashima, E. (27 June 2017). \"Pharmaceutical giant rocked by ransomware attack\". The Washington Post. https:\/\/www.washingtonpost.com\/news\/the-switch\/wp\/2017\/06\/27\/pharmaceutical-giant-rocked-by-ransomware-attack .   \n\n\u2191 Blue Ribbon Study Panel on Biodefense (28 October 2015). \"A National Blueprint for Biodefense: Leadership and Major Reform Needed To Optimize Efforts\". Hudson Institute. https:\/\/www.hudson.org\/research\/11824-a-national-blueprint-for-biodefense-leadership-and-major-reform-needed-to-optimize-efforts .   \n\n\u2191 Center for the Study of Weapons of Mass Destruction (2017). Emergence and Convergence Deep Dive: the Age of Genomic Data. Technical Executive Summary. National Defense University.   \n\n\u2191 Beckett, A. (25 November 2009). \"The dark side of the internet\". The Guardian. https:\/\/www.theguardian.com\/technology\/2009\/nov\/26\/dark-side-internet-freenet .   \n\n\u2191 INTERPOL (24 February 2015). \"Pharmaceutical Crime on the Darknet\" (PDF). https:\/\/www.gwern.net\/docs\/sr\/2015-interpol-pharmaceuticals.pdf .   \n\n\u2191 Langewiesche, W. (11 September 2016). \"Welcome to the dark net, a wilderness where invisible world wars are fought and hackers roam free\". Vanity Fair. https:\/\/www.vanityfair.com\/news\/2016\/09\/welcome-to-the-dark-net?verso=true .   \n\n\u2191 Adam, L.; Kozar, M.; Letort, G. et al. (2011). \"Strengths and limitations of the federal guidance on synthetic DNA\". Nature Biotechnology 29 (3): 208-10. doi:10.1038\/nbt.1802. PMID 21390018.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\">https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on agricultureLIMSwiki journal articles on cybersecurity\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 22 April 2019, at 19:45.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 211 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","d53838ebf5172963d6c6496807f9617e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Cyberbiosecurity_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Cyberbiosecurity: An emerging new discipline to help safeguard the bioeconomy<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Cyberbiosecurity is being proposed as a formal new enterprise which encompasses cybersecurity, cyber-physical security, and biosecurity as applied to biological and biomedical-based systems. In recent years, an array of important meetings and public discussions, commentaries, and publications have occurred that highlight numerous vulnerabilities. While necessary first steps, they do not provide a systematized structure for effectively promoting communication, education and training, elucidation, and prioritization for analysis, research, development, testing and evaluation, and implementation of scientific and technological standards of practice, policy, or regulatory or legal considerations for protecting the bioeconomy. Further, experts in biosecurity and cybersecurity are generally not aware of each other's domains, expertise, perspectives, priorities, or where mutually supported opportunities exist for which positive outcomes could result. Creating, promoting, and advancing a new discipline can assist with formal, beneficial, and continuing engagements. Recent key activities and publications that inform the creation of cyberbiosecurity are briefly reviewed, as is the expansion of cyberbiosecurity to include biomanufacturing, which is supported by a rigorous analysis of a biomanufacturing facility. Recommendations are provided to initialize cyberbiosecurity and place it on a trajectory to establish a structured and sustainable discipline, forum, and enterprise.\n<\/p><p><b>Keywords<\/b>: cyberbiosecurity, bioeconomy, biosecurity, biomanufacturing, cybersecurity, cyber-physical security, supply chain\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>We propose \u201ccyberbiosecurity\u201d as an emerging hybridized discipline at the interface of cybersecurity, cyber-physical security, and biosecurity. Initially, we define this term as \u201cunderstanding the vulnerabilities to unwanted surveillance, intrusions, and malicious and harmful activities which can occur within or at the interfaces of commingled life and medical sciences, cyber, cyber-physical, supply chain, and infrastructure systems, and developing and instituting measures to prevent, protect against, mitigate, investigate, and attribute such threats as it pertains to security, competitiveness, and resilience.\u201d We emphasize this is an initial definition; we fully expect that the definition and the landscape will rapidly evolve, requiring the definition to be revised. We also contend that, because of its diversity and extent, cyberbiosecurity needs its own systematics, so that it can be better communicated, organized, explored, advanced, and implemented. Here, we also posit that cyberbiosecurity contributes to a larger strategic objective of \u201csafeguarding the bioeconomy,\u201d<sup id=\"rdp-ebb-cite_ref-NASEMMeeting14_1-0\" class=\"reference\"><a href=\"#cite_note-NASEMMeeting14-1\">[1]<\/a><\/sup> a concept advanced in the U.S. which seeks to increase security and resilience of the bioeconomy to protect its rapidly changing cyber-life science topology.\n<\/p><p>Thus far, what we are proposing to call cyberbiosecurity has primarily been initiated out of two principal sets of activities. The first set of activities involved a study<sup id=\"rdp-ebb-cite_ref-AAASNat14_2-0\" class=\"reference\"><a href=\"#cite_note-AAASNat14-2\">[2]<\/a><\/sup> and three workshops<sup id=\"rdp-ebb-cite_ref-NASEMMeeting14_1-1\" class=\"reference\"><a href=\"#cite_note-NASEMMeeting14-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NASEMSafe15_3-0\" class=\"reference\"><a href=\"#cite_note-NASEMSafe15-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NASEMSafe16_4-0\" class=\"reference\"><a href=\"#cite_note-NASEMSafe16-4\">[4]<\/a><\/sup>, which were primarily focused on security issues with respect to \u201cbig data\u201d and the relationship with the \u201cbioeconomy.\u201d The second set was a first-ever systems analysis of a biomanufacturing facility, which expands the view to include a different \u201ctarget set\u201d and approach to understanding vulnerabilities with sharp acuity. This tasked study was conducted to comprehensively understand the vulnerabilities with respect to a wide range of unwanted intrusions and nefarious activities in the life science, cyber, cyber-physical, infrastructure, and supply chain aspects, and determine what measures could be taken or developed and implemented to anticipate, detect, identify, prevent, mitigate, respond to, and attribute such potential exploitation. The first published paper on cyberbiosecurity primarily focuses on the security of the biotechnology interface with cyberspace.<sup id=\"rdp-ebb-cite_ref-PeccoudCyber18_5-0\" class=\"reference\"><a href=\"#cite_note-PeccoudCyber18-5\">[5]<\/a><\/sup> In addition to the system analysis as part of the second set, a small workshop was held in the U.S. that sought to scope and stimulate interest in the government, academic, corporate, and non-profit sectors, create a core constituency, understand what topics and themes could constitute cyberbiosecurity, identify priorities, and begin to develop a campaign and timeline. The workshop was highly successful. These endeavors, together with additional recent activities and publications<sup id=\"rdp-ebb-cite_ref-KozminskiBio15_6-0\" class=\"reference\"><a href=\"#cite_note-KozminskiBio15-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PauwelsHow16_7-0\" class=\"reference\"><a href=\"#cite_note-PauwelsHow16-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PauwelsWho17_8-0\" class=\"reference\"><a href=\"#cite_note-PauwelsWho17-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PauwelsTheInt17_9-0\" class=\"reference\"><a href=\"#cite_note-PauwelsTheInt17-9\">[9]<\/a><\/sup>, have added to scoping the future of cyberbiosecurity yet to come.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Simply stated, since its inception, biosecurity has been primarily focused on reducing the risks associated with the misuse of science which could cause harm to humans, animals, plants, and the environment through the creation, production, and deliberate or accidental release of infectious disease agents or their byproducts (e.g., toxins). Cybersecurity has been a separate field which has been primarily focused on the security of information technology systems, from personal computers and communications devices to large infrastructures and networks. Up until just the past few years, the \u201ccyber\u201d overlaps with biosecurity have not been realized or fleshed out. The important interrelationship between biosecurity and cybersecurity is gaining increasing attention. We posit that the two must work collaboratively and will not be effective working separately. Cyberbiosecurity actually started with thinking about a particular set of problems being confronted by the life sciences. As a result of our recent work, described below, other dimensions are being added. Establishing a unifying discipline, crafting its systematics, and identifying an evolutionary path forward are within reach.\n<\/p><p>The economic strength and growth of the United States have been due to a culture and environment that foster innovation. Those developments could not be possible without significant contributions by science and engineering. The intersection among economic growth and the biological sciences contributions\u2014the bioeconomy\u2014has recently been recognized as an important component of national security. For the U.S., the bioeconomy accounts for an estimated $4 trillion annually, nearly 25% of the GDP. That contribution ranges from pharmaceuticals to renewable energy, from environmental remediation to public health resilience, and from agriculture to emerging disease response. As part of the U.S. national security architecture, \u201csafeguarding the sciences\u201d is a priority. In doing so, the U.S. Federal Bureau of Investigation (FBI) and other federal agencies also fulfill the U.S. obligation to the Biological Toxins and Weapons Convention (BTWC) and compliance to the United Nations Security Council Resolution (UNSCR) 1540, preventing the misuse of biological material, technology, and expertise, and encouraging the enforcement of the related statutes. The FBI also sponsors and actively engages the International Genetically Engineered Machine (iGEM) competition to inculcate a culture of security among international students, who will become leaders of research, industry, and policymaking. At the same time, the FBI works with U.S. policymakers<sup id=\"rdp-ebb-cite_ref-YouSafe17_10-0\" class=\"reference\"><a href=\"#cite_note-YouSafe17-10\">[10]<\/a><\/sup> to redefine the scope of the biosecurity spectrum for the twenty-first century, a century with an unprecedented pace of biological research and innovation, and the use of diverse and large datasets (big data) to assist global scientific and societal priorities and opportunities. Concomitant to both realized and future benefits and growth, the life sciences are becoming increasingly digitized\u2014while at the same time intellectual property protection, cyber intrusion, and the protection of personal medical and <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomic<\/a> information becoming more important\u2014and the impacts on science, trade, and commerce loom large. Engagements with the science media<sup id=\"rdp-ebb-cite_ref-KozminskiBio15_6-1\" class=\"reference\"><a href=\"#cite_note-KozminskiBio15-6\">[6]<\/a><\/sup> and testimonies<sup id=\"rdp-ebb-cite_ref-YouSafe17_10-1\" class=\"reference\"><a href=\"#cite_note-YouSafe17-10\">[10]<\/a><\/sup> have raised these issues to advance both U.S. competitiveness and national security.\n<\/p><p>In 2014, the American Association for the Advancement of Science (AAAS), FBI, and the United Nations Interregional Crime and Justice Research Institute (UNICRI) published a report entitled \u201cNational and Transnational Security Implications of Big Data in the Life Sciences.<sup id=\"rdp-ebb-cite_ref-AAASNat14_2-1\" class=\"reference\"><a href=\"#cite_note-AAASNat14-2\">[2]<\/a><\/sup> Briefly, this report starts by helping to understand \u201cbig data\u201d; massive, diverse data sets that are created, reside, are analyzed in, and move in information ecosystems. For the life sciences, big data refers to datasets including \u201craw data, combined data, or published data from the health-care system, pharmaceutical industry, genomics and other \u2013omics fields, clinical research, environment, agriculture, and microbiome efforts.\u201d Further, they state that big data also includes analytic technologies and outputs, such as from \u201cdata integration, data mining, data fusion, image and speech recognition, natural language processing, machine learning, social media analysis, and Bayesian analysis.\u201d A number of areas that have drawn and need attention are pointed out, such as the security of the cyber infrastructure and data repositories, and the privacy and confidentiality of individuals. In our view, their focus on the security risks of big data in the life sciences falls into just two major categories, i.e., inappropriate access to data and analytic technologies through vulnerabilities in the data and cyber infrastructure. As such, the use of big data technologies to integrate current data and enable the design of a harmful biological agent should be revisited and refined. Thanks to this team's efforts, not only do we have a useful topology of big data, the beginnings of a structure for thinking about security implications at the bio-cyber interface (technical, legal, institutional, and individual) and a set of high-level recommendations for a path forward.\n<\/p><p>From 2014 to 2016, three workshops were organized by the U.S. National Academies on behalf of the FBI under the theme of \u201cSafeguarding the Bioeconomy.\u201d The first<sup id=\"rdp-ebb-cite_ref-NASEMMeeting14_1-2\" class=\"reference\"><a href=\"#cite_note-NASEMMeeting14-1\">[1]<\/a><\/sup> laid the foundation for the next two. Presentations and discussions focused on the security implications of the convergence in the life and chemical sciences with physical, mathematical, computational, engineering, and social and behavioral sciences. In addition to broader contexts, two specific technologies received focus: neuromorphic computing and 3-D bioprinting. The second workshop<sup id=\"rdp-ebb-cite_ref-NASEMSafe15_3-1\" class=\"reference\"><a href=\"#cite_note-NASEMSafe15-3\">[3]<\/a><\/sup> introduced a range of new threats to and vulnerabilities of the bioeconomy, which at the time had not received focused consideration with respect to U.S. \u201ccompetiveness, security, economic growth, and global leadership in research and innovation.\u201d This workshop was built on three major themes: the role of <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">informatics<\/a> in the bioeconomy, criminal threats and vulnerabilities in the existing and near-future bioeconomy, and securing and flourishing the bioeconomy for the future. Rapid growth of this sector creates increasing security risks to proprietary materials and informatics, brings about an increase in frequency in industrial espionage and data hacks, and decreases the effectiveness of traditional security measures. Still, alternative and adaptive security measures could be implemented even with the inherent openness of emerging technologies upon which the bioeconomy is dependent. Workshop participants not only provided more detail on the threats and vulnerabilities but also both comprehensive categories and specific approaches that could be taken to address the problems and concerns identified. The third workshop<sup id=\"rdp-ebb-cite_ref-NASEMSafe16_4-1\" class=\"reference\"><a href=\"#cite_note-NASEMSafe16-4\">[4]<\/a><\/sup> principally focused on data generation and access with respect to the bioeconomy within several categories of both clinical and non-clinical data, from the perspectives of biosecurity, data policy and regulation, future implications, technology advances, data sovereignty and sharing, cybersecurity, and international implications. Taken together, these events significantly expanded the view of what the emerging discipline of cyberbiosecurity could encompass.\n<\/p><p>Pauwels and her co-authors also raise important concerns and recommendations for the security of biotechnology in cyberspace. In the first, she and Vidyarthi<sup id=\"rdp-ebb-cite_ref-PauwelsHow16_7-1\" class=\"reference\"><a href=\"#cite_note-PauwelsHow16-7\">[7]<\/a><\/sup> raise concerns over data breaches of health care <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> and what it means for the biotechnology industry. Protecting digital DNA and personal medical information is highlighted, as well as the fact that a then recent U.S. Presidential cybersecurity initiative put significant resources into shoring up cyberinfrastructure. Unfortunately, the need for improvements to protecting the bioeconomy, which is heavily dependent on information systems and infrastructure, was not recognized. The report outlined the implications of not protecting the bioeconomy dimension. Their recommendations were primarily focused on protecting genomic data. In the second report, Pauwels and Dunlap<sup id=\"rdp-ebb-cite_ref-PauwelsTheInt17_9-1\" class=\"reference\"><a href=\"#cite_note-PauwelsTheInt17-9\">[9]<\/a><\/sup> go into more depth framing potential cyber-vulnerabilities for specific types of biotechnologies: genome-editing; DNA assembly, synthesis and printing; portable genomic sequencers; artificial intelligence for understanding biological complexity; autonomous systems and robotics in cloud labs; and lab-on-a-chip and microfluidic technologies, all of which have cyber-physical interfaces. These authors also suggest governance systems and policy recommendations which might be harnessed to address the lab-focused concerns they raise.\n<\/p><p>Other recent publications also highlight the complexity of the enterprise we are terming \u201ccyberbiosecurity\u201d and concerns over security, robustness, and resiliency. These include:\n<\/p>\n<ul><li> security of personal genomic data when foreign companies purchase all or part of a U.S. company or are contracted for genomic or health care data services, which provides access to sensitive personal information<sup id=\"rdp-ebb-cite_ref-PauwelsWho17_8-1\" class=\"reference\"><a href=\"#cite_note-PauwelsWho17-8\">[8]<\/a><\/sup>;<\/li>\n<li> the continuing vulnerability of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a><sup id=\"rdp-ebb-cite_ref-WeiseMillions15_11-0\" class=\"reference\"><a href=\"#cite_note-WeiseMillions15-11\">[11]<\/a><\/sup> and health care systems<sup id=\"rdp-ebb-cite_ref-HackettUCLA15_12-0\" class=\"reference\"><a href=\"#cite_note-HackettUCLA15-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WintonHolly16_13-0\" class=\"reference\"><a href=\"#cite_note-WintonHolly16-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GriffinNHS17_14-0\" class=\"reference\"><a href=\"#cite_note-GriffinNHS17-14\">[14]<\/a><\/sup>;<\/li>\n<li> imposing control over DNA sequencing through DNA-encoded malware<sup id=\"rdp-ebb-cite_ref-GreenbergBio17_15-0\" class=\"reference\"><a href=\"#cite_note-GreenbergBio17-15\">[15]<\/a><\/sup>;<\/li>\n<li> synthetic biology supply chain vulnerabilities<sup id=\"rdp-ebb-cite_ref-FrazarDefining17_16-0\" class=\"reference\"><a href=\"#cite_note-FrazarDefining17-16\">[16]<\/a><\/sup>;<\/li>\n<li> cybersecurity compromise of large industrial biopharma companies<sup id=\"rdp-ebb-cite_ref-CollierMerck17_17-0\" class=\"reference\"><a href=\"#cite_note-CollierMerck17-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShabanPharma17_18-0\" class=\"reference\"><a href=\"#cite_note-ShabanPharma17-18\">[18]<\/a><\/sup>; and<\/li>\n<li> high-level studies which are systematically examining U.S. biodefense programs and capabilities.<sup id=\"rdp-ebb-cite_ref-BlueANat15_19-0\" class=\"reference\"><a href=\"#cite_note-BlueANat15-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CenterEmergence17_20-0\" class=\"reference\"><a href=\"#cite_note-CenterEmergence17-20\">[20]<\/a><\/sup><\/li><\/ul>\n<p>The darkweb\/darknet<sup id=\"rdp-ebb-cite_ref-BeckettTheDark09_21-0\" class=\"reference\"><a href=\"#cite_note-BeckettTheDark09-21\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-InterpolPharma15_22-0\" class=\"reference\"><a href=\"#cite_note-InterpolPharma15-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LangewiescheWelcome16_23-0\" class=\"reference\"><a href=\"#cite_note-LangewiescheWelcome16-23\">[23]<\/a><\/sup> could be included as it interfaces with dual use life science endeavors and biopharma research, development, intellectual property, and products, compromising the integrity of critical life science and health cyber-supported technologies and infrastructures. Because of the reliance on <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a>, the security of synthetic DNA could also be included, as well.<sup id=\"rdp-ebb-cite_ref-AdamStrengths11_24-0\" class=\"reference\"><a href=\"#cite_note-AdamStrengths11-24\">[24]<\/a><\/sup> Clearly, this rapidly expanding galaxy does needs a universally accepted definition, common terms of reference, and defined boundaries and structure for best value, ordered evolution, and impact.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Adding_another_dimension:_Cyberbiosecurity_systems_analysis_of_a_biomanufacturing_facility\">Adding another dimension: Cyberbiosecurity systems analysis of a biomanufacturing facility<\/span><\/h2>\n<p>Now we add another dimension to cyberbiosecurity and take an approach that we posit that should be incorporated with other aspects discussed earlier. The biopharma industry itself has its own substantial equities and investments in the research, development, production, and sale of vaccines, therapeutics, and prophylactics for the global market. The U.S. Government has substantial investments in the development and production of critical vaccines and biotherapeutics for both civilian and military purposes. Concomitantly, experts are increasingly recognizing that biomanufacturing itself is potentially vulnerable to unwanted or illicit activities which could result in damaging outcomes. These could include the theft of intellectual property, disruption of the supply chain, manipulation of the bioprocess development and bioproduction, cyberattacks on key information technology components and cyberphysical interfaces, the corruption of critical data, and manipulation of security systems and infrastructure upon which secure and safe facility operations are dependent. Our sponsor was not interested in generalizations or esoteric approximations about the security vulnerabilities of a biomanufacturing facility but wanted a comprehensive, detailed, actionable analysis.\n<\/p><p>Thus, we undertook an in-depth, multidimensional analysis of an existing biomanufacturing facility to identify and project security gaps and vulnerabilities, make recommendations with respect to addressing those identified and projected, and set the stage for more specific and comprehensive measures to be undertaken, whether they exist or have to be developed and validated. The systems analysis approach used was designed to assess the state of security at present, determine what an acceptable state of security would be, and provide guidance and recommendations to take the facility from its current state to the desired state.\n<\/p><p>The bioprocess development\/bioproduction facility used as the \u201ctest bed\u201d for this analysis designs, develops, and produces clinical trial quantities of protein-based biotherapeutics and the associated documentation for commercial and government clients. If the outputs from this \u201ctest bed\u201d meet client expectations and the client receives government approval, the client scales up production and the product is marketed. This facility was studied as a system, consisting of four key, interrelated subsystems: end-to-end bioprocess development\/biomanufacturing; the supply chain; the supporting information systems infrastructure and cyber-physical interfaces with bioprocess development and biomanufacturing; and facility infrastructure, including its relationship to the facility's host infrastructure. The systems analysis was a phased process with project management methods applied. The facility or any of its components or operations were not compromised, corrupted, or altered during this project in any manner or form. Rather, it was studied thoroughly yet benignly. The analysis included human factors and \u201cdownstream\u201d considerations, as well.\n<\/p><p>The systems aspects of a biomanufacturing facility which are potentially vulnerable to security threats and the solutions required are summarized in Figure 1.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Murch_FrontBioengBiotech2019_6.jpg\" class=\"image wiki-link\" data-key=\"acd3b441c43eec2b6a958a34ea0264a0\"><img alt=\"Fig1 Murch FrontBioengBiotech2019 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/9c\/Fig1_Murch_FrontBioengBiotech2019_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> A systems view of protecting a biomanufacturing facility. For each defensive set identified, multiple threats and impacts were identified and potentially more than one pathway or technique could be used by an adversary to achieve their objectives. GMP, Good Manufacturing Practice; IT, Information Technology; IS, Information Systems; VPN, Virtual Private Network; IP, Intellectual Property.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Due to space limitations, we provide only a top-level view of the analysis. Key overarching findings include:\n<\/p>\n<ul><li> Vulnerabilities can exist across the entire system, from bioprocess development and GMP to supply chain, to cyber-physical and infrastructure; there are potentially more than one might anticipate <i>a priori<\/i>.<\/li>\n<li> Successful exploitation of vulnerabilities can occur through passive and active means for passive and active purposes, depending upon adversaries' intentions, objectives, accesses, knowledge and resources, and outcomes sought.<\/li>\n<li> Exploitation of some vulnerabilities require direct access to facilities or components; personnel and physical security aspects should not be overlooked.<\/li>\n<li> Adversaries can use combinations and sequences of methods and targeting, both subtle and not, to attempt to and achieve their objectives.<\/li>\n<li> The operational capabilities of adversaries, not just technical, must be considered and accounted for in planning for and implementing security measures.<\/li><\/ul>\n<p>We emphasize that, while there are general principles that apply and observations that will derive from such analyses, the analysis design and execution and the resulting solution set, should be tailored on a facility-by-facility basis. We note that the defensive areas noted may not be singular, but rather require combinations of defensive approaches and techniques to be identified and implemented to ensure optimal security robustness.\n<\/p><p>What is considerably important from this analysis is that a rigorous study of a facility such as this can result in the identification and characterization of discrete vulnerabilities, gaps, shortfalls, and opportunities for which readily-available solutions can be implemented, or otherwise can be developed, tested, and implemented. We did not conduct detailed studies regarding how genomics can be compromised as it relates to biomanufacturing because we were directed not to, but we are well aware of plausible scenarios and what the effects could be.\n<\/p><p>Our analysis demonstrates that biomanufacturing facilities can benefit from comprehensive, multidisciplinary analyses to identify security vulnerabilities, leading to solutions to mitigate or address them. This, in turn, raises the prospect of the development and validation of a set of methods or protocols would be in order which could be used by facility staff or external service providers to shore up individual facilities, from do-it-yourself to large biopharma. Walking this out, guidelines or standards could be developed, established, and accepted to ensure consistency and quality of the analyses conducted, the credentials of the personnel doing so, and the quality and effectiveness of measures undertaken. While sophisticated adversaries could design and execute sophisticated attacks, it is likely in many instances that relatively straightforward methods and practices could raise the bar considerably to reduce risk. Lastly, combining analyses of this sort could be used as a basis for informed investments in research, development, testing, and evaluation for solutions to the most worrisome current and future threats.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Moving_cyberbiosecurity_forward\">Moving cyberbiosecurity forward<\/span><\/h2>\n<p>Many other critical cyber-enabled life science and biomedical technologies, systems, and applications naturally lend themselves to inclusion within cyberbiosecurity. These include, but are not limited to, personalized genomics, medical and fitness technologies, 3-D printing of critical personalized medical devices, and medical laboratory and surgical robotics. A more comprehensive system is warranted. Cyberbiosecurity could be expanded to include cyber-bio systems within agriculture and farm-to-table food production, processing and distribution systems, and within natural resource and environmental management. Direct and ordered engagements of the pertinent sectors of the life sciences, biosecurity, and cyber-cybersecurity communities should occur. Academia, industry, government, and non-profits (including policy, regulatory, and legal experts) need to begin to learn to communicate with and educate each other, harmoniously identify and develop priorities and opportunities, and specify \u201cnext steps.\u201d A major opportunity exists right now to propose a unified structure and common vernacular. Lastly, while definition and assemblage of cyberbiosecurity is occurring, national or international strategies should be pursued to harmonize the emerging enterprise and foster measurable value, success, and sustainability.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>RM: Co-originator of cyberbiosecurity concept; Lead author, responsible for structure, content and figure; responsible for considering, incorporating co-author contributions and suggested modifications; responsible for final version. WS: Co-originator of Safeguarding the Bioeconomy concept and campaign; contributed or reviewed and edited paragraphs related to the AAAS-FBI-UNICRI Big Data report, the US National Academies workshops and the relevant FBI programs and initiatives. SR: Co-originator of cyberbiosecurity concept; overall quality assurance and readability reviews and modifications; contributions to section on cyberbiosecurity applied to biomanufacturing. WB: Contributions to section on cyberbiosecurity applied to biomanufacturing; reviews to ensure content quality and readability of paper. JP: Co-originator of cyberbiosecurity concept; review and critique of manuscript to ensure complementarity and alignment with first paper published on cyberbiosecurity (biotechnology focus) in another journal (he is the lead author; RM, SR, and WB are co-authors).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>The funding for the systems analysis of the Test Bed Facility was provided through a contract award (Contract FA4600-12-D-9000, Task Order 0065) from the U. S. Department of Defense, United States Strategic Command to the National Strategic Research Institute, University of Nebraska.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-NASEMMeeting14-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NASEMMeeting14_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-NASEMMeeting14_1-1\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-NASEMMeeting14_1-2\">1.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Board on Chemical Sciences and Technology; Board on Life Sciences (2014). <i>Meeting Recap: Workshop - Convergence: Safeguarding Technology in the Bioeconomy<\/i>. The National Academies of Sciences, Engineering, and Medicine.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Meeting+Recap%3A+Workshop+-+Convergence%3A+Safeguarding+Technology+in+the+Bioeconomy&rft.aulast=Board+on+Chemical+Sciences+and+Technology%3B+Board+on+Life+Sciences&rft.au=Board+on+Chemical+Sciences+and+Technology%3B+Board+on+Life+Sciences&rft.date=2014&rft.pub=The+National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AAASNat14-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AAASNat14_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-AAASNat14_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">FBI WMD Directorate, American Association for the Advancement of Science, United Nations Interregional Crime and Justice Research Institute (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.aaas.org\/sites\/default\/files\/AAAS-FBI-UNICRI_Big_Data_Report_111014.pdf\" target=\"_blank\">\"National and Transnational Security Implications of Big Data in the Life Sciences\"<\/a> (PDF). AAAS<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.aaas.org\/sites\/default\/files\/AAAS-FBI-UNICRI_Big_Data_Report_111014.pdf\" target=\"_blank\">http:\/\/www.aaas.org\/sites\/default\/files\/AAAS-FBI-UNICRI_Big_Data_Report_111014.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=National+and+Transnational+Security+Implications+of+Big+Data+in+the+Life+Sciences&rft.atitle=&rft.aulast=FBI+WMD+Directorate%2C+American+Association+for+the+Advancement+of+Science%2C+United+Nations+Interregional+Crime+and+Justice+Research+Institute&rft.au=FBI+WMD+Directorate%2C+American+Association+for+the+Advancement+of+Science%2C+United+Nations+Interregional+Crime+and+Justice+Research+Institute&rft.date=2014&rft.pub=AAAS&rft_id=http%3A%2F%2Fwww.aaas.org%2Fsites%2Fdefault%2Ffiles%2FAAAS-FBI-UNICRI_Big_Data_Report_111014.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NASEMSafe15-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NASEMSafe15_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-NASEMSafe15_3-1\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Board on Chemical Sciences and Technology (2015) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf\" target=\"_blank\"><i>Meeting Recap: Safeguarding the Bioeconomy: Applications and Implications of Emerging Science<\/i><\/a>. The National Academies of Sciences, Engineering, and Medicine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf\" target=\"_blank\">https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Meeting+Recap%3A+Safeguarding+the+Bioeconomy%3A+Applications+and+Implications+of+Emerging+Science&rft.aulast=Board+on+Chemical+Sciences+and+Technology&rft.au=Board+on+Chemical+Sciences+and+Technology&rft.date=2015&rft.pub=The+National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rft_id=https%3A%2F%2Fwww.ehidc.org%2Fsites%2Fdefault%2Ffiles%2Fresources%2Ffiles%2FSafeguarding%2520the%2520Bioeconomy_II_Recap%2520Final%2520090815.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NASEMSafe16-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NASEMSafe16_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-NASEMSafe16_4-1\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Board on Life Sciences and Board on Chemical Sciences and Technology (2016) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ibpforum.org\/sites\/default\/files\/Safeguarding_the_Bioeconomy_III_Recap.pdf\" target=\"_blank\"><i>Meeting Recap: Safeguarding the Bioeconomy III: Securing Life Sciences Data<\/i><\/a>. The National Academies of Sciences, Engineering, and Medicine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ibpforum.org\/sites\/default\/files\/Safeguarding_the_Bioeconomy_III_Recap.pdf\" target=\"_blank\">https:\/\/www.ibpforum.org\/sites\/default\/files\/Safeguarding_the_Bioeconomy_III_Recap.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Meeting+Recap%3A+Safeguarding+the+Bioeconomy+III%3A+Securing+Life+Sciences+Data&rft.aulast=Board+on+Life+Sciences+and+Board+on+Chemical+Sciences+and+Technology&rft.au=Board+on+Life+Sciences+and+Board+on+Chemical+Sciences+and+Technology&rft.date=2016&rft.pub=The+National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rft_id=https%3A%2F%2Fwww.ibpforum.org%2Fsites%2Fdefault%2Ffiles%2FSafeguarding_the_Bioeconomy_III_Recap.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PeccoudCyber18-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PeccoudCyber18_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". <i>Trends in Biotechnology<\/i> <b>36<\/b> (1): 4\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.tibtech.2017.10.012\" target=\"_blank\">10.1016\/j.tibtech.2017.10.012<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29224719\" target=\"_blank\">29224719<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyberbiosecurity%3A+From+Naive+Trust+to+Risk+Awareness&rft.jtitle=Trends+in+Biotechnology&rft.aulast=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.au=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.date=2018&rft.volume=36&rft.issue=1&rft.pages=4%E2%80%937&rft_id=info:doi\/10.1016%2Fj.tibtech.2017.10.012&rft_id=info:pmid\/29224719&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KozminskiBio15-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KozminskiBio15_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-KozminskiBio15_6-1\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kozminski, K.G.; Drubin, D.G. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4710219\" target=\"_blank\">\"Biosecurity in the age of Big Data: A conversation with the FBI\"<\/a>. <i>Molecular Biology of the Cell<\/i> <b>26<\/b> (22): 3894\u201397. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1091%2Fmbc.E14-01-0027\" target=\"_blank\">10.1091\/mbc.E14-01-0027<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4710219\/\" target=\"_blank\">PMC4710219<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26543195\" target=\"_blank\">26543195<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4710219\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4710219<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Biosecurity+in+the+age+of+Big+Data%3A+A+conversation+with+the+FBI&rft.jtitle=Molecular+Biology+of+the+Cell&rft.aulast=Kozminski%2C+K.G.%3B+Drubin%2C+D.G.&rft.au=Kozminski%2C+K.G.%3B+Drubin%2C+D.G.&rft.date=2015&rft.volume=26&rft.issue=22&rft.pages=3894%E2%80%9397&rft_id=info:doi\/10.1091%2Fmbc.E14-01-0027&rft_id=info:pmc\/PMC4710219&rft_id=info:pmid\/26543195&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4710219&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PauwelsHow16-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PauwelsHow16_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-PauwelsHow16_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pauwels, E.; Vidyarthi, A. (29 March 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wilsoncenter.org\/publication\/how-our-unhealthy-cybersecurity-infrastructure-hurting-biotechnology\" target=\"_blank\">\"How Our Unhealthy Cybersecurity Infrastructure Is Hurting Biotechnology\"<\/a>. <i>Wilson Briefs<\/i>. Wilson Center<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wilsoncenter.org\/publication\/how-our-unhealthy-cybersecurity-infrastructure-hurting-biotechnology\" target=\"_blank\">https:\/\/www.wilsoncenter.org\/publication\/how-our-unhealthy-cybersecurity-infrastructure-hurting-biotechnology<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+Our+Unhealthy+Cybersecurity+Infrastructure+Is+Hurting+Biotechnology&rft.atitle=Wilson+Briefs&rft.aulast=Pauwels%2C+E.%3B+Vidyarthi%2C+A.&rft.au=Pauwels%2C+E.%3B+Vidyarthi%2C+A.&rft.date=29+March+2016&rft.pub=Wilson+Center&rft_id=https%3A%2F%2Fwww.wilsoncenter.org%2Fpublication%2Fhow-our-unhealthy-cybersecurity-infrastructure-hurting-biotechnology&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PauwelsWho17-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PauwelsWho17_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-PauwelsWho17_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pauwels, E.; Vidyarthi, A. (19 November 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics\" target=\"_blank\">\"Who Will Own The Secrets In Our Genes? A U.S. \u2013 China Race in Artificial Intelligence and Genomics\"<\/a>. <i>Wilson Briefs<\/i>. Wilson Center<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics\" target=\"_blank\">https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Who+Will+Own+The+Secrets+In+Our+Genes%3F+A+U.S.+%E2%80%93+China+Race+in+Artificial+Intelligence+and+Genomics&rft.atitle=Wilson+Briefs&rft.aulast=Pauwels%2C+E.%3B+Vidyarthi%2C+A.&rft.au=Pauwels%2C+E.%3B+Vidyarthi%2C+A.&rft.date=19+November+2017&rft.pub=Wilson+Center&rft_id=https%3A%2F%2Fwww.wilsoncenter.org%2Fpublication%2Fwho-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PauwelsTheInt17-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PauwelsTheInt17_9-0\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-PauwelsTheInt17_9-1\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pauwels, E.; Dunlap, E. (07 September 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wilsoncenter.org\/publication\/the-intelligent-and-connected-bio-labs-the-future-promise-and-peril-the-fourth\" target=\"_blank\">\"The Intelligent and Connected Bio-Labs of the Future: Promise and Peril in the Fourth Industrial Revolution\"<\/a>. <i>Wilson Briefs<\/i>. Wilson Center<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wilsoncenter.org\/publication\/the-intelligent-and-connected-bio-labs-the-future-promise-and-peril-the-fourth\" target=\"_blank\">https:\/\/www.wilsoncenter.org\/publication\/the-intelligent-and-connected-bio-labs-the-future-promise-and-peril-the-fourth<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Intelligent+and+Connected+Bio-Labs+of+the+Future%3A+Promise+and+Peril+in+the+Fourth+Industrial+Revolution&rft.atitle=Wilson+Briefs&rft.aulast=Pauwels%2C+E.%3B+Dunlap%2C+E.&rft.au=Pauwels%2C+E.%3B+Dunlap%2C+E.&rft.date=07+September+2017&rft.pub=Wilson+Center&rft_id=https%3A%2F%2Fwww.wilsoncenter.org%2Fpublication%2Fthe-intelligent-and-connected-bio-labs-the-future-promise-and-peril-the-fourth&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YouSafe17-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-YouSafe17_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-YouSafe17_10-1\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">You, E.H. (16 March 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf\" target=\"_blank\">\"Safeguarding the Bioeconomy: U.S. Opportunities and Challenges - Testimony for the U.S.-China Economic and Security Review Commission\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf\" target=\"_blank\">https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Safeguarding+the+Bioeconomy%3A+U.S.+Opportunities+and+Challenges+-+Testimony+for+the+U.S.-China+Economic+and+Security+Review+Commission&rft.atitle=&rft.aulast=You%2C+E.H.&rft.au=You%2C+E.H.&rft.date=16+March+2017&rft_id=https%3A%2F%2Fwww.ehidc.org%2Fsites%2Fdefault%2Ffiles%2Fresources%2Ffiles%2FEd_You_Testimony_USCC.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeiseMillions15-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WeiseMillions15_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Weise, E. (05 February 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.usatoday.com\/story\/tech\/2015\/02\/05\/anthem-health-care-computer-security-breach\/22917635\/\" target=\"_blank\">\"Millions of Anthem customers alerted to hack\"<\/a>. <i>USA Today<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.usatoday.com\/story\/tech\/2015\/02\/05\/anthem-health-care-computer-security-breach\/22917635\/\" target=\"_blank\">https:\/\/www.usatoday.com\/story\/tech\/2015\/02\/05\/anthem-health-care-computer-security-breach\/22917635\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Millions+of+Anthem+customers+alerted+to+hack&rft.atitle=USA+Today&rft.aulast=Weise%2C+E.&rft.au=Weise%2C+E.&rft.date=05+February+2015&rft_id=https%3A%2F%2Fwww.usatoday.com%2Fstory%2Ftech%2F2015%2F02%2F05%2Fanthem-health-care-computer-security-breach%2F22917635%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HackettUCLA15-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HackettUCLA15_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hackett, R. (17 July 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/fortune.com\/2015\/07\/17\/ucla-health-system-data-breach\/\" target=\"_blank\">\"UCLA Health System data breach may affect millions\"<\/a>. <i>Fortune<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/fortune.com\/2015\/07\/17\/ucla-health-system-data-breach\/\" target=\"_blank\">http:\/\/fortune.com\/2015\/07\/17\/ucla-health-system-data-breach\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=UCLA+Health+System+data+breach+may+affect+millions&rft.atitle=Fortune&rft.aulast=Hackett%2C+R.&rft.au=Hackett%2C+R.&rft.date=17+July+2015&rft_id=http%3A%2F%2Ffortune.com%2F2015%2F07%2F17%2Fucla-health-system-data-breach%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WintonHolly16-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WintonHolly16_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Winton, R. (18 February 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.latimes.com\/business\/technology\/la-me-ln-hollywood-hospital-bitcoin-20160217-story.html\" target=\"_blank\">\"Hollywood hospital pays $17,000 in bitcoin to hackers; FBI investigating\"<\/a>. <i>Los Angeles Times<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.latimes.com\/business\/technology\/la-me-ln-hollywood-hospital-bitcoin-20160217-story.html\" target=\"_blank\">https:\/\/www.latimes.com\/business\/technology\/la-me-ln-hollywood-hospital-bitcoin-20160217-story.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Hollywood+hospital+pays+%2417%2C000+in+bitcoin+to+hackers%3B+FBI+investigating&rft.atitle=Los+Angeles+Times&rft.aulast=Winton%2C+R.&rft.au=Winton%2C+R.&rft.date=18+February+2016&rft_id=https%3A%2F%2Fwww.latimes.com%2Fbusiness%2Ftechnology%2Fla-me-ln-hollywood-hospital-bitcoin-20160217-story.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GriffinNHS17-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GriffinNHS17_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Griffin, A. (12 May 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.independent.co.uk\/news\/uk\/home-news\/nhs-cyber-attack-hack-hospitals-16-patients-turned-away-wanna-decryptor-a7733196.html\" target=\"_blank\">\"NHS hack: Cyber attack takes 16 hospitals offline as patients are turned away\"<\/a>. <i>Independent<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.independent.co.uk\/news\/uk\/home-news\/nhs-cyber-attack-hack-hospitals-16-patients-turned-away-wanna-decryptor-a7733196.html\" target=\"_blank\">https:\/\/www.independent.co.uk\/news\/uk\/home-news\/nhs-cyber-attack-hack-hospitals-16-patients-turned-away-wanna-decryptor-a7733196.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NHS+hack%3A+Cyber+attack+takes+16+hospitals+offline+as+patients+are+turned+away&rft.atitle=Independent&rft.aulast=Griffin%2C+A.&rft.au=Griffin%2C+A.&rft.date=12+May+2017&rft_id=https%3A%2F%2Fwww.independent.co.uk%2Fnews%2Fuk%2Fhome-news%2Fnhs-cyber-attack-hack-hospitals-16-patients-turned-away-wanna-decryptor-a7733196.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GreenbergBio17-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GreenbergBio17_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Greenberg, A. (10 August 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wired.com\/story\/malware-dna-hack\/\" target=\"_blank\">\"Biohackers encoded malware in a strand of DNA\"<\/a>. <i>Wired<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wired.com\/story\/malware-dna-hack\/\" target=\"_blank\">https:\/\/www.wired.com\/story\/malware-dna-hack\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Biohackers+encoded+malware+in+a+strand+of+DNA&rft.atitle=Wired&rft.aulast=Greenberg%2C+A.&rft.au=Greenberg%2C+A.&rft.date=10+August+2017&rft_id=https%3A%2F%2Fwww.wired.com%2Fstory%2Fmalware-dna-hack%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FrazarDefining17-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FrazarDefining17_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Frazar, S.L.; Hund, G.E.; Bonheyo, G.T. et al. (2017). \"Defining the synthetic biology supply chain\". <i>Health Security<\/i> <b>15<\/b> (4): 392-400. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fhs.2016.0083\" target=\"_blank\">10.1089\/hs.2016.0083<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28767286\" target=\"_blank\">28767286<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Defining+the+synthetic+biology+supply+chain&rft.jtitle=Health+Security&rft.aulast=Frazar%2C+S.L.%3B+Hund%2C+G.E.%3B+Bonheyo%2C+G.T.+et+al.&rft.au=Frazar%2C+S.L.%3B+Hund%2C+G.E.%3B+Bonheyo%2C+G.T.+et+al.&rft.date=2017&rft.volume=15&rft.issue=4&rft.pages=392-400&rft_id=info:doi\/10.1089%2Fhs.2016.0083&rft_id=info:pmid\/28767286&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CollierMerck17-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CollierMerck17_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Collier, K. (30 June 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cyberscoop.com\/merck-petya-ransomware-ukraine\/\" target=\"_blank\">\"Merck IT systems still crippled in Petya's aftermath\"<\/a>. <i>CyberScoop<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cyberscoop.com\/merck-petya-ransomware-ukraine\/\" target=\"_blank\">https:\/\/www.cyberscoop.com\/merck-petya-ransomware-ukraine\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Merck+IT+systems+still+crippled+in+Petya%27s+aftermath&rft.atitle=CyberScoop&rft.aulast=Collier%2C+K.&rft.au=Collier%2C+K.&rft.date=30+June+2017&rft_id=https%3A%2F%2Fwww.cyberscoop.com%2Fmerck-petya-ransomware-ukraine%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShabanPharma17-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShabanPharma17_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Shaban, H.; Nakashima, E. (27 June 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.washingtonpost.com\/news\/the-switch\/wp\/2017\/06\/27\/pharmaceutical-giant-rocked-by-ransomware-attack\" target=\"_blank\">\"Pharmaceutical giant rocked by ransomware attack\"<\/a>. <i>The Washington Post<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.washingtonpost.com\/news\/the-switch\/wp\/2017\/06\/27\/pharmaceutical-giant-rocked-by-ransomware-attack\" target=\"_blank\">https:\/\/www.washingtonpost.com\/news\/the-switch\/wp\/2017\/06\/27\/pharmaceutical-giant-rocked-by-ransomware-attack<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Pharmaceutical+giant+rocked+by+ransomware+attack&rft.atitle=The+Washington+Post&rft.aulast=Shaban%2C+H.%3B+Nakashima%2C+E.&rft.au=Shaban%2C+H.%3B+Nakashima%2C+E.&rft.date=27+June+2017&rft_id=https%3A%2F%2Fwww.washingtonpost.com%2Fnews%2Fthe-switch%2Fwp%2F2017%2F06%2F27%2Fpharmaceutical-giant-rocked-by-ransomware-attack&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlueANat15-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlueANat15_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Blue Ribbon Study Panel on Biodefense (28 October 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.hudson.org\/research\/11824-a-national-blueprint-for-biodefense-leadership-and-major-reform-needed-to-optimize-efforts\" target=\"_blank\">\"A National Blueprint for Biodefense: Leadership and Major Reform Needed To Optimize Efforts\"<\/a>. Hudson Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hudson.org\/research\/11824-a-national-blueprint-for-biodefense-leadership-and-major-reform-needed-to-optimize-efforts\" target=\"_blank\">https:\/\/www.hudson.org\/research\/11824-a-national-blueprint-for-biodefense-leadership-and-major-reform-needed-to-optimize-efforts<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+National+Blueprint+for+Biodefense%3A+Leadership+and+Major+Reform+Needed+To+Optimize+Efforts&rft.atitle=&rft.aulast=Blue+Ribbon+Study+Panel+on+Biodefense&rft.au=Blue+Ribbon+Study+Panel+on+Biodefense&rft.date=28+October+2015&rft.pub=Hudson+Institute&rft_id=https%3A%2F%2Fwww.hudson.org%2Fresearch%2F11824-a-national-blueprint-for-biodefense-leadership-and-major-reform-needed-to-optimize-efforts&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CenterEmergence17-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CenterEmergence17_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Center for the Study of Weapons of Mass Destruction (2017). <i>Emergence and Convergence Deep Dive: the Age of Genomic Data. Technical Executive Summary<\/i>. National Defense University.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Emergence+and+Convergence+Deep+Dive%3A+the+Age+of+Genomic+Data.+Technical+Executive+Summary&rft.aulast=Center+for+the+Study+of+Weapons+of+Mass+Destruction&rft.au=Center+for+the+Study+of+Weapons+of+Mass+Destruction&rft.date=2017&rft.pub=National+Defense+University&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BeckettTheDark09-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BeckettTheDark09_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Beckett, A. (25 November 2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.theguardian.com\/technology\/2009\/nov\/26\/dark-side-internet-freenet\" target=\"_blank\">\"The dark side of the internet\"<\/a>. <i>The Guardian<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.theguardian.com\/technology\/2009\/nov\/26\/dark-side-internet-freenet\" target=\"_blank\">https:\/\/www.theguardian.com\/technology\/2009\/nov\/26\/dark-side-internet-freenet<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+dark+side+of+the+internet&rft.atitle=The+Guardian&rft.aulast=Beckett%2C+A.&rft.au=Beckett%2C+A.&rft.date=25+November+2009&rft_id=https%3A%2F%2Fwww.theguardian.com%2Ftechnology%2F2009%2Fnov%2F26%2Fdark-side-internet-freenet&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-InterpolPharma15-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-InterpolPharma15_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">INTERPOL (24 February 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gwern.net\/docs\/sr\/2015-interpol-pharmaceuticals.pdf\" target=\"_blank\">\"Pharmaceutical Crime on the Darknet\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.gwern.net\/docs\/sr\/2015-interpol-pharmaceuticals.pdf\" target=\"_blank\">https:\/\/www.gwern.net\/docs\/sr\/2015-interpol-pharmaceuticals.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Pharmaceutical+Crime+on+the+Darknet&rft.atitle=&rft.aulast=INTERPOL&rft.au=INTERPOL&rft.date=24+February+2015&rft_id=https%3A%2F%2Fwww.gwern.net%2Fdocs%2Fsr%2F2015-interpol-pharmaceuticals.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LangewiescheWelcome16-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LangewiescheWelcome16_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Langewiesche, W. (11 September 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.vanityfair.com\/news\/2016\/09\/welcome-to-the-dark-net?verso=true\" target=\"_blank\">\"Welcome to the dark net, a wilderness where invisible world wars are fought and hackers roam free\"<\/a>. <i>Vanity Fair<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.vanityfair.com\/news\/2016\/09\/welcome-to-the-dark-net?verso=true\" target=\"_blank\">https:\/\/www.vanityfair.com\/news\/2016\/09\/welcome-to-the-dark-net?verso=true<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Welcome+to+the+dark+net%2C+a+wilderness+where+invisible+world+wars+are+fought+and+hackers+roam+free&rft.atitle=Vanity+Fair&rft.aulast=Langewiesche%2C+W.&rft.au=Langewiesche%2C+W.&rft.date=11+September+2016&rft_id=https%3A%2F%2Fwww.vanityfair.com%2Fnews%2F2016%2F09%2Fwelcome-to-the-dark-net%3Fverso%3Dtrue&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AdamStrengths11-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AdamStrengths11_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Adam, L.; Kozar, M.; Letort, G. et al. (2011). \"Strengths and limitations of the federal guidance on synthetic DNA\". <i>Nature Biotechnology<\/i> <b>29<\/b> (3): 208-10. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt.1802\" target=\"_blank\">10.1038\/nbt.1802<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21390018\" target=\"_blank\">21390018<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Strengths+and+limitations+of+the+federal+guidance+on+synthetic+DNA&rft.jtitle=Nature+Biotechnology&rft.aulast=Adam%2C+L.%3B+Kozar%2C+M.%3B+Letort%2C+G.+et+al.&rft.au=Adam%2C+L.%3B+Kozar%2C+M.%3B+Letort%2C+G.+et+al.&rft.date=2011&rft.volume=29&rft.issue=3&rft.pages=208-10&rft_id=info:doi\/10.1038%2Fnbt.1802&rft_id=info:pmid\/21390018&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165455\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.614 seconds\nReal time usage: 0.644 seconds\nPreprocessor visited node count: 17604\/1000000\nPreprocessor generated node count: 35454\/1000000\nPost\u2010expand include size: 129757\/2097152 bytes\nTemplate argument size: 52119\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 617.943 1 - -total\n 83.52% 516.136 1 - Template:Reflist\n 72.53% 448.193 24 - Template:Citation\/core\n 48.04% 296.838 16 - Template:Cite_web\n 15.51% 95.858 4 - Template:Cite_journal\n 14.08% 86.990 4 - Template:Cite_book\n 10.63% 65.693 1 - Template:Infobox_journal_article\n 10.15% 62.707 1 - Template:Infobox\n 6.00% 37.093 80 - Template:Infobox\/row\n 5.13% 31.727 36 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11034-0!*!0!!en!5!* and timestamp 20190701165454 and revision id 35478\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy\">https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","d53838ebf5172963d6c6496807f9617e_images":["https:\/\/www.limswiki.org\/images\/9\/9c\/Fig1_Murch_FrontBioengBiotech2019_6.jpg"],"d53838ebf5172963d6c6496807f9617e_timestamp":1562000094,"c35e087aee1f0ed0d5c790bbd387ef29_type":"article","c35e087aee1f0ed0d5c790bbd387ef29_title":"Smart information systems in cybersecurity: An ethical analysis (Macnish et al. 2019)","c35e087aee1f0ed0d5c790bbd387ef29_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis","c35e087aee1f0ed0d5c790bbd387ef29_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Smart information systems in cybersecurity: An ethical analysis\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nSmart information systems in cybersecurity: An ethical analysisJournal\n \nORBIT JournalAuthor(s)\n \nMacnish, Kevin; Fernandez-Inguanzo, Ana; Kirichenko, AlexeyAuthor affiliation(s)\n \nUniversity of Twente, F-SecurePrimary contact\n \nEmail: k dot macnish at utwente dot nlYear published\n \n2019Volume and issue\n \n2(2)Page(s)\n \n105DOI\n \n10.29297\/orbit.v2i2.105ISSN\n \n2515-8562Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.orbit-rri.org\/ojs\/index.php\/orbit\/article\/view\/105Download\n \nhttps:\/\/www.orbit-rri.org\/ojs\/index.php\/orbit\/article\/view\/105\/117 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 The use of smart information systems in cybersecurity \n4 Literature review: Ethical issues of using SIS in cybersecurity \n\n4.1 Informed consent \n4.2 Protection from harm \n4.3 Privacy and control of data \n4.4 Vulnerabilities and disclosure \n4.5 Competence of research ethics committees \n4.6 Security issues \n4.7 Trust and transparency \n4.8 Risk \n4.9 Responsibility \n4.10 Business interests and codes of conduct \n\n\n5 The case study of a cybersecurity company using SIS \n\n5.1 Description of SIS technologies being used in Company A \n5.2 The effectiveness of using SIS by Company A \n\n\n6 Ethical implications in cybersecurity \n\n6.1 Privacy \n6.2 Internationalization, standardization, and legal aspects \n6.3 Monetization issues \n6.4 Anomalies \n6.5 Policy issues, awareness, and knowledge \n6.6 Security \n6.7 Risk assessment \n6.8 Mechanisms to address ethical issues \n\n\n7 Conclusion \n\n7.1 Implications of this report \n7.2 Future research \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nThis report provides an overview of the current implementation of smart information systems (SIS) in the field of cybersecurity. It also identifies the positive and negative aspects of using SIS in cybersecurity, including ethical issues which could arise while using SIS in this area. One company working in the industry of telecommunications (Company A) is analysed in this report. Further specific ethical issues that arise when using SIS technologies in Company A are critically evaluated. Finally, conclusions are drawn on the case study, and areas for improvement are suggested. \nKeywords: cybersecurity, ethics, smart information systems, big data\n\nIntroduction \nIncreasing numbers of items are becoming connected to the internet. Cisco\u2014a global leader in information technology, networking, and cybersecurity\u2014estimates that more than 8.7 billion devices were connected to the internet by the end of 2012, a number that will likely rise to over 40 billion in 2020.[1] Cybersecurity has therefore become an important concern both publicly and privately. In the public sector, governments have created and enlarged cybersecurity divisions such as the U.S. Cyber Command and the Chinese \u201cInformation Security Base,\u201d whose mission is to provide security to critical national security assets.[1]\nIn the private sphere, companies are struggling to keep up with the required need for security in the face of increasingly sophisticated attacks from a variety of sources. In 2017, there were \u201cover 130 large-scale, targeted breaches [by hackers of computer networks] in the U.S.,\u201d and \u201cbetween January 1, 2005 and April 18, 2018 there have been 8,854 recorded breaches.\u201d[2] Furthermore, cyberattacks affect not only the online world, but also lead to vulnerabilities in the physical world, particularly when an attack threatens industries such as healthcare, communications, energy, or military networks, putting large swathes of society at risk. Indeed, it has been argued that some cyberattacks could constitute legitimate grounds for declarations of (physical) war.[3]\nCybersecurity is therefore a complex and multi-disciplinary issue. Security has been defined in the international relations and security studies spheres both as \u201cthe absence of threats to acquired values\u201d[4] and \u201cthe \u201cabsence of harm to acquired values.\u201d[5] Within the profession, cybersecurity is more commonly defined in terms of confidentiality, integrity, and availability of information.[6] A 2014 literature review on the meanings attributed to cybersecurity has led to the broader definition of cybersecurity as \"the organization and collection of resources, processes, and structures used to protect cyberspace and cyberspace-enabled systems.\u201d[7]\nCybersecurity therefore can be seen to encompass property rights of ownership of networks that could come under attack, as well as other concerns attributed with these, such as issues of access, extraction, contribution, removal, management, exclusion, and alienation.[8] Hence cybersecurity fulfills a similar role to physical security in protecting property from some level of intrusion. Craigen et al. also argue that cybersecurity refers not only to a technical domain, but also that the values underlying that domain should be included in the description of cybersecurity.[7] Seen this way, ethical issues and values form bedrock to cybersecurity research as identifying the values which cybersecurity seeks to protect.\nThe case study is divided into four main sections. The next two sections focus on the technical aspects of cybersecurity and a literature review of academic articles concerning ethical issues in cybersecurity, respectively. Then the practice of cybersecurity research is presented through an interview conducted with four employees at a major telecommunications software and hardware company, Company A. Finally, the last section critically evaluates ethical issues that have arisen in the use of SIS technologies in cybersecurity.\n\nThe use of smart information systems in cybersecurity \nThe introduction of big data and artificial intelligence (AI) (representations of smart information systems, or SIS) in cybersecurity is still in its early phase. Currently there is comparatively little work carried out on cybersecurity using SIS for several reasons. These include the remarkable diversity of cyberattacks (e.g., different approaches to hacking systems and introducing malware), the danger of false positives and false negatives, and the relatively low intelligence of existing SIS.\nTaking these in turn, the diversity of attacks\u2014both in the source of the attack, the focus of the attack, and the motivation of the attack\u2014is significant. Attacks can be launched from outside an organization (e.g., from a hacking collective, such as Anonymous) or from an insider (e.g., a disaffected employee looking to damage a system). They may come from a single source, typically masked through using the darknet, or from a source who has engaged in a number of \u201chops\u201d (moving from one computer on a network to another, thus masking the original source) such that the originator could appear to be in a hospital or in a military base. If an attack were to appear to come from a military base, this might encourage the attacked party to \u201chack back.\u201d However, if the military base were an artificial screen presented in front of a hospital, the reverse hack could bring down that hospital\u2019s computer networks. The focus of the attack could be on imitating a user or system administrator (local IT expert) or on exploiting a security flaw in unpatched code (programming in a network that has a flaw which has not yet been fixed, also known as a zero-day exploit). The motivation of the attack can range from state security and intelligence gathering (e.g., U.S. Intelligence spying on Chinese military installations), to financial incentives through blackmail (e.g., encrypting a company\u2019s files and agreeing to decrypt them only when the company has paid the hacker a certain sum of money). This diversity means that it is extremely difficulty to develop a SIS that will effectively recognize an attack for what it is.\nSecondly, the danger of false positives and false negatives is significant in light of the difficulty of recognizing an attack. If an attack is not recognized by a SIS as a false negative, it may be successful. This is particularly the case if security personnel have come to place undue trust in the automation and do not provide quality assurance of the SIS, a behavior known as \u201cautomation bias.\u201d[9][10] By contrast, the SIS could be so cautious that it may lead to an excessive number of false positives in which a legitimate interaction is falsely labelled an attack and not permitted to continue. This leads to frustration and could entail the eventual disabling of the SIS.[11]\nThirdly, and despite some hype in the media, SIS are still at a relatively unintelligent stage of development. Computer vision systems designed to identify people loitering, for example, recognize that a person has not left a circle with radius x in y number of seconds, but they cannot determine why the person is there or what their intent may be. As such, the inability to determine intentions from actions renders automated systems relatively impotent.\nDespite these concerns, there are some potential grounds for use of SIS in cybersecurity. The most effective is in scanning systems for known attacks, or known abnormal patterns of behavior that have a very high likelihood of being an attack. When coupled with a human operator to scan any alerts and so determine whether to take action, the combined human-machine security system can prove to be effective, albeit still facing the above problems of automation bias and excessive false positives.[12]\n\nLiterature review: Ethical issues of using SIS in cybersecurity \nIn this section we will conduct a literature review of the most fundamental ethical issues in cybersecurity that are being proposed in the academic environment. Our goal is to compare them with the interview that has been conducted in a major telecommunications software and hardware company, Company A, in order to give an overview on the ethical issues in cybersecurity. \nThe literature review was carried out through a combination of online search using generic engines, such as Google and Google Scholar, and discipline-specific search engines on websites such as PhilPapers.org and The Philosopher's Index. Selected papers were then read and, where appropriate, the bibliographic references were used to locate further literature. Generic search on Google also provided links to trade publications and websites that were a further source of background information. \nThe ethical issues to arise from the literature review were informed consent, protection from harm, privacy and control of data, vulnerabilities and disclosure, competence of research ethics committees, security issues, trust and transparency, risk, responsibility, and business interests and codes of conduct.\n\nInformed consent \nAcquiring informed consent is an important activity for cybersecurity, and one that has been at the heart of research ethics and practice for decades.[13][14] Consent is variously valued as the respect for autonomy[14] or the minimization of harm.[15] As such, the justification for informed consent is a considerable challenge for data analytics where anonymized data may be used without explicit consent of the person from whom it originates. This is also true within global cybersecurity, where a number of complicating issues arise, such as the complexity of informing users about detailed technical aspects in order to provide necessary information, as well as language barriers.[16] This, though, is the case for many other areas of research such as medical or social sciences, and the scripts need not be different in cybersecurity.[17]\nNonetheless, challenges of complexity, and of conveying that complexity in a manner that is sufficiently informative for a non-expert to make a decision, remain. Wolter Pieters notes that information provision does not correspond merely to the amount of information communicated, but to how it is presented, and that the type of information given is justified and appropriate. \u201cOne cannot speak about informed consent if one gives too little information, but one cannot speak about informed consent either if one gives too much. Indeed, giving too much information might lead to uninformed dissent, as distrust is invited by superfluous information.\u201d[18]\n\nProtection from harm \nCybersecurity has the potential to cause harm to its users, even when that harm is not intended. Concerns exist regarding the disclosure of vulnerabilities (such as a flaw in a security program which would allow for a hacker to break into the network with relative ease), for example, such as whether they should be disclosed publicly once a company has failed to address them. If not, then the vulnerability entails that a person may be at risk of attack, which is particularly concerning if the device at risk is medical in nature, such as a pacemaker.[19][20] However, disclosure could bring the vulnerability to the awareness of potential attackers who had not considered it previously. This is true of cybersecurity generally, whether involving SIS or not.\n\nPrivacy and control of data \nPrivacy is a central issue in cybersecurity, as increasing amounts of personal data are gathered and stored in the cloud. Furthermore, these data can be highly sensitive, such as health or bank records.[21] While the data at risk from attack is private, in order to identify an attack, particularly when SIS are involved, an effective cybersecurity system must maintain an awareness of \u201ctypical\u201d behavior so that \u201catypical\u201d behavior stands out more obviously. However, doing this requires ongoing development of personal profiles of users of a particular system, which in turn involves monitoring their behavior online. In cases of both attack and prevention of attacks, users\u2019 privacy risks are compromised.\nA related issues is that of control of data, which may be seen as an aspect of privacy[22][23] or additional to privacy concerns.[24][25] In either case, the control of data is a critical factor, as once an attack is successful, control is lost. The data may then be used for a variety of ends, not only relating to violations of privacy but also for political or other gain, as was the case with Cambridge Analytica[26], where the problem was not only privacy concerns, but also the control of users\u2019 data, which enabled discrete, targeted political advertising concerning the U.K.\u2019s referendum on membership of the European Union and the United States presidential election, both in 2016.[27]\nWhile the E.U. has sought to resolve concerns with privacy and control of data through the introduction of the General Data Protection Regulation[28], this has raised its own concerns. While European companies must follow strict regulations in developing SIS-related algorithms when it comes to accessing personal data, the same only applies to non-European companies when they practice in Europe. This leads to a concern of \u201cdata dumping, in which research is carried out in countries with lower barriers for use of personal data, rather than jump through bureaucratic hurdles in Europe. The result is that the data of non-European citizens is placed at higher risk than that of Europeans.\u201d[17]\nIncidental findings also fall under this category, as data derived from regular scans with the goal of profile-building can uncover new information about an individual which they did not want to reveal. Decisions should be made in advance on how to reveal that information and to whom it should be revealed; for example, the discovery that an employee is looking for another job.\n\nVulnerabilities and disclosure \nAn awareness or a duty to find vulnerabilities in a network which leave it open to an attack can help cybersecurity professionals understand the magnitude of a particular attack. However, disclosure of vulnerabilities to a particular authority, such as the company responsible, also risks the leak of that vulnerability from the responsible authority to communities of hackers so that that network or others may be exploited.[17] If vulnerabilities are made public, then the public visibility of a system and therefore its commercial viability may be threatened. For example, Wolter Pieters has pointed out the challenge of exposing vulnerabilities in e-voting systems: prior to an election and the systems will not be trusted; after an election and the election result will be called into question. However, if the vulnerability is not disclosed, then an attack may occur, which genuinely compromises the election. A related issue here is whether cybersecurity researchers looking at the techniques and practices of hackers should have a duty to expose vulnerabilities as an act of professional whistle-blowing. By rendering this a duty, there is less pressure on the professional to have to decide what is the right thing to do in a particular case, such as when competing financial interests may argue against such revelations.[29] As noted above, ethical issues arising from vulnerability disclosure are true of cybersecurity generally, whether involving SIS or not.\n\nCompetence of research ethics committees \nWithin universities and many research institutions, research ethics committees (RECs) or institutional review boards oversee applications for research to provide protection for research participants. However, RECs are often composed of experts in ethics who have limited awareness of cybersecurity practice, or computer scientists who lack ethical expertise. An example of this occurred when potentially harmful research was carried out on non-consenting individuals in totalitarian states which effectively tested the firewalls of those states.[16] While this research clearly put individuals at risk without their consent, at least two RECs determined that the research was not of relevance for ethical review because it did not concern human participants or personal data. It did, however, concern IP addresses, which could easily be linked to a person, putting that person at risk.[17] In the case of research using SIS, the potential for obscurity of the data could render the link with individuals more difficult to recognize still. Furthermore, it should be noted that these are concerns which arise in institutions with access to an REC. As pointed out by Macnish and van der Ham[17], many private companies do not have any ethical oversight facilities.\n\nSecurity issues \nGiven the aforementioned definition of security as the absence of threat to acquired values, the maintenance of good security is an ethical issue, as without it commonly held values may be compromised. \u201cInsufficient funding, poor oversight of systems, late or no installation of 'patches' (fixes to security flaws), how and where data are stored, how those data are accessed, and poor training of staff in security awareness\u201d[17] are therefore all instances of ethical concern.\n\nTrust and transparency \nTrust is an issue which connects the cybersecurity expert to the users who are being protected. Relating back to concerns regarding the risks inherent in publicizing vulnerabilities, there are pressing issues concerning transparency, such as \u201chow far to push transparency: should it extend to government agencies or even other companies? On one hand sharing information increases vulnerability as one\u2019s defenses are known, and one\u2019s experience of attacks shared, but on the other it is arguably only by pooling experience that an effective defense can be mounted.\u201d[17] \nPieters argues that trust in a person goes hand-in-hand with the explanation that a person gives. Artificial agents hence need to explain their decisions to the user, such as how security is maintained in online transactions.[18] He argues that there is a need for better understanding of the relationship between explanation and trust in AI and information security. Glass et al. concluded that trust depends on both the detail of explanations provided and on the transparency of the system.[30] From a cybersecurity perspective, what matters is how to communicate whether the system is secure, why it is secure, or how it is secure. In SIS, explanations are typically provided by the system itself, while in information security the explanations are provided by the designer.[31] Pieters argues that the role of explanations consists, at least in part, in acquiring and maintaining users\u2019 trust. He further exposes the concept of \u201cblack boxes\u201d which, together with trust and explanation, is a fundamental concept in cybersecurity, where the precise algorithm and associated decision-making techniques may become invisible within SIS systems.[18]\nFurthermore, through applying Bruno Latour's actor-network theory[32], Pieters highlights several issues with explanations and trust in information systems. He notes that explanations can be different depending on the actors who are explaining the system or technology. For example, a government seeking to protect the democratic credentials of an election, or a business with a commercial interest in keeping the source code secret, will have different explanations for an e-voting system.[18] In the same way, Pieter notes that delegation of technical aspects relating to the SIS will lead to a new actor who will not necessarily have the same abilities to explain the system as the designer.\nPieters also notes that explanations can have different goals, such as transparency versus justification. He argues[18]: \n\nExplanation-for-trust is explanation of how a system works, by revealing details of its internal operations. Explanation-for-confidence is explanation that makes the user feel comfortable in using the system, by providing information on its external communications. In explanation-for-trust, the black box of the system is opened; in explanation-for-confidence, it is not.\nIn the field of cybersecurity, as elsewhere in security, explanation of the security capabilities of the system to the user is an important requirement. \u201cThis is especially true because security is not instantly visible in using a system, as security of a system is not a functional requirement.\u201d[18] For example, it is not possible to infer that if a system gives good results then that system is secure. As Pieters warns, a criminal might have changed the results of voting without anyone noticing. Uncertainty is a feature within these systems, and given that security is often added to the system without being integral to it, it is feasible that the system can function without compromise being detected. The challenges of trust are exacerbated when the system operates using data analytics and potentially opaque algorithms that cannot be understood, still less challenged, by those affected.[33]\n\nRisk \nConsideration of who will decide what risks will be taken, what are the acceptable risks, and how risk is calculated[34][35] is important in cybersecurity. One of the arguments given for not requesting informed consent in the case described by Burnett and Feamster[16] regarding the non-consensual importing of malware onto users' computers to test firewalls was that, in the opinion of the researchers, there was only a limited risk of harm to the subject. However, it does not take much reflection to identify the risk to users who live in states where censorship is an issue, leading to potentially difficult situations.[36][17] Furthermore, it has been demonstrated that different groups of society tend to assess risk differently, with the acceptable risk threshold of white men being significantly higher than that of women or ethnic minorities.[37][38]\n\nResponsibility \nThe locus of responsibility for protecting against, and paying for protection against, cyber attacks is an ongoing issue.[39] It is not clear whether companies should be left to fend for themselves against hostile state-sponsored attacks, or whether governments should provide at least some financial support for them. Given the aforementioned potential to view cyber attacks as justification for declaring war, it is important to ask the degree to which the state should shoulder \u201cresponsibility for protecting its own economy on the internet as it does in physical space, by providing safe places to trade.\u201d[17] \nCybersecurity is usually taken to concern attacks from outside an entity rather than inside, for example using firewalls against incoming traffic.[40] Yet the development of technology allows for a global environment in which many businesses provide third parties access to their own networks, thus expanding the boundaries of what, or who, may be seen as \u201cinside.\u201d This extends to \u201cmobile devices [that] can access data from anywhere, and smart buildings [which] are being equipped with microchips that constantly communicate with each other.\u201d[40] Cleeff et al. refer to this as \u201cdeperimeterization,\u201d implying that not only is the border of the organization\u2019s IT blurred, but also that the accountability for that border is dispersed (a problem exacerbated in data analytics and AI where responsibility for decision-making is not always clear.[41] For example, \u201cif the organization makes a decision to apply a certain data protection policy in its software, the data may in fact be managed by a different organization. How will the organization that actually manages the data implement and verify this?\u201d[40]\n\nBusiness interests and codes of conduct \nCompeting interests are frequently perceived in security and profit. This may be seen as a zero-sum game in which any money spent on security is money which cannot be spent on increasing profit. However, this is clearly a flawed approach given the financial costs incurred in suffering a successful cyberattack. An example here is the decision of Marissa Meier, then CEO of Yahoo, not to inform the public of attacks in 2013 and 2014 regarding their accounts, most likely because such a revelation could have led to a loss in profit. Yet, when it became known, it devastated the company.[42] In response to similar concerns, Macnish and van der Ham argue for the necessity of guidance on disclosure of vulnerabilities, declaring \"public-spirited motivations should be protected from predatory practices by companies seeking to paper over cracks in their own security through legal action. However, current conventions as to how to proceed with disclosure of vulnerabilities seem to be skewed in the favor of corporations and against the interests of the public.\u201d[17] \nThey note that ethical problems cannot be solved easily, yet proposing the creation of a code of conduct for cybersecurity to provide guidance and a degree of consensus within the cybersecurity community regarding appropriate action in the face of attacks.\n\nThe case study of a cybersecurity company using SIS \nThe literature review demonstrates a variety of ethical issues in cybersecurity. In this section our goal is to present the ethical problems that arise in practice. We aim to compare practice with academic literature concerning ethical issues of SIS in cybersecurity. This will help to inform both sides if there is a lack of understanding of the problems, and to enable mutual learning.\nThis case study focuses on the ethical challenges that SIS bring in cybersecurity to shed some light on the risks of this sector and how they are currently minimized. The interview was conducted with four employees as a group at the headquarters of Company A in Scandinavia. All are experts in the Company A cybersecurity research team: Interviewee 1, a doctoral student; Interviewee 2, a researcher who focuses on core network security; Interviewee 3, a researcher who focuses on trusted computing; and Interviewee 4, a researcher with a background in machine learning (see Table 1, below). The methodology employed for the interview can be found in Understanding Ethics and Human Rights in Smart Information Systems: A Multi Case Study Approach by Macnish et al..[43]\n\n\n\n\n\n\n\nTable 1.\n\n\n\nDescription\n\nOrganization 1\n\n\nOrganization\n\nCompany A\n\n\nLocation\n\nScandinavia\n\n\nSector\n\nCybersecurity\/Telecommunications\n\n\nName\n\nInterviewee 1\u20134\n\n\nLength\n\n136 minutes\n\n\n\nDescription of SIS technologies being used in Company A \nBackground research was initially conducted through investigating Company A\u2019s website and public documents from conferences. This was then supplemented by the interviewees\u2019 explanations of the technical capabilities of the technologies used at Company A.\nCompany A is a global digital communications company. It is involved in cloud computing, artificial intelligence, machine learning, internet of things, and the infrastructure of mobile networks, including 5G. Company A\u2019s website refers to a combination of analytics and augmented intelligence, but the company also specializes in research and development (R&D) through Bell Labs, where it conducts research. Marcus Weldon, president of Bell Labs, in his book The Future X Network, shows the development of technology and the relation with global economy and society, by acknowledging the \u201cscale of changes wrought by a nexus of global, high-speed connectivity, billions of connected devices (IoT), cloud services, and non-stop data streaming, collection, and big data analytics.\u201d[44]\nThese technologies are changing our world, and Company A sees itself as driving innovation and the future of technology to power this digital age and transform how people live, work, and communicate. These technologies use data, including personal data from customers and metadata from phone networks. During the interview, Interviewee 1 argued that they do not use AI, but they do use statistics and analytics, such as products that use machine learning (ML) and data collection to identify malware. They also use analytics to create rules for developing effective firewalls for the network. However, Interviewee 3 noted that AI is still part of the research and the internal projects:\n\n[W]e do not sell a brain... or the giant quantum computing brain that solves all the problems, but for a very long time, planning has been used in many products, you can consider some configuration algorithms that can be considered as AI, these things exist, but not in the futuristic sense. (Interviewee 3, 2018)\nThe term \u201ccybersecurity\u201d appears in different articles across Company A\u2019s website. The cybersecurity research team at Company A developed a report on security for 5G networks which has served as guidance for the European Union. They analyze bulk datasets to help clients (communications providers rather than end users) maximize efficiency and thus profit, while at the same time providing security such as malware detection to protect the end user from attacks.\nSIS applications vary due to the amount and variety of data that Company A gathers from its customers, as well as the diverse needs of those customers. Many of these needs could not be met without SIS technology, as they would be impossible to perform by hand. For the most part, Company A\u2019s cybersecurity research team uses rule-based applications for sorting information, which is then evaluated by a person. Interestingly from an ethical point of view, Interviewee 1 pointed out that clients\u2019 data gathering capability has expanded faster than their data analysis capability, so that they increasingly gather data that has no obvious purpose.\n\nThe effectiveness of using SIS by Company A \nAs noted above, the use of AI and ML is due to the complexity and amount of data retrieved from clients\u2019 systems. According to Company A\u2019s website, cloud computing, AI, ML, IoT, and 5G Networks are changing the world, and they have the power to transform how we live, work, and communicate. Much of this is due to the fact that the operations now performed would previously have been impossible, owing to the sheer volume and complexity of the data.\nCompany A has been using SIS in cybersecurity for some time. SIS allows the team to discover attempted hacks or other misuse such as fraud, or the use of fake base stations (imitating a legitimate mobile phone tower in order to collect personal data). Current technology allows pre-filtering and sorting but is less effective at identifying or responding to targeted attacks, which are more sophisticated than bulk attacks. Interviewee 4 described a detection system they had worked on:\n\n[O]ne of their security teams was working on malware detection for telecom software for operators. That software ended up in systems that will protect end-users from malware that could be installed into phones. This is more at the operator level, not like an antivirus which is for a phone users-level. (Interviewee 4, 2018)\nEthical implications in cybersecurity \nIn this section we will look in greater depth at the ethical issues discussed during the interview conducted with the four employees at Company A. The issues which were uncovered in the interview widely reflect those found within the literature. It is, however, important to note that SIS use is growing rapidly: the technology is evolving and huge amounts of data are being collected. Generally, the interviewees explained that there is a lack of joint efforts from the ethical review boards within Company A, and there is a need to continue and improve the dialogue between the ethical and technical fields.\nThe ethical issues discussed in the interview comprised of privacy; internationalization, standardization, and legal aspects; monetization issues; anomalies; policy issues, awareness, and knowledge; security; risk assessment; and mechanisms to address ethical issues. Each of these will be discussed in greater depth in this section.\n\nPrivacy \nCompany A takes privacy seriously. Interviewee 2 pointed out that they were involved in drafting the document for 5G networks concerning privacy and the future of 5G security, which became a guideline for the European Parliament and for national legislatures. Privacy was seen during the interview as one of the most important underlying ethical issues. Concerns about users\u2019 and companies\u2019 privacy were evident. Some discussion was held around the issue of \u201cquantifying privacy\u201d (how does one measure privacy?). However, further problems arise in sharing data with customers, which to Company A are telecommunications providers rather than end users, as the team often does not know what the customer knows. Hence, data that may be anonymous in one dataset may be re-identified when cross-referenced with another dataset which is proprietary to the customer.\n\nSometimes if you manage to monetize your data, whatever data we\u2019re talking about, not just telco, and a buyer also has access to other sources of data that cross-correlate with your data, or have similar identifiers, you can never predict this as a seller of data. The end result is that your customer basically gets access to something that he can just map back to the original data, pretty much, by just looking at two fields and just cross-correlating. And you can never predict this. In that sense, it\u2019s already doomed from that point of view, but it\u2019s a best effort sort of thing, and within a narrow context it still works. (Interviewee 4, 2018)\nDifferential privacy, a technical \u201cfix\u201d for privacy concerns employed by Apple[45], among others, was also discussed. The team noted that differential privacy does not work with complete reliability because you can never be sure of what the data can lead to. Hence, uncertainty also becomes an important issue in relation to privacy. Furthermore, Interviewee 3 considered that we should have a numerical measurement for privacy; however, they suggested that would not be possible.\n\nInternationalization, standardization, and legal aspects \nGiven the global nature of telecommunications, international cloud computing, and the IoT, there is an increasing need for global regulation. Interviewee 4 introduced the problem of an application on mobile phones that sends data to China every five minutes. In such cases, the application needs to know which state\u2019s laws should be followed: those of the country where the user currently is, those of the state in which the user is registered as a citizen, those of the country where the operator is located, or those of the country of origin of the application operator (in this case, China). Interviewee 2 argued that one of the issues that they have encountered is that the customer data comes from everywhere in the world. As Company A is a global company, it works also in places such as the Middle East or Asia, and not only receives information from European customers but from other parts of the world. She raised the question as to whether it would be ethical to see data from everywhere in the world when there are no clear guidelines. Interviewee 3 also pointed out the issues with different regulations:\n\nNorthern Europe is doing well; Germany is most strict. Italy, Spain, [and] Portugal [are] strict. [Some others do not] really care. (Interviewee 3, 2018)\nInterviewee 2 explained that European laws are much stricter than most other nations, and in following the European laws, Company A restricts data sharing. It hence does not share data with third parties and has just one person looking at data unless there is a clear need for more. Interviewee 4 also pointed out that there is a Company A \u201csensitive data handling policy,\u201d which involves rules for data encryption and storage, which is closely monitored. Furthermore, special clearances are required to access some data, although the cybersecurity research team is in a \u201cprivileged\u201d position to receive such data. Interviewee 4 noted that some data is not allowed to be copied, just processed on the server.\nInterviewee 3 added that governments are also involved, and there is a need for standardized practices:\n\nIn telco, we have some interesting issues that are coming up. It\u2019s not just telco versus attacker. You have two other players. Standardization, where you try and make a level playing field for everyone. Then you\u2019ve got governments, [say] security services, who might say, \u201cWell, let\u2019s get rid of encryption, because bad guys use encryption. (Interviewee 3, 2018).\nInterviewee 3 explained that the spirit of GDPR is not about compliance but about risk management, and companies have to show that they are doing due diligence and minimizing the risks as much as possible. As an example of this, Interviewee 2 suggested that in order to review data, you can ask for one group of phones instead of having access to the whole network, which would compromise a large number of people. In contrast, Interviewee 3 argues that according to U.S. laws, the National Security Agency (NSA) are allowed to collect data of domestic individuals which they then send to the U.K. for analysis. There was also general agreement that what mattered was not just being compliant with the letter of the law, but also the spirit. The team noted that Finnish regulators in particular are not only concerned with compliance but also the motivations behind activities, and where the boundaries lie as to the limits of acceptable practice, which speaks of a high ethical standard.\n\nMonetization issues \nThe team felt that the existence of public clouds and data sharing with different companies such as Amazon increases the potential for monetization of data. Different stakeholders are looking to monetize data, which is very privacy-sensitive. Interviewee 1 argued that these new advances and technologies, such as targeted advertisements, are helping to monetize customer\u2019s data. Interviewee 4 added that some companies are seeking to monetize data within the current regulations, which is something that, according to Interviewee 4, must be questioned:\n\n[A]re we doing the best we can before we monetize it, selling it, whether using it for mining\u2014is anonymization and privacy worth it? Can we prove to certain knowledge, mathematically, that this is anonymized ... can we quantify that point? (Interviewee 4, 2018)\nHowever, the team agreed that not all operators have cybersecurity people, and not many people are working on telecommunications cybersecurity within operators. Thus, people that have expert knowledge are rare in this field. As Interviewee 2 pointed out, there are relatively few European security teams; companies such as KPN and Orange have one, but not every operator does.\nFurthermore, and related to the lack of security expertise, the team felt that there is a need to manage customers\u2019 expectations. Many customers place a high value on SIS even though they do not understand it or the level of security it can engender. Some customers \u201cwant perfect security right from the start\u201d said Interviewee 3. In addition, these expectations also hold true among some operators and senior managers who are guilty of \u201coff-loading perfect expectations to machines\u201d said Interviewee 1.\n\nAnomalies \nInterviewee 4 pointed out that in cybersecurity there is a need to search actively for anomalies. These have arisen for the team in the case of identifying fake base stations. Interestingly, Interviewee 4 mentioned that the U.S. has been trying to stop the news about these fake base stations because knowledge of their existence may damage the trust that people put in the networks:\n\n[I]n China you have fake antennas or fake base stations which can push advertisements etc. to people\u2019s phones, and there have been thousands in China ... In France, these fake base stations are used by the police to catch all the phones, not to do something malicious because is kind of the police enforcement, these are the so-called anomalies, when you have for a short period of time a phone for which service is delayed. (Interviewee 4, 2018)\nInterviewee 2 explained that they did not encounter many fake stations, but rather, they see attacks which seem to come from other network operators, e.g., a telecommunications provider in Barbados asking another telecommunications provider in Finland for the location of a Finnish subscriber, when there is no obvious technical need (such as to enable roaming). In such cases there is clearly no reason to give that information. Company A also makes use of firewalls to prevent attacks, but these need to be tailored to avoid creating false positives and blocking too much legitimate traffic.\n\nPolicy issues, awareness, and knowledge \nCompany A holds mandatory ethics training for all staff, which covers privacy compliance. However, Interviewee 3 suggested that it could be far more effective than is currently the case:\n\n[I]t appeals to the lowest common denominator for everyone, when it says things like \"you should apply privacy by design, you should use methods and processes.\" (Interviewee 3, 2018)\nHowever, Interviewee 2 offers a more positive perspective, arguing that it is making both companies and users aware of the problem: \n\n[A]t least the message gets through to every employee, that somehow we care, that you should think about that. (Interviewee 2, 2018)\nInterviewee 3 noted that customer data is strictly regulated at Company A, with codes of conduct and legal frameworks to guide behavior. The company\u2019s legal framework also provides a base from which to determine ethical decisions. Interviewee 4 explained that they had a data security course which was mandatory, and so there are serious attempts to deal with the ethical implications of the work. Moreover, Interviewee 3 argued that users should also have technical knowledge and the technical competence regarding practicing safe behavior online.\nThe team agreed that there is a need for more regulation. Interviewee 3 argued that privacy and data analytics should become regulated industries, similar to car management software, or software for medical devices, in which industries you have to keep the source code for 50 years, and it has to be documented and signed before it can be used. Interviewee 3 also mentioned that it is worth paying attention to the level of training for engineers regarding the need for an ethical background. Interviewee 3 explained that every engineer has to make ethical decisions at some point. As such, it is important that engineers are free to object and refuse to participate in certain projects. Interviewee 2 added that they have an ethics department in Company A that helps with these issues, providing support to employees who may have concerns. Furthermore, they stated that there is no code of conduct for cybersecurity.\n\nSecurity \nInterviewee 3 described how IT departments in some companies send internal \u201cphishing mails\u201d (emails attempting to trick the recipient into giving private information) to test their security, and the problem is that employees tend to have a high record on clicking on them, demonstrating a weak level of security awareness. Interviewee 3 also explained that Company A, among other companies, has a \u201chackathon\u201d every year to discover security flaws. Interviewee 4 mentioned that they have company-wide encryption policies for some sensitive materials, which is easy to use now, but that was not the case in the past. Interviewee 4 felt that security is of importance at Company A, but, as Interviewee 1 pointed out, most research is conducted internally, resulting in a lack of publications, at least for the public space. This leaves a number of unanswered questions:\n\n[W]ho is attacking your system and what are they after? This hasn't been researched properly, or has been researched but not [made] publicly available. (Interviewee 3, 2018)\nRisk assessment \nInterviewee 4 noted that there is a lack of risk assessment regarding some key aspects of security, such as the risk of not having security protocols, or the comparative risk of predictive versus reactive strategies. Interviewee 3 said that they had a PhD student currently studying cybersecurity attacks, and one of the things that came out of this research is that the attackers do not necessarily go for the weakest part of the system, because that is not where \u201cthe big game are.\u201d Therefore, this shows the need to have cybersecurity teams that will look for security pitfalls in every part of the system, even in the parts that are considered more secure by design.\nInterviewee 3 further stated that there is a problem in that the technology they work with can be misused, e.g., used for spying on different countries. Interviewee 2 continued that even if the government has access to this information, the question still remains as to the extent to which citizens can be sure that no one else has the same access. What if a government's position changes, such as that of Germany in the 1920s and \u201830s? There is very little that can be done under such circumstances.\n\nMechanisms to address ethical issues \nDuring the interview it was noted that there is a need for a culture of openness and challenge in organizations, and that the current paradigm of ethical standards in the use of SIS in cybersecurity is present but not developed. While the GDPR has improved general levels of awareness of cybersecurity and the importance of privacy, there is a need for ethical training for current engineers, as well as to develop stricter codes of conduct for this sector. The external regulations of, for example, targeted advertising and the issues of internationalization require consideration. Furthermore, while GDPR has a strong impact on privacy in Europe, other countries allow companies to gather data more freely.\nCompany A has a number of security strategies which go some way towards addressing ethical concerns. Mandatory training sessions are held annually and policy documents provide guidance. These are supplemented by a culture of challenge and openness in which employees feel free to share their concerns and step back from working on a project with which they have ethical concerns. There are also security measures put in place to keep sensitive data secure, such as limiting the machines on which the data can sit and operating a security clearance system such that only certain people are cleared to access the data.\nEngagement with different stakeholders, such as the internal Company A units, the academic community, regulators, as well as government agencies, clients. and end users was deemed both desirable and beneficial for all.\n\nConclusion \nThe literature review and the interview highlight a correlation between academic understanding of the ethical issues in cybersecurity and those working for the cybersecurity industry. However, both have also shown a lack of joint efforts from academia and engineering, as well as a need to improve the dialogue between the two. There is concern that the level of technical abstraction of university-based development stifles ethical oversight of the development of new SIS technologies in computer science. At the same time, there is a need to include ethical oversight in industry, with clearer codes of conduct for the cybersecurity community. One of the strongest arguments from the team at Company A was the lack of clear codes for international practice. As SIS technology is being developed with cloud computing, and the facility to acquire data from all over the world grows, so there is a need to improve ethical protocols for companies.\nOverall, it was shown that ethical concerns regarding SIS in cybersecurity go further than mere privacy issues. As it is a sector that will grow in the coming years, incorporating ML and the IoT, the importance of cybersecurity, and thereby the ethics of cybersecurity, will become more important.\nAmong the ethical issues we found the following: informed consent; protection from harm; disclosure of vulnerabilities; biases; the nature of hacking; trust; transparency; the necessity for a risk assessment in cybersecurity; and the responsibility between companies, governments, and users. Interestingly, the issue of monetization (how far can one ethically go to monetize customer\u2019s data) appeared in the interview, but the topic is not one that has been widely discussed in the academic literature (see Table 2, below).\n\n\n\n\n\n\n\nTable 2.\n\n\n\nIssues arising in literature review\n\nIssues arising in interview\n\n\nSimilarities\n\n\nProtection from harm\n\nProtection from harm\n\n\nPrivacy and control of data\n\nPrivacy and control of data\n\n\nCompetence of research ethics committees\n\nCompetence of research ethics committees\n\n\nSecurity issues\n\nSecurity issues\n\n\nRisk\n\nRisk assessment\n\n\nCodes of conduct\n\nPolicy issues (awareness and knowledge) and mechanisms to address ethical issues\n\n\nResponsibility\n\nInternationalization, standardization, and legal aspects\n\n\nDifferences\n\n\nVulnerabilities and disclosure\n\nAnomalies\n\n\nTrust and transparency\n\n\n\n\nInformed consent\n\n\n\n\n\nImplications of this report \nThis report exposes some of the weakest part of SIS technology and the importance of cybersecurity, by supporting the claim that there is a need to improve the ethics of research in SIS. The cyber world is forming an important part of society, and in some areas at least, albeit not among the interviewees for this case study, there is a lack of understanding of the ethical problems that come with this, which can bring damage to many stakeholders.\n\nFuture research \nThis report argues for the need for multi-disciplinary studies between academia and the technical community to prevent ethical concerns from being undervalued. Future research goes hand in hand with legal implications, particularly at the international level, as well the need to create clearer codes of conduct for businesses and international practices, and the necessity to increase the cybersecurity teams within companies. \n\nReferences \n\n\n\u2191 1.0 1.1 Singer, P.W.; Friedman, A. (2014). Cybersecurity and Cyberwar: What Everyone Needs to Know (1st ed.). Oxford University Press. ISBN 9780199918119. https:\/\/books.google.com\/books?id=9VDSAQAAQBAJ .   \n\n\u2191 Sobers, R. (18 May 2018). \"60 Must-Know Cybersecurity Statistics for 2018\". Varonis Blog. Archived from the original on 08 November 2018. https:\/\/web.archive.org\/web\/20181108122758\/https:\/\/www.varonis.com\/blog\/cybersecurity-statistics\/ . Retrieved 17 December 2018 .   \n\n\u2191 Smith, P.T. (2018). \"Cyberattacks as Casus Belli: A Sovereignty\u2010Based Account\". Journal of Applied Philosophy 35 (2): 222\u201341. doi:10.1111\/japp.12169.   \n\n\u2191 Wolters, A. (1952). \"\"National Security\" as an Ambiguous Symbol\". Political Science Quarterly 67 (4): 481\u2013502. doi:10.2307\/2145138.   \n\n\u2191 Baldwin, D.A. (1997). \"The Concept of Security\". Review of International Studies 23 (1): 5\u201326. https:\/\/www.cambridge.org\/core\/journals\/review-of-international-studies\/article\/concept-of-security\/67188B6038200A97C0B0A370FDC9D6B8 .   \n\n\u2191 Lundgren, B.; M\u00f6ller, N. (2019). \"Defining Information Security\". Science and Engineering Ethics 25 (2): 419\u201341. doi:10.1007\/s11948-017-9992-1.   \n\n\u2191 7.0 7.1 Craigen, D.; Diakun\u2014Thibault, N.; Purse, R. (2014). \"Defining Cybersecurity\". Technology Innovation Management Review 4 (10): 13\u201321. doi:10.22215\/timreview\/835.   \n\n\u2191 Hess, C.; Ostrom, E. (2006). Understanding Knowledge as a Commons: From Theory to Practice. MIT Press. ISBN 9780262083577.   \n\n\u2191 Bainbridge, L. (1983). \"Ironies of automation\". Automatica 19 (6): 775\u201379. doi:10.1016\/0005-1098(83)90046-8.   \n\n\u2191 Goddard, K.; Roudsari, A.; Wyatt, J.C. (2012). \"Automation bias: A systematic review of frequency, effect mediators, and mitigators\". JAMIA 19 (1): 121\u20137. doi:10.1136\/amiajnl-2011-000089. PMC PMC3240751. PMID 21685142. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3240751 .   \n\n\u2191 Tucker, E. (July 2018). \"Cyber security \u2013 why you\u2019re doing it all wrong\". Computer Weekly. https:\/\/www.computerweekly.com\/opinion\/Cyber-security-why-youre-doing-it-all-wrong . Retrieved 17 December 2018 .   \n\n\u2191 Macnish, K. (2012). \"Unblinking eyes: The ethics of automating surveillance\". Ethics and Information Technology 14 (2): 151\u201367. doi:10.1007\/s10676-012-9291-0.   \n\n\u2191 Johnson M.L.; Bellovin S.M.; Keromytis A.D. (2012). \"Computer Security Research with Human Subjects: Risks, Benefits and Informed Consent\". In Danezis G.; Dietrich S.; Sako K.. Financial Cryptography and Data Security. Springer. pp. 131\u201337. doi:10.1007\/978-3-642-29889-9_11. ISBN 9783642298899.   \n\n\u2191 14.0 14.1 Miller, F.; Wertheimer, A., ed. (2009). The Ethics of Consent. Oxford University Press. ISBN 9780195335149.   \n\n\u2191 Manson, N.C.; O'Neill, O. (2007). Rethinking Informed Consent in Bioethics. Cambridge University Press. doi:10.1017\/CBO9780511814600. ISBN 9780511814600.   \n\n\u2191 16.0 16.1 16.2 Burnett, S.; Feamster, N. (2015). \"Encore: Lightweight Measurement of Web Censorship with Cross-Origin Requests\". Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication: 653\u201367. doi:10.1145\/2785956.2787485.   \n\n\u2191 17.0 17.1 17.2 17.3 17.4 17.5 17.6 17.7 17.8 17.9 van der Ham, J. (14 September 2018). \"jeroenh\/Ethics-and-Cyber-Security\/template.tex\". GitHub. https:\/\/github.com\/jeroenh\/Ethics-and-Cyber-Security\/blob\/master\/template.tex .   \n\n\u2191 18.0 18.1 18.2 18.3 18.4 18.5 Pieters, W. (2011). \"Explanation and trust: what to tell the user in security and AI?\". Ethics and Information Technology 13 (1): 53\u201364. doi:10.1007\/s10676-010-9253-3.   \n\n\u2191 Nichols, S. (07 September 2016). \"St Jude sues short-selling MedSec over pacemaker 'hack' report\". The Register. https:\/\/www.theregister.co.uk\/2016\/09\/07\/st_jude_sues_over_hacking_claim\/ . Retrieved 04 July 2018 .   \n\n\u2191 Spring, T. (31 August 2016). \"Researchers: MedSec, Muddy Waters Set Bad Precedent With St. Jude Medical Short\". Threat Post. https:\/\/threatpost.com\/researchers-medsec-muddy-waters-set-bad-precedent-with-st-jude-medical-short\/120266\/ . Retrieved 04 July 2018 .   \n\n\u2191 Manjikian, M. (2017). Cybersecurity Ethics. Routledge. pp. 81\u2013112. ISBN 9781138717527.   \n\n\u2191 Moore, A.D. (2015). Privacy, Security and Accountability: Ethics, Law and Policy. Rowman and Littlefield. ISBN 9781783484768.   \n\n\u2191 Moore, A.D. (2003). \"Privacy: Its Meaning and Value\". American Philosophical Quarterly 40: 215\u201327. https:\/\/ssrn.com\/abstract=1980880 .   \n\n\u2191 Allen, A.L. (1999). \"Privacy-as-Data Control: Conceptual, Practical, and Moral Limits of the Paradigm\". Connecticut Law Review 32: 861\u201375. https:\/\/heinonline.org\/HOL\/LandingPage?collection=journals&handle=hein.journals\/conlr32&div=35 .   \n\n\u2191 Macnish, K. (2018). \"Government Surveillance and Why Defining Privacy Matters in a Post\u2010Snowden World\". Journal of Applied Philosophy 35 (2): 417\u201332. doi:10.1111\/japp.12219.   \n\n\u2191 Cadwalladr, C.; Graham-Harrison, E. (17 March 2018). \"Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in major data breach\". The Guardian. https:\/\/www.theguardian.com\/news\/2018\/mar\/17\/cambridge-analytica-facebook-influence-us-election .   \n\n\u2191 Ienca, M.; Vayena, E. (30 March 2018). \"Cambridge Analytica and Online Manipulation\". Scientific American. https:\/\/blogs.scientificamerican.com\/observations\/cambridge-analytica-and-online-manipulation\/ . Retrieved 10 July 2018 .   \n\n\u2191 Council of the European Union, European Parliament (27 April 2016). \"Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation) (Text with EEA relevance)\". European Union. https:\/\/publications.europa.eu\/en\/publication-detail\/-\/publication\/3e485e15-11bd-11e6-ba9a-01aa75ed71a1\/language-en .   \n\n\u2191 Davis, M. (1991). \"Thinking like an engineer: The place of a code of ethics in the practice of a profession\". Philosophy & Public Affairs 20 (2): 150\u201367. https:\/\/www.jstor.org\/stable\/2265293 .   \n\n\u2191 Glass, A.; McGuinness, D.L.; Wolverton, M. (2008). \"Toward establishing trust in adaptive agents\". Proceedings of the 13th International Conference on Intelligent User Interfaces: 227\u201336. doi:10.1145\/1378773.1378804.   \n\n\u2191 Bederson, B.B.; Lee, B.; Sherman, R.M. et al. (2003). \"Electronic voting system usability issues\". Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: 145\u201352. doi:10.1145\/642611.642638.   \n\n\u2191 Latour, B. (2005). Reassembling the Social: An Introduction to Actor-Network-Theory. Oxford University Press. ISBN 9780199256044.   \n\n\u2191 O'Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown. ISBN 9780553418811.   \n\n\u2191 Hansson, S. (2013). The Ethics of Risk: Ethical Analysis in an Uncertain World. Palgrave MacMillan. ISBN 9781137333650.   \n\n\u2191 Wolff, J. (2010). \"Five Types of Risky Situation\". Law, Innovation and Technology 2 (2): 151\u201363. doi:10.5235\/175799610794046177.   \n\n\u2191 Byers, J.W. (2015). \"Encore: Lightweight Measurement of Web Censorship with Cross-Origin Requests \u2013 Public Review\" (PDF). https:\/\/conferences.sigcomm.org\/sigcomm\/2015\/pdf\/reviews\/226pr.pdf .   \n\n\u2191 Hermansson, H. (2010). \"Towards a fair procedure for risk management\". Journal of Risk Research 13 (4): 501\u201315. doi:10.1080\/13669870903305903.   \n\n\u2191 Hermansson, H. (2005). \"Consistent risk management: Three models outlined\". Journal of Risk Research 8 (7\u20138): 557\u201368. doi:10.1080\/13669870500085189.   \n\n\u2191 Guiora, A.N. (2017). Cybersecurity: Geopolitics, Law, and Policy. Routledge. pp. 89\u2013111. ISBN 9781138033290.   \n\n\u2191 40.0 40.1 40.2 van Cleeff, A.; Pieters, W.; Wieringa, R.J. (2009). \"Security Implications of Virtualization: A Literature Study\". Proceedings from the 2009 International Conference on Computational Science and Engineering: 353-358. doi:10.1109\/CSE.2009.267.   \n\n\u2191 Sparrow, R. (2007). \"Killer Robots\". Journal of Applied Philosophy 24 (1): 62\u201377. doi:10.1111\/j.1468-5930.2007.00346.x.   \n\n\u2191 Stone, N. (07 April 2019). \"The Yahoo Cyber Attack & What should you learn from it?\". CashFloat Blog. https:\/\/www.cashfloat.co.uk\/blog\/technology-innovation\/yahoo-cyber-attack\/ .   \n\n\u2191 Macnish, K.; Ryan, M.; Stahl, B. (2019). \"Understanding Ethics and Human Rights in Smart Information Systems: A Multi Case Study Approach\". ORBIT Journal 2 (2): 102. doi:10.29297\/orbit.v2i1.102.   \n\n\u2191 Marko, K. (27 October 2015). \"The Omni-Connected World: Bell Labs Plans For Future Of Connected Everything\". Forbes. https:\/\/www.forbes.com\/sites\/kurtmarko\/2015\/10\/27\/omni-connected-world\/ .   \n\n\u2191 \"Our Approach to Privacy\". Apple, Inc. 2018. https:\/\/www.apple.com\/lae\/privacy\/approach-to-privacy\/ . Retrieved 17 December 2018 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation (e.g., British English to American English). In some cases important information was missing from the references, and that information was added. The 2018 article by Sobers on 60 must-know cybersecurity facts has been updated in 2019; an archived version from 2018 is used in this version. The Stone article on the Yahoo cyber attack also appears to have been updated in 2019, though no archived version of the article from 2017 exists. The Lundgren and M\u00f6ller citation has changed since the original article published online; this version represents the new information. The original cites an article by Macnish and van der Ham, but the research doesn't appear to be published yet; found a draft on GitHub to cite. The original has an inline citation for Marko 2015 but doesn't include it in the closing references; found the supposed reference online and included it here. Non-figured \"flavor\" images from the original were not included here.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on artificial intelligenceLIMSwiki journal articles on big dataLIMSwiki journal articles on cybersecurity\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 4 June 2019, at 17:58.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 175 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","c35e087aee1f0ed0d5c790bbd387ef29_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Smart_information_systems_in_cybersecurity_An_ethical_analysis skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Smart information systems in cybersecurity: An ethical analysis<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>This report provides an overview of the current implementation of smart information systems (SIS) in the field of <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a>. It also identifies the positive and negative aspects of using SIS in cybersecurity, including ethical issues which could arise while using SIS in this area. One company working in the industry of telecommunications (Company A) is analysed in this report. Further specific ethical issues that arise when using SIS technologies in Company A are critically evaluated. Finally, conclusions are drawn on the case study, and areas for improvement are suggested. \n<\/p><p><b>Keywords<\/b>: cybersecurity, ethics, smart information systems, big data\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Increasing numbers of items are becoming connected to the internet. Cisco\u2014a global leader in information technology, networking, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a>\u2014estimates that more than 8.7 billion devices were connected to the internet by the end of 2012, a number that will likely rise to over 40 billion in 2020.<sup id=\"rdp-ebb-cite_ref-SingerCyber14_1-0\" class=\"reference\"><a href=\"#cite_note-SingerCyber14-1\">[1]<\/a><\/sup> Cybersecurity has therefore become an important concern both publicly and privately. In the public sector, governments have created and enlarged cybersecurity divisions such as the U.S. Cyber Command and the Chinese \u201cInformation Security Base,\u201d whose mission is to provide security to critical national security assets.<sup id=\"rdp-ebb-cite_ref-SingerCyber14_1-1\" class=\"reference\"><a href=\"#cite_note-SingerCyber14-1\">[1]<\/a><\/sup>\n<\/p><p>In the private sphere, companies are struggling to keep up with the required need for security in the face of increasingly sophisticated attacks from a variety of sources. In 2017, there were \u201cover 130 large-scale, targeted breaches [by hackers of computer networks] in the U.S.,\u201d and \u201cbetween January 1, 2005 and April 18, 2018 there have been 8,854 recorded breaches.\u201d<sup id=\"rdp-ebb-cite_ref-Sobers60Must18Arch_2-0\" class=\"reference\"><a href=\"#cite_note-Sobers60Must18Arch-2\">[2]<\/a><\/sup> Furthermore, cyberattacks affect not only the online world, but also lead to vulnerabilities in the physical world, particularly when an attack threatens industries such as healthcare, communications, energy, or military networks, putting large swathes of society at risk. Indeed, it has been argued that some cyberattacks could constitute legitimate grounds for declarations of (physical) war.<sup id=\"rdp-ebb-cite_ref-SmithCyber18_3-0\" class=\"reference\"><a href=\"#cite_note-SmithCyber18-3\">[3]<\/a><\/sup>\n<\/p><p>Cybersecurity is therefore a complex and multi-disciplinary issue. Security has been defined in the international relations and security studies spheres both as \u201cthe absence of threats to acquired values\u201d<sup id=\"rdp-ebb-cite_ref-WolfersNational52_4-0\" class=\"reference\"><a href=\"#cite_note-WolfersNational52-4\">[4]<\/a><\/sup> and \u201cthe \u201cabsence of harm to acquired values.\u201d<sup id=\"rdp-ebb-cite_ref-BaldwinTheCon97_5-0\" class=\"reference\"><a href=\"#cite_note-BaldwinTheCon97-5\">[5]<\/a><\/sup> Within the profession, cybersecurity is more commonly defined in terms of confidentiality, integrity, and availability of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>.<sup id=\"rdp-ebb-cite_ref-LundgrenDefin19_6-0\" class=\"reference\"><a href=\"#cite_note-LundgrenDefin19-6\">[6]<\/a><\/sup> A 2014 literature review on the meanings attributed to cybersecurity has led to the broader definition of cybersecurity as \"the organization and collection of resources, processes, and structures used to protect cyberspace and cyberspace-enabled systems.\u201d<sup id=\"rdp-ebb-cite_ref-CraigenDefining14_7-0\" class=\"reference\"><a href=\"#cite_note-CraigenDefining14-7\">[7]<\/a><\/sup>\n<\/p><p>Cybersecurity therefore can be seen to encompass property rights of ownership of networks that could come under attack, as well as other concerns attributed with these, such as issues of access, extraction, contribution, removal, management, exclusion, and alienation.<sup id=\"rdp-ebb-cite_ref-HessUnder06_8-0\" class=\"reference\"><a href=\"#cite_note-HessUnder06-8\">[8]<\/a><\/sup> Hence cybersecurity fulfills a similar role to physical security in protecting property from some level of intrusion. Craigen <i>et al.<\/i> also argue that cybersecurity refers not only to a technical domain, but also that the values underlying that domain should be included in the description of cybersecurity.<sup id=\"rdp-ebb-cite_ref-CraigenDefining14_7-1\" class=\"reference\"><a href=\"#cite_note-CraigenDefining14-7\">[7]<\/a><\/sup> Seen this way, ethical issues and values form bedrock to cybersecurity research as identifying the values which cybersecurity seeks to protect.\n<\/p><p>The case study is divided into four main sections. The next two sections focus on the technical aspects of cybersecurity and a literature review of academic articles concerning ethical issues in cybersecurity, respectively. Then the practice of cybersecurity research is presented through an interview conducted with four employees at a major telecommunications software and hardware company, Company A. Finally, the last section critically evaluates ethical issues that have arisen in the use of SIS technologies in cybersecurity.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_use_of_smart_information_systems_in_cybersecurity\">The use of smart information systems in cybersecurity<\/span><\/h2>\n<p>The introduction of big data and <a href=\"https:\/\/www.limswiki.org\/index.php\/Artificial_intelligence\" title=\"Artificial intelligence\" class=\"wiki-link\" data-key=\"0c45a597361ca47e1cd8112af676276e\">artificial intelligence<\/a> (AI) (representations of smart information systems, or SIS) in cybersecurity is still in its early phase. Currently there is comparatively little work carried out on cybersecurity using SIS for several reasons. These include the remarkable diversity of cyberattacks (e.g., different approaches to hacking systems and introducing malware), the danger of false positives and false negatives, and the relatively low intelligence of existing SIS.\n<\/p><p>Taking these in turn, the diversity of attacks\u2014both in the source of the attack, the focus of the attack, and the motivation of the attack\u2014is significant. Attacks can be launched from outside an organization (e.g., from a hacking collective, such as Anonymous) or from an insider (e.g., a disaffected employee looking to damage a system). They may come from a single source, typically masked through using the darknet, or from a source who has engaged in a number of \u201chops\u201d (moving from one computer on a network to another, thus masking the original source) such that the originator could appear to be in a <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> or in a military base. If an attack were to appear to come from a military base, this might encourage the attacked party to \u201chack back.\u201d However, if the military base were an artificial screen presented in front of a hospital, the reverse hack could bring down that hospital\u2019s computer networks. The focus of the attack could be on imitating a user or system administrator (local IT expert) or on exploiting a security flaw in unpatched code (programming in a network that has a flaw which has not yet been fixed, also known as a zero-day exploit). The motivation of the attack can range from state security and intelligence gathering (e.g., U.S. Intelligence spying on Chinese military installations), to financial incentives through blackmail (e.g., encrypting a company\u2019s files and agreeing to decrypt them only when the company has paid the hacker a certain sum of money). This diversity means that it is extremely difficulty to develop a SIS that will effectively recognize an attack for what it is.\n<\/p><p>Secondly, the danger of false positives and false negatives is significant in light of the difficulty of recognizing an attack. If an attack is not recognized by a SIS as a false negative, it may be successful. This is particularly the case if security personnel have come to place undue trust in the automation and do not provide quality assurance of the SIS, a behavior known as \u201cautomation bias.\u201d<sup id=\"rdp-ebb-cite_ref-BainbridgeIronies83_9-0\" class=\"reference\"><a href=\"#cite_note-BainbridgeIronies83-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GoddardAuto12_10-0\" class=\"reference\"><a href=\"#cite_note-GoddardAuto12-10\">[10]<\/a><\/sup> By contrast, the SIS could be so cautious that it may lead to an excessive number of false positives in which a legitimate interaction is falsely labelled an attack and not permitted to continue. This leads to frustration and could entail the eventual disabling of the SIS.<sup id=\"rdp-ebb-cite_ref-TuckerCyber18_11-0\" class=\"reference\"><a href=\"#cite_note-TuckerCyber18-11\">[11]<\/a><\/sup>\n<\/p><p>Thirdly, and despite some hype in the media, SIS are still at a relatively unintelligent stage of development. Computer vision systems designed to identify people loitering, for example, recognize that a person has not left a circle with radius x in y number of seconds, but they cannot determine why the person is there or what their intent may be. As such, the inability to determine intentions from actions renders automated systems relatively impotent.\n<\/p><p>Despite these concerns, there are some potential grounds for use of SIS in cybersecurity. The most effective is in scanning systems for known attacks, or known abnormal patterns of behavior that have a very high likelihood of being an attack. When coupled with a human operator to scan any alerts and so determine whether to take action, the combined human-machine security system can prove to be effective, albeit still facing the above problems of automation bias and excessive false positives.<sup id=\"rdp-ebb-cite_ref-MacnishUnblink12_12-0\" class=\"reference\"><a href=\"#cite_note-MacnishUnblink12-12\">[12]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Literature_review:_Ethical_issues_of_using_SIS_in_cybersecurity\">Literature review: Ethical issues of using SIS in cybersecurity<\/span><\/h2>\n<p>In this section we will conduct a literature review of the most fundamental ethical issues in cybersecurity that are being proposed in the academic environment. Our goal is to compare them with the interview that has been conducted in a major telecommunications software and hardware company, Company A, in order to give an overview on the ethical issues in cybersecurity. \n<\/p><p>The literature review was carried out through a combination of online search using generic engines, such as Google and Google Scholar, and discipline-specific search engines on websites such as PhilPapers.org and <i>The Philosopher's Index<\/i>. Selected papers were then read and, where appropriate, the bibliographic references were used to locate further literature. Generic search on Google also provided links to trade publications and websites that were a further source of background information. \n<\/p><p>The ethical issues to arise from the literature review were informed consent, protection from harm, privacy and control of data, vulnerabilities and disclosure, competence of research ethics committees, security issues, trust and transparency, risk, responsibility, and business interests and codes of conduct.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Informed_consent\">Informed consent<\/span><\/h3>\n<p>Acquiring informed consent is an important activity for cybersecurity, and one that has been at the heart of research ethics and practice for decades.<sup id=\"rdp-ebb-cite_ref-JohnsonComp12_13-0\" class=\"reference\"><a href=\"#cite_note-JohnsonComp12-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MillerTheEthics09_14-0\" class=\"reference\"><a href=\"#cite_note-MillerTheEthics09-14\">[14]<\/a><\/sup> Consent is variously valued as the respect for autonomy<sup id=\"rdp-ebb-cite_ref-MillerTheEthics09_14-1\" class=\"reference\"><a href=\"#cite_note-MillerTheEthics09-14\">[14]<\/a><\/sup> or the minimization of harm.<sup id=\"rdp-ebb-cite_ref-MansonRethink07_15-0\" class=\"reference\"><a href=\"#cite_note-MansonRethink07-15\">[15]<\/a><\/sup> As such, the justification for informed consent is a considerable challenge for data analytics where anonymized data may be used without explicit consent of the person from whom it originates. This is also true within global cybersecurity, where a number of complicating issues arise, such as the complexity of informing users about detailed technical aspects in order to provide necessary information, as well as language barriers.<sup id=\"rdp-ebb-cite_ref-BurnettEncore15_16-0\" class=\"reference\"><a href=\"#cite_note-BurnettEncore15-16\">[16]<\/a><\/sup> This, though, is the case for many other areas of research such as medical or social sciences, and the scripts need not be different in cybersecurity.<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-0\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup>\n<\/p><p>Nonetheless, challenges of complexity, and of conveying that complexity in a manner that is sufficiently informative for a non-expert to make a decision, remain. Wolter Pieters notes that information provision does not correspond merely to the amount of information communicated, but to how it is presented, and that the type of information given is justified and appropriate. \u201cOne cannot speak about informed consent if one gives too little information, but one cannot speak about informed consent either if one gives too much. Indeed, giving too much information might lead to uninformed dissent, as distrust is invited by superfluous information.\u201d<sup id=\"rdp-ebb-cite_ref-PietersExplan11_18-0\" class=\"reference\"><a href=\"#cite_note-PietersExplan11-18\">[18]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Protection_from_harm\">Protection from harm<\/span><\/h3>\n<p>Cybersecurity has the potential to cause harm to its users, even when that harm is not intended. Concerns exist regarding the disclosure of vulnerabilities (such as a flaw in a security program which would allow for a hacker to break into the network with relative ease), for example, such as whether they should be disclosed publicly once a company has failed to address them. If not, then the vulnerability entails that a person may be at risk of attack, which is particularly concerning if the device at risk is <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_device\" title=\"Medical device\" class=\"wiki-link\" data-key=\"8e821122daa731f0fa8782fae57831fa\">medical in nature<\/a>, such as a <a href=\"https:\/\/www.limswiki.org\/index.php\/Pacemaker_failure\" title=\"Pacemaker failure\" class=\"wiki-link\" data-key=\"c3815549e959ce0ac4d82574f4d56a80\">pacemaker<\/a>.<sup id=\"rdp-ebb-cite_ref-NicholsStJude16_19-0\" class=\"reference\"><a href=\"#cite_note-NicholsStJude16-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SpringRes16_20-0\" class=\"reference\"><a href=\"#cite_note-SpringRes16-20\">[20]<\/a><\/sup> However, disclosure could bring the vulnerability to the awareness of potential attackers who had not considered it previously. This is true of cybersecurity generally, whether involving SIS or not.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Privacy_and_control_of_data\">Privacy and control of data<\/span><\/h3>\n<p>Privacy is a central issue in cybersecurity, as increasing amounts of personal data are gathered and stored in the cloud. Furthermore, these data can be highly sensitive, such as health or bank records.<sup id=\"rdp-ebb-cite_ref-ManjikianCyber17_21-0\" class=\"reference\"><a href=\"#cite_note-ManjikianCyber17-21\">[21]<\/a><\/sup> While the data at risk from attack is private, in order to identify an attack, particularly when SIS are involved, an effective cybersecurity system must maintain an awareness of \u201ctypical\u201d behavior so that \u201catypical\u201d behavior stands out more obviously. However, doing this requires ongoing development of personal profiles of users of a particular system, which in turn involves monitoring their behavior online. In cases of both attack and prevention of attacks, users\u2019 privacy risks are compromised.\n<\/p><p>A related issues is that of control of data, which may be seen as an aspect of privacy<sup id=\"rdp-ebb-cite_ref-MoorePriv15_22-0\" class=\"reference\"><a href=\"#cite_note-MoorePriv15-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MoorePrivacy12_23-0\" class=\"reference\"><a href=\"#cite_note-MoorePrivacy12-23\">[23]<\/a><\/sup> or additional to privacy concerns.<sup id=\"rdp-ebb-cite_ref-AllenPrivacy99_24-0\" class=\"reference\"><a href=\"#cite_note-AllenPrivacy99-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MacnishGov16_25-0\" class=\"reference\"><a href=\"#cite_note-MacnishGov16-25\">[25]<\/a><\/sup> In either case, the control of data is a critical factor, as once an attack is successful, control is lost. The data may then be used for a variety of ends, not only relating to violations of privacy but also for political or other gain, as was the case with Cambridge Analytica<sup id=\"rdp-ebb-cite_ref-CadwalladrRevealed18_26-0\" class=\"reference\"><a href=\"#cite_note-CadwalladrRevealed18-26\">[26]<\/a><\/sup>, where the problem was not only privacy concerns, but also the control of users\u2019 data, which enabled discrete, targeted political advertising concerning the U.K.\u2019s referendum on membership of the European Union and the United States presidential election, both in 2016.<sup id=\"rdp-ebb-cite_ref-IencaCambridge18_27-0\" class=\"reference\"><a href=\"#cite_note-IencaCambridge18-27\">[27]<\/a><\/sup>\n<\/p><p>While the E.U. has sought to resolve concerns with privacy and control of data through the introduction of the General Data Protection Regulation<sup id=\"rdp-ebb-cite_ref-EUReg16_28-0\" class=\"reference\"><a href=\"#cite_note-EUReg16-28\">[28]<\/a><\/sup>, this has raised its own concerns. While European companies must follow strict regulations in developing SIS-related algorithms when it comes to accessing personal data, the same only applies to non-European companies when they practice in Europe. This leads to a concern of \u201cdata dumping, in which research is carried out in countries with lower barriers for use of personal data, rather than jump through bureaucratic hurdles in Europe. The result is that the data of non-European citizens is placed at higher risk than that of Europeans.\u201d<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-1\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup>\n<\/p><p>Incidental findings also fall under this category, as data derived from regular scans with the goal of profile-building can uncover new information about an individual which they did not want to reveal. Decisions should be made in advance on how to reveal that information and to whom it should be revealed; for example, the discovery that an employee is looking for another job.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Vulnerabilities_and_disclosure\">Vulnerabilities and disclosure<\/span><\/h3>\n<p>An awareness or a duty to find vulnerabilities in a network which leave it open to an attack can help cybersecurity professionals understand the magnitude of a particular attack. However, disclosure of vulnerabilities to a particular authority, such as the company responsible, also risks the leak of that vulnerability from the responsible authority to communities of hackers so that that network or others may be exploited.<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-2\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup> If vulnerabilities are made public, then the public visibility of a system and therefore its commercial viability may be threatened. For example, Wolter Pieters has pointed out the challenge of exposing vulnerabilities in e-voting systems: prior to an election and the systems will not be trusted; after an election and the election result will be called into question. However, if the vulnerability is not disclosed, then an attack may occur, which genuinely compromises the election. A related issue here is whether cybersecurity researchers looking at the techniques and practices of hackers should have a duty to expose vulnerabilities as an act of professional whistle-blowing. By rendering this a duty, there is less pressure on the professional to have to decide what is the right thing to do in a particular case, such as when competing financial interests may argue against such revelations.<sup id=\"rdp-ebb-cite_ref-DavisThink91_29-0\" class=\"reference\"><a href=\"#cite_note-DavisThink91-29\">[29]<\/a><\/sup> As noted above, ethical issues arising from vulnerability disclosure are true of cybersecurity generally, whether involving SIS or not.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competence_of_research_ethics_committees\">Competence of research ethics committees<\/span><\/h3>\n<p>Within universities and many research institutions, research ethics committees (RECs) or institutional review boards oversee applications for research to provide protection for research participants. However, RECs are often composed of experts in ethics who have limited awareness of cybersecurity practice, or computer scientists who lack ethical expertise. An example of this occurred when potentially harmful research was carried out on non-consenting individuals in totalitarian states which effectively tested the firewalls of those states.<sup id=\"rdp-ebb-cite_ref-BurnettEncore15_16-1\" class=\"reference\"><a href=\"#cite_note-BurnettEncore15-16\">[16]<\/a><\/sup> While this research clearly put individuals at risk without their consent, at least two RECs determined that the research was not of relevance for ethical review because it did not concern human participants or personal data. It did, however, concern IP addresses, which could easily be linked to a person, putting that person at risk.<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-3\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup> In the case of research using SIS, the potential for obscurity of the data could render the link with individuals more difficult to recognize still. Furthermore, it should be noted that these are concerns which arise in institutions with access to an REC. As pointed out by Macnish and van der Ham<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-4\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup>, many private companies do not have any ethical oversight facilities.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Security_issues\">Security issues<\/span><\/h3>\n<p>Given the aforementioned definition of security as the absence of threat to acquired values, the maintenance of good security is an ethical issue, as without it commonly held values may be compromised. \u201cInsufficient funding, poor oversight of systems, late or no installation of 'patches' (fixes to security flaws), how and where data are stored, how those data are accessed, and poor training of staff in security awareness\u201d<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-5\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup> are therefore all instances of ethical concern.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Trust_and_transparency\">Trust and transparency<\/span><\/h3>\n<p>Trust is an issue which connects the cybersecurity expert to the users who are being protected. Relating back to concerns regarding the risks inherent in publicizing vulnerabilities, there are pressing issues concerning transparency, such as \u201chow far to push transparency: should it extend to government agencies or even other companies? On one hand sharing information increases vulnerability as one\u2019s defenses are known, and one\u2019s experience of attacks shared, but on the other it is arguably only by pooling experience that an effective defense can be mounted.\u201d<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-6\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup> \n<\/p><p>Pieters argues that trust in a person goes hand-in-hand with the explanation that a person gives. Artificial agents hence need to explain their decisions to the user, such as how security is maintained in online transactions.<sup id=\"rdp-ebb-cite_ref-PietersExplan11_18-1\" class=\"reference\"><a href=\"#cite_note-PietersExplan11-18\">[18]<\/a><\/sup> He argues that there is a need for better understanding of the relationship between explanation and trust in AI and information security. Glass <i>et al.<\/i> concluded that trust depends on both the detail of explanations provided and on the transparency of the system.<sup id=\"rdp-ebb-cite_ref-GlassToward08_30-0\" class=\"reference\"><a href=\"#cite_note-GlassToward08-30\">[30]<\/a><\/sup> From a cybersecurity perspective, what matters is how to communicate whether the system is secure, why it is secure, or how it is secure. In SIS, explanations are typically provided by the system itself, while in information security the explanations are provided by the designer.<sup id=\"rdp-ebb-cite_ref-BedersonElect03_31-0\" class=\"reference\"><a href=\"#cite_note-BedersonElect03-31\">[31]<\/a><\/sup> Pieters argues that the role of explanations consists, at least in part, in acquiring and maintaining users\u2019 trust. He further exposes the concept of \u201cblack boxes\u201d which, together with trust and explanation, is a fundamental concept in cybersecurity, where the precise algorithm and associated decision-making techniques may become invisible within SIS systems.<sup id=\"rdp-ebb-cite_ref-PietersExplan11_18-2\" class=\"reference\"><a href=\"#cite_note-PietersExplan11-18\">[18]<\/a><\/sup>\n<\/p><p>Furthermore, through applying Bruno Latour's actor-network theory<sup id=\"rdp-ebb-cite_ref-LatourReass05_32-0\" class=\"reference\"><a href=\"#cite_note-LatourReass05-32\">[32]<\/a><\/sup>, Pieters highlights several issues with explanations and trust in information systems. He notes that explanations can be different depending on the actors who are explaining the system or technology. For example, a government seeking to protect the democratic credentials of an election, or a business with a commercial interest in keeping the source code secret, will have different explanations for an e-voting system.<sup id=\"rdp-ebb-cite_ref-PietersExplan11_18-3\" class=\"reference\"><a href=\"#cite_note-PietersExplan11-18\">[18]<\/a><\/sup> In the same way, Pieter notes that delegation of technical aspects relating to the SIS will lead to a new actor who will not necessarily have the same abilities to explain the system as the designer.\n<\/p><p>Pieters also notes that explanations can have different goals, such as transparency versus justification. He argues<sup id=\"rdp-ebb-cite_ref-PietersExplan11_18-4\" class=\"reference\"><a href=\"#cite_note-PietersExplan11-18\">[18]<\/a><\/sup>: \n<\/p>\n<blockquote>Explanation-for-trust is explanation of how a system works, by revealing details of its internal operations. Explanation-for-confidence is explanation that makes the user feel comfortable in using the system, by providing information on its external communications. In explanation-for-trust, the black box of the system is opened; in explanation-for-confidence, it is not.<\/blockquote>\n<p>In the field of cybersecurity, as elsewhere in security, explanation of the security capabilities of the system to the user is an important requirement. \u201cThis is especially true because security is not instantly visible in using a system, as security of a system is not a functional requirement.\u201d<sup id=\"rdp-ebb-cite_ref-PietersExplan11_18-5\" class=\"reference\"><a href=\"#cite_note-PietersExplan11-18\">[18]<\/a><\/sup> For example, it is not possible to infer that if a system gives good results then that system is secure. As Pieters warns, a criminal might have changed the results of voting without anyone noticing. Uncertainty is a feature within these systems, and given that security is often added to the system without being integral to it, it is feasible that the system can function without compromise being detected. The challenges of trust are exacerbated when the system operates using data analytics and potentially opaque algorithms that cannot be understood, still less challenged, by those affected.<sup id=\"rdp-ebb-cite_ref-ONeilWeapons16_33-0\" class=\"reference\"><a href=\"#cite_note-ONeilWeapons16-33\">[33]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Risk\">Risk<\/span><\/h3>\n<p>Consideration of who will decide what risks will be taken, what are the acceptable risks, and how risk is calculated<sup id=\"rdp-ebb-cite_ref-HanssonTheEthics13_34-0\" class=\"reference\"><a href=\"#cite_note-HanssonTheEthics13-34\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WolffFive10_35-0\" class=\"reference\"><a href=\"#cite_note-WolffFive10-35\">[35]<\/a><\/sup> is important in cybersecurity. One of the arguments given for not requesting informed consent in the case described by Burnett and Feamster<sup id=\"rdp-ebb-cite_ref-BurnettEncore15_16-2\" class=\"reference\"><a href=\"#cite_note-BurnettEncore15-16\">[16]<\/a><\/sup> regarding the non-consensual importing of malware onto users' computers to test firewalls was that, in the opinion of the researchers, there was only a limited risk of harm to the subject. However, it does not take much reflection to identify the risk to users who live in states where censorship is an issue, leading to potentially difficult situations.<sup id=\"rdp-ebb-cite_ref-ByersEncore15_36-0\" class=\"reference\"><a href=\"#cite_note-ByersEncore15-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-7\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup> Furthermore, it has been demonstrated that different groups of society tend to assess risk differently, with the acceptable risk threshold of white men being significantly higher than that of women or ethnic minorities.<sup id=\"rdp-ebb-cite_ref-HermanssonTowards10_37-0\" class=\"reference\"><a href=\"#cite_note-HermanssonTowards10-37\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HermanssonConsist05_38-0\" class=\"reference\"><a href=\"#cite_note-HermanssonConsist05-38\">[38]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Responsibility\">Responsibility<\/span><\/h3>\n<p>The locus of responsibility for protecting against, and paying for protection against, cyber attacks is an ongoing issue.<sup id=\"rdp-ebb-cite_ref-GuioraCyber17_39-0\" class=\"reference\"><a href=\"#cite_note-GuioraCyber17-39\">[39]<\/a><\/sup> It is not clear whether companies should be left to fend for themselves against hostile state-sponsored attacks, or whether governments should provide at least some financial support for them. Given the aforementioned potential to view cyber attacks as justification for declaring war, it is important to ask the degree to which the state should shoulder \u201cresponsibility for protecting its own economy on the internet as it does in physical space, by providing safe places to trade.\u201d<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-8\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup> \n<\/p><p>Cybersecurity is usually taken to concern attacks from outside an entity rather than inside, for example using firewalls against incoming traffic.<sup id=\"rdp-ebb-cite_ref-vanCleeffSecurity09_40-0\" class=\"reference\"><a href=\"#cite_note-vanCleeffSecurity09-40\">[40]<\/a><\/sup> Yet the development of technology allows for a global environment in which many businesses provide third parties access to their own networks, thus expanding the boundaries of what, or who, may be seen as \u201cinside.\u201d This extends to \u201cmobile devices [that] can access data from anywhere, and smart buildings [which] are being equipped with microchips that constantly communicate with each other.\u201d<sup id=\"rdp-ebb-cite_ref-vanCleeffSecurity09_40-1\" class=\"reference\"><a href=\"#cite_note-vanCleeffSecurity09-40\">[40]<\/a><\/sup> Cleeff <i>et al.<\/i> refer to this as \u201cdeperimeterization,\u201d implying that not only is the border of the organization\u2019s IT blurred, but also that the accountability for that border is dispersed (a problem exacerbated in data analytics and AI where responsibility for decision-making is not always clear.<sup id=\"rdp-ebb-cite_ref-SparrowKiller07_41-0\" class=\"reference\"><a href=\"#cite_note-SparrowKiller07-41\">[41]<\/a><\/sup> For example, \u201cif the organization makes a decision to apply a certain data protection policy in its software, the data may in fact be managed by a different organization. How will the organization that actually manages the data implement and verify this?\u201d<sup id=\"rdp-ebb-cite_ref-vanCleeffSecurity09_40-2\" class=\"reference\"><a href=\"#cite_note-vanCleeffSecurity09-40\">[40]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Business_interests_and_codes_of_conduct\">Business interests and codes of conduct<\/span><\/h3>\n<p>Competing interests are frequently perceived in security and profit. This may be seen as a zero-sum game in which any money spent on security is money which cannot be spent on increasing profit. However, this is clearly a flawed approach given the financial costs incurred in suffering a successful cyberattack. An example here is the decision of Marissa Meier, then CEO of Yahoo, not to inform the public of attacks in 2013 and 2014 regarding their accounts, most likely because such a revelation could have led to a loss in profit. Yet, when it became known, it devastated the company.<sup id=\"rdp-ebb-cite_ref-StoneTheYahoo19_42-0\" class=\"reference\"><a href=\"#cite_note-StoneTheYahoo19-42\">[42]<\/a><\/sup> In response to similar concerns, Macnish and van der Ham argue for the necessity of guidance on disclosure of vulnerabilities, declaring \"public-spirited motivations should be protected from predatory practices by companies seeking to paper over cracks in their own security through legal action. However, current conventions as to how to proceed with disclosure of vulnerabilities seem to be skewed in the favor of corporations and against the interests of the public.\u201d<sup id=\"rdp-ebb-cite_ref-MacnishEthics19_17-9\" class=\"reference\"><a href=\"#cite_note-MacnishEthics19-17\">[17]<\/a><\/sup> \n<\/p><p>They note that ethical problems cannot be solved easily, yet proposing the creation of a code of conduct for cybersecurity to provide guidance and a degree of consensus within the cybersecurity community regarding appropriate action in the face of attacks.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_case_study_of_a_cybersecurity_company_using_SIS\">The case study of a cybersecurity company using SIS<\/span><\/h2>\n<p>The literature review demonstrates a variety of ethical issues in cybersecurity. In this section our goal is to present the ethical problems that arise in practice. We aim to compare practice with academic literature concerning ethical issues of SIS in cybersecurity. This will help to inform both sides if there is a lack of understanding of the problems, and to enable mutual learning.\n<\/p><p>This case study focuses on the ethical challenges that SIS bring in cybersecurity to shed some light on the risks of this sector and how they are currently minimized. The interview was conducted with four employees as a group at the headquarters of Company A in Scandinavia. All are experts in the Company A cybersecurity research team: Interviewee 1, a doctoral student; Interviewee 2, a researcher who focuses on core network security; Interviewee 3, a researcher who focuses on trusted computing; and Interviewee 4, a researcher with a background in machine learning (see Table 1, below). The methodology employed for the interview can be found in <i>Understanding Ethics and Human Rights in Smart Information Systems: A Multi Case Study Approach<\/i> by Macnish <i>et al.<\/i>.<sup id=\"rdp-ebb-cite_ref-MacnishUnder19_43-0\" class=\"reference\"><a href=\"#cite_note-MacnishUnder19-43\">[43]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1.<\/b>\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Description\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Organization 1\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Organization\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Company A\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Location\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Scandinavia\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sector\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cybersecurity\/Telecommunications\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Name\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Interviewee 1\u20134\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Length\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">136 minutes\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Description_of_SIS_technologies_being_used_in_Company_A\">Description of SIS technologies being used in Company A<\/span><\/h3>\n<p>Background research was initially conducted through investigating Company A\u2019s website and public documents from conferences. This was then supplemented by the interviewees\u2019 explanations of the technical capabilities of the technologies used at Company A.\n<\/p><p>Company A is a global digital communications company. It is involved in <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud computing<\/a>, artificial intelligence, machine learning, <a href=\"https:\/\/www.limswiki.org\/index.php\/Internet_of_things\" title=\"Internet of things\" class=\"wiki-link\" data-key=\"13e0b826fa1770fe4bea72e3cb942f0f\">internet of things<\/a>, and the infrastructure of mobile networks, including 5G. Company A\u2019s website refers to a combination of analytics and augmented intelligence, but the company also specializes in research and development (R&D) through Bell Labs, where it conducts research. Marcus Weldon, president of Bell Labs, in his book <i>The Future X Network<\/i>, shows the development of technology and the relation with global economy and society, by acknowledging the \u201cscale of changes wrought by a nexus of global, high-speed connectivity, billions of connected devices (IoT), cloud services, and non-stop data streaming, collection, and big data analytics.\u201d<sup id=\"rdp-ebb-cite_ref-MarkoTheOmni15_44-0\" class=\"reference\"><a href=\"#cite_note-MarkoTheOmni15-44\">[44]<\/a><\/sup>\n<\/p><p>These technologies are changing our world, and Company A sees itself as driving innovation and the future of technology to power this digital age and transform how people live, work, and communicate. These technologies use data, including personal data from customers and metadata from phone networks. During the interview, Interviewee 1 argued that they do not use AI, but they do use statistics and analytics, such as products that use machine learning (ML) and data collection to identify malware. They also use analytics to create rules for developing effective firewalls for the network. However, Interviewee 3 noted that AI is still part of the research and the internal projects:\n<\/p>\n<blockquote>[W]e do not sell a brain... or the giant quantum computing brain that solves all the problems, but for a very long time, planning has been used in many products, you can consider some configuration algorithms that can be considered as AI, these things exist, but not in the futuristic sense. (Interviewee 3, 2018)<\/blockquote>\n<p>The term \u201ccybersecurity\u201d appears in different articles across Company A\u2019s website. The cybersecurity research team at Company A developed a report on security for 5G networks which has served as guidance for the European Union. They analyze bulk datasets to help clients (communications providers rather than end users) maximize efficiency and thus profit, while at the same time providing security such as malware detection to protect the end user from attacks.\n<\/p><p>SIS applications vary due to the amount and variety of data that Company A gathers from its customers, as well as the diverse needs of those customers. Many of these needs could not be met without SIS technology, as they would be impossible to perform by hand. For the most part, Company A\u2019s cybersecurity research team uses rule-based applications for sorting information, which is then evaluated by a person. Interestingly from an ethical point of view, Interviewee 1 pointed out that clients\u2019 data gathering capability has expanded faster than their data analysis capability, so that they increasingly gather data that has no obvious purpose.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_effectiveness_of_using_SIS_by_Company_A\">The effectiveness of using SIS by Company A<\/span><\/h3>\n<p>As noted above, the use of AI and ML is due to the complexity and amount of data retrieved from clients\u2019 systems. According to Company A\u2019s website, cloud computing, AI, ML, IoT, and 5G Networks are changing the world, and they have the power to transform how we live, work, and communicate. Much of this is due to the fact that the operations now performed would previously have been impossible, owing to the sheer volume and complexity of the data.\n<\/p><p>Company A has been using SIS in cybersecurity for some time. SIS allows the team to discover attempted hacks or other misuse such as fraud, or the use of fake base stations (imitating a legitimate mobile phone tower in order to collect personal data). Current technology allows pre-filtering and sorting but is less effective at identifying or responding to targeted attacks, which are more sophisticated than bulk attacks. Interviewee 4 described a detection system they had worked on:\n<\/p>\n<blockquote>[O]ne of their security teams was working on malware detection for telecom software for operators. That software ended up in systems that will protect end-users from malware that could be installed into phones. This is more at the operator level, not like an antivirus which is for a phone users-level. (Interviewee 4, 2018)<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Ethical_implications_in_cybersecurity\">Ethical implications in cybersecurity<\/span><\/h2>\n<p>In this section we will look in greater depth at the ethical issues discussed during the interview conducted with the four employees at Company A. The issues which were uncovered in the interview widely reflect those found within the literature. It is, however, important to note that SIS use is growing rapidly: the technology is evolving and huge amounts of data are being collected. Generally, the interviewees explained that there is a lack of joint efforts from the ethical review boards within Company A, and there is a need to continue and improve the dialogue between the ethical and technical fields.\n<\/p><p>The ethical issues discussed in the interview comprised of privacy; internationalization, standardization, and legal aspects; monetization issues; anomalies; policy issues, awareness, and knowledge; security; risk assessment; and mechanisms to address ethical issues. Each of these will be discussed in greater depth in this section.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Privacy\">Privacy<\/span><\/h3>\n<p>Company A takes privacy seriously. Interviewee 2 pointed out that they were involved in drafting the document for 5G networks concerning privacy and the future of 5G security, which became a guideline for the European Parliament and for national legislatures. Privacy was seen during the interview as one of the most important underlying ethical issues. Concerns about users\u2019 and companies\u2019 privacy were evident. Some discussion was held around the issue of \u201cquantifying privacy\u201d (how does one measure privacy?). However, further problems arise in sharing data with customers, which to Company A are telecommunications providers rather than end users, as the team often does not know what the customer knows. Hence, data that may be anonymous in one dataset may be re-identified when cross-referenced with another dataset which is proprietary to the customer.\n<\/p>\n<blockquote>Sometimes if you manage to monetize your data, whatever data we\u2019re talking about, not just telco, and a buyer also has access to other sources of data that cross-correlate with your data, or have similar identifiers, you can never predict this as a seller of data. The end result is that your customer basically gets access to something that he can just map back to the original data, pretty much, by just looking at two fields and just cross-correlating. And you can never predict this. In that sense, it\u2019s already doomed from that point of view, but it\u2019s a best effort sort of thing, and within a narrow context it still works. (Interviewee 4, 2018)<\/blockquote>\n<p>Differential privacy, a technical \u201cfix\u201d for privacy concerns employed by Apple<sup id=\"rdp-ebb-cite_ref-ApplePrivacy_45-0\" class=\"reference\"><a href=\"#cite_note-ApplePrivacy-45\">[45]<\/a><\/sup>, among others, was also discussed. The team noted that differential privacy does not work with complete reliability because you can never be sure of what the data can lead to. Hence, uncertainty also becomes an important issue in relation to privacy. Furthermore, Interviewee 3 considered that we should have a numerical measurement for privacy; however, they suggested that would not be possible.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Internationalization.2C_standardization.2C_and_legal_aspects\">Internationalization, standardization, and legal aspects<\/span><\/h3>\n<p>Given the global nature of telecommunications, international cloud computing, and the IoT, there is an increasing need for global regulation. Interviewee 4 introduced the problem of an application on mobile phones that sends data to China every five minutes. In such cases, the application needs to know which state\u2019s laws should be followed: those of the country where the user currently is, those of the state in which the user is registered as a citizen, those of the country where the operator is located, or those of the country of origin of the application operator (in this case, China). Interviewee 2 argued that one of the issues that they have encountered is that the customer data comes from everywhere in the world. As Company A is a global company, it works also in places such as the Middle East or Asia, and not only receives information from European customers but from other parts of the world. She raised the question as to whether it would be ethical to see data from everywhere in the world when there are no clear guidelines. Interviewee 3 also pointed out the issues with different regulations:\n<\/p>\n<blockquote>Northern Europe is doing well; Germany is most strict. Italy, Spain, [and] Portugal [are] strict. [Some others do not] really care. (Interviewee 3, 2018)<\/blockquote>\n<p>Interviewee 2 explained that European laws are much stricter than most other nations, and in following the European laws, Company A restricts data sharing. It hence does not share data with third parties and has just one person looking at data unless there is a clear need for more. Interviewee 4 also pointed out that there is a Company A \u201csensitive data handling policy,\u201d which involves rules for data encryption and storage, which is closely monitored. Furthermore, special clearances are required to access some data, although the cybersecurity research team is in a \u201cprivileged\u201d position to receive such data. Interviewee 4 noted that some data is not allowed to be copied, just processed on the server.\n<\/p><p>Interviewee 3 added that governments are also involved, and there is a need for standardized practices:\n<\/p>\n<blockquote>In telco, we have some interesting issues that are coming up. It\u2019s not just telco versus attacker. You have two other players. Standardization, where you try and make a level playing field for everyone. Then you\u2019ve got governments, [say] security services, who might say, \u201cWell, let\u2019s get rid of encryption, because bad guys use encryption. (Interviewee 3, 2018).<\/blockquote>\n<p>Interviewee 3 explained that the spirit of GDPR is not about compliance but about risk management, and companies have to show that they are doing due diligence and minimizing the risks as much as possible. As an example of this, Interviewee 2 suggested that in order to review data, you can ask for one group of phones instead of having access to the whole network, which would compromise a large number of people. In contrast, Interviewee 3 argues that according to U.S. laws, the National Security Agency (NSA) are allowed to collect data of domestic individuals which they then send to the U.K. for analysis. There was also general agreement that what mattered was not just being compliant with the letter of the law, but also the spirit. The team noted that Finnish regulators in particular are not only concerned with compliance but also the motivations behind activities, and where the boundaries lie as to the limits of acceptable practice, which speaks of a high ethical standard.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Monetization_issues\">Monetization issues<\/span><\/h3>\n<p>The team felt that the existence of public clouds and data sharing with different companies such as Amazon increases the potential for monetization of data. Different stakeholders are looking to monetize data, which is very privacy-sensitive. Interviewee 1 argued that these new advances and technologies, such as targeted advertisements, are helping to monetize customer\u2019s data. Interviewee 4 added that some companies are seeking to monetize data within the current regulations, which is something that, according to Interviewee 4, must be questioned:\n<\/p>\n<blockquote>[A]re we doing the best we can before we monetize it, selling it, whether using it for mining\u2014is anonymization and privacy worth it? Can we prove to certain knowledge, mathematically, that this is anonymized ... can we quantify that point? (Interviewee 4, 2018)<\/blockquote>\n<p>However, the team agreed that not all operators have cybersecurity people, and not many people are working on telecommunications cybersecurity within operators. Thus, people that have expert knowledge are rare in this field. As Interviewee 2 pointed out, there are relatively few European security teams; companies such as KPN and Orange have one, but not every operator does.\n<\/p><p>Furthermore, and related to the lack of security expertise, the team felt that there is a need to manage customers\u2019 expectations. Many customers place a high value on SIS even though they do not understand it or the level of security it can engender. Some customers \u201cwant perfect security right from the start\u201d said Interviewee 3. In addition, these expectations also hold true among some operators and senior managers who are guilty of \u201coff-loading perfect expectations to machines\u201d said Interviewee 1.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Anomalies\">Anomalies<\/span><\/h3>\n<p>Interviewee 4 pointed out that in cybersecurity there is a need to search actively for anomalies. These have arisen for the team in the case of identifying fake base stations. Interestingly, Interviewee 4 mentioned that the U.S. has been trying to stop the news about these fake base stations because knowledge of their existence may damage the trust that people put in the networks:\n<\/p>\n<blockquote>[I]n China you have fake antennas or fake base stations which can push advertisements etc. to people\u2019s phones, and there have been thousands in China ... In France, these fake base stations are used by the police to catch all the phones, not to do something malicious because is kind of the police enforcement, these are the so-called anomalies, when you have for a short period of time a phone for which service is delayed. (Interviewee 4, 2018)<\/blockquote>\n<p>Interviewee 2 explained that they did not encounter many fake stations, but rather, they see attacks which seem to come from other network operators, e.g., a telecommunications provider in Barbados asking another telecommunications provider in Finland for the location of a Finnish subscriber, when there is no obvious technical need (such as to enable roaming). In such cases there is clearly no reason to give that information. Company A also makes use of firewalls to prevent attacks, but these need to be tailored to avoid creating false positives and blocking too much legitimate traffic.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Policy_issues.2C_awareness.2C_and_knowledge\">Policy issues, awareness, and knowledge<\/span><\/h3>\n<p>Company A holds mandatory ethics training for all staff, which covers privacy compliance. However, Interviewee 3 suggested that it could be far more effective than is currently the case:\n<\/p>\n<blockquote>[I]t appeals to the lowest common denominator for everyone, when it says things like \"you should apply privacy by design, you should use methods and processes.\" (Interviewee 3, 2018)<\/blockquote>\n<p>However, Interviewee 2 offers a more positive perspective, arguing that it is making both companies and users aware of the problem: \n<\/p>\n<blockquote>[A]t least the message gets through to every employee, that somehow we care, that you should think about that. (Interviewee 2, 2018)<\/blockquote>\n<p>Interviewee 3 noted that customer data is strictly regulated at Company A, with codes of conduct and legal frameworks to guide behavior. The company\u2019s legal framework also provides a base from which to determine ethical decisions. Interviewee 4 explained that they had a data security course which was mandatory, and so there are serious attempts to deal with the ethical implications of the work. Moreover, Interviewee 3 argued that users should also have technical knowledge and the technical competence regarding practicing safe behavior online.\n<\/p><p>The team agreed that there is a need for more regulation. Interviewee 3 argued that privacy and data analytics should become regulated industries, similar to car management software, or software for medical devices, in which industries you have to keep the source code for 50 years, and it has to be documented and signed before it can be used. Interviewee 3 also mentioned that it is worth paying attention to the level of training for engineers regarding the need for an ethical background. Interviewee 3 explained that every engineer has to make ethical decisions at some point. As such, it is important that engineers are free to object and refuse to participate in certain projects. Interviewee 2 added that they have an ethics department in Company A that helps with these issues, providing support to employees who may have concerns. Furthermore, they stated that there is no code of conduct for cybersecurity.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Security\">Security<\/span><\/h3>\n<p>Interviewee 3 described how IT departments in some companies send internal \u201cphishing mails\u201d (emails attempting to trick the recipient into giving private information) to test their security, and the problem is that employees tend to have a high record on clicking on them, demonstrating a weak level of security awareness. Interviewee 3 also explained that Company A, among other companies, has a \u201chackathon\u201d every year to discover security flaws. Interviewee 4 mentioned that they have company-wide encryption policies for some sensitive materials, which is easy to use now, but that was not the case in the past. Interviewee 4 felt that security is of importance at Company A, but, as Interviewee 1 pointed out, most research is conducted internally, resulting in a lack of publications, at least for the public space. This leaves a number of unanswered questions:\n<\/p>\n<blockquote>[W]ho is attacking your system and what are they after? This hasn't been researched properly, or has been researched but not [made] publicly available. (Interviewee 3, 2018)<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Risk_assessment\">Risk assessment<\/span><\/h3>\n<p>Interviewee 4 noted that there is a lack of risk assessment regarding some key aspects of security, such as the risk of not having security protocols, or the comparative risk of predictive versus reactive strategies. Interviewee 3 said that they had a PhD student currently studying cybersecurity attacks, and one of the things that came out of this research is that the attackers do not necessarily go for the weakest part of the system, because that is not where \u201cthe big game are.\u201d Therefore, this shows the need to have cybersecurity teams that will look for security pitfalls in every part of the system, even in the parts that are considered more secure by design.\n<\/p><p>Interviewee 3 further stated that there is a problem in that the technology they work with can be misused, e.g., used for spying on different countries. Interviewee 2 continued that even if the government has access to this information, the question still remains as to the extent to which citizens can be sure that no one else has the same access. What if a government's position changes, such as that of Germany in the 1920s and \u201830s? There is very little that can be done under such circumstances.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Mechanisms_to_address_ethical_issues\">Mechanisms to address ethical issues<\/span><\/h3>\n<p>During the interview it was noted that there is a need for a culture of openness and challenge in organizations, and that the current paradigm of ethical standards in the use of SIS in cybersecurity is present but not developed. While the GDPR has improved general levels of awareness of cybersecurity and the importance of privacy, there is a need for ethical training for current engineers, as well as to develop stricter codes of conduct for this sector. The external regulations of, for example, targeted advertising and the issues of internationalization require consideration. Furthermore, while GDPR has a strong impact on privacy in Europe, other countries allow companies to gather data more freely.\n<\/p><p>Company A has a number of security strategies which go some way towards addressing ethical concerns. Mandatory training sessions are held annually and policy documents provide guidance. These are supplemented by a culture of challenge and openness in which employees feel free to share their concerns and step back from working on a project with which they have ethical concerns. There are also security measures put in place to keep sensitive data secure, such as limiting the machines on which the data can sit and operating a security clearance system such that only certain people are cleared to access the data.\n<\/p><p>Engagement with different stakeholders, such as the internal Company A units, the academic community, regulators, as well as government agencies, clients. and end users was deemed both desirable and beneficial for all.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The literature review and the interview highlight a correlation between academic understanding of the ethical issues in cybersecurity and those working for the cybersecurity industry. However, both have also shown a lack of joint efforts from academia and engineering, as well as a need to improve the dialogue between the two. There is concern that the level of technical abstraction of university-based development stifles ethical oversight of the development of new SIS technologies in computer science. At the same time, there is a need to include ethical oversight in industry, with clearer codes of conduct for the cybersecurity community. One of the strongest arguments from the team at Company A was the lack of clear codes for international practice. As SIS technology is being developed with cloud computing, and the facility to acquire data from all over the world grows, so there is a need to improve ethical protocols for companies.\n<\/p><p>Overall, it was shown that ethical concerns regarding SIS in cybersecurity go further than mere privacy issues. As it is a sector that will grow in the coming years, incorporating ML and the IoT, the importance of cybersecurity, and thereby the ethics of cybersecurity, will become more important.\n<\/p><p>Among the ethical issues we found the following: informed consent; protection from harm; disclosure of vulnerabilities; biases; the nature of hacking; trust; transparency; the necessity for a risk assessment in cybersecurity; and the responsibility between companies, governments, and users. Interestingly, the issue of monetization (how far can one ethically go to monetize customer\u2019s data) appeared in the interview, but the topic is not one that has been widely discussed in the academic literature (see Table 2, below).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 2.<\/b>\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Issues arising in literature review\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Issues arising in interview\n<\/th><\/tr>\n<tr>\n<th style=\"background-color:white; padding-left:10px; padding-right:10px; text-align:center;\" colspan=\"2\">Similarities\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Protection from harm\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Protection from harm\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Privacy and control of data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Privacy and control of data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Competence of research ethics committees\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Competence of research ethics committees\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security issues\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security issues\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Risk\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Risk assessment\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Codes of conduct\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Policy issues (awareness and knowledge) and mechanisms to address ethical issues\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Responsibility\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Internationalization, standardization, and legal aspects\n<\/td><\/tr>\n<tr>\n<th style=\"background-color:white; padding-left:10px; padding-right:10px; text-align:center;\" colspan=\"2\">Differences\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Vulnerabilities and disclosure\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Anomalies\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Trust and transparency\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Informed consent\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Implications_of_this_report\">Implications of this report<\/span><\/h3>\n<p>This report exposes some of the weakest part of SIS technology and the importance of cybersecurity, by supporting the claim that there is a need to improve the ethics of research in SIS. The cyber world is forming an important part of society, and in some areas at least, albeit not among the interviewees for this case study, there is a lack of understanding of the ethical problems that come with this, which can bring damage to many stakeholders.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Future_research\">Future research<\/span><\/h3>\n<p>This report argues for the need for multi-disciplinary studies between academia and the technical community to prevent ethical concerns from being undervalued. Future research goes hand in hand with legal implications, particularly at the international level, as well the need to create clearer codes of conduct for businesses and international practices, and the necessity to increase the cybersecurity teams within companies. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-SingerCyber14-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SingerCyber14_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-SingerCyber14_1-1\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Singer, P.W.; Friedman, A. (2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/books.google.com\/books?id=9VDSAQAAQBAJ\" target=\"_blank\"><i>Cybersecurity and Cyberwar: What Everyone Needs to Know<\/i><\/a> (1st ed.). Oxford University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780199918119<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/books.google.com\/books?id=9VDSAQAAQBAJ\" target=\"_blank\">https:\/\/books.google.com\/books?id=9VDSAQAAQBAJ<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Cybersecurity+and+Cyberwar%3A+What+Everyone+Needs+to+Know&rft.aulast=Singer%2C+P.W.%3B+Friedman%2C+A.&rft.au=Singer%2C+P.W.%3B+Friedman%2C+A.&rft.date=2014&rft.edition=1st&rft.pub=Oxford+University+Press&rft.isbn=9780199918119&rft_id=https%3A%2F%2Fbooks.google.com%2Fbooks%3Fid%3D9VDSAQAAQBAJ&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Sobers60Must18Arch-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Sobers60Must18Arch_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Sobers, R. (18 May 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20181108122758\/https:\/\/www.varonis.com\/blog\/cybersecurity-statistics\/\" target=\"_blank\">\"60 Must-Know Cybersecurity Statistics for 2018\"<\/a>. <i>Varonis Blog<\/i>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.varonis.com\/blog\/cybersecurity-statistics\/\" target=\"_blank\">the original<\/a> on 08 November 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20181108122758\/https:\/\/www.varonis.com\/blog\/cybersecurity-statistics\/\" target=\"_blank\">https:\/\/web.archive.org\/web\/20181108122758\/https:\/\/www.varonis.com\/blog\/cybersecurity-statistics\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=60+Must-Know+Cybersecurity+Statistics+for+2018&rft.atitle=Varonis+Blog&rft.aulast=Sobers%2C+R.&rft.au=Sobers%2C+R.&rft.date=18+May+2018&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20181108122758%2Fhttps%3A%2F%2Fwww.varonis.com%2Fblog%2Fcybersecurity-statistics%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithCyber18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmithCyber18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smith, P.T. (2018). \"Cyberattacks as Casus Belli: A Sovereignty\u2010Based Account\". <i>Journal of Applied Philosophy<\/i> <b>35<\/b> (2): 222\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjapp.12169\" target=\"_blank\">10.1111\/japp.12169<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyberattacks+as+Casus+Belli%3A+A+Sovereignty%E2%80%90Based+Account&rft.jtitle=Journal+of+Applied+Philosophy&rft.aulast=Smith%2C+P.T.&rft.au=Smith%2C+P.T.&rft.date=2018&rft.volume=35&rft.issue=2&rft.pages=222%E2%80%9341&rft_id=info:doi\/10.1111%2Fjapp.12169&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WolfersNational52-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WolfersNational52_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wolters, A. (1952). \"\"National Security\" as an Ambiguous Symbol\". <i>Political Science Quarterly<\/i> <b>67<\/b> (4): 481\u2013502. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2307%2F2145138\" target=\"_blank\">10.2307\/2145138<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%22National+Security%22+as+an+Ambiguous+Symbol&rft.jtitle=Political+Science+Quarterly&rft.aulast=Wolters%2C+A.&rft.au=Wolters%2C+A.&rft.date=1952&rft.volume=67&rft.issue=4&rft.pages=481%E2%80%93502&rft_id=info:doi\/10.2307%2F2145138&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaldwinTheCon97-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaldwinTheCon97_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Baldwin, D.A. (1997). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cambridge.org\/core\/journals\/review-of-international-studies\/article\/concept-of-security\/67188B6038200A97C0B0A370FDC9D6B8\" target=\"_blank\">\"The Concept of Security\"<\/a>. <i>Review of International Studies<\/i> <b>23<\/b> (1): 5\u201326<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cambridge.org\/core\/journals\/review-of-international-studies\/article\/concept-of-security\/67188B6038200A97C0B0A370FDC9D6B8\" target=\"_blank\">https:\/\/www.cambridge.org\/core\/journals\/review-of-international-studies\/article\/concept-of-security\/67188B6038200A97C0B0A370FDC9D6B8<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Concept+of+Security&rft.jtitle=Review+of+International+Studies&rft.aulast=Baldwin%2C+D.A.&rft.au=Baldwin%2C+D.A.&rft.date=1997&rft.volume=23&rft.issue=1&rft.pages=5%E2%80%9326&rft_id=https%3A%2F%2Fwww.cambridge.org%2Fcore%2Fjournals%2Freview-of-international-studies%2Farticle%2Fconcept-of-security%2F67188B6038200A97C0B0A370FDC9D6B8&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LundgrenDefin19-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LundgrenDefin19_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lundgren, B.; M\u00f6ller, N. (2019). \"Defining Information Security\". <i>Science and Engineering Ethics<\/i> <b>25<\/b> (2): 419\u201341. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11948-017-9992-1\" target=\"_blank\">10.1007\/s11948-017-9992-1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Defining+Information+Security&rft.jtitle=Science+and+Engineering+Ethics&rft.aulast=Lundgren%2C+B.%3B+M%C3%B6ller%2C+N.&rft.au=Lundgren%2C+B.%3B+M%C3%B6ller%2C+N.&rft.date=2019&rft.volume=25&rft.issue=2&rft.pages=419%E2%80%9341&rft_id=info:doi\/10.1007%2Fs11948-017-9992-1&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CraigenDefining14-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CraigenDefining14_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-CraigenDefining14_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Craigen, D.; Diakun\u2014Thibault, N.; Purse, R. (2014). \"Defining Cybersecurity\". <i>Technology Innovation Management Review<\/i> <b>4<\/b> (10): 13\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.22215%2Ftimreview%2F835\" target=\"_blank\">10.22215\/timreview\/835<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Defining+Cybersecurity&rft.jtitle=Technology+Innovation+Management+Review&rft.aulast=Craigen%2C+D.%3B+Diakun%E2%80%94Thibault%2C+N.%3B+Purse%2C+R.&rft.au=Craigen%2C+D.%3B+Diakun%E2%80%94Thibault%2C+N.%3B+Purse%2C+R.&rft.date=2014&rft.volume=4&rft.issue=10&rft.pages=13%E2%80%9321&rft_id=info:doi\/10.22215%2Ftimreview%2F835&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HessUnder06-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HessUnder06_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hess, C.; Ostrom, E. (2006). <i>Understanding Knowledge as a Commons: From Theory to Practice<\/i>. MIT Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780262083577.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Understanding+Knowledge+as+a+Commons%3A+From+Theory+to+Practice&rft.aulast=Hess%2C+C.%3B+Ostrom%2C+E.&rft.au=Hess%2C+C.%3B+Ostrom%2C+E.&rft.date=2006&rft.pub=MIT+Press&rft.isbn=9780262083577&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BainbridgeIronies83-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BainbridgeIronies83_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bainbridge, L. (1983). \"Ironies of automation\". <i>Automatica<\/i> <b>19<\/b> (6): 775\u201379. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2F0005-1098%2883%2990046-8\" target=\"_blank\">10.1016\/0005-1098(83)90046-8<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ironies+of+automation&rft.jtitle=Automatica&rft.aulast=Bainbridge%2C+L.&rft.au=Bainbridge%2C+L.&rft.date=1983&rft.volume=19&rft.issue=6&rft.pages=775%E2%80%9379&rft_id=info:doi\/10.1016%2F0005-1098%2883%2990046-8&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoddardAuto12-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoddardAuto12_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goddard, K.; Roudsari, A.; Wyatt, J.C. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3240751\" target=\"_blank\">\"Automation bias: A systematic review of frequency, effect mediators, and mitigators\"<\/a>. <i>JAMIA<\/i> <b>19<\/b> (1): 121\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2011-000089\" target=\"_blank\">10.1136\/amiajnl-2011-000089<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3240751\/\" target=\"_blank\">PMC3240751<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21685142\" target=\"_blank\">21685142<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3240751\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3240751<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automation+bias%3A+A+systematic+review+of+frequency%2C+effect+mediators%2C+and+mitigators&rft.jtitle=JAMIA&rft.aulast=Goddard%2C+K.%3B+Roudsari%2C+A.%3B+Wyatt%2C+J.C.&rft.au=Goddard%2C+K.%3B+Roudsari%2C+A.%3B+Wyatt%2C+J.C.&rft.date=2012&rft.volume=19&rft.issue=1&rft.pages=121%E2%80%937&rft_id=info:doi\/10.1136%2Famiajnl-2011-000089&rft_id=info:pmc\/PMC3240751&rft_id=info:pmid\/21685142&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3240751&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TuckerCyber18-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TuckerCyber18_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tucker, E. (July 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.computerweekly.com\/opinion\/Cyber-security-why-youre-doing-it-all-wrong\" target=\"_blank\">\"Cyber security \u2013 why you\u2019re doing it all wrong\"<\/a>. <i>Computer Weekly<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.computerweekly.com\/opinion\/Cyber-security-why-youre-doing-it-all-wrong\" target=\"_blank\">https:\/\/www.computerweekly.com\/opinion\/Cyber-security-why-youre-doing-it-all-wrong<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cyber+security+%E2%80%93+why+you%E2%80%99re+doing+it+all+wrong&rft.atitle=Computer+Weekly&rft.aulast=Tucker%2C+E.&rft.au=Tucker%2C+E.&rft.date=July+2018&rft_id=https%3A%2F%2Fwww.computerweekly.com%2Fopinion%2FCyber-security-why-youre-doing-it-all-wrong&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MacnishUnblink12-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MacnishUnblink12_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Macnish, K. (2012). \"Unblinking eyes: The ethics of automating surveillance\". <i>Ethics and Information Technology<\/i> <b>14<\/b> (2): 151\u201367. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10676-012-9291-0\" target=\"_blank\">10.1007\/s10676-012-9291-0<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Unblinking+eyes%3A+The+ethics+of+automating+surveillance&rft.jtitle=Ethics+and+Information+Technology&rft.aulast=Macnish%2C+K.&rft.au=Macnish%2C+K.&rft.date=2012&rft.volume=14&rft.issue=2&rft.pages=151%E2%80%9367&rft_id=info:doi\/10.1007%2Fs10676-012-9291-0&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JohnsonComp12-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JohnsonComp12_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Johnson M.L.; Bellovin S.M.; Keromytis A.D. (2012). \"Computer Security Research with Human Subjects: Risks, Benefits and Informed Consent\". In Danezis G.; Dietrich S.; Sako K.. <i>Financial Cryptography and Data Security<\/i>. Springer. pp. 131\u201337. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-29889-9_11\" target=\"_blank\">10.1007\/978-3-642-29889-9_11<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783642298899.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Computer+Security+Research+with+Human+Subjects%3A+Risks%2C+Benefits+and+Informed+Consent&rft.atitle=Financial+Cryptography+and+Data+Security&rft.aulast=Johnson+M.L.%3B+Bellovin+S.M.%3B+Keromytis+A.D.&rft.au=Johnson+M.L.%3B+Bellovin+S.M.%3B+Keromytis+A.D.&rft.date=2012&rft.pages=pp.%26nbsp%3B131%E2%80%9337&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-642-29889-9_11&rft.isbn=9783642298899&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MillerTheEthics09-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MillerTheEthics09_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-MillerTheEthics09_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Miller, F.; Wertheimer, A., ed. (2009). <i>The Ethics of Consent<\/i>. Oxford University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780195335149.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Ethics+of+Consent&rft.date=2009&rft.pub=Oxford+University+Press&rft.isbn=9780195335149&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MansonRethink07-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MansonRethink07_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Manson, N.C.; O'Neill, O. (2007). <i>Rethinking Informed Consent in Bioethics<\/i>. Cambridge University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1017%2FCBO9780511814600\" target=\"_blank\">10.1017\/CBO9780511814600<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780511814600.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Rethinking+Informed+Consent+in+Bioethics&rft.aulast=Manson%2C+N.C.%3B+O%27Neill%2C+O.&rft.au=Manson%2C+N.C.%3B+O%27Neill%2C+O.&rft.date=2007&rft.pub=Cambridge+University+Press&rft_id=info:doi\/10.1017%2FCBO9780511814600&rft.isbn=9780511814600&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BurnettEncore15-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BurnettEncore15_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-BurnettEncore15_16-1\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-BurnettEncore15_16-2\">16.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Burnett, S.; Feamster, N. (2015). \"Encore: Lightweight Measurement of Web Censorship with Cross-Origin Requests\". <i>Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication<\/i>: 653\u201367. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2785956.2787485\" target=\"_blank\">10.1145\/2785956.2787485<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Encore%3A+Lightweight+Measurement+of+Web+Censorship+with+Cross-Origin+Requests&rft.jtitle=Proceedings+of+the+2015+ACM+Conference+on+Special+Interest+Group+on+Data+Communication&rft.aulast=Burnett%2C+S.%3B+Feamster%2C+N.&rft.au=Burnett%2C+S.%3B+Feamster%2C+N.&rft.date=2015&rft.pages=653%E2%80%9367&rft_id=info:doi\/10.1145%2F2785956.2787485&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MacnishEthics19-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MacnishEthics19_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-1\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-2\">17.2<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-3\">17.3<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-4\">17.4<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-5\">17.5<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-6\">17.6<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-7\">17.7<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-8\">17.8<\/a><\/sup> <sup><a href=\"#cite_ref-MacnishEthics19_17-9\">17.9<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">van der Ham, J. (14 September 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/jeroenh\/Ethics-and-Cyber-Security\/blob\/master\/template.tex\" target=\"_blank\">\"jeroenh\/Ethics-and-Cyber-Security\/template.tex\"<\/a>. <i>GitHub<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/jeroenh\/Ethics-and-Cyber-Security\/blob\/master\/template.tex\" target=\"_blank\">https:\/\/github.com\/jeroenh\/Ethics-and-Cyber-Security\/blob\/master\/template.tex<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=jeroenh%2FEthics-and-Cyber-Security%2Ftemplate.tex&rft.atitle=GitHub&rft.aulast=van+der+Ham%2C+J.&rft.au=van+der+Ham%2C+J.&rft.date=14+September+2018&rft_id=https%3A%2F%2Fgithub.com%2Fjeroenh%2FEthics-and-Cyber-Security%2Fblob%2Fmaster%2Ftemplate.tex&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PietersExplan11-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PietersExplan11_18-0\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-PietersExplan11_18-1\">18.1<\/a><\/sup> <sup><a href=\"#cite_ref-PietersExplan11_18-2\">18.2<\/a><\/sup> <sup><a href=\"#cite_ref-PietersExplan11_18-3\">18.3<\/a><\/sup> <sup><a href=\"#cite_ref-PietersExplan11_18-4\">18.4<\/a><\/sup> <sup><a href=\"#cite_ref-PietersExplan11_18-5\">18.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pieters, W. (2011). \"Explanation and trust: what to tell the user in security and AI?\". <i>Ethics and Information Technology<\/i> <b>13<\/b> (1): 53\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10676-010-9253-3\" target=\"_blank\">10.1007\/s10676-010-9253-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Explanation+and+trust%3A+what+to+tell+the+user+in+security+and+AI%3F&rft.jtitle=Ethics+and+Information+Technology&rft.aulast=Pieters%2C+W.&rft.au=Pieters%2C+W.&rft.date=2011&rft.volume=13&rft.issue=1&rft.pages=53%E2%80%9364&rft_id=info:doi\/10.1007%2Fs10676-010-9253-3&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NicholsStJude16-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NicholsStJude16_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Nichols, S. (07 September 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.theregister.co.uk\/2016\/09\/07\/st_jude_sues_over_hacking_claim\/\" target=\"_blank\">\"St Jude sues short-selling MedSec over pacemaker 'hack' report\"<\/a>. <i>The Register<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.theregister.co.uk\/2016\/09\/07\/st_jude_sues_over_hacking_claim\/\" target=\"_blank\">https:\/\/www.theregister.co.uk\/2016\/09\/07\/st_jude_sues_over_hacking_claim\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 July 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=St+Jude+sues+short-selling+MedSec+over+pacemaker+%27hack%27+report&rft.atitle=The+Register&rft.aulast=Nichols%2C+S.&rft.au=Nichols%2C+S.&rft.date=07+September+2016&rft_id=https%3A%2F%2Fwww.theregister.co.uk%2F2016%2F09%2F07%2Fst_jude_sues_over_hacking_claim%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpringRes16-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpringRes16_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Spring, T. (31 August 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/threatpost.com\/researchers-medsec-muddy-waters-set-bad-precedent-with-st-jude-medical-short\/120266\/\" target=\"_blank\">\"Researchers: MedSec, Muddy Waters Set Bad Precedent With St. Jude Medical Short\"<\/a>. <i>Threat Post<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/threatpost.com\/researchers-medsec-muddy-waters-set-bad-precedent-with-st-jude-medical-short\/120266\/\" target=\"_blank\">https:\/\/threatpost.com\/researchers-medsec-muddy-waters-set-bad-precedent-with-st-jude-medical-short\/120266\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 04 July 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Researchers%3A+MedSec%2C+Muddy+Waters+Set+Bad+Precedent+With+St.+Jude+Medical+Short&rft.atitle=Threat+Post&rft.aulast=Spring%2C+T.&rft.au=Spring%2C+T.&rft.date=31+August+2016&rft_id=https%3A%2F%2Fthreatpost.com%2Fresearchers-medsec-muddy-waters-set-bad-precedent-with-st-jude-medical-short%2F120266%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ManjikianCyber17-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ManjikianCyber17_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Manjikian, M. (2017). <i>Cybersecurity Ethics<\/i>. Routledge. pp. 81\u2013112. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781138717527.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Cybersecurity+Ethics&rft.aulast=Manjikian%2C+M.&rft.au=Manjikian%2C+M.&rft.date=2017&rft.pages=pp.%26nbsp%3B81%E2%80%93112&rft.pub=Routledge&rft.isbn=9781138717527&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoorePriv15-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoorePriv15_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Moore, A.D. (2015). <i>Privacy, Security and Accountability: Ethics, Law and Policy<\/i>. Rowman and Littlefield. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781783484768.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Privacy%2C+Security+and+Accountability%3A+Ethics%2C+Law+and+Policy&rft.aulast=Moore%2C+A.D.&rft.au=Moore%2C+A.D.&rft.date=2015&rft.pub=Rowman+and+Littlefield&rft.isbn=9781783484768&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoorePrivacy12-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoorePrivacy12_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moore, A.D. (2003). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ssrn.com\/abstract=1980880\" target=\"_blank\">\"Privacy: Its Meaning and Value\"<\/a>. <i>American Philosophical Quarterly<\/i> <b>40<\/b>: 215\u201327<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ssrn.com\/abstract=1980880\" target=\"_blank\">https:\/\/ssrn.com\/abstract=1980880<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Privacy%3A+Its+Meaning+and+Value&rft.jtitle=American+Philosophical+Quarterly&rft.aulast=Moore%2C+A.D.&rft.au=Moore%2C+A.D.&rft.date=2003&rft.volume=40&rft.pages=215%E2%80%9327&rft_id=https%3A%2F%2Fssrn.com%2Fabstract%3D1980880&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AllenPrivacy99-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AllenPrivacy99_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Allen, A.L. (1999). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/heinonline.org\/HOL\/LandingPage?collection=journals&handle=hein.journals\/conlr32&div=35\" target=\"_blank\">\"Privacy-as-Data Control: Conceptual, Practical, and Moral Limits of the Paradigm\"<\/a>. <i>Connecticut Law Review<\/i> <b>32<\/b>: 861\u201375<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/heinonline.org\/HOL\/LandingPage?collection=journals&handle=hein.journals\/conlr32&div=35\" target=\"_blank\">https:\/\/heinonline.org\/HOL\/LandingPage?collection=journals&handle=hein.journals\/conlr32&div=35<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Privacy-as-Data+Control%3A+Conceptual%2C+Practical%2C+and+Moral+Limits+of+the+Paradigm&rft.jtitle=Connecticut+Law+Review&rft.aulast=Allen%2C+A.L.&rft.au=Allen%2C+A.L.&rft.date=1999&rft.volume=32&rft.pages=861%E2%80%9375&rft_id=https%3A%2F%2Fheinonline.org%2FHOL%2FLandingPage%3Fcollection%3Djournals%26handle%3Dhein.journals%2Fconlr32%26div%3D35&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MacnishGov16-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MacnishGov16_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Macnish, K. (2018). \"Government Surveillance and Why Defining Privacy Matters in a Post\u2010Snowden World\". <i>Journal of Applied Philosophy<\/i> <b>35<\/b> (2): 417\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fjapp.12219\" target=\"_blank\">10.1111\/japp.12219<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Government+Surveillance+and+Why+Defining+Privacy+Matters+in+a+Post%E2%80%90Snowden+World&rft.jtitle=Journal+of+Applied+Philosophy&rft.aulast=Macnish%2C+K.&rft.au=Macnish%2C+K.&rft.date=2018&rft.volume=35&rft.issue=2&rft.pages=417%E2%80%9332&rft_id=info:doi\/10.1111%2Fjapp.12219&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CadwalladrRevealed18-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CadwalladrRevealed18_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Cadwalladr, C.; Graham-Harrison, E. (17 March 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.theguardian.com\/news\/2018\/mar\/17\/cambridge-analytica-facebook-influence-us-election\" target=\"_blank\">\"Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in major data breach\"<\/a>. <i>The Guardian<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.theguardian.com\/news\/2018\/mar\/17\/cambridge-analytica-facebook-influence-us-election\" target=\"_blank\">https:\/\/www.theguardian.com\/news\/2018\/mar\/17\/cambridge-analytica-facebook-influence-us-election<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Revealed%3A+50+million+Facebook+profiles+harvested+for+Cambridge+Analytica+in+major+data+breach&rft.atitle=The+Guardian&rft.aulast=Cadwalladr%2C+C.%3B+Graham-Harrison%2C+E.&rft.au=Cadwalladr%2C+C.%3B+Graham-Harrison%2C+E.&rft.date=17+March+2018&rft_id=https%3A%2F%2Fwww.theguardian.com%2Fnews%2F2018%2Fmar%2F17%2Fcambridge-analytica-facebook-influence-us-election&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IencaCambridge18-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IencaCambridge18_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Ienca, M.; Vayena, E. (30 March 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/blogs.scientificamerican.com\/observations\/cambridge-analytica-and-online-manipulation\/\" target=\"_blank\">\"Cambridge Analytica and Online Manipulation\"<\/a>. <i>Scientific American<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/blogs.scientificamerican.com\/observations\/cambridge-analytica-and-online-manipulation\/\" target=\"_blank\">https:\/\/blogs.scientificamerican.com\/observations\/cambridge-analytica-and-online-manipulation\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 July 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cambridge+Analytica+and+Online+Manipulation&rft.atitle=Scientific+American&rft.aulast=Ienca%2C+M.%3B+Vayena%2C+E.&rft.au=Ienca%2C+M.%3B+Vayena%2C+E.&rft.date=30+March+2018&rft_id=https%3A%2F%2Fblogs.scientificamerican.com%2Fobservations%2Fcambridge-analytica-and-online-manipulation%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EUReg16-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EUReg16_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Council of the European Union, European Parliament (27 April 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/publications.europa.eu\/en\/publication-detail\/-\/publication\/3e485e15-11bd-11e6-ba9a-01aa75ed71a1\/language-en\" target=\"_blank\">\"Regulation (EU) 2016\/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation) (Text with EEA relevance)\"<\/a>. European Union<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/publications.europa.eu\/en\/publication-detail\/-\/publication\/3e485e15-11bd-11e6-ba9a-01aa75ed71a1\/language-en\" target=\"_blank\">https:\/\/publications.europa.eu\/en\/publication-detail\/-\/publication\/3e485e15-11bd-11e6-ba9a-01aa75ed71a1\/language-en<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Regulation+%28EU%29+2016%2F679+of+the+European+Parliament+and+of+the+Council+of+27+April+2016+on+the+protection+of+natural+persons+with+regard+to+the+processing+of+personal+data+and+on+the+free+movement+of+such+data%2C+and+repealing+Directive+95%2F46%2FEC+%28General+Data+Protection+Regulation%29+%28Text+with+EEA+relevance%29&rft.atitle=&rft.aulast=Council+of+the+European+Union%2C+European+Parliament&rft.au=Council+of+the+European+Union%2C+European+Parliament&rft.date=27+April+2016&rft.pub=European+Union&rft_id=https%3A%2F%2Fpublications.europa.eu%2Fen%2Fpublication-detail%2F-%2Fpublication%2F3e485e15-11bd-11e6-ba9a-01aa75ed71a1%2Flanguage-en&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DavisThink91-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DavisThink91_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Davis, M. (1991). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.jstor.org\/stable\/2265293\" target=\"_blank\">\"Thinking like an engineer: The place of a code of ethics in the practice of a profession\"<\/a>. <i>Philosophy & Public Affairs<\/i> <b>20<\/b> (2): 150\u201367<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.jstor.org\/stable\/2265293\" target=\"_blank\">https:\/\/www.jstor.org\/stable\/2265293<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Thinking+like+an+engineer%3A+The+place+of+a+code+of+ethics+in+the+practice+of+a+profession&rft.jtitle=Philosophy+%26+Public+Affairs&rft.aulast=Davis%2C+M.&rft.au=Davis%2C+M.&rft.date=1991&rft.volume=20&rft.issue=2&rft.pages=150%E2%80%9367&rft_id=https%3A%2F%2Fwww.jstor.org%2Fstable%2F2265293&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GlassToward08-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GlassToward08_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Glass, A.; McGuinness, D.L.; Wolverton, M. (2008). \"Toward establishing trust in adaptive agents\". <i>Proceedings of the 13th International Conference on Intelligent User Interfaces<\/i>: 227\u201336. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1378773.1378804\" target=\"_blank\">10.1145\/1378773.1378804<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+establishing+trust+in+adaptive+agents&rft.jtitle=Proceedings+of+the+13th+International+Conference+on+Intelligent+User+Interfaces&rft.aulast=Glass%2C+A.%3B+McGuinness%2C+D.L.%3B+Wolverton%2C+M.&rft.au=Glass%2C+A.%3B+McGuinness%2C+D.L.%3B+Wolverton%2C+M.&rft.date=2008&rft.pages=227%E2%80%9336&rft_id=info:doi\/10.1145%2F1378773.1378804&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BedersonElect03-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BedersonElect03_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bederson, B.B.; Lee, B.; Sherman, R.M. et al. (2003). \"Electronic voting system usability issues\". <i>Proceedings of the SIGCHI Conference on Human Factors in Computing Systems<\/i>: 145\u201352. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F642611.642638\" target=\"_blank\">10.1145\/642611.642638<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+voting+system+usability+issues&rft.jtitle=Proceedings+of+the+SIGCHI+Conference+on+Human+Factors+in+Computing+Systems&rft.aulast=Bederson%2C+B.B.%3B+Lee%2C+B.%3B+Sherman%2C+R.M.+et+al.&rft.au=Bederson%2C+B.B.%3B+Lee%2C+B.%3B+Sherman%2C+R.M.+et+al.&rft.date=2003&rft.pages=145%E2%80%9352&rft_id=info:doi\/10.1145%2F642611.642638&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LatourReass05-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LatourReass05_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Latour, B. (2005). <i>Reassembling the Social: An Introduction to Actor-Network-Theory<\/i>. Oxford University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780199256044.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Reassembling+the+Social%3A+An+Introduction+to+Actor-Network-Theory&rft.aulast=Latour%2C+B.&rft.au=Latour%2C+B.&rft.date=2005&rft.pub=Oxford+University+Press&rft.isbn=9780199256044&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ONeilWeapons16-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ONeilWeapons16_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">O'Neil, C. (2016). <i>Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy<\/i>. Crown. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780553418811.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Weapons+of+Math+Destruction%3A+How+Big+Data+Increases+Inequality+and+Threatens+Democracy&rft.aulast=O%27Neil%2C+C.&rft.au=O%27Neil%2C+C.&rft.date=2016&rft.pub=Crown&rft.isbn=9780553418811&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HanssonTheEthics13-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HanssonTheEthics13_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hansson, S. (2013). <i>The Ethics of Risk: Ethical Analysis in an Uncertain World<\/i>. Palgrave MacMillan. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781137333650.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Ethics+of+Risk%3A+Ethical+Analysis+in+an+Uncertain+World&rft.aulast=Hansson%2C+S.&rft.au=Hansson%2C+S.&rft.date=2013&rft.pub=Palgrave+MacMillan&rft.isbn=9781137333650&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WolffFive10-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WolffFive10_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wolff, J. (2010). \"Five Types of Risky Situation\". <i>Law, Innovation and Technology<\/i> <b>2<\/b> (2): 151\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5235%2F175799610794046177\" target=\"_blank\">10.5235\/175799610794046177<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Five+Types+of+Risky+Situation&rft.jtitle=Law%2C+Innovation+and+Technology&rft.aulast=Wolff%2C+J.&rft.au=Wolff%2C+J.&rft.date=2010&rft.volume=2&rft.issue=2&rft.pages=151%E2%80%9363&rft_id=info:doi\/10.5235%2F175799610794046177&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ByersEncore15-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ByersEncore15_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Byers, J.W. (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/conferences.sigcomm.org\/sigcomm\/2015\/pdf\/reviews\/226pr.pdf\" target=\"_blank\">\"Encore: Lightweight Measurement of Web Censorship with Cross-Origin Requests \u2013 Public Review\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/conferences.sigcomm.org\/sigcomm\/2015\/pdf\/reviews\/226pr.pdf\" target=\"_blank\">https:\/\/conferences.sigcomm.org\/sigcomm\/2015\/pdf\/reviews\/226pr.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Encore%3A+Lightweight+Measurement+of+Web+Censorship+with+Cross-Origin+Requests+%E2%80%93+Public+Review&rft.atitle=&rft.aulast=Byers%2C+J.W.&rft.au=Byers%2C+J.W.&rft.date=2015&rft_id=https%3A%2F%2Fconferences.sigcomm.org%2Fsigcomm%2F2015%2Fpdf%2Freviews%2F226pr.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HermanssonTowards10-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HermanssonTowards10_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hermansson, H. (2010). \"Towards a fair procedure for risk management\". <i>Journal of Risk Research<\/i> <b>13<\/b> (4): 501\u201315. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F13669870903305903\" target=\"_blank\">10.1080\/13669870903305903<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+a+fair+procedure+for+risk+management&rft.jtitle=Journal+of+Risk+Research&rft.aulast=Hermansson%2C+H.&rft.au=Hermansson%2C+H.&rft.date=2010&rft.volume=13&rft.issue=4&rft.pages=501%E2%80%9315&rft_id=info:doi\/10.1080%2F13669870903305903&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HermanssonConsist05-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HermanssonConsist05_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hermansson, H. (2005). \"Consistent risk management: Three models outlined\". <i>Journal of Risk Research<\/i> <b>8<\/b> (7\u20138): 557\u201368. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F13669870500085189\" target=\"_blank\">10.1080\/13669870500085189<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Consistent+risk+management%3A+Three+models+outlined&rft.jtitle=Journal+of+Risk+Research&rft.aulast=Hermansson%2C+H.&rft.au=Hermansson%2C+H.&rft.date=2005&rft.volume=8&rft.issue=7%E2%80%938&rft.pages=557%E2%80%9368&rft_id=info:doi\/10.1080%2F13669870500085189&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuioraCyber17-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuioraCyber17_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Guiora, A.N. (2017). <i>Cybersecurity: Geopolitics, Law, and Policy<\/i>. Routledge. pp. 89\u2013111. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781138033290.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Cybersecurity%3A+Geopolitics%2C+Law%2C+and+Policy&rft.aulast=Guiora%2C+A.N.&rft.au=Guiora%2C+A.N.&rft.date=2017&rft.pages=pp.%26nbsp%3B89%E2%80%93111&rft.pub=Routledge&rft.isbn=9781138033290&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vanCleeffSecurity09-40\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-vanCleeffSecurity09_40-0\">40.0<\/a><\/sup> <sup><a href=\"#cite_ref-vanCleeffSecurity09_40-1\">40.1<\/a><\/sup> <sup><a href=\"#cite_ref-vanCleeffSecurity09_40-2\">40.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">van Cleeff, A.; Pieters, W.; Wieringa, R.J. (2009). \"Security Implications of Virtualization: A Literature Study\". <i>Proceedings from the 2009 International Conference on Computational Science and Engineering<\/i>: 353-358. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCSE.2009.267\" target=\"_blank\">10.1109\/CSE.2009.267<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Security+Implications+of+Virtualization%3A+A+Literature+Study&rft.jtitle=Proceedings+from+the+2009+International+Conference+on+Computational+Science+and+Engineering&rft.aulast=van+Cleeff%2C+A.%3B+Pieters%2C+W.%3B+Wieringa%2C+R.J.&rft.au=van+Cleeff%2C+A.%3B+Pieters%2C+W.%3B+Wieringa%2C+R.J.&rft.date=2009&rft.pages=353-358&rft_id=info:doi\/10.1109%2FCSE.2009.267&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SparrowKiller07-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SparrowKiller07_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sparrow, R. (2007). \"Killer Robots\". <i>Journal of Applied Philosophy<\/i> <b>24<\/b> (1): 62\u201377. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1468-5930.2007.00346.x\" target=\"_blank\">10.1111\/j.1468-5930.2007.00346.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Killer+Robots&rft.jtitle=Journal+of+Applied+Philosophy&rft.aulast=Sparrow%2C+R.&rft.au=Sparrow%2C+R.&rft.date=2007&rft.volume=24&rft.issue=1&rft.pages=62%E2%80%9377&rft_id=info:doi\/10.1111%2Fj.1468-5930.2007.00346.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StoneTheYahoo19-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StoneTheYahoo19_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Stone, N. (07 April 2019). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cashfloat.co.uk\/blog\/technology-innovation\/yahoo-cyber-attack\/\" target=\"_blank\">\"The Yahoo Cyber Attack & What should you learn from it?\"<\/a>. <i>CashFloat Blog<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cashfloat.co.uk\/blog\/technology-innovation\/yahoo-cyber-attack\/\" target=\"_blank\">https:\/\/www.cashfloat.co.uk\/blog\/technology-innovation\/yahoo-cyber-attack\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Yahoo+Cyber+Attack+%26+What+should+you+learn+from+it%3F&rft.atitle=CashFloat+Blog&rft.aulast=Stone%2C+N.&rft.au=Stone%2C+N.&rft.date=07+April+2019&rft_id=https%3A%2F%2Fwww.cashfloat.co.uk%2Fblog%2Ftechnology-innovation%2Fyahoo-cyber-attack%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MacnishUnder19-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MacnishUnder19_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Macnish, K.; Ryan, M.; Stahl, B. (2019). \"Understanding Ethics and Human Rights in Smart Information Systems: A Multi Case Study Approach\". <i>ORBIT Journal<\/i> <b>2<\/b> (2): 102. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.29297%2Forbit.v2i1.102\" target=\"_blank\">10.29297\/orbit.v2i1.102<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+Ethics+and+Human+Rights+in+Smart+Information+Systems%3A+A+Multi+Case+Study+Approach&rft.jtitle=ORBIT+Journal&rft.aulast=Macnish%2C+K.%3B+Ryan%2C+M.%3B+Stahl%2C+B.&rft.au=Macnish%2C+K.%3B+Ryan%2C+M.%3B+Stahl%2C+B.&rft.date=2019&rft.volume=2&rft.issue=2&rft.pages=102&rft_id=info:doi\/10.29297%2Forbit.v2i1.102&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarkoTheOmni15-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarkoTheOmni15_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Marko, K. (27 October 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.forbes.com\/sites\/kurtmarko\/2015\/10\/27\/omni-connected-world\/\" target=\"_blank\">\"The Omni-Connected World: Bell Labs Plans For Future Of Connected Everything\"<\/a>. <i>Forbes<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.forbes.com\/sites\/kurtmarko\/2015\/10\/27\/omni-connected-world\/\" target=\"_blank\">https:\/\/www.forbes.com\/sites\/kurtmarko\/2015\/10\/27\/omni-connected-world\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Omni-Connected+World%3A+Bell+Labs+Plans+For+Future+Of+Connected+Everything&rft.atitle=Forbes&rft.aulast=Marko%2C+K.&rft.au=Marko%2C+K.&rft.date=27+October+2015&rft_id=https%3A%2F%2Fwww.forbes.com%2Fsites%2Fkurtmarko%2F2015%2F10%2F27%2Fomni-connected-world%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ApplePrivacy-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ApplePrivacy_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.apple.com\/lae\/privacy\/approach-to-privacy\/\" target=\"_blank\">\"Our Approach to Privacy\"<\/a>. Apple, Inc. 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.apple.com\/lae\/privacy\/approach-to-privacy\/\" target=\"_blank\">https:\/\/www.apple.com\/lae\/privacy\/approach-to-privacy\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 December 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Our+Approach+to+Privacy&rft.atitle=&rft.date=2018&rft.pub=Apple%2C+Inc&rft_id=https%3A%2F%2Fwww.apple.com%2Flae%2Fprivacy%2Fapproach-to-privacy%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation (e.g., British English to American English). In some cases important information was missing from the references, and that information was added. The 2018 article by Sobers on 60 must-know cybersecurity facts has been updated in 2019; an archived version from 2018 is used in this version. The Stone article on the Yahoo cyber attack also appears to have been updated in 2019, though no archived version of the article from 2017 exists. The Lundgren and M\u00f6ller citation has changed since the original article published online; this version represents the new information. The original cites an article by Macnish and van der Ham, but the research doesn't appear to be published yet; found a draft on GitHub to cite. The original has an inline citation for Marko 2015 but doesn't include it in the closing references; found the supposed reference online and included it here. Non-figured \"flavor\" images from the original were not included here.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165454\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.982 seconds\nReal time usage: 1.009 seconds\nPreprocessor visited node count: 32583\/1000000\nPreprocessor generated node count: 39872\/1000000\nPost\u2010expand include size: 204887\/2097152 bytes\nTemplate argument size: 70829\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 979.396 1 - -total\n 87.49% 856.827 1 - Template:Reflist\n 76.93% 753.405 45 - Template:Citation\/core\n 39.70% 388.807 22 - Template:Cite_journal\n 22.14% 216.791 11 - Template:Cite_book\n 19.50% 190.988 12 - Template:Cite_web\n 6.60% 64.642 1 - Template:Infobox_journal_article\n 6.32% 61.866 1 - Template:Infobox\n 4.71% 46.140 33 - Template:Citation\/identifier\n 4.31% 42.218 56 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11068-0!*!0!!en!*!* and timestamp 20190701165453 and revision id 35786\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","c35e087aee1f0ed0d5c790bbd387ef29_images":[],"c35e087aee1f0ed0d5c790bbd387ef29_timestamp":1562000093,"e69b956327faef5b93c17a708e883f73_type":"article","e69b956327faef5b93c17a708e883f73_title":"Cyberbiosecurity: A new perspective on protecting U.S. food and agricultural system (Duncan et al. 2019)","e69b956327faef5b93c17a708e883f73_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system","e69b956327faef5b93c17a708e883f73_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Cyberbiosecurity: A new perspective on protecting U.S. food and agricultural system\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nCyberbiosecurity: A new perspective on protecting U.S. food and agricultural systemJournal\n \nFrontiers in Bioengineering and BiotechnologyAuthor(s)\n \nDuncan, Susan E.; Reinhard, Robert; Williams, Robert C.; Ramsey, Ford; Thomason, Wade;\r\nLee, Kiho; Dudek, Nancy; Mostaghimi, Saied; Colbert, Edward; Murch, RandallAuthor affiliation(s)\n \nVirginia Tech, Tyson FoodsPrimary contact\n \nEmail: duncans at vt dot eduEditors\n \nMorse, Stephen AllenYear published\n \n2019Volume and issue\n \n7Page(s)\n \n63DOI\n \n10.3389\/fbioe.2019.00063ISSN\n \n2296-4185Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00063\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00063\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction: Food and agriculture cyberbiosecurity at the interface of biosecurity and cybersecurity \n3 Vulnerability of the food and agricultural system and the bioeconomy \n4 Risk assessment, critical control points, and regulatory options \n5 Considering the diversity within and across plant, animal, and environmental sectors of the food and agricultural system \n\n5.1 Dairy \n5.2 Food animals \n5.3 Row crops \n5.4 Fruits and vegetables \n5.5 Environmental resources (water) \n\n\n6 Conclusions: Moving Toward Solutions \n7 Supplementary material \n8 Acknowledgements \n\n8.1 Author Contributions \n8.2 Conflict of interest \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nOur national data and infrastructure security issues affecting the \u201cbioeconomy\u201d are evolving rapidly. Simultaneously, the conversation about cybersecurity of the U.S. food and agricultural system (cyber biosecurity) is incomplete and disjointed. The food and agricultural production sectors influence over 20% of the nation's economy ($6.7T) and 15% of U.S. employment (43.3M jobs). The food and agricultural sectors are immensely diverse, and they require advanced technologies and efficiencies that rely on computer technologies, big data, cloud-based data storage, and internet accessibility. There is a critical need to safeguard the cyber biosecurity of our bioeconomy, but currently protections are minimal and do not broadly exist across the food and agricultural system. Using the food safety management Hazard Analysis Critical Control Point (HACCP) system concept as an introductory point of reference, we identify important features in broad food and agricultural production and food systems: dairy, food animals, row crops, fruits and vegetables, and environmental resources (water). This analysis explores the relevant concepts of cyber biosecurity from food production to the end product user (such as the consumer) and considers the integration of diverse transportation, supplier, and retailer networks. We describe common challenges and unique barriers across these systems and recommend solutions to advance the role of cyber biosecurity in the food and agricultural sectors.\nKeywords: plant, animal, food, cyber biosecurity, biosecurity, cybersecurity, agriculture, bioeconomy\n\nIntroduction: Food and agriculture cyberbiosecurity at the interface of biosecurity and cybersecurity \nPublic trust and confidence in the food supply are critical and influential on acceptance of data-driven innovations and technologies within the food and agriculture systems. Cyberbiosecurity is a nascent paradigm and discipline at the interface of biosafety\/biosecurity, cybersecurity, and cyber-physical security (Figure 1).[1] This new discipline has emerged alongside \u201cbig data\u201d with the extensive and ever-increasing reliance of the life sciences on information systems technologies, rapid and profitable expansion of life science discoveries, and the growth of the U.S. bioeconomy. Protecting biological data and information within the life sciences has unique differences from the more familiar biosafety and biosecurity approaches.[2] While the latter two categories address biological risks and threats, they do not protect against harm created when computational and information technology-dependent systems are threatened or corrupted. Just as food safety regulations target the protection of human health, incorporating cyber biosecurity strategies for the food and agriculture industries is a protective step in securing the food supply. Such efforts have the power to positively influence lives and protect the bioeconomy. Cyberbiosecurity can improve the security and stability of domestic and global food and agriculture systems. United States innovation in this realm is routinely studied and adopted around the globe, and as such, the U.S. can provide insight and leadership in cyber biosecurity of global food and agriculture systems.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 Cyberbiosecurity is an emerging discipline for protecting life sciences data, functions and operations (or infrastructure), and the bioeconomy\n\n\n\nIntegrated scientific, mathematical, computational, and engineering advancements in regenerative biology, genetics and breeding technologies, plant-derived vaccine and animal therapies, biological design and testing automation, and other activities are rapidly leading to development of biotechnological and agricultural applications of direct relevance to the food and agriculture system.[3][4] The translation and application of data-driven technologies for precision agriculture, autonomous systems, bio-automated processing and data recording, and other technologies yields large data sets of economic and bio-based information for agribusinesses.[5] Such advances require high throughput processing, data management and integration, bio-automation, and other computer-based management of biological data. These advances increase efficiencies, decision processes, and output within the food and agricultural system. However, such information is susceptible to ownership policy challenges, theft, and cyberattack as users may not be alert to potential vulnerabilities nor be trained in effective protections and security strategies.[5][6] Unprotected or weakly protected systems are susceptible to unwanted surveillance, intrusions into data systems, and cyber-activities targeted toward malicious attack. Cyberbiosecurity threats include inappropriate access to systems, data, or analytical technologies and the use or corruption of the information accessed to cause harm within life science-focused research, production, processing, and use. Examples of data-driven, high-value food and agricultural products susceptible to cyber threat include high-yielding and specialty agricultural crops, high performance livestock, biopharma fermented molecules developed through advanced breeding and genomics, biotechnology advancements, and \u201cbig data\u201d analyses.[7] As technology advances, all parts of society, from governmental agencies to public health and manufacturing, rely more on advanced biological systems with big data and technologies that utilize such information. The identification and mitigation of cyber biosecurity threats will become increasingly important.\n\nVulnerability of the food and agricultural system and the bioeconomy \nThe U.S. food and agriculture system, influencing 20% ($6.7T) of the domestic bioeconomy[8], represents a significant risk to global food security. The data science market value for agriculture is estimated in excess of $20B.[5] The food and agriculture system is composed of many sectors that are not well-integrated, is widely dispersed geographically, and has huge diversity in size (number of employees) and capacity. Most of the economic value is generated by large, multinational corporate enterprises. Conversely, small family-owned farming operations account for 90% of U.S. farms, which yield 24% of the value of agricultural production.[9]\nThe family small-business agricultural enterprise (family farm) has economic and social distinctions from corporate farms. Small farm producers view their data with a sense of personal privacy and protection.[5] Small businesses often use their internet-linked home computer for both personal and business activities, increasing the risk of cyber-attack[10][11]; over 20% of small businesses get hacked.[11] Generally, small farms and agribusinesses are not comfortable adopting computer security technology (selecting, configuring, managing) although they recognize its relevance and value. Moderate-sized agribusinesses, including many food processing companies and supporting industries, are vulnerable since cyberattacks are often targeted against organizations with more than 100 employees.[11] Additionally, the food and agriculture system includes military food production\u2014such as the manufacturing of packaged meals for soldiers\u2014which has a high potential for sabotage.[12] It is important to note that attackers need not know details of the food manufacturing process. Attackers need only know technical methods for exploiting the machinery or the process, such as lowering the temperature on meat cookers before packaging.[13][14]\nThe incorporation of cyber-based technologies and data driven solutions in farm production, food processing, supplier industries, transport of goods, regulatory oversight, and marketing sales and communication with consumers creates a paradigm shift.[6] Cloud-based storage of large data sets, use of open-sourced or internet\/cloud-based software, and corporate management of proprietary software each increase opportunities for data access by unauthorized users. Within the food and agriculture system, the use of biological and genetic analytical technologies within research laboratories is widespread for the evaluation of food quality, identification of zoonotic disease, and animal and plant health. Additionally, the use of bioinformatics and genetic technologies is enhancing the rate of development of new products and crops. Public trust and acceptance are key to incorporating advanced technologies into the food and agriculture system.[4][15] Interdependency of information technology with biological output creates opportunities for new bio-threats, which can harm public trust; transparency is valued.[7] When public opinion is turned against a technical advancement, policy and protection strategies may cause more harm than the actual threat itself.[4]\nHolistically, the ramifications of a failure to provide cyber biosecurity of the food and agriculture system fall into several general categories[6]:\n\n Threats to confidentiality\u2014data privacy\n Data exposure (e.g., na\u00efve exposure of data by individuals, cybersecurity gaps in small businesses, or laboratories to potential threats)\n Capturing private data with intent to aggregate data for profit or predictive advantage\n Threats to integrity\u2014theft or destruction of intellectual property\/productivity disruptions, and safety risks\n Intellectual property theft (e.g., advances in plant and animal varieties and genetics)\n Manipulation of critical automated (computer-based) processes (e.g., thermal processing time and temperature for food safety)\n Seizing control of robotics or autonomous vehicles (e.g., failure to perform, overriding of precise function)\n Threats to availability\u2014disruption of agricultural\/food production and supply\n Misinformation influencing trust and cooperation within the food and agriculture system and\/or consumers\n Lack of equipment, supplies, or end-products to meet expectations\n Lack of ability to perform vulnerability assessments and develop emergency response plans (e.g., protection of rivers, surface waters, and drinking water supplies)\nThe food and agricultural industries are at a critical point as the development and use of biological, genetic, precision, and information technologies expand and intersect. Collectively, there is a need to evaluate potential liabilities and understand the vulnerabilities of biological and genetic data systems.\n\nRisk assessment, critical control points, and regulatory options \nCybersecurity risk assessment for industrial control systems (ICS) is advancing rapidly. Cherdantseva et al.[16] reviewed 24 different cybersecurity risk assessment methods relevant to ICS. Applications of such risk assessment approaches in the food and agriculture sectors have not been evaluated, and the complexity and diversity of those sectors may not conform to the current cybersecurity risk assessment methods. Cyberbiosecurity risk assessment strategies that address the unique security challenges at the intersection of the biological, physical, and cyberspace are important for protecting the food and agriculture system.\nFood manufacturers use the principles of Hazard Analysis and Critical Control Points (HACCP) to assure the production of safe products. HACCP is a familiar risk assessment process within the food and agriculture system. This management system looks at the likely occurrence of a chemical, biological, or physical food safety hazard in the manufacturing process and the controls that can be put in place to reduce, eliminate, or control the potential hazard. HACCP principles use critical control points (CCPs) as steps in a process where specific controls can be implemented to control, reduce, or eliminate a hazard. HACCP principles are used around the world for the production of safe food products and are required by the U.S. Department of Agriculture's Food Safety Inspection Service and the U.S. Food and Drug Administration. A risk matrix (see Table 1 in the Supplemental Material) may be used to identify potential vulnerabilities and estimate likelihood of occurrence with the potential public health and financial consequences. An example using HACCP principles for an assessment of an industrial laboratory processing biological and genetic materials is presented in the Supplemental Material. In this specific example, two CCPs (alternative supplier verification of biological and genetic materials program, and cyber biosecurity data verification program) were identified to mitigate potential risks. Four control point programs (supplier approval, employee training, security programs, and good laboratory standard operating procedures) were identified to support the overarching process for cyber biosecurity.\nSeveral economic problems confront policymakers when addressing cyber biosecurity in the food and agriculture sector. The most pressing concerns are externalities caused by the networked nature of the system and the misaligned incentives of individual agents. The risks associated with cyber biosecurity threats and harm to society are likely to be larger than the losses suffered by an individual entity; individual firms may not have incentives to provide socially optimal levels of security for the network. Furthermore, if agents know that their own protection depends on security investments made by others, they may become free-riders. Again, this results in inadequate private provision of the public good or security of the network.[17]\nMultiple regulatory and policy options exist to counter threats to the food and agriculture system. In some cases, it may be easier to implement protections within these sectors because agribusinesses are already subject to relatively strict disclosure regulations. Information disclosure provides regulators with the data necessary to align individual incentives with the security of the system as a whole. This could be done with top-down regulation, changes to the assignment of liability, or the development of market-based systems for the control of cyber biosecurity risks. For instance, the development of cyber biosecurity insurance markets could be encouraged. Regardless of eventual policy measures, it will be important to ensure that the costs of protecting the system are properly aligned with the probabilities of loss and magnitudes of loss associated with cyber biosecurity threats. The most efficient methods of securing the food and agriculture system are likely to rely on a variety of regulatory approaches.\n\nConsidering the diversity within and across plant, animal, and environmental sectors of the food and agricultural system \nThe HACCP concept assesses risk and establishes CCPs for a specific facility and cannot be generalized effectively to all food manufacturing plants. Applying this concept to cyber biosecurity risk, control points, and CCPs, therefore, is challenged by the diversity of enterprises within a sector and across the food and agriculture system. Within each sector are unique suppliers providing biological material, chemicals and ingredients, robotics and machinery, software, data, and data storage systems. Some of the security measures are encompassed by cybersecurity, cyberphysical security, and biosecurity\/biosafety practices, at least for large corporate entities with sufficient resources. However, an unsecured system from a small agribusiness supplier, producer, processor, or commodity cooperative could introduce risk.\nWe use the illustration of a train with multiple boxcars as an example of various sectors within one commodity sector of the food and agriculture system (Figure 2, top). The various cars represent the transition from genetics and breeding through production, processing, distribution, and consumer purchase\/use. The exchange of information between the different sectors is often limited, as illustrated by the couplings. The role of the federal government policies and programs provide support and guidance (tracks). Suppliers and other support systems access one or more sectors within a commodity system. The system is driven (engine) by general public (consumers) acceptance of practices and goods, or their fear and mistrust if a risk or threat is perceived. If any stage \u201cderails\u201d or if any supporting agency or organization \u201cbuckles\u201d due to a cyber biosecurity threat or attack, the entire system is at risk, with subsequent risk to the U.S. food supply and the bioeconomy (Figure 2, bottom). Currently, the cybersecurity industry is not visibly involved in protecting biological data interfacing with the cyber-physical infrastructure supporting the food and agriculture system.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 (Top) Food and agriculture system for each commodity sector is a sequence of stages, with limited communications and sharing of data between each; (Bottom) if a cyber-biosecurity event occurs, it can have catastrophic effect on the entire food and agriculture system.\n\n\n\nSome potential mitigations to the issues are possible. Cyber biosecurity planning and implementation are needed to protect the intellectual and physical (data) property associated with such food and agriculture priorities. Examples include:\n\n Plant and animal germplasm, such as old world corn germplasm, microbiology collection (pathogens, fermentation, microbiome) repositories, including economic assessment and protection of data sharing;\n Biocontrolled systems or processes, such as \u201csmart\u201d technology greenhouse data;\n Animal and plant disease diagnostic networks and information sharing;\n Fermentation processing and thermal processing control parameters; and\n Freshwater and drinking water supplies and treatment systems.\nWe further illustrate by outlining some unique considerations for various food and agriculture commodities.\n\nDairy \nSelection of genetics for breeding is key to the high milk production in the U.S. dairy industry. Genetic data is highly evaluated as part of the process for breeding. Milk production records are important for establishing high-performance animals. While there are some very large dairy herds (>2,000 animals), the U.S. dairy industry is dominated by small to medium farms, many of whom sell their milk through a cooperative structure. Herd health records and drug use are regulated. Data security is variable, and often limited. Fluid milk and dairy food processors do not have detailed records of individual cow production or farm production practices, creating a gap in tracing of information and potential for data breach. Processors utilize computer systems for maintaining processing temperatures, ingredient additions, sanitizing, and cleaning steps.\n\nFood animals \nSelective breeding is critical to maximize genetic gain during food animal production. For instance, multiple lines of breeds are incorporated into swine production to enhance heterogeneity. Pedigree information of the breeds significantly influences selection of founders for the production system. Breach or manipulation of the information can lead to a devastating loss to producers. Recent development in genomic-based selection strategies[18] may also be vulnerable to cyber biosecurity threats as the genomic information can be targeted or exploited. Potential application of genome editing technology in food animals[19] may also generate novel genetic information that could dramatically improve productivity of food animals.\n\nRow crops \nSimilar to the dairy industry, the row crop sector consists of a large number of farms of varying size. Grain is typically commingled at the first point of sale and often aggregated further during the process of storage and handling, greatly limiting traceability.[20] Modern farms using precision agriculture technologies generate enormous amounts of data, about everything from soil conditions to machinery performance and location; such information is often controlled by agriculture technology providers.[5][6] Securing data and preventing breaches across all these systems is difficult and is frequently an afterthought by the actual users.[21] Individual producer data is often sent directly to a third-party entity for data storage, cleaning, and processing. Many aggregate data and use this as market information or sell it to other companies who do. Commodity traders may use some data streams to guide investment. Anonymization typically occurs at the time of aggregation, but questions exist about the effectiveness of these techniques. After transfer, data security becomes the responsibility of the third-party data management company, but these entities are themselves not immune from security breaches and would be vulnerable to security issues inserted upstream at the farm or machinery level. Finally, commodity markets are strongly influenced by crop production estimates generated by surveys of farmers and the agriculture industry.\n\nFruits and vegetables \nFresh fruits and vegetables are leading sources for foodborne illness in the United States.[22] Furthermore, even in the absence of foodborne illness outbreaks, fresh produce recalls occur regularly due to the presence of potential harmful microorganisms. Fresh produce available for sale in local markets may have been produced in one of many locations throughout the nation or from one of many countries around the world. The production, sorting, grading, commingling, transporting, marketing, and sale of fresh fruits and vegetables is complex and involves numerous industry actors with varying roles. Tracking fresh produce from initial production through consumption is critical to limit the potential for and impact of foodborne illness outbreaks. Accurate product information and rapid access to data is essential to identify contaminated product in the market, prevent or limit foodborne illness, limit the damage to non-implicated producers, and maintain consumer confidence. Access to product tracking and microbiological data is increasing in the fresh produce industry.\n\nEnvironmental resources (water) \nDrinking water safety is extremely important on-farm, for food processing, ensuring consumers' health and proper functioning of the ecosystem. The proportion of the world's population consuming drinking water from certified and controlled water sources is about 90% and still increasing.[23] However, 2.3 billion people worldwide suffer from diseases related to drinking water. Over the past three decades, significant drinking water contamination incidents have occurred in developing as well as developed countries, creating health problems for consumers.[24][25] Traditional risk management systems, based on addressing and correcting the failure after its occurrence, are inadequate to deal with potential cyber biosecurity threats (as the cybersecurity landscape is changing rapidly as technology continues to advance). Given the severity of risk and potential harm, cyber biosecurity must be given a high priority for the drinking water management and treatment sector.[26]\n\nConclusions: Moving Toward Solutions \nThe complex and vastly diverse enterprises within the food and agriculture system increases vulnerability of our food supply and threatens our ability to contribute to the global food supply. Rapid advancements in technologies and adoption into the food and agriculture sectors increase the risks for cyber biosecurity threats and attacks. The current food and agriculture workforce has limited knowledge or training appropriate to evaluate and protect the vast amount of data generated by these technologies. The cybersecurity industry is not well-prepared to address the unique structure and functions within food and agriculture sectors. Protecting them includes (1) developing and characterizing effective cyber biosecurity risk assessment and mitigation strategies; (2) developing and preparing the current and future workforce to identify, address, and adopt effective cyber biosecurity strategies; (3) considering policy and regulations, including insurance, for protection within and across the entire system; and (4) effectively communicating within the sector and across the food and agriculture system.[15] Awareness, knowledge, adoption, and frequent evaluation of cyber biosecurity plans and strategies among and within all sectors is essential. A multidisciplinary approach integrating expertise in agriculture, food, engineering, computer science, and cybersecurity is needed for filling this gap. The USDA, in consultation with academic, public, and private sector experts and representation from sectors within the food and agriculture system, should lead an initiative for developing a planned approach to addressing cyber biosecurity. Private and public funding is needed to support research priorities and implementation strategies. Checkoff funding mechanisms or cooperative agreements, which are common within commodity systems, may be options for assisting small to moderate-sized agribusinesses. Workforce development, effective communication strategies, and cooperation across sectors and industries will help increase support and compliance, reducing the risks and providing increased protection for the U.S. bioeconomy and our domestic and global food supply.\n\nSupplementary material \nThe Supplementary Material for this article can be found online at: https:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00063\/full#supplementary-material\n\nAcknowledgements \nThe authors acknowledge the Virginia Agricultural Experiment Station, Blacksburg, VA for financial support for publishing this manuscript. The authors gratefully acknowledge Michael J. Stamper, Data Visualization Designer and lecturer at the University Libraries, Data Services at Virginia Tech, for his contributions to the figures.\n\nAuthor Contributions \nSD lead author, responsible for structure, content, and figure; responsible for considering, incorporation co-author contributions and suggested edits; responsible for final version. RR provided draft content related to HACCP and post-harvest processing and cyber biosecurity; contributed, reviewed, and edited the manuscript. RW contributed, reviewed, and edited content related to HACCP and post-harvest processing and biosecurity; reviewed and critiqued manuscript to ensure quality and flow. FR contributed, reviewed, and edited content related to food and agriculture system influence on bioeconomy; reviewed and critiqued manuscript to ensure quality and flow. WT contributed, reviewed, and edited content related to cyber biosecurity in agriculture (pre-harvest; crop, soil, and environment); reviewed and critiqued manuscript to ensure quality and flow. KL contributed, reviewed, and edited content related to cyber biosecurity in agriculture (pre-harvest; animal breeding and genetics); reviewed and critiqued manuscript to ensure quality and flow. ND contributions to sections relating to biotechnology; overall quality assurance and readability reviews and modifications. SM contributed, reviewed, and edited content related to cyber biosecurity in food and agricultural system and the environment; reviewed and critiqued manuscript to ensure quality and flow. EC contributed, reviewed, and edited content related to cybersecurity, data sources, and integration into the food and agricultural system; reviewed and critiqued manuscript to ensure quality and flow. RM co-originator of the cyber biosecurity concept; co-originator of the concepts relating to food and agricultural system; contributed, reviewed, and edited content related to cyber biosecurity, data sources, and integration into the food and agricultural system; reviewed and critiqued manuscript to ensure quality, flow, and relevance to the targeted audience.\n\nConflict of interest \nRR is employed by Tyson Foods.\nThe remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Murch, R.S.; So, W.K.; Buchholz, W.G. et al. (2018). \"Cyberbiosecurity: An Emerging New Discipline to Help Safeguard the Bioeconomy\". Frontiers in Bioengineering and Biotechnology 6: 39. doi:10.3389\/fbioe.2018.00039.   \n\n\u2191 Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". Trends in Biotechnology 36 (1): 4\u20137. doi:10.1016\/j.tibtech.2017.10.012. PMID 29224719.   \n\n\u2191 Board on Chemical Sciences and Technology; Board on Life Sciences (2014). Meeting Recap: Workshop - Convergence: Safeguarding Technology in the Bioeconomy. The National Academies of Sciences, Engineering, and Medicine.   \n\n\u2191 4.0 4.1 4.2 Wintle, B.C.; Boehm, C.R.; Rhodes, C. et al. (2017). \"A transatlantic perspective on 20 emerging issues in biological engineering\". eLife 3: e30247. doi:10.7554\/eLife.30247. PMC PMC5685469. PMID 29132504. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469 .   \n\n\u2191 5.0 5.1 5.2 5.3 5.4 Sykuta, M.E. (2016). \"Big data in agriculture: Property rights, privacy and competition in ag data services\". International Food and Agribusiness Management Review 19 (A): 57\u201374. https:\/\/www.ifama.org\/Volume-19-Issue-A .   \n\n\u2191 6.0 6.1 6.2 6.3 Boghossian, A.; Linsky, S.; Brown, A. et al. (2018). \"Threats to Precision Agriculture\" (PDF). U.S. Department of Homeland Security. https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/2018%20AEP_Threats_to_Precision_Agriculture.pdf . Retrieved 08 January 2019 .   \n\n\u2191 7.0 7.1 Board on Chemical Sciences and Technology (2015) (PDF). Meeting Recap: Safeguarding the Bioeconomy: Applications and Implications of Emerging Science. The National Academies of Sciences, Engineering, and Medicine. https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf .   \n\n\u2191 \"What is the Food and Ag Industries\u2019 Impact in Your Community?\". Feeding the Economy. 2018. https:\/\/feedingtheeconomy.com\/ . Retrieved 28 October 2018 .   \n\n\u2191 MacDonald, J.M.; Hoppe, R.A. (06 March 2017). \"Large Family Farms Continue To Dominate U.S. Agricultural Production\". Amber Waves. U.S. Department of Agriculture. https:\/\/www.ers.usda.gov\/amber-waves\/2017\/march\/large-family-farms-continue-to-dominate-us-agricultural-production\/ . Retrieved 28 October 2018 .   \n\n\u2191 National Agricultural Statistics Service (20 August 2013). \"Farm Computer Usage and Ownership\". U.S. Department of Agriculture. https:\/\/usda.library.cornell.edu\/concern\/publications\/h128nd689?locale=en . Retrieved 27 October 2018 .   \n\n\u2191 11.0 11.1 11.2 Geil, A.; Sagers, G.; Spaulding, A.D. et al.. \"Cyber security on the farm: An assessment of cyber security practices in the United States agriculture industry\". International Food and Agribusiness Management Review 21: 317\u201334. doi:10.22434\/IFAMR2017.0045.   \n\n\u2191 Colbert, E.J.M.; Kott, A.; Knachel, L.P.. \"The game-theoretic model and experimental investigation of cyber wargaming\". The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology: 1\u201318. doi:10.1177\/1548512918795061.   \n\n\u2191 Colbert, E.; Sullivan, D.; Wong, K. et al. (2015). Table-Top Exercise Final Report: Intrusion Detection Capabilities for US Army SCADA Systems: Information Packet. US Army Research Lab Technical Report ARL-TR-7498.   \n\n\u2191 Colbert, E.; Sullivan, D.; Wong, K. et al. (2015). RED and BLUE Teaming of a US Army SCADA System: Table-Top Exercise Final Report. US Army Research Lab Technical Report ARL-TR-7497.   \n\n\u2191 15.0 15.1 National Institute of Food and Agriculture (01 November 2016). \"Data Summit: Changing the Face, Place, and Space of Agriculture\" (PDF). U.S. Department of Agriculture. https:\/\/nifa.usda.gov\/sites\/default\/files\/resource\/Stakeholder%20Ideas%20Engine%20Input%20-%20Summary%5B1%5D.pdf . Retrieved 28 October 2018 .   \n\n\u2191 Cherdantseva, Y.; Burnap, P.; Blyth, A. et al. (2016). \"A review of cybersecurity risk assessment methods for SCADA systems\". Computers & Security 56: 1\u201327. doi:10.1016\/j.cose.2015.09.009.   \n\n\u2191 Varian, H. (2004). \"System Reliability and Free Riding\". In Camp, L.J.; Lewis, S.. Economics of Information Security. Advances in Information Security. 12. Springer. pp. 1\u201315. doi:10.1007\/1-4020-8090-5_1. ISBN 9781402080906.   \n\n\u2191 Sellner, E.M.; Kim, J.W.; McClure, M.C. et al. (2007). \"Board-invited review: Applications of genomic information in livestock\". Journal of Animal Science 85: 12. doi:10.2527\/jas.2007-0291. PMID 17709778.   \n\n\u2191 Telugu, B.P.; Park, K.E.; Park, C.H. (2017). \"Genome editing and genetic engineering in livestock for advancing agricultural and biomedical applications\". Mammalian Genome 28 (7\u20138): 338\u201347. doi:10.1007\/s00335-017-9709-4. PMID 28712062.   \n\n\u2191 Golan, E.; Krissoff, B.; Kuchler, F. et al. (March 2004). \"Traceability in the U.S. Food Supply: Economic Theory and Industry Studies\". USDA Economic Research Service. https:\/\/www.ers.usda.gov\/publications\/pub-details\/?pubid=41632 .   \n\n\u2191 Ferris, J.L. (2017). \"Data Privacy and Protection in the Agriculture Industry: Is Federal Regulation Necessary?\". Minnesota Journal of Law, Science & Technology 18 (1): 309\u201342. https:\/\/scholarship.law.umn.edu\/mjlst\/vol18\/iss1\/6 .   \n\n\u2191 Callej\u00f3n, R.M.; Rodr\u00edguez-Naranjo, M.I.; Ubeda, C. et al. (2015). \"Reported foodborne outbreaks due to fresh produce in the United States and European Union: Trends and causes\". Foodborne Pathogens and Disease 12 (1): 32\u20138. doi:10.1089\/fpd.2014.1821. PMID 25587926.   \n\n\u2191 Vierira, J.M. (2011). \"A strategic approach for Water Safety Plans implementation in Portugal\". Journal of Water and Health 9 (1): 107\u201316. doi:10.2166\/wh.2010.150. PMID 21301119.   \n\n\u2191 Hamilton, P.D.; Gale, P.; Pollard, S.J. (2006). \"A commentary on recent water safety initiatives in the context of water utility risk management\". Environment International 32 (8): 958\u201366. doi:10.1016\/j.envint.2006.06.001. PMID 16870255.   \n\n\u2191 Tsoukalas, D.S.; Tsitsifli, S. (2018). \"A Critical Evaluation of Water Safety Plans (WSPs) and HACCP Implementation in Water Utilities\". Proceedings 2 (11): 600. doi:10.3390\/proceedings2110600.   \n\n\u2191 Germano, J.H. (2018). \"Cybersecurity Risk and Responsibility in the Water Sector\" (PDF). American Water Works Association. https:\/\/www.awwa.org\/Portals\/0\/AWWA\/Government\/AWWACybersecurityRiskandResponsibility.pdf?ver=2018-12-05 . Retrieved 08 January 2019 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\">https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on agricultureLIMSwiki journal articles on cybersecurity\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 9 April 2019, at 15:02.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 406 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","e69b956327faef5b93c17a708e883f73_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Cyberbiosecurity_A_new_perspective_on_protecting_U_S_food_and_agricultural_system skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Cyberbiosecurity: A new perspective on protecting U.S. food and agricultural system<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Our national data and infrastructure security issues affecting the \u201cbioeconomy\u201d are evolving rapidly. Simultaneously, the conversation about cybersecurity of the U.S. <a href=\"https:\/\/www.limswiki.org\/index.php\/Agriculture_industry\" title=\"Agriculture industry\" class=\"wiki-link\" data-key=\"4882fd1b1f6fb6017adf6f0c0741eafc\">food and agricultural system<\/a> (cyber biosecurity) is incomplete and disjointed. The food and agricultural production sectors influence over 20% of the nation's economy ($6.7T) and 15% of U.S. employment (43.3M jobs). The food and agricultural sectors are immensely diverse, and they require advanced technologies and efficiencies that rely on computer technologies, big data, <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud-based<\/a> data storage, and internet accessibility. There is a critical need to safeguard the cyber biosecurity of our bioeconomy, but currently protections are minimal and do not broadly exist across the food and agricultural system. Using the food safety management Hazard Analysis Critical Control Point (HACCP) system concept as an introductory point of reference, we identify important features in broad food and agricultural production and food systems: dairy, food animals, row crops, fruits and vegetables, and environmental resources (water). This analysis explores the relevant concepts of cyber biosecurity from food production to the end product user (such as the consumer) and considers the integration of diverse transportation, supplier, and retailer networks. We describe common challenges and unique barriers across these systems and recommend solutions to advance the role of cyber biosecurity in the food and agricultural sectors.\n<\/p><p><b>Keywords<\/b>: plant, animal, food, cyber biosecurity, biosecurity, cybersecurity, agriculture, bioeconomy\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction:_Food_and_agriculture_cyberbiosecurity_at_the_interface_of_biosecurity_and_cybersecurity\">Introduction: Food and agriculture cyberbiosecurity at the interface of biosecurity and cybersecurity<\/span><\/h2>\n<p>Public trust and confidence in the food supply are critical and influential on acceptance of data-driven innovations and technologies within the food and agriculture systems. Cyberbiosecurity is a nascent paradigm and discipline at the interface of biosafety\/biosecurity, cybersecurity, and cyber-physical security (Figure 1).<sup id=\"rdp-ebb-cite_ref-MurchCyber18_1-0\" class=\"reference\"><a href=\"#cite_note-MurchCyber18-1\">[1]<\/a><\/sup> This new discipline has emerged alongside \u201cbig data\u201d with the extensive and ever-increasing reliance of the life sciences on information systems technologies, rapid and profitable expansion of life science discoveries, and the growth of the U.S. bioeconomy. Protecting biological data and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> within the life sciences has unique differences from the more familiar biosafety and biosecurity approaches.<sup id=\"rdp-ebb-cite_ref-PeccoudCyber18_2-0\" class=\"reference\"><a href=\"#cite_note-PeccoudCyber18-2\">[2]<\/a><\/sup> While the latter two categories address biological risks and threats, they do not protect against harm created when computational and information technology-dependent systems are threatened or corrupted. Just as food safety regulations target the protection of human health, incorporating cyber biosecurity strategies for the food and agriculture industries is a protective step in securing the food supply. Such efforts have the power to positively influence lives and protect the bioeconomy. Cyberbiosecurity can improve the security and stability of domestic and global food and agriculture systems. United States innovation in this realm is routinely studied and adopted around the globe, and as such, the U.S. can provide insight and leadership in cyber biosecurity of global food and agriculture systems.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Duncan_FrontBioengBiotech2019_7.jpg\" class=\"image wiki-link\" data-key=\"8c95db2f356d21ae7e1d6a2206decfeb\"><img alt=\"Fig1 Duncan FrontBioengBiotech2019 7.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/37\/Fig1_Duncan_FrontBioengBiotech2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> Cyberbiosecurity is an emerging discipline for protecting life sciences data, functions and operations (or infrastructure), and the bioeconomy<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Integrated scientific, mathematical, computational, and engineering advancements in regenerative biology, genetics and breeding technologies, plant-derived vaccine and animal therapies, biological design and testing automation, and other activities are rapidly leading to development of biotechnological and agricultural applications of direct relevance to the food and agriculture system.<sup id=\"rdp-ebb-cite_ref-NASEMMeeting14_3-0\" class=\"reference\"><a href=\"#cite_note-NASEMMeeting14-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WintleATrans17_4-0\" class=\"reference\"><a href=\"#cite_note-WintleATrans17-4\">[4]<\/a><\/sup> The translation and application of data-driven technologies for precision agriculture, autonomous systems, bio-automated processing and data recording, and other technologies yields large data sets of economic and bio-based information for agribusinesses.<sup id=\"rdp-ebb-cite_ref-SykutaBig16_5-0\" class=\"reference\"><a href=\"#cite_note-SykutaBig16-5\">[5]<\/a><\/sup> Such advances require high throughput processing, data management and integration, bio-automation, and other computer-based management of biological data. These advances increase efficiencies, decision processes, and output within the food and agricultural system. However, such information is susceptible to ownership policy challenges, theft, and cyberattack as users may not be alert to potential vulnerabilities nor be trained in effective protections and security strategies.<sup id=\"rdp-ebb-cite_ref-SykutaBig16_5-1\" class=\"reference\"><a href=\"#cite_note-SykutaBig16-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BoghossianThreats18_6-0\" class=\"reference\"><a href=\"#cite_note-BoghossianThreats18-6\">[6]<\/a><\/sup> Unprotected or weakly protected systems are susceptible to unwanted surveillance, intrusions into data systems, and cyber-activities targeted toward malicious attack. Cyberbiosecurity threats include inappropriate access to systems, data, or analytical technologies and the use or corruption of the information accessed to cause harm within life science-focused research, production, processing, and use. Examples of data-driven, high-value food and agricultural products susceptible to cyber threat include high-yielding and specialty agricultural crops, high performance livestock, biopharma fermented molecules developed through advanced breeding and genomics, biotechnology advancements, and \u201cbig data\u201d analyses.<sup id=\"rdp-ebb-cite_ref-NASEMSafe15_7-0\" class=\"reference\"><a href=\"#cite_note-NASEMSafe15-7\">[7]<\/a><\/sup> As technology advances, all parts of society, from governmental agencies to public health and manufacturing, rely more on advanced biological systems with big data and technologies that utilize such information. The identification and mitigation of cyber biosecurity threats will become increasingly important.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Vulnerability_of_the_food_and_agricultural_system_and_the_bioeconomy\">Vulnerability of the food and agricultural system and the bioeconomy<\/span><\/h2>\n<p>The U.S. food and agriculture system, influencing 20% ($6.7T) of the domestic bioeconomy<sup id=\"rdp-ebb-cite_ref-FEWhatIs18_8-0\" class=\"reference\"><a href=\"#cite_note-FEWhatIs18-8\">[8]<\/a><\/sup>, represents a significant risk to global food security. The data science market value for agriculture is estimated in excess of $20B.<sup id=\"rdp-ebb-cite_ref-SykutaBig16_5-2\" class=\"reference\"><a href=\"#cite_note-SykutaBig16-5\">[5]<\/a><\/sup> The food and agriculture system is composed of many sectors that are not well-integrated, is widely dispersed geographically, and has huge diversity in size (number of employees) and capacity. Most of the economic value is generated by large, multinational corporate enterprises. Conversely, small family-owned farming operations account for 90% of U.S. farms, which yield 24% of the value of agricultural production.<sup id=\"rdp-ebb-cite_ref-MacDonaldLarge17_9-0\" class=\"reference\"><a href=\"#cite_note-MacDonaldLarge17-9\">[9]<\/a><\/sup>\n<\/p><p>The family small-business agricultural enterprise (family farm) has economic and social distinctions from corporate farms. Small farm producers view their data with a sense of personal privacy and protection.<sup id=\"rdp-ebb-cite_ref-SykutaBig16_5-3\" class=\"reference\"><a href=\"#cite_note-SykutaBig16-5\">[5]<\/a><\/sup> Small businesses often use their internet-linked home computer for both personal and business activities, increasing the risk of cyber-attack<sup id=\"rdp-ebb-cite_ref-NASSFarm13_10-0\" class=\"reference\"><a href=\"#cite_note-NASSFarm13-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GeilCyber18_11-0\" class=\"reference\"><a href=\"#cite_note-GeilCyber18-11\">[11]<\/a><\/sup>; over 20% of small businesses get hacked.<sup id=\"rdp-ebb-cite_ref-GeilCyber18_11-1\" class=\"reference\"><a href=\"#cite_note-GeilCyber18-11\">[11]<\/a><\/sup> Generally, small farms and agribusinesses are not comfortable adopting computer security technology (selecting, configuring, managing) although they recognize its relevance and value. Moderate-sized agribusinesses, including many food processing companies and supporting industries, are vulnerable since cyberattacks are often targeted against organizations with more than 100 employees.<sup id=\"rdp-ebb-cite_ref-GeilCyber18_11-2\" class=\"reference\"><a href=\"#cite_note-GeilCyber18-11\">[11]<\/a><\/sup> Additionally, the food and agriculture system includes military food production\u2014such as the manufacturing of packaged meals for soldiers\u2014which has a high potential for sabotage.<sup id=\"rdp-ebb-cite_ref-ColbertTheGame18_12-0\" class=\"reference\"><a href=\"#cite_note-ColbertTheGame18-12\">[12]<\/a><\/sup> It is important to note that attackers need not know details of the food manufacturing process. Attackers need only know technical methods for exploiting the machinery or the process, such as lowering the temperature on meat cookers before packaging.<sup id=\"rdp-ebb-cite_ref-ColbertTable15_13-0\" class=\"reference\"><a href=\"#cite_note-ColbertTable15-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ColbertRED15_14-0\" class=\"reference\"><a href=\"#cite_note-ColbertRED15-14\">[14]<\/a><\/sup>\n<\/p><p>The incorporation of cyber-based technologies and data driven solutions in farm production, food processing, supplier industries, transport of goods, regulatory oversight, and marketing sales and communication with consumers creates a paradigm shift.<sup id=\"rdp-ebb-cite_ref-BoghossianThreats18_6-1\" class=\"reference\"><a href=\"#cite_note-BoghossianThreats18-6\">[6]<\/a><\/sup> Cloud-based storage of large data sets, use of open-sourced or internet\/cloud-based software, and corporate management of proprietary software each increase opportunities for data access by unauthorized users. Within the food and agriculture system, the use of biological and genetic analytical technologies within research <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratories<\/a> is widespread for the evaluation of food quality, identification of zoonotic disease, and animal and plant health. Additionally, the use of <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> and genetic technologies is enhancing the rate of development of new products and crops. Public trust and acceptance are key to incorporating advanced technologies into the food and agriculture system.<sup id=\"rdp-ebb-cite_ref-WintleATrans17_4-1\" class=\"reference\"><a href=\"#cite_note-WintleATrans17-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NIFAData16_15-0\" class=\"reference\"><a href=\"#cite_note-NIFAData16-15\">[15]<\/a><\/sup> Interdependency of information technology with biological output creates opportunities for new bio-threats, which can harm public trust; transparency is valued.<sup id=\"rdp-ebb-cite_ref-NASEMSafe15_7-1\" class=\"reference\"><a href=\"#cite_note-NASEMSafe15-7\">[7]<\/a><\/sup> When public opinion is turned against a technical advancement, policy and protection strategies may cause more harm than the actual threat itself.<sup id=\"rdp-ebb-cite_ref-WintleATrans17_4-2\" class=\"reference\"><a href=\"#cite_note-WintleATrans17-4\">[4]<\/a><\/sup>\n<\/p><p>Holistically, the ramifications of a failure to provide cyber biosecurity of the food and agriculture system fall into several general categories<sup id=\"rdp-ebb-cite_ref-BoghossianThreats18_6-2\" class=\"reference\"><a href=\"#cite_note-BoghossianThreats18-6\">[6]<\/a><\/sup>:\n<\/p>\n<ul><li> Threats to confidentiality\u2014data privacy\n<ul><li> Data exposure (e.g., na\u00efve exposure of data by individuals, cybersecurity gaps in small businesses, or laboratories to potential threats)<\/li>\n<li> Capturing private data with intent to aggregate data for profit or predictive advantage<\/li><\/ul><\/li>\n<li> Threats to integrity\u2014theft or destruction of intellectual property\/productivity disruptions, and safety risks\n<ul><li> Intellectual property theft (e.g., advances in plant and animal varieties and genetics)<\/li>\n<li> Manipulation of critical automated (computer-based) processes (e.g., thermal processing time and temperature for food safety)<\/li>\n<li> Seizing control of robotics or autonomous vehicles (e.g., failure to perform, overriding of precise function)<\/li><\/ul><\/li>\n<li> Threats to availability\u2014disruption of agricultural\/food production and supply<\/li>\n<li> Misinformation influencing trust and cooperation within the food and agriculture system and\/or consumers<\/li>\n<li> Lack of equipment, supplies, or end-products to meet expectations<\/li>\n<li> Lack of ability to perform vulnerability assessments and develop emergency response plans (e.g., protection of rivers, surface waters, and drinking water supplies)<\/li><\/ul>\n<p>The food and agricultural industries are at a critical point as the development and use of biological, genetic, precision, and information technologies expand and intersect. Collectively, there is a need to evaluate potential liabilities and understand the vulnerabilities of biological and genetic data systems.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Risk_assessment.2C_critical_control_points.2C_and_regulatory_options\">Risk assessment, critical control points, and regulatory options<\/span><\/h2>\n<p>Cybersecurity risk assessment for industrial control systems (ICS) is advancing rapidly. Cherdantseva <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-CherdantsevaARev16_16-0\" class=\"reference\"><a href=\"#cite_note-CherdantsevaARev16-16\">[16]<\/a><\/sup> reviewed 24 different cybersecurity risk assessment methods relevant to ICS. Applications of such risk assessment approaches in the food and agriculture sectors have not been evaluated, and the complexity and diversity of those sectors may not conform to the current cybersecurity risk assessment methods. Cyberbiosecurity risk assessment strategies that address the unique security challenges at the intersection of the biological, physical, and cyberspace are important for protecting the food and agriculture system.\n<\/p><p>Food manufacturers use the principles of Hazard Analysis and Critical Control Points (HACCP) to assure the production of safe products. HACCP is a familiar risk assessment process within the food and agriculture system. This management system looks at the likely occurrence of a chemical, biological, or physical food safety hazard in the manufacturing process and the controls that can be put in place to reduce, eliminate, or control the potential hazard. HACCP principles use critical control points (CCPs) as steps in a process where specific controls can be implemented to control, reduce, or eliminate a hazard. HACCP principles are used around the world for the production of safe food products and are required by the <a href=\"https:\/\/www.limswiki.org\/index.php\/United_States_Department_of_Agriculture\" title=\"United States Department of Agriculture\" class=\"wiki-link\" data-key=\"6044177f939c200248410b8053502910\">U.S. Department of Agriculture<\/a>'s Food Safety Inspection Service and the U.S. <a href=\"https:\/\/www.limswiki.org\/index.php\/Food_and_Drug_Administration\" title=\"Food and Drug Administration\" class=\"wiki-link\" data-key=\"e2be8927071ac419c0929f7aa1ede7fe\">Food and Drug Administration<\/a>. A risk matrix (see Table 1 in the Supplemental Material) may be used to identify potential vulnerabilities and estimate likelihood of occurrence with the potential public health and financial consequences. An example using HACCP principles for an assessment of an industrial laboratory processing biological and genetic materials is presented in the Supplemental Material. In this specific example, two CCPs (alternative supplier verification of biological and genetic materials program, and cyber biosecurity data verification program) were identified to mitigate potential risks. Four control point programs (supplier approval, employee training, security programs, and good laboratory standard operating procedures) were identified to support the overarching process for cyber biosecurity.\n<\/p><p>Several economic problems confront policymakers when addressing cyber biosecurity in the food and agriculture sector. The most pressing concerns are externalities caused by the networked nature of the system and the misaligned incentives of individual agents. The risks associated with cyber biosecurity threats and harm to society are likely to be larger than the losses suffered by an individual entity; individual firms may not have incentives to provide socially optimal levels of security for the network. Furthermore, if agents know that their own protection depends on security investments made by others, they may become free-riders. Again, this results in inadequate private provision of the public good or security of the network.<sup id=\"rdp-ebb-cite_ref-VarianSystem04_17-0\" class=\"reference\"><a href=\"#cite_note-VarianSystem04-17\">[17]<\/a><\/sup>\n<\/p><p>Multiple regulatory and policy options exist to counter threats to the food and agriculture system. In some cases, it may be easier to implement protections within these sectors because agribusinesses are already subject to relatively strict disclosure regulations. Information disclosure provides regulators with the data necessary to align individual incentives with the security of the system as a whole. This could be done with top-down regulation, changes to the assignment of liability, or the development of market-based systems for the control of cyber biosecurity risks. For instance, the development of cyber biosecurity insurance markets could be encouraged. Regardless of eventual policy measures, it will be important to ensure that the costs of protecting the system are properly aligned with the probabilities of loss and magnitudes of loss associated with cyber biosecurity threats. The most efficient methods of securing the food and agriculture system are likely to rely on a variety of regulatory approaches.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Considering_the_diversity_within_and_across_plant.2C_animal.2C_and_environmental_sectors_of_the_food_and_agricultural_system\">Considering the diversity within and across plant, animal, and environmental sectors of the food and agricultural system<\/span><\/h2>\n<p>The HACCP concept assesses risk and establishes CCPs for a specific facility and cannot be generalized effectively to all food manufacturing plants. Applying this concept to cyber biosecurity risk, control points, and CCPs, therefore, is challenged by the diversity of enterprises within a sector and across the food and agriculture system. Within each sector are unique suppliers providing biological material, chemicals and ingredients, robotics and machinery, software, data, and data storage systems. Some of the security measures are encompassed by cybersecurity, cyberphysical security, and biosecurity\/biosafety practices, at least for large corporate entities with sufficient resources. However, an unsecured system from a small agribusiness supplier, producer, processor, or commodity cooperative could introduce risk.\n<\/p><p>We use the illustration of a train with multiple boxcars as an example of various sectors within one commodity sector of the food and agriculture system (Figure 2, top). The various cars represent the transition from genetics and breeding through production, processing, distribution, and consumer purchase\/use. The exchange of information between the different sectors is often limited, as illustrated by the couplings. The role of the federal government policies and programs provide support and guidance (tracks). Suppliers and other support systems access one or more sectors within a commodity system. The system is driven (engine) by general public (consumers) acceptance of practices and goods, or their fear and mistrust if a risk or threat is perceived. If any stage \u201cderails\u201d or if any supporting agency or organization \u201cbuckles\u201d due to a cyber biosecurity threat or attack, the entire system is at risk, with subsequent risk to the U.S. food supply and the bioeconomy (Figure 2, bottom). Currently, the cybersecurity industry is not visibly involved in protecting biological data interfacing with the cyber-physical infrastructure supporting the food and agriculture system.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Duncan_FrontBioengBiotech2019_7.jpg\" class=\"image wiki-link\" data-key=\"e29a5911be116fd7d7cd13ca34875433\"><img alt=\"Fig2 Duncan FrontBioengBiotech2019 7.jpg\" src=\"https:\/\/www.limswiki.org\/images\/d\/db\/Fig2_Duncan_FrontBioengBiotech2019_7.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> <b>(Top)<\/b> Food and agriculture system for each commodity sector is a sequence of stages, with limited communications and sharing of data between each; <b>(Bottom)<\/b> if a cyber-biosecurity event occurs, it can have catastrophic effect on the entire food and agriculture system.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Some potential mitigations to the issues are possible. Cyber biosecurity planning and implementation are needed to protect the intellectual and physical (data) property associated with such food and agriculture priorities. Examples include:\n<\/p>\n<ul><li> Plant and animal germplasm, such as old world corn germplasm, microbiology collection (pathogens, fermentation, microbiome) repositories, including economic assessment and protection of data sharing;<\/li>\n<li> Biocontrolled systems or processes, such as \u201csmart\u201d technology greenhouse data;<\/li>\n<li> Animal and plant disease diagnostic networks and information sharing;<\/li>\n<li> Fermentation processing and thermal processing control parameters; and<\/li>\n<li> Freshwater and drinking water supplies and treatment systems.<\/li><\/ul>\n<p>We further illustrate by outlining some unique considerations for various food and agriculture commodities.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Dairy\">Dairy<\/span><\/h3>\n<p>Selection of genetics for breeding is key to the high milk production in the U.S. dairy industry. Genetic data is highly evaluated as part of the process for breeding. Milk production records are important for establishing high-performance animals. While there are some very large dairy herds (>2,000 animals), the U.S. dairy industry is dominated by small to medium farms, many of whom sell their milk through a cooperative structure. Herd health records and drug use are regulated. Data security is variable, and often limited. Fluid milk and dairy food processors do not have detailed records of individual cow production or farm production practices, creating a gap in tracing of information and potential for data breach. Processors utilize computer systems for maintaining processing temperatures, ingredient additions, sanitizing, and cleaning steps.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Food_animals\">Food animals<\/span><\/h3>\n<p>Selective breeding is critical to maximize genetic gain during food animal production. For instance, multiple lines of breeds are incorporated into swine production to enhance heterogeneity. Pedigree information of the breeds significantly influences selection of founders for the production system. Breach or manipulation of the information can lead to a devastating loss to producers. Recent development in genomic-based selection strategies<sup id=\"rdp-ebb-cite_ref-SellnerBoard07_18-0\" class=\"reference\"><a href=\"#cite_note-SellnerBoard07-18\">[18]<\/a><\/sup> may also be vulnerable to cyber biosecurity threats as the genomic information can be targeted or exploited. Potential application of genome editing technology in food animals<sup id=\"rdp-ebb-cite_ref-TeluguGenome17_19-0\" class=\"reference\"><a href=\"#cite_note-TeluguGenome17-19\">[19]<\/a><\/sup> may also generate novel genetic information that could dramatically improve productivity of food animals.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Row_crops\">Row crops<\/span><\/h3>\n<p>Similar to the dairy industry, the row crop sector consists of a large number of farms of varying size. Grain is typically commingled at the first point of sale and often aggregated further during the process of storage and handling, greatly limiting traceability.<sup id=\"rdp-ebb-cite_ref-GolanTrace04_20-0\" class=\"reference\"><a href=\"#cite_note-GolanTrace04-20\">[20]<\/a><\/sup> Modern farms using precision agriculture technologies generate enormous amounts of data, about everything from soil conditions to machinery performance and location; such information is often controlled by agriculture technology providers.<sup id=\"rdp-ebb-cite_ref-SykutaBig16_5-4\" class=\"reference\"><a href=\"#cite_note-SykutaBig16-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BoghossianThreats18_6-3\" class=\"reference\"><a href=\"#cite_note-BoghossianThreats18-6\">[6]<\/a><\/sup> Securing data and preventing breaches across all these systems is difficult and is frequently an afterthought by the actual users.<sup id=\"rdp-ebb-cite_ref-FerrisData17_21-0\" class=\"reference\"><a href=\"#cite_note-FerrisData17-21\">[21]<\/a><\/sup> Individual producer data is often sent directly to a third-party entity for data storage, cleaning, and processing. Many aggregate data and use this as market information or sell it to other companies who do. Commodity traders may use some data streams to guide investment. Anonymization typically occurs at the time of aggregation, but questions exist about the effectiveness of these techniques. After transfer, data security becomes the responsibility of the third-party data management company, but these entities are themselves not immune from security breaches and would be vulnerable to security issues inserted upstream at the farm or machinery level. Finally, commodity markets are strongly influenced by crop production estimates generated by surveys of farmers and the agriculture industry.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Fruits_and_vegetables\">Fruits and vegetables<\/span><\/h3>\n<p>Fresh fruits and vegetables are leading sources for foodborne illness in the United States.<sup id=\"rdp-ebb-cite_ref-Callej.C3.B3nRepo15_22-0\" class=\"reference\"><a href=\"#cite_note-Callej.C3.B3nRepo15-22\">[22]<\/a><\/sup> Furthermore, even in the absence of foodborne illness outbreaks, fresh produce recalls occur regularly due to the presence of potential harmful microorganisms. Fresh produce available for sale in local markets may have been produced in one of many locations throughout the nation or from one of many countries around the world. The production, sorting, grading, commingling, transporting, marketing, and sale of fresh fruits and vegetables is complex and involves numerous industry actors with varying roles. Tracking fresh produce from initial production through consumption is critical to limit the potential for and impact of foodborne illness outbreaks. Accurate product information and rapid access to data is essential to identify contaminated product in the market, prevent or limit foodborne illness, limit the damage to non-implicated producers, and maintain consumer confidence. Access to product tracking and microbiological data is increasing in the fresh produce industry.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Environmental_resources_.28water.29\">Environmental resources (water)<\/span><\/h3>\n<p>Drinking water safety is extremely important on-farm, for food processing, ensuring consumers' health and proper functioning of the ecosystem. The proportion of the world's population consuming drinking water from certified and controlled water sources is about 90% and still increasing.<sup id=\"rdp-ebb-cite_ref-VieiraAStrat11_23-0\" class=\"reference\"><a href=\"#cite_note-VieiraAStrat11-23\">[23]<\/a><\/sup> However, 2.3 billion people worldwide suffer from diseases related to drinking water. Over the past three decades, significant drinking water contamination incidents have occurred in developing as well as developed countries, creating health problems for consumers.<sup id=\"rdp-ebb-cite_ref-HamiltonAComm06_24-0\" class=\"reference\"><a href=\"#cite_note-HamiltonAComm06-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TsoukalasACrit18_25-0\" class=\"reference\"><a href=\"#cite_note-TsoukalasACrit18-25\">[25]<\/a><\/sup> Traditional risk management systems, based on addressing and correcting the failure after its occurrence, are inadequate to deal with potential cyber biosecurity threats (as the cybersecurity landscape is changing rapidly as technology continues to advance). Given the severity of risk and potential harm, cyber biosecurity must be given a high priority for the drinking water management and treatment sector.<sup id=\"rdp-ebb-cite_ref-GermanoCyber18_26-0\" class=\"reference\"><a href=\"#cite_note-GermanoCyber18-26\">[26]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions:_Moving_Toward_Solutions\">Conclusions: Moving Toward Solutions<\/span><\/h2>\n<p>The complex and vastly diverse enterprises within the food and agriculture system increases vulnerability of our food supply and threatens our ability to contribute to the global food supply. Rapid advancements in technologies and adoption into the food and agriculture sectors increase the risks for cyber biosecurity threats and attacks. The current food and agriculture workforce has limited knowledge or training appropriate to evaluate and protect the vast amount of data generated by these technologies. The cybersecurity industry is not well-prepared to address the unique structure and functions within food and agriculture sectors. Protecting them includes (1) developing and characterizing effective cyber biosecurity risk assessment and mitigation strategies; (2) developing and preparing the current and future workforce to identify, address, and adopt effective cyber biosecurity strategies; (3) considering policy and regulations, including insurance, for protection within and across the entire system; and (4) effectively communicating within the sector and across the food and agriculture system.<sup id=\"rdp-ebb-cite_ref-NIFAData16_15-1\" class=\"reference\"><a href=\"#cite_note-NIFAData16-15\">[15]<\/a><\/sup> Awareness, knowledge, adoption, and frequent evaluation of cyber biosecurity plans and strategies among and within all sectors is essential. A multidisciplinary approach integrating expertise in agriculture, food, engineering, computer science, and cybersecurity is needed for filling this gap. The USDA, in consultation with academic, public, and private sector experts and representation from sectors within the food and agriculture system, should lead an initiative for developing a planned approach to addressing cyber biosecurity. Private and public funding is needed to support research priorities and implementation strategies. Checkoff funding mechanisms or cooperative agreements, which are common within commodity systems, may be options for assisting small to moderate-sized agribusinesses. Workforce development, effective communication strategies, and cooperation across sectors and industries will help increase support and compliance, reducing the risks and providing increased protection for the U.S. bioeconomy and our domestic and global food supply.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_material\">Supplementary material<\/span><\/h2>\n<p>The Supplementary Material for this article can be found online at: <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00063\/full#supplementary-material\" target=\"_blank\">https:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00063\/full#supplementary-material<\/a>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors acknowledge the Virginia Agricultural Experiment Station, Blacksburg, VA for financial support for publishing this manuscript. The authors gratefully acknowledge Michael J. Stamper, Data Visualization Designer and lecturer at the University Libraries, Data Services at Virginia Tech, for his contributions to the figures.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_Contributions\">Author Contributions<\/span><\/h3>\n<p>SD lead author, responsible for structure, content, and figure; responsible for considering, incorporation co-author contributions and suggested edits; responsible for final version. RR provided draft content related to HACCP and post-harvest processing and cyber biosecurity; contributed, reviewed, and edited the manuscript. RW contributed, reviewed, and edited content related to HACCP and post-harvest processing and biosecurity; reviewed and critiqued manuscript to ensure quality and flow. FR contributed, reviewed, and edited content related to food and agriculture system influence on bioeconomy; reviewed and critiqued manuscript to ensure quality and flow. WT contributed, reviewed, and edited content related to cyber biosecurity in agriculture (pre-harvest; crop, soil, and environment); reviewed and critiqued manuscript to ensure quality and flow. KL contributed, reviewed, and edited content related to cyber biosecurity in agriculture (pre-harvest; animal breeding and genetics); reviewed and critiqued manuscript to ensure quality and flow. ND contributions to sections relating to biotechnology; overall quality assurance and readability reviews and modifications. SM contributed, reviewed, and edited content related to cyber biosecurity in food and agricultural system and the environment; reviewed and critiqued manuscript to ensure quality and flow. EC contributed, reviewed, and edited content related to cybersecurity, data sources, and integration into the food and agricultural system; reviewed and critiqued manuscript to ensure quality and flow. RM co-originator of the cyber biosecurity concept; co-originator of the concepts relating to food and agricultural system; contributed, reviewed, and edited content related to cyber biosecurity, data sources, and integration into the food and agricultural system; reviewed and critiqued manuscript to ensure quality, flow, and relevance to the targeted audience.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>RR is employed by Tyson Foods.\n<\/p><p>The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-MurchCyber18-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MurchCyber18_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Murch, R.S.; So, W.K.; Buchholz, W.G. et al. (2018). \"Cyberbiosecurity: An Emerging New Discipline to Help Safeguard the Bioeconomy\". <i>Frontiers in Bioengineering and Biotechnology<\/i> <b>6<\/b>: 39. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffbioe.2018.00039\" target=\"_blank\">10.3389\/fbioe.2018.00039<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyberbiosecurity%3A+An+Emerging+New+Discipline+to+Help+Safeguard+the+Bioeconomy&rft.jtitle=Frontiers+in+Bioengineering+and+Biotechnology&rft.aulast=Murch%2C+R.S.%3B+So%2C+W.K.%3B+Buchholz%2C+W.G.+et+al.&rft.au=Murch%2C+R.S.%3B+So%2C+W.K.%3B+Buchholz%2C+W.G.+et+al.&rft.date=2018&rft.volume=6&rft.pages=39&rft_id=info:doi\/10.3389%2Ffbioe.2018.00039&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PeccoudCyber18-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PeccoudCyber18_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". <i>Trends in Biotechnology<\/i> <b>36<\/b> (1): 4\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.tibtech.2017.10.012\" target=\"_blank\">10.1016\/j.tibtech.2017.10.012<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29224719\" target=\"_blank\">29224719<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyberbiosecurity%3A+From+Naive+Trust+to+Risk+Awareness&rft.jtitle=Trends+in+Biotechnology&rft.aulast=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.au=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.date=2018&rft.volume=36&rft.issue=1&rft.pages=4%E2%80%937&rft_id=info:doi\/10.1016%2Fj.tibtech.2017.10.012&rft_id=info:pmid\/29224719&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NASEMMeeting14-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NASEMMeeting14_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Board on Chemical Sciences and Technology; Board on Life Sciences (2014). <i>Meeting Recap: Workshop - Convergence: Safeguarding Technology in the Bioeconomy<\/i>. The National Academies of Sciences, Engineering, and Medicine.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Meeting+Recap%3A+Workshop+-+Convergence%3A+Safeguarding+Technology+in+the+Bioeconomy&rft.aulast=Board+on+Chemical+Sciences+and+Technology%3B+Board+on+Life+Sciences&rft.au=Board+on+Chemical+Sciences+and+Technology%3B+Board+on+Life+Sciences&rft.date=2014&rft.pub=The+National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WintleATrans17-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WintleATrans17_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-WintleATrans17_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-WintleATrans17_4-2\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wintle, B.C.; Boehm, C.R.; Rhodes, C. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469\" target=\"_blank\">\"A transatlantic perspective on 20 emerging issues in biological engineering\"<\/a>. <i>eLife<\/i> <b>3<\/b>: e30247. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7554%2FeLife.30247\" target=\"_blank\">10.7554\/eLife.30247<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5685469\/\" target=\"_blank\">PMC5685469<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29132504\" target=\"_blank\">29132504<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5685469<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+transatlantic+perspective+on+20+emerging+issues+in+biological+engineering&rft.jtitle=eLife&rft.aulast=Wintle%2C+B.C.%3B+Boehm%2C+C.R.%3B+Rhodes%2C+C.+et+al.&rft.au=Wintle%2C+B.C.%3B+Boehm%2C+C.R.%3B+Rhodes%2C+C.+et+al.&rft.date=2017&rft.volume=3&rft.pages=e30247&rft_id=info:doi\/10.7554%2FeLife.30247&rft_id=info:pmc\/PMC5685469&rft_id=info:pmid\/29132504&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5685469&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SykutaBig16-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SykutaBig16_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-SykutaBig16_5-1\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-SykutaBig16_5-2\">5.2<\/a><\/sup> <sup><a href=\"#cite_ref-SykutaBig16_5-3\">5.3<\/a><\/sup> <sup><a href=\"#cite_ref-SykutaBig16_5-4\">5.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sykuta, M.E. (2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ifama.org\/Volume-19-Issue-A\" target=\"_blank\">\"Big data in agriculture: Property rights, privacy and competition in ag data services\"<\/a>. <i>International Food and Agribusiness Management Review<\/i> <b>19<\/b> (A): 57\u201374<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ifama.org\/Volume-19-Issue-A\" target=\"_blank\">https:\/\/www.ifama.org\/Volume-19-Issue-A<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+data+in+agriculture%3A+Property+rights%2C+privacy+and+competition+in+ag+data+services&rft.jtitle=International+Food+and+Agribusiness+Management+Review&rft.aulast=Sykuta%2C+M.E.&rft.au=Sykuta%2C+M.E.&rft.date=2016&rft.volume=19&rft.issue=A&rft.pages=57%E2%80%9374&rft_id=https%3A%2F%2Fwww.ifama.org%2FVolume-19-Issue-A&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BoghossianThreats18-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BoghossianThreats18_6-0\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-BoghossianThreats18_6-1\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-BoghossianThreats18_6-2\">6.2<\/a><\/sup> <sup><a href=\"#cite_ref-BoghossianThreats18_6-3\">6.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Boghossian, A.; Linsky, S.; Brown, A. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/2018%20AEP_Threats_to_Precision_Agriculture.pdf\" target=\"_blank\">\"Threats to Precision Agriculture\"<\/a> (PDF). U.S. Department of Homeland Security<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/2018%20AEP_Threats_to_Precision_Agriculture.pdf\" target=\"_blank\">https:\/\/www.dhs.gov\/sites\/default\/files\/publications\/2018%20AEP_Threats_to_Precision_Agriculture.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Threats+to+Precision+Agriculture&rft.atitle=&rft.aulast=Boghossian%2C+A.%3B+Linsky%2C+S.%3B+Brown%2C+A.+et+al.&rft.au=Boghossian%2C+A.%3B+Linsky%2C+S.%3B+Brown%2C+A.+et+al.&rft.date=2018&rft.pub=U.S.+Department+of+Homeland+Security&rft_id=https%3A%2F%2Fwww.dhs.gov%2Fsites%2Fdefault%2Ffiles%2Fpublications%2F2018%2520AEP_Threats_to_Precision_Agriculture.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NASEMSafe15-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NASEMSafe15_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-NASEMSafe15_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Board on Chemical Sciences and Technology (2015) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf\" target=\"_blank\"><i>Meeting Recap: Safeguarding the Bioeconomy: Applications and Implications of Emerging Science<\/i><\/a>. The National Academies of Sciences, Engineering, and Medicine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf\" target=\"_blank\">https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Meeting+Recap%3A+Safeguarding+the+Bioeconomy%3A+Applications+and+Implications+of+Emerging+Science&rft.aulast=Board+on+Chemical+Sciences+and+Technology&rft.au=Board+on+Chemical+Sciences+and+Technology&rft.date=2015&rft.pub=The+National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rft_id=https%3A%2F%2Fwww.ehidc.org%2Fsites%2Fdefault%2Ffiles%2Fresources%2Ffiles%2FSafeguarding%2520the%2520Bioeconomy_II_Recap%2520Final%2520090815.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FEWhatIs18-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FEWhatIs18_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/feedingtheeconomy.com\/\" target=\"_blank\">\"What is the Food and Ag Industries\u2019 Impact in Your Community?\"<\/a>. <i>Feeding the Economy<\/i>. 2018<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/feedingtheeconomy.com\/\" target=\"_blank\">https:\/\/feedingtheeconomy.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+is+the+Food+and+Ag+Industries%E2%80%99+Impact+in+Your+Community%3F&rft.atitle=Feeding+the+Economy&rft.date=2018&rft_id=https%3A%2F%2Ffeedingtheeconomy.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MacDonaldLarge17-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MacDonaldLarge17_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">MacDonald, J.M.; Hoppe, R.A. (06 March 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ers.usda.gov\/amber-waves\/2017\/march\/large-family-farms-continue-to-dominate-us-agricultural-production\/\" target=\"_blank\">\"Large Family Farms Continue To Dominate U.S. Agricultural Production\"<\/a>. <i>Amber Waves<\/i>. U.S. Department of Agriculture<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ers.usda.gov\/amber-waves\/2017\/march\/large-family-farms-continue-to-dominate-us-agricultural-production\/\" target=\"_blank\">https:\/\/www.ers.usda.gov\/amber-waves\/2017\/march\/large-family-farms-continue-to-dominate-us-agricultural-production\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Large+Family+Farms+Continue+To+Dominate+U.S.+Agricultural+Production&rft.atitle=Amber+Waves&rft.aulast=MacDonald%2C+J.M.%3B+Hoppe%2C+R.A.&rft.au=MacDonald%2C+J.M.%3B+Hoppe%2C+R.A.&rft.date=06+March+2017&rft.pub=U.S.+Department+of+Agriculture&rft_id=https%3A%2F%2Fwww.ers.usda.gov%2Famber-waves%2F2017%2Fmarch%2Flarge-family-farms-continue-to-dominate-us-agricultural-production%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NASSFarm13-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NASSFarm13_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Agricultural Statistics Service (20 August 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/usda.library.cornell.edu\/concern\/publications\/h128nd689?locale=en\" target=\"_blank\">\"Farm Computer Usage and Ownership\"<\/a>. U.S. Department of Agriculture<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/usda.library.cornell.edu\/concern\/publications\/h128nd689?locale=en\" target=\"_blank\">https:\/\/usda.library.cornell.edu\/concern\/publications\/h128nd689?locale=en<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Farm+Computer+Usage+and+Ownership&rft.atitle=&rft.aulast=National+Agricultural+Statistics+Service&rft.au=National+Agricultural+Statistics+Service&rft.date=20+August+2013&rft.pub=U.S.+Department+of+Agriculture&rft_id=https%3A%2F%2Fusda.library.cornell.edu%2Fconcern%2Fpublications%2Fh128nd689%3Flocale%3Den&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeilCyber18-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GeilCyber18_11-0\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-GeilCyber18_11-1\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-GeilCyber18_11-2\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Geil, A.; Sagers, G.; Spaulding, A.D. et al.. \"Cyber security on the farm: An assessment of cyber security practices in the United States agriculture industry\". <i>International Food and Agribusiness Management Review<\/i> <b>21<\/b>: 317\u201334. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.22434%2FIFAMR2017.0045\" target=\"_blank\">10.22434\/IFAMR2017.0045<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyber+security+on+the+farm%3A+An+assessment+of+cyber+security+practices+in+the+United+States+agriculture+industry&rft.jtitle=International+Food+and+Agribusiness+Management+Review&rft.aulast=Geil%2C+A.%3B+Sagers%2C+G.%3B+Spaulding%2C+A.D.+et+al.&rft.au=Geil%2C+A.%3B+Sagers%2C+G.%3B+Spaulding%2C+A.D.+et+al.&rft.volume=21&rft.pages=317%E2%80%9334&rft_id=info:doi\/10.22434%2FIFAMR2017.0045&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ColbertTheGame18-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ColbertTheGame18_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Colbert, E.J.M.; Kott, A.; Knachel, L.P.. \"The game-theoretic model and experimental investigation of cyber wargaming\". <i>The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology<\/i>: 1\u201318. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1548512918795061\" target=\"_blank\">10.1177\/1548512918795061<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+game-theoretic+model+and+experimental+investigation+of+cyber+wargaming&rft.jtitle=The+Journal+of+Defense+Modeling+and+Simulation%3A+Applications%2C+Methodology%2C+Technology&rft.aulast=Colbert%2C+E.J.M.%3B+Kott%2C+A.%3B+Knachel%2C+L.P.&rft.au=Colbert%2C+E.J.M.%3B+Kott%2C+A.%3B+Knachel%2C+L.P.&rft.pages=1%E2%80%9318&rft_id=info:doi\/10.1177%2F1548512918795061&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ColbertTable15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ColbertTable15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Colbert, E.; Sullivan, D.; Wong, K. et al. (2015). <i>Table-Top Exercise Final Report: Intrusion Detection Capabilities for US Army SCADA Systems: Information Packet<\/i>. US Army Research Lab Technical Report ARL-TR-7498.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Table-Top+Exercise+Final+Report%3A+Intrusion+Detection+Capabilities+for+US+Army+SCADA+Systems%3A+Information+Packet&rft.aulast=Colbert%2C+E.%3B+Sullivan%2C+D.%3B+Wong%2C+K.+et+al.&rft.au=Colbert%2C+E.%3B+Sullivan%2C+D.%3B+Wong%2C+K.+et+al.&rft.date=2015&rft.series=US+Army+Research+Lab+Technical+Report+ARL-TR-7498&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ColbertRED15-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ColbertRED15_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Colbert, E.; Sullivan, D.; Wong, K. et al. (2015). <i>RED and BLUE Teaming of a US Army SCADA System: Table-Top Exercise Final Report<\/i>. US Army Research Lab Technical Report ARL-TR-7497.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=RED+and+BLUE+Teaming+of+a+US+Army+SCADA+System%3A+Table-Top+Exercise+Final+Report&rft.aulast=Colbert%2C+E.%3B+Sullivan%2C+D.%3B+Wong%2C+K.+et+al.&rft.au=Colbert%2C+E.%3B+Sullivan%2C+D.%3B+Wong%2C+K.+et+al.&rft.date=2015&rft.series=US+Army+Research+Lab+Technical+Report+ARL-TR-7497&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NIFAData16-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NIFAData16_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-NIFAData16_15-1\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Institute of Food and Agriculture (01 November 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/nifa.usda.gov\/sites\/default\/files\/resource\/Stakeholder%20Ideas%20Engine%20Input%20-%20Summary%5B1%5D.pdf\" target=\"_blank\">\"Data Summit: Changing the Face, Place, and Space of Agriculture\"<\/a> (PDF). U.S. Department of Agriculture<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/nifa.usda.gov\/sites\/default\/files\/resource\/Stakeholder%20Ideas%20Engine%20Input%20-%20Summary%5B1%5D.pdf\" target=\"_blank\">https:\/\/nifa.usda.gov\/sites\/default\/files\/resource\/Stakeholder%20Ideas%20Engine%20Input%20-%20Summary%5B1%5D.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 October 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+Summit%3A+Changing+the+Face%2C+Place%2C+and+Space+of+Agriculture&rft.atitle=&rft.aulast=National+Institute+of+Food+and+Agriculture&rft.au=National+Institute+of+Food+and+Agriculture&rft.date=01+November+2016&rft.pub=U.S.+Department+of+Agriculture&rft_id=https%3A%2F%2Fnifa.usda.gov%2Fsites%2Fdefault%2Ffiles%2Fresource%2FStakeholder%2520Ideas%2520Engine%2520Input%2520-%2520Summary%255B1%255D.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CherdantsevaARev16-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CherdantsevaARev16_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cherdantseva, Y.; Burnap, P.; Blyth, A. et al. (2016). \"A review of cybersecurity risk assessment methods for SCADA systems\". <i>Computers & Security<\/i> <b>56<\/b>: 1\u201327. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cose.2015.09.009\" target=\"_blank\">10.1016\/j.cose.2015.09.009<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+review+of+cybersecurity+risk+assessment+methods+for+SCADA+systems&rft.jtitle=Computers+%26+Security&rft.aulast=Cherdantseva%2C+Y.%3B+Burnap%2C+P.%3B+Blyth%2C+A.+et+al.&rft.au=Cherdantseva%2C+Y.%3B+Burnap%2C+P.%3B+Blyth%2C+A.+et+al.&rft.date=2016&rft.volume=56&rft.pages=1%E2%80%9327&rft_id=info:doi\/10.1016%2Fj.cose.2015.09.009&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VarianSystem04-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VarianSystem04_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Varian, H. (2004). \"System Reliability and Free Riding\". In Camp, L.J.; Lewis, S.. <i>Economics of Information Security<\/i>. Advances in Information Security. <b>12<\/b>. Springer. pp. 1\u201315. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F1-4020-8090-5_1\" target=\"_blank\">10.1007\/1-4020-8090-5_1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781402080906.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=System+Reliability+and+Free+Riding&rft.atitle=Economics+of+Information+Security&rft.aulast=Varian%2C+H.&rft.au=Varian%2C+H.&rft.date=2004&rft.series=Advances+in+Information+Security&rft.volume=12&rft.pages=pp.%26nbsp%3B1%E2%80%9315&rft.pub=Springer&rft_id=info:doi\/10.1007%2F1-4020-8090-5_1&rft.isbn=9781402080906&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SellnerBoard07-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SellnerBoard07_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sellner, E.M.; Kim, J.W.; McClure, M.C. et al. (2007). \"Board-invited review: Applications of genomic information in livestock\". <i>Journal of Animal Science<\/i> <b>85<\/b>: 12. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2527%2Fjas.2007-0291\" target=\"_blank\">10.2527\/jas.2007-0291<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17709778\" target=\"_blank\">17709778<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Board-invited+review%3A+Applications+of+genomic+information+in+livestock&rft.jtitle=Journal+of+Animal+Science&rft.aulast=Sellner%2C+E.M.%3B+Kim%2C+J.W.%3B+McClure%2C+M.C.+et+al.&rft.au=Sellner%2C+E.M.%3B+Kim%2C+J.W.%3B+McClure%2C+M.C.+et+al.&rft.date=2007&rft.volume=85&rft.pages=12&rft_id=info:doi\/10.2527%2Fjas.2007-0291&rft_id=info:pmid\/17709778&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TeluguGenome17-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TeluguGenome17_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Telugu, B.P.; Park, K.E.; Park, C.H. (2017). \"Genome editing and genetic engineering in livestock for advancing agricultural and biomedical applications\". <i>Mammalian Genome<\/i> <b>28<\/b> (7\u20138): 338\u201347. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00335-017-9709-4\" target=\"_blank\">10.1007\/s00335-017-9709-4<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28712062\" target=\"_blank\">28712062<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genome+editing+and+genetic+engineering+in+livestock+for+advancing+agricultural+and+biomedical+applications&rft.jtitle=Mammalian+Genome&rft.aulast=Telugu%2C+B.P.%3B+Park%2C+K.E.%3B+Park%2C+C.H.&rft.au=Telugu%2C+B.P.%3B+Park%2C+K.E.%3B+Park%2C+C.H.&rft.date=2017&rft.volume=28&rft.issue=7%E2%80%938&rft.pages=338%E2%80%9347&rft_id=info:doi\/10.1007%2Fs00335-017-9709-4&rft_id=info:pmid\/28712062&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GolanTrace04-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GolanTrace04_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Golan, E.; Krissoff, B.; Kuchler, F. et al. (March 2004). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ers.usda.gov\/publications\/pub-details\/?pubid=41632\" target=\"_blank\">\"Traceability in the U.S. Food Supply: Economic Theory and Industry Studies\"<\/a>. USDA Economic Research Service<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ers.usda.gov\/publications\/pub-details\/?pubid=41632\" target=\"_blank\">https:\/\/www.ers.usda.gov\/publications\/pub-details\/?pubid=41632<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Traceability+in+the+U.S.+Food+Supply%3A+Economic+Theory+and+Industry+Studies&rft.atitle=&rft.aulast=Golan%2C+E.%3B+Krissoff%2C+B.%3B+Kuchler%2C+F.+et+al.&rft.au=Golan%2C+E.%3B+Krissoff%2C+B.%3B+Kuchler%2C+F.+et+al.&rft.date=March+2004&rft.pub=USDA+Economic+Research+Service&rft_id=https%3A%2F%2Fwww.ers.usda.gov%2Fpublications%2Fpub-details%2F%3Fpubid%3D41632&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FerrisData17-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FerrisData17_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ferris, J.L. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/scholarship.law.umn.edu\/mjlst\/vol18\/iss1\/6\" target=\"_blank\">\"Data Privacy and Protection in the Agriculture Industry: Is Federal Regulation Necessary?\"<\/a>. <i>Minnesota Journal of Law, Science & Technology<\/i> <b>18<\/b> (1): 309\u201342<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/scholarship.law.umn.edu\/mjlst\/vol18\/iss1\/6\" target=\"_blank\">https:\/\/scholarship.law.umn.edu\/mjlst\/vol18\/iss1\/6<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+Privacy+and+Protection+in+the+Agriculture+Industry%3A+Is+Federal+Regulation+Necessary%3F&rft.jtitle=Minnesota+Journal+of+Law%2C+Science+%26+Technology&rft.aulast=Ferris%2C+J.L.&rft.au=Ferris%2C+J.L.&rft.date=2017&rft.volume=18&rft.issue=1&rft.pages=309%E2%80%9342&rft_id=https%3A%2F%2Fscholarship.law.umn.edu%2Fmjlst%2Fvol18%2Fiss1%2F6&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Callej.C3.B3nRepo15-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Callej.C3.B3nRepo15_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Callej\u00f3n, R.M.; Rodr\u00edguez-Naranjo, M.I.; Ubeda, C. et al. (2015). \"Reported foodborne outbreaks due to fresh produce in the United States and European Union: Trends and causes\". <i>Foodborne Pathogens and Disease<\/i> <b>12<\/b> (1): 32\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Ffpd.2014.1821\" target=\"_blank\">10.1089\/fpd.2014.1821<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25587926\" target=\"_blank\">25587926<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reported+foodborne+outbreaks+due+to+fresh+produce+in+the+United+States+and+European+Union%3A+Trends+and+causes&rft.jtitle=Foodborne+Pathogens+and+Disease&rft.aulast=Callej%C3%B3n%2C+R.M.%3B+Rodr%C3%ADguez-Naranjo%2C+M.I.%3B+Ubeda%2C+C.+et+al.&rft.au=Callej%C3%B3n%2C+R.M.%3B+Rodr%C3%ADguez-Naranjo%2C+M.I.%3B+Ubeda%2C+C.+et+al.&rft.date=2015&rft.volume=12&rft.issue=1&rft.pages=32%E2%80%938&rft_id=info:doi\/10.1089%2Ffpd.2014.1821&rft_id=info:pmid\/25587926&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VieiraAStrat11-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VieiraAStrat11_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vierira, J.M. (2011). \"A strategic approach for Water Safety Plans implementation in Portugal\". <i>Journal of Water and Health<\/i> <b>9<\/b> (1): 107\u201316. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2166%2Fwh.2010.150\" target=\"_blank\">10.2166\/wh.2010.150<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21301119\" target=\"_blank\">21301119<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+strategic+approach+for+Water+Safety+Plans+implementation+in+Portugal&rft.jtitle=Journal+of+Water+and+Health&rft.aulast=Vierira%2C+J.M.&rft.au=Vierira%2C+J.M.&rft.date=2011&rft.volume=9&rft.issue=1&rft.pages=107%E2%80%9316&rft_id=info:doi\/10.2166%2Fwh.2010.150&rft_id=info:pmid\/21301119&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HamiltonAComm06-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HamiltonAComm06_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hamilton, P.D.; Gale, P.; Pollard, S.J. (2006). \"A commentary on recent water safety initiatives in the context of water utility risk management\". <i>Environment International<\/i> <b>32<\/b> (8): 958\u201366. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.envint.2006.06.001\" target=\"_blank\">10.1016\/j.envint.2006.06.001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16870255\" target=\"_blank\">16870255<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+commentary+on+recent+water+safety+initiatives+in+the+context+of+water+utility+risk+management&rft.jtitle=Environment+International&rft.aulast=Hamilton%2C+P.D.%3B+Gale%2C+P.%3B+Pollard%2C+S.J.&rft.au=Hamilton%2C+P.D.%3B+Gale%2C+P.%3B+Pollard%2C+S.J.&rft.date=2006&rft.volume=32&rft.issue=8&rft.pages=958%E2%80%9366&rft_id=info:doi\/10.1016%2Fj.envint.2006.06.001&rft_id=info:pmid\/16870255&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TsoukalasACrit18-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TsoukalasACrit18_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tsoukalas, D.S.; Tsitsifli, S. (2018). \"A Critical Evaluation of Water Safety Plans (WSPs) and HACCP Implementation in Water Utilities\". <i>Proceedings<\/i> <b>2<\/b> (11): 600. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3390%2Fproceedings2110600\" target=\"_blank\">10.3390\/proceedings2110600<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Critical+Evaluation+of+Water+Safety+Plans+%28WSPs%29+and+HACCP+Implementation+in+Water+Utilities&rft.jtitle=Proceedings&rft.aulast=Tsoukalas%2C+D.S.%3B+Tsitsifli%2C+S.&rft.au=Tsoukalas%2C+D.S.%3B+Tsitsifli%2C+S.&rft.date=2018&rft.volume=2&rft.issue=11&rft.pages=600&rft_id=info:doi\/10.3390%2Fproceedings2110600&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GermanoCyber18-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GermanoCyber18_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Germano, J.H. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.awwa.org\/Portals\/0\/AWWA\/Government\/AWWACybersecurityRiskandResponsibility.pdf?ver=2018-12-05\" target=\"_blank\">\"Cybersecurity Risk and Responsibility in the Water Sector\"<\/a> (PDF). American Water Works Association<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.awwa.org\/Portals\/0\/AWWA\/Government\/AWWACybersecurityRiskandResponsibility.pdf?ver=2018-12-05\" target=\"_blank\">https:\/\/www.awwa.org\/Portals\/0\/AWWA\/Government\/AWWACybersecurityRiskandResponsibility.pdf?ver=2018-12-05<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cybersecurity+Risk+and+Responsibility+in+the+Water+Sector&rft.atitle=&rft.aulast=Germano%2C+J.H.&rft.au=Germano%2C+J.H.&rft.date=2018&rft.pub=American+Water+Works+Association&rft_id=https%3A%2F%2Fwww.awwa.org%2FPortals%2F0%2FAWWA%2FGovernment%2FAWWACybersecurityRiskandResponsibility.pdf%3Fver%3D2018-12-05&rfr_id=info:sid\/en.wikipedia.org:Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165453\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.624 seconds\nReal time usage: 0.657 seconds\nPreprocessor visited node count: 19956\/1000000\nPreprocessor generated node count: 36392\/1000000\nPost\u2010expand include size: 144000\/2097152 bytes\nTemplate argument size: 50448\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 622.083 1 - -total\n 83.11% 517.012 1 - Template:Reflist\n 71.47% 444.602 26 - Template:Citation\/core\n 44.76% 278.423 14 - Template:Cite_journal\n 17.52% 108.997 7 - Template:Cite_web\n 14.24% 88.595 5 - Template:Cite_book\n 10.66% 66.344 1 - Template:Infobox_journal_article\n 10.21% 63.542 1 - Template:Infobox\n 6.16% 38.301 80 - Template:Infobox\/row\n 5.54% 34.478 22 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11024-0!*!0!!en!5!* and timestamp 20190701165452 and revision id 35423\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system\">https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","e69b956327faef5b93c17a708e883f73_images":["https:\/\/www.limswiki.org\/images\/3\/37\/Fig1_Duncan_FrontBioengBiotech2019_7.jpg","https:\/\/www.limswiki.org\/images\/d\/db\/Fig2_Duncan_FrontBioengBiotech2019_7.jpg"],"e69b956327faef5b93c17a708e883f73_timestamp":1562000092,"484090b9c88bd26fd5f834f6ab7cbd8a_type":"article","484090b9c88bd26fd5f834f6ab7cbd8a_title":"Defending our public biological databases as a global critical infrastructure (Caswell et al. 2019)","484090b9c88bd26fd5f834f6ab7cbd8a_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure","484090b9c88bd26fd5f834f6ab7cbd8a_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Defending our public biological databases as a global critical infrastructure\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nDefending our public biological databases as a global critical infrastructureJournal\n \nFrontiers in Bioengineering and BiotechnologyAuthor(s)\n \nCaswell, Jacob; Gans, Jason D.; Generous, Nicolas; Hudson, Corey M.; Merkley, Eric; Johnson, Curtis; Oehmen, Christopher;\r\nOmberg, Kristin; Purvine, Emilie; Taylor, Karen; Ting, Christina L.; Wolinsky, Murray; Xie, GaryAuthor affiliation(s)\n \nSandia National Laboratories, Los Alamos National Laboratory, Pacific Northwest National LaboratoryPrimary contact\n \nEmail: karen at pnnl dot govEditors\n \nMurch, Randall S.Year published\n \n2019Volume and issue\n \n7Page(s)\n \n58DOI\n \n10.3389\/fbioe.2019.00058ISSN\n \n2296-4185Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00058\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00058\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Background: Problems with public biological databases \n\n3.1 Data integrity \n3.2 Vulnerabilities and intentional tampering \n\n\n4 Approaches for improving biological databases \n\n4.1 Automated approaches for detecting anomalies \n4.2 Protections against intentional errors \n\n\n5 Preliminary conclusions \n6 Acknowledgements \n\n6.1 Author contributions \n6.2 Funding \n6.3 Conflict of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nProgress in modern biology is being driven, in part, by the large amounts of freely available data in public resources such as the International Nucleotide Sequence Database Collaboration (INSDC), the world's primary database of biological sequence (and related) information. INSDC and similar databases have dramatically increased the pace of fundamental biological discovery and enabled a host of innovative therapeutic, diagnostic, and forensic applications. However, as high-value, openly shared resources with a high degree of assumed trust, these repositories share compelling similarities to the early days of the internet. Consequently, as public biological databases continue to increase in size and importance, we expect that they will face the same threats as undefended cyberspace. There is a unique opportunity, before a significant breach and loss of trust occurs, to ensure they evolve with quality and security as a design philosophy rather than costly \u201cretrofitted\u201d mitigations. This perspective article surveys some potential quality assurance and security weaknesses in existing open genomic and proteomic repositories, describes methods to mitigate the likelihood of both intentional and unintentional errors, and offers recommendations for risk mitigation based on lessons learned from cybersecurity.\nKeywords: cyberbiosecurity, biosecurity, cybersecurity, biological databases, machine learning, bioeconomy\n\nIntroduction \nAlthough an openly shared interaction platform confers great value to the biological research community, it may also introduce quality and security risks. Without a system for trusted correction and revision, these shared resources may facilitate widespread dissemination and use of low-quality content, for instance, taxonomically misclassified or erroneous sequences. Furthermore, as these public databases increase in size and importance, they may fall victim to the same security issues and abuses that plague cyberspace to this day. If we act now by developing the databases with quality and security as a design philosophy, we can protect these databases at a much lower cost and with fewer challenges than we currently face with the internet.\nIn this perspective article, the authors aim to outline some potential quality assurance and security weaknesses in existing public biological repositories. In the background section we provide a discussion of errors present in public biological databases and discuss possible security vulnerabilities inherent in their access, publication, and distribution models and systems. Both unintentional and intentional errors are discussed, the latter of which has not been given significant consideration in literature.[1] Afterwards, we attempt to introduce greater trust in the data and analyses by providing recommendations to mitigate or account for these errors and vulnerabilities and point to approaches used by other internet databases. Finally, we summarize our recommendations in the conclusions section.\nThis article focuses on databases which contain public and freely available data. We recognize that other biological databases exist which contain private, sensitive, or otherwise valuable data (e.g., human genomes). While unauthorized disclosure is not a formal concern in public, non-human databases, safeguarding against intentional or unintentional erroneous content is. Some approaches have been proposed to protect unauthorized disclosure[2][3][4] and, while we don't survey these approaches in this perspective, we note that the public database community may benefit from these ideas as well.\n\nBackground: Problems with public biological databases \nData integrity \nAn important goal for bioinformatics is the continuous improvement of biological databases. Given the rapid nature of this improvement and the rate of data production though, the content of these repositories is not without error. For example, the problem of contaminated sequences has been recognized for nearly two decades, with evidence stating that bacteria and human error are the two most common sources of contamination.[5][6] Ancient DNA is also particularly affected by human contamination.[7] These contaminants are frequently introduced during experiments[5][8] from natural associations and insufficient purification.[9] In the past few years, additional reports have highlighted cases of DNA contamination in published genome data[10][11], suggesting that DNA contamination may be more widespread than previously thought. We recognize that errors and omissions can occur in open databases both at the sequence and at the metadata levels, but for this article we mainly focus on sequence and taxonomic data concerns for the purposes of illustrating some of the many data integrity challenges possible.\nIn addition to contamination, two high-profile examples of sequence errors include the reassembly of a misassembled Francisella tularensis genome[12] and the identification of single nucleotide errors in a reference Tobacco mosaic virus (TMV) genome.[13] Without a way to flag or remove the erroneous entries, future researchers are left to continually rediscover them. The errors in the reference TMV sequence are particularly disturbing. The taxonomic assignment corresponds to a pathogenic strain, but due to two erroneous single nucleotide polymorphisms (SNPs), virions synthesized from the published reference sequence are atypically not infectious. Overlooked contamination in reference genomes can thereby lead to wrong or confusing results and may have major detrimental effects on biological conclusions.[14][15] While resequencing could be used to identify and correct sequence errors, it is only possible when the original source material is available. For the given example of single nucleotide errors in the TMV genome, the biological sample (sequenced in 1982) no longer exists. In addition to missing samples, samples of high consequence human and agricultural pathogens may not be available for resequencing.\nDatabase integrity considerations for proteomics are generally similar to those for genomics because databases of protein sequences are derived from genome sequencing, via genome annotation and in silico translation. A sequence database error is unlikely to result in spurious detection of a protein that is present in the sample (false positive), but it could easily lead to a failure to detect a protein that is present (false negative). This is particularly concerning for discovery of accurate peptide signatures for use in targeted assays, a rapidly growing area of research.\nIn this section we discussed the issue of errors in genomic and proteomic databases and their impacts for research and application. Sources of these errors may include, among others, entry errors derived from data transfer, original errors derived from source data, and metadata errors (typically provenance-related) derived from the analysis pipeline. Original errors can arise from sequencing and sample preparation instrumentation chemistry, hardware, and software. Metadata errors can arise from bioinformatics software and faulty human interpretation. Each of these errors may be considered noise or the result of some other unintentional cause, but the key problem to note is that each element of the analytical process introduces some level of artifact when creating the analytical product, i.e., what is defined as a peak or a spot, what is the gene scaffold, what is the closed genome, etc. Any difference in process would therefore by its nature have some impact on the final genome. Our goal here is to start drawing connections between these process elements and genome anomalies.\n\nVulnerabilities and intentional tampering \nIn contrast to the data integrity issues discussed in the prior section, errors may also be intentionally introduced into a biological database. For example, consider the hypothetical scenario discussed by Peccoud et al.[16] whereby a graduate student reads an article and subsequently requests the plasmids described, but receives a faulty sample. It may be that the published sequences were fabricated, or that the source laboratory unwittingly sent faulty plasmids. One could also imagine a scenario where an intentionally mislabeled or harmful sequence is submitted to an open database that could later be unknowingly synthesized in a research setting or, more seriously, in a production capacity. Furthermore, depending on how sequences could be submitted to the database, the adversary may be able to keep the pathogenic sequence from being detected by certain anomaly detection heuristics.\nIndividuals may also exploit the vulnerabilities inherent in the database as a cybersystem, leading to errors introduced after publication of data despite manipulation and deletion controls. As with any database, biological databases can be compromised, enabling data integrity issues related to insertion, manipulation, exfiltration, and deletion of data, as well as providing a platform for privilege escalation, unauthorized surveillance, or distribution of malware. Ultimately, the effects of the operating environment and the tools used to deliver databases will inform the most appropriate threat model.\n\nApproaches for improving biological databases \nIn 2000, a workshop titled Bioinformatics: Converting Data to Knowledge[17] tackled the question of biological database integrity as one of its focus areas. At that time, suggested solutions included building organism-type (e.g., eukaryote) specific grammar-based tools, enabling database self-validation through specialized ontologies, advocating for quality control in laboratories to minimize likelihood of errors, and authorizing only trained curators and annotators to enter data. They also recommended that data provenance be maintained so that the data history and evolution can be understood over time. These approaches broadly fall into two categories: ensuring integrity before or during data entry and analyzing data already in a database. Nearly 20 years later, we still emphasize the importance of quality control in laboratories and standardized data entry procedures, but it is clear that errors continue to make their way into databases for a variety of reasons. In this section, we highlight several categories of existing methods to detect data integrity issues in biological databases and outline the strengths and weaknesses of each. We also provide recommendations for improving biological database security.\n\nAutomated approaches for detecting anomalies \nSome biological databases take the manual curation approach, such as the SwissProt subset of the UniProt (Universal Protein Resource Database). This effort requires significant resources to maintain, consisting of three principal investigators, a large staff, and external advisory board.[18] Given the complexity and exponential growth of biological data, automatic methods are needed.\nSome tools have been developed to assess the technical quality of genome assemblies (e.g., QUAST[19]), their completeness in terms of gene content (e.g., BUSCO[20], ProDeGe[21]) and even their contamination level (e.g., acdc[22], CheckM[23]). Currently there are several analysis pipelines and search methodologies to detect potentially contaminated sequences in the published and assembled genome, such as Taxoblast[24], GenomePeek[25], homology search[26], and a multi-step cleaning process followed by a consensus of rankings.[27][28] All these tools and methods require human review or use of additional tools to distinguish true positive from true negative and are therefore not feasible at scale.\nAnother database quality issue is the automated identification of taxonomically anomalous, questionable, or erroneous GenBank taxonomic assignments. Automated error identification of taxonomic assignments now draws on methods such as anomaly detection, classification, and prediction techniques. These methods have proved impactful in areas like computer vision[29] and natural language processing.[30] They have also been adopted by bioinformatics and computational biology.[31] Much of the work in applying machine learning to biological data is for classification and prediction of metadata, e.g., gene or taxonomy prediction in genomics, and structure and function prediction in proteomics. Verification of sequence metadata contained in a database is then performed by comparing with the predicted metadata from the sequence.\nSequence-based methods to detect taxonomically misclassified bacterial genome sequences tend to be based either on distance measures between pairs of sequences or on consistency with a reference 16S rRNA phylogeny. Common distance metrics include the average nucleotide identity (ANI), digital DNA-DNA hybridization (dDDH), multi-locus sequence analysis (MLSA), k-mer overlap (summarized by Federhen et al.[32]), and information theoretic distances.[33] Given a genome distance, taxonomic misclassifications have been discovered by identifying outlier genomes that exceed a manually determined distance threshold to trusted reference genomes.[34][35][36][37][38][32][39] The need for reference genomes is problematic, since approximately 20 percent of the bacterial genome sequences in GenBank currently (as of August, 2017) do not have a reference (or \u201ctype\u201d) genome available (NCBI).[40] The lack of bacterial genomes with a \u201ctype\u201d designation is not due to the cost of sequencing but rather the need to satisfy a specific set of formal requirements[41], which includes submitting culturable isolates to more than one culture collection. This poses a significant challenge for unculturable bacteria.\nDistinct from these pairwise distance-based methods, a recent method for identifying taxonomically mislabeled sequences[42] uses consistency between a given set of taxonomic labels and a phylogenetic tree computed from a multiple sequence alignment of 16S rRNA sequences. This approach uses a single model of evolution to identify sequences whose taxonomic placement is most likely incorrect. However, there are multiple, competing methods for assigning bacterial taxonomy and, in particular, multiple sequence alignment of 16S rRNA can fail to resolve closely related species.[43][44][45]\nMachine learning has been applied to understand the sequences themselves. For example, the tools DeepBind[46] and DeepSEA[47] take sequences as input and learn how variations in the sequences can predict function. The successes of these tools coupled with recent research on sequence anomaly detection using long short-term memory (LSTM) recurrent neural networks (RNNs) in cybersecurity[48] could enable a new technique for biological sequence anomaly detection. Finally, if available, machine learning could potentially be applied to data concerning the sequence sources or data submitters themselves to evaluate quality and trustworthiness. However, that discussion is beyond the scope of this article.\n\nProtections against intentional errors \nIf a trusted method does not exist to ensure the continued quality and revision of content in biological databases, those who use the data should be aware of this risk and account for it in their analysis appropriately. In what follows, we outline previous efforts to develop analytics to detect and mitigate the impact of deliberately introduced database errors, both known and unknown.\nAny machine learning analytic is necessarily a product of the data it observes. In an open data environment, an adversary can directly control any subsequent analysis by changing the data to change an algorithm's underlying model.[49] The focus of a \u201ccounter adversarial\u201d approach to data analytics is to harden machine learning methods against the effects of inputs that are designed to mislead supervised[50][51][52] and unsupervised[53][54] algorithms. It has been shown that there exist label tampering attacks which significantly decrease the accuracy of a classifier, while being nearly undetectable by standard cross validation tests.[55] In other words, the defender does not know the performance of the classifier has been corrupted. To protect against label tampering, an \u201censembles of outlier measures\u201d (EOM) method has been proposed to identify label tampering. The approach relies on a set of attributes that capture the \u201coutlierness\u201d of a sample to predict whether a sample has been tampered with. Tampered samples can then be remediated by changing the sample class label. In the context of a biological database, these labels may be metadata attributes associated with an entry. In the unsupervised machine learning scenario, an adversary may try to subvert a clustering algorithm by, for example, heuristically inserting data points to arbitrarily poison (i.e., merge)[54] clusters. In the context of a genomics database, poisoning of clusters would significantly reduce the ability to detect anomalous genomic sequences. Kegelmeyer et al. demonstrate that their remediation methodology based on an EOM applies equally well in the unsupervised context.[55]\nAs vulnerable cyber systems, best cyber practices can also be leveraged to protect biological databases. However, in the context of intentional manipulation of biological databases, special consideration must be given to the ability of these databases to enable production of dangerous biological material. The International Gene Synthesis Consortium (IGSC), for instance, provides two principal protections against the manufacture of malicious genetic material\u2014known as the Harmonized Screening Protocol.[56] The first is a customer screening, and the second is a screening of DNA sequences against the Regulated Pathogen Database (RPD). This database is built from data on the U.S. Select Agent List, the Australian Group List, and other national lists of regulated pathogens. Members of the IGSC agree to translate each synthetic gene into an amino acid sequence and test for homology. These are then accepted, reviewed, or rejected. The RPD is updated annually and provided to members.\nThe Harmonized Screening Protocol requires at least two difficult processes: sharing the database and updating the database. Sharing the database requires the maintenance of authentication. Providers and users are part of a shared environment where they need to trust that everyone has an authentic and up-to-date version of the database. Updating the database requires maintenance to avoid \u201calert fatigue\u201d from false positives and the dangerous potential case of false negatives resulting in malicious manufacture. Maintaining the security of this requires an environment of authentication and active database inspection and curation. For the former, there may be opportunities to incorporate advanced encryption and authentication algorithms being considered in the cyber domain such as blockchain. However, significant computational resource costs must be contended with.\n\nPreliminary conclusions \nThis survey of concerns with biological databases and methods for ensuring database integrity is certainly not exhaustive but represents broad capabilities within data science and cybersecurity today that have shown promise either within computational biology already, or in tackling similar problems in other domains. A goal of the authors is to illuminate these concerns for a wide audience in the context of the historical lessons learned in cyberspace. In the early days of the internet, the emphasis was on functionality and enabling the actions of largely well-intentioned communities of users. This functionality pervaded every element of our critical infrastructure. However, the same fabric that supports this infrastructure also represents a significant risk. Mitigating this risk after the wide penetration of open functionality is much more difficult than it might have been if the internet had been created with integrity and security in mind. As biological data becomes a bedrock critical infrastructure for the entire bioeconomy\u2014and follows the same exponential trends of size, pervasiveness, and importance as the internet\u2014we have a unique opportunity to ensure that this capability mitigates current and future risks from a worldwide set of actors. This paper calls out several existing research areas that can be leveraged to protect against accidental and intentional modifications and misuse of public biological databases.\n\nAcknowledgements \nThe US Department of Energy (DOE) Laboratories are managed by contracted private entities as Federally Funded Research and Development Companies, and as such each laboratory is operated on behalf of DOE, independently from the operating company. Los Alamos National Laboratory is operated for DOE's National Nuclear Security Administration by Triad National Security, LLC under Contract 89233218CNA000001. Pacific Northwest National Laboratory is operated for DOE by Battelle Memorial Institute under contract DE-AC05-76RL01830. Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for DOE's National Nuclear Security Administration under contract DE-NA0003525.\nAny subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.\n\nAuthor contributions \nJC, JG, CH, EM, CO, EP, CT, and GX wrote sections of the manuscript. JC, JG, CH, EM, CJ, CO, KO, EP, KT, CT, and GX contributed to the formulation of the perspectives therein. NG, KT, and MW worked on document conception and assembly. All authors contributed to manuscript revision and approved of the submitted content.\n\nFunding \nThis research was supported in part by the Department of Homeland Security Science and Technology Directorate, award HSHQPM-14-X-00069.\n\nConflict of interest \nThe authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n\nReferences \n\n\n\u2191 Moussouni, F.; Berti\u2010\u00c9quille, L. (2014). \"Cleaning, Integrating, and Warehousing Genomic Data From Biomedical Resources\". In Elloumi, M.; Zomaya, A.Y.. Biological Knowledge Discovery Handbook. John Wiley & Sons. pp. 35\u201358. doi:10.1002\/9781118617151.ch02. ISBN 9781118617151.   \n\n\u2191 Kim, M.; Lauter, K. (2015). \"Private genome analysis through homomorphic encryption\". BMC Medical Informatics and Decision Making 15 (Suppl 5): S3. doi:10.1186\/1472-6947-15-S5-S3. PMC PMC4699052. PMID 26733152. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699052 .   \n\n\u2191 Mandal, A.; Mitchell, J.C.; Montgomery, H.; Roy, A. (2018). \"Data oblivious genome variants search on Intel SGX\". In Garcia-Alfaro, J.; Herrera-Joancomart\u00ed, J.; Livraga, G.; Rios, R.. Data Privacy Management, Cryptocurrencies and Blockchain Technology. Lecture Notes in Computer Science. Springer International Publishing. pp. 21. doi:10.1007\/978-3-030-00305-0_21. ISBN 9783030003050.   \n\n\u2191 Ozercan, H.I.; Ileri, A.M.; Ayday, E.; Alkan, C. (2018). \"Realizing the potential of blockchain technologies in genomics\". Gemone Research 28 (9): 1255\u201363. doi:10.1101\/gr.207464.116. PMC PMC6120626. PMID 30076130. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6120626 .   \n\n\u2191 5.0 5.1 Merchant, S.; Wood, D.E.; Salzberg, S.L. (2014). \"Unexpected cross-species contamination in genome sequencing projects\". PeerJ 2: e675. doi:10.7717\/peerj.675. PMC PMC4243333. PMID 25426337. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4243333 .   \n\n\u2191 Strong, M.J.; Xu, G.; Morici, L. et al. (2014). \"Microbial contamination in next generation sequencing: implications for sequence-based analysis of clinical samples\". PLoS Pathogens 10 (11): e1004437. doi:10.1371\/journal.ppat.1004437. PMC PMC4239086. PMID 25412476. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4239086 .   \n\n\u2191 Pilli, E.; Modi, A.; Serpico, C. et al. (2013). \"Monitoring DNA contamination in handled vs. directly excavated ancient human skeletal remains\". PLoS One 8 (1): e52524. doi:10.1371\/journal.pone.0052524. PMC PMC3556025. PMID 23372650. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3556025 .   \n\n\u2191 Ballenghien, M.; Faivre, N;. Galtier, N. (2017). \"Patterns of cross-contamination in a multispecies population genomic project: Detection, quantification, impact, and solutions\". BMC Biology 15 (1): 25. doi:10.1186\/s12915-017-0366-6. PMC PMC5370491. PMID 28356154. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5370491 .   \n\n\u2191 Simion, P.; Philippe, H.; Baurain, D. et al. (2017). \"A Large and Consistent Phylogenomic Dataset Supports Sponges as the Sister Group to All Other Animals\". Current Biology 27 (7): 958-967. doi:10.1016\/j.cub.2017.02.031. PMID 28318975.   \n\n\u2191 Witt, N.; Rodger, G.; Vandesompele, J. et al. (2009). \"An assessment of air as a source of DNA contamination encountered when performing PCR\". Journal of Biomolecular Techniques 20 (5): 236\u201340. PMC PMC2777341. PMID 19949694. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2777341 .   \n\n\u2191 Longo, M.S.; O'Neill, M.J.; O'Neill, R.J. (2011). \"Abundant human DNA contamination identified in non-primate genome databases\". PLoS One 6 (2): e16410. doi:10.1371\/journal.pone.0016410. PMC PMC3040168. PMID 21358816. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040168 .   \n\n\u2191 Puiu, D.; Salzberg, S.L. (2008). \"Re-assembly of the genome of Francisella tularensis subsp. holarctica OSU18\". PLoS One 3 (10): e3427. doi:10.1371\/journal.pone.0003427. PMC PMC2561293. PMID 18927608. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2561293 .   \n\n\u2191 Cooper, P. (2014). \"Proof by synthesis of Tobacco mosaic virus\". Genome Biology 15 (5): R67. doi:10.1186\/gb-2014-15-5-r67. PMC PMC4072989. PMID 24887356. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4072989 .   \n\n\u2191 Philippe, H;. Brinkmann, H.; Lavrov, D.V. et al. (2011). \"Resolving difficult phylogenetic questions: Why more sequences are not enough\". PLoS Biology 9 (3): e1000602. doi:10.1371\/journal.pbio.1000602. PMC PMC3057953. PMID 21423652. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3057953 .   \n\n\u2191 Laurin-Lemay, S.; Brinkmann, H.; Philippe, H. (2012). \"Resolving difficult phylogenetic questions: Why more sequences are not enough\". PLoS Biology 22 (15): R593\u20134. doi:10.1016\/j.cub.2012.06.013. PMID 22877776.   \n\n\u2191 Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". Trends in Biotechnology 36 (1): 4\u20137. doi:10.1016\/j.tibtech.2017.10.012. PMID 29224719.   \n\n\u2191 National Research Council (2000). \"Bioinformatics: Converting Data to Knowledge\". National Academies Press. doi:10.17226\/9990. https:\/\/www.nap.edu\/catalog\/9990\/bioinformatics-converting-data-to-knowledge-workshop-summary .   \n\n\u2191 Pundir, S;. Martin, M.J.; O'Donovan, C. (2017). \"UniProt Protein Knowledgebase\". Methods in Molecular Biology 1558: 41\u201355. doi:10.1007\/978-1-4939-6783-4_2. PMC PMC5565770. PMID 28150232. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5565770 .   \n\n\u2191 Gurevich, A.; Saveliev, V.; Vyahhi, N.; Tesler, G. (2013). \"QUAST: Quality assessment tool for genome assemblies\". Bioinformatics 29 (8): 1072\u20135. doi:10.1093\/bioinformatics\/btt086. PMC PMC3624806. PMID 23422339. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3624806 .   \n\n\u2191 Sim\u00e3o, F.A.; Waterhouse, R.M.; Ioannidis, P. et al. (2015). \"BUSCO: Assessing genome assembly and annotation completeness with single-copy orthologs\". Bioinformatics 31 (19): 3210\u20132. doi:10.1093\/bioinformatics\/btv351. PMID 26059717.   \n\n\u2191 Tennessen, K.; Andersen, E.; Clingenpeel, S. et al. (2016). \"ProDeGe: A computational protocol for fully automated decontamination of genomes\". The ISME Journal 10 (1): 269\u201372. doi:10.1038\/ismej.2015.100. PMC PMC4681846. PMID 26057843. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4681846 .   \n\n\u2191 Lux, M.; Kr\u00fcger, J.; Rinke, C. et al. (2016). \"acdc - Automated Contamination Detection and Confidence estimation for single-cell genome data\". BMC Bioinformatics 17 (1): 543. doi:10.1186\/s12859-016-1397-7. PMC PMC5168860. PMID 27998267. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5168860 .   \n\n\u2191 Parks, D.H.; Imelfort, M.; Skennerton, C.T. et al. (2015). \"CheckM: Assessing the quality of microbial genomes recovered from isolates, single cells, and metagenomes\". Genome Research 25 (7): 1043\u201355. doi:10.1101\/gr.186072.114. PMC PMC4484387. PMID 25977477. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4484387 .   \n\n\u2191 Dittami, S.M.; Corre, E. (2017). \"Detection of bacterial contaminants and hybrid sequences in the genome of the kelp Saccharina japonica using Taxoblast\". PeerJ 5: e4073. doi:10.7717\/peerj.4073. PMC PMC5695246. PMID 29158994. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5695246 .   \n\n\u2191 McNair, K.; Edwards, R.A. (2015). \"GenomePeek: An online tool for prokaryotic genome and metagenome analysis\". PeerJ 3: e1025. doi:10.7717\/peerj.1025. PMC PMC4476108. PMID 26157610. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4476108 .   \n\n\u2191 Kryukov, K.; Imanishi, T. (2016). \"Human Contamination in Public Genome Assemblies\". PLoS One 11 (9): e0162424. doi:10.1371\/journal.pone.0162424. PMC PMC5017631. PMID 27611326. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5017631 .   \n\n\u2191 Cornet, L.; Meunier, L.; Van Vlierberghe, M. et al. (2018). \"Consensus assessment of the contamination level of publicly available cyanobacterial genomes\". PLoS One 13 (7): e0200323. doi:10.1371\/journal.pone.0200323. PMC PMC6059444. PMID 30044797. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6059444 .   \n\n\u2191 Lu, J.; Salzberg, S.L. (2018). \"Removing contaminants from databases of draft genomes\". PLoS Computational Biology 14 (6): e1006277. doi:10.1371\/journal.pcbi.1006277. PMC PMC6034898. PMID 29939994. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6034898 .   \n\n\u2191 Krizhevsky, A.; Sutskever, U.; Hinton, G.E. (2017). \"ImageNet classification with deep convolutional neural networks\". Communications of the ACM 60 (6): 84\u201390. doi:10.1145\/3065386.   \n\n\u2191 Sutskever, U.; Vinyals, O.; Le, Q.V. (2014). \"Sequence to Sequence Learning with Neural Networks\". Proceedings from Advances in Neural Information Processing Systems 2014. http:\/\/papers.nips.cc\/paper\/5346-sequence-to-sequence-learning-with-neural .   \n\n\u2191 Larra\u00f1aga, P.; Calvo, B.; Santana, R. et al. (2006). \"Machine learning in bioinformatics\". Briefings in Bioinformatics 7 (1): 86\u2013112. doi:10.1093\/bib\/bbk007. PMID 16761367.   \n\n\u2191 32.0 32.1 Federhen, S.; Rossello-Mora, R.; Klenk, H.-P. et al. (2016). \"Meeting report: GenBank microbial genomic taxonomy workshop (12\u201313 May, 2015)\". Standards in Genomic Sciences 11: 15. doi:10.1186\/s40793-016-0134-1. PMC PMC4748488. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4748488 .   \n\n\u2191 Li, M.; Chen, X.; Li, X. et al. (2004). \"The Similarity Metric\". IEEE Transactions on Information Theory 50 (12): 3250\u201364. doi:10.1109\/TIT.2004.838101.   \n\n\u2191 Goris, J.; Konstantinudis, K.T.; Klappenbach, J.A. et al. (2007). \"DNA-DNA hybridization values and their relationship to whole-genome sequence similarities\". International Journal of Systematic and Evolutionary Microbiology 57 (Pt 1): 81\u201391. doi:10.1099\/ijs.0.64483-0. PMID 17220447.   \n\n\u2191 Colston, S.M.; Fullmer, M.S.; Beka, L. et al. (2014). \"Bioinformatic genome comparisons for taxonomic and phylogenetic assignments using Aeromonas as a test case\". mBio 5 (6): e02136. doi:10.1128\/mBio.02136-14. PMC PMC4251997. PMID 25406383. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4251997 .   \n\n\u2191 Figueras, M.J.; Beaz-Hidalgo, R.; Hossain, M.J.; Liles, M.R. (2014). \"Taxonomic affiliation of new genomes should be verified using average nucleotide identity and multilocus phylogenetic analysis\". Genome Announcements 2 (6): e00927-14. doi:10.1128\/genomeA.00927-14. PMC PMC4256179. PMID 25477398. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4256179 .   \n\n\u2191 Kim, M.; Oh, H.S.; Park, S.C.; Chun, J. (2014). \"Towards a taxonomic coherence between average nucleotide identity and 16S rRNA gene sequence similarity for species demarcation of prokaryotes\". International Journal of Systematic and Evolutionary Microbiology 64 (Pt 2): 346-51. doi:10.1099\/ijs.0.059774-0. PMID 24505072.   \n\n\u2191 Beaz-Hidalgo, R;. Hossain, M.J.; Liles, M.R.; Figueras, M.J. (2015). \"Strategies to avoid wrongly labelled genomes using as example the detected wrong taxonomic affiliation for aeromonas genomes in the GenBank database\". PLoS One 10 (1): e0115813. doi:10.1371\/journal.pone.0115813. PMC PMC4301921. PMID 25607802. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4301921 .   \n\n\u2191 Tanizawa, Y.; Fujisawa, T.; Kaminuma, E. et al. (2016). \"DFAST and DAGA: web-based integrated genome annotation tools and resources\". Bioscience of Microbiota, Food and Health 35 (4): 173-184. doi:10.12938\/bmfh.16-003. PMC PMC5107635. PMID 27867804. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5107635 .   \n\n\u2191 \"Bacterial ANI Report\". NCBI. ftp:\/\/ftp.ncbi.nlm.nih.gov\/genomes\/ASSEMBLY_REPORTS\/ANI_report_bacteria.txt .   \n\n\u2191 Federhen, S. (2015). \"Type material in the NCBI Taxonomy Database\". Nucleic Acids Research 43 (DB1): D1086\u201398. doi:10.1093\/nar\/gku1127. PMC PMC4383940. PMID 25398905. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383940 .   \n\n\u2191 Kozlov, A.M.; Zhang, J.; Yilmaz, P. et al. (2016). \"Phylogeny-aware identification and correction of taxonomically mislabeled sequences\". Nucleic Acids Research 44 (11): 5022-33. doi:10.1093\/nar\/gkw396. PMC PMC4914121. PMID 27166378. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4914121 .   \n\n\u2191 Richter, M.; Rossell\u00f3-M\u00f3ra, R. (2009). \"Shifting the genomic gold standard for the prokaryotic species definition\". Proceedings of the National Academy of Sciences of the United States of America 106 (45): 19126-31. doi:10.1073\/pnas.0906412106. PMC PMC2776425. PMID 19855009. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2776425 .   \n\n\u2191 K\u00e4mpfer, P.; Glaeser, S.P. (2012). \"Prokaryotic taxonomy in the sequencing era--the polyphasic approach revisited\". Environmental Microbiology 14 (2): 291-317. doi:10.1111\/j.1462-2920.2011.02615.x. PMID 22040009.   \n\n\u2191 Larsen, M.V.; Cosentino, S.; Lukjancenko, O. et al. (2014). \"Benchmarking of methods for genomic taxonomy\". Journal of Clinical Microbiology 52 (5): 1529-39. doi:10.1128\/JCM.02981-13. PMC PMC3993634. PMID 24574292. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3993634 .   \n\n\u2191 Alipanahi, B.; Delong, A.; Weirauch, M.T.; Frey, B.J. (2015). \"Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning\". Nature Biotechnology 33 (8): 831\u20138. doi:10.1038\/nbt.3300. PMID 26213851.   \n\n\u2191 Zhou, J.; Troyanskaya, O.G. (2015). \"Predicting effects of noncoding variants with deep learning-based sequence model\". Nature Methods 12 (10): 931\u20134. doi:10.1038\/nmeth.3547. PMC PMC4768299. PMID 26301843. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4768299 .   \n\n\u2191 Brown, A.; Tuor, A.; Hutchinson, B.; Nichols, N. (2018). \"Recurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detection\". Proceedings of the First Workshop on Machine Learning for Computing Systems: 1. doi:10.1145\/3217871.3217872.   \n\n\u2191 Goodfellow, I.; Pouget-Abadie, J.; Mirza, M. et al. (2014). \"Generative Adversarial Nets\". Proceedings from Advances in Neural Information Processing Systems 2014. http:\/\/papers.nips.cc\/paper\/5423-generative-adversarial-nets .   \n\n\u2191 Dalvi, N.; Domingos, P.; Mausam, S. et al. (2004). \"Adversarial classification\". Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining: 99\u2013108. doi:10.1145\/1014052.1014066.   \n\n\u2191 Kantarc\u0131o\u011flu, M.; Xi, B.; Clifton, C. (2011). \"Classifier evaluation and attribute selection against active adversaries\". Data Mining and Knowledge Discovery 22 (1\u20132): 291\u2013335. doi:10.1007\/s10618-010-0197-3.   \n\n\u2191 Biggio, B.; Corona, I.; Maiorca, D. et al. (2013). \"Evasion Attacks against Machine Learning at Test Time\". In Blockeel, H.; Kersting, K.; Nijssen, S.; \u017delezn\u00fd, F.. Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Science. Springer. pp. 387\u2013402. doi:10.1007\/978-3-642-40994-3_25. ISBN 9783642409943.   \n\n\u2191 Dutrisac, J.G.; Skillicorn, D.B. (2008). \"Hiding clusters in adversarial settings\". Proceedings from the 2008 IEEE International Conference on Intelligence and Security Informatics: 185\u201387. doi:10.1109\/ISI.2008.4565051.   \n\n\u2191 54.0 54.1 Biggio, B.; Pillai, I.; Bul\u00f2, S.R. et al. (2013). \"Is data clustering in adversarial settings secure?\". Proceedings of the 2013 ACM Workshop on Artificial Intelligence and Security: 87\u201398. doi:10.1145\/2517312.2517321.   \n\n\u2191 55.0 55.1 Kegelmeyer, P.; Shead, T.M.; Crussell, J. et al. (08 May 2015). \"Counter Adversarial Data Analytics\" (PDF). Sandia Report SAND2015-3711. Sandia National Laboratory. https:\/\/www.sandia.gov\/~wpk\/pubs\/publications\/cada-full-uur.pdf .   \n\n\u2191 International Gene Synthesis Consortium (19 November 2017). \"Harmonized Screening Protocol v2.0\" (PDF). https:\/\/genesynthesisconsortium.org\/wp-content\/uploads\/IGSCHarmonizedProtocol11-21-17.pdf .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The footnote in the original material were turned into an inline references for this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\">https:\/\/www.limswiki.org\/index.php\/Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on cybersecurityLIMSwiki journal articles on data management and sharing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 19 June 2019, at 01:23.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 131 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","484090b9c88bd26fd5f834f6ab7cbd8a_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Defending_our_public_biological_databases_as_a_global_critical_infrastructure skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Defending our public biological databases as a global critical infrastructure<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Progress in modern biology is being driven, in part, by the large amounts of freely available data in public resources such as the International Nucleotide Sequence Database Collaboration (INSDC), the world's primary database of biological sequence (and related) <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>. INSDC and similar databases have dramatically increased the pace of fundamental biological discovery and enabled a host of innovative therapeutic, diagnostic, and forensic applications. However, as high-value, openly shared resources with a high degree of assumed trust, these repositories share compelling similarities to the early days of the internet. Consequently, as public biological databases continue to increase in size and importance, we expect that they will face the same threats as undefended cyberspace. There is a unique opportunity, before a significant breach and loss of trust occurs, to ensure they evolve with quality and security as a design philosophy rather than costly \u201cretrofitted\u201d mitigations. This perspective article surveys some potential quality assurance and security weaknesses in existing open <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomic<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Proteomics\" title=\"Proteomics\" class=\"wiki-link\" data-key=\"ac19f0aedfd86ae1fa91c52fad65ab21\">proteomic<\/a> repositories, describes methods to mitigate the likelihood of both intentional and unintentional errors, and offers recommendations for risk mitigation based on lessons learned from <a href=\"https:\/\/www.limswiki.org\/index.php\/Cybersecurity\" title=\"Cybersecurity\" class=\"mw-redirect wiki-link\" data-key=\"ba653dc2a1384e5f9f6ac9dc1a740109\">cybersecurity<\/a>.\n<\/p><p><b>Keywords<\/b>: cyberbiosecurity, biosecurity, cybersecurity, biological databases, machine learning, bioeconomy\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Although an openly shared interaction platform confers great value to the biological research community, it may also introduce quality and security risks. Without a system for trusted correction and revision, these shared resources may facilitate widespread dissemination and use of low-quality content, for instance, taxonomically misclassified or erroneous <a href=\"https:\/\/www.limswiki.org\/index.php\/Sequencing\" title=\"Sequencing\" class=\"mw-disambig wiki-link\" data-key=\"e36167a9eb152ca16a0c4c4e6d13f323\">sequences<\/a>. Furthermore, as these public databases increase in size and importance, they may fall victim to the same security issues and abuses that plague cyberspace to this day. If we act now by developing the databases with quality and security as a design philosophy, we can protect these databases at a much lower cost and with fewer challenges than we currently face with the internet.\n<\/p><p>In this perspective article, the authors aim to outline some potential quality assurance and security weaknesses in existing public biological repositories. In the background section we provide a discussion of errors present in public biological databases and discuss possible security vulnerabilities inherent in their access, publication, and distribution models and systems. Both unintentional and intentional errors are discussed, the latter of which has not been given significant consideration in literature.<sup id=\"rdp-ebb-cite_ref-MoussouniClean13_1-0\" class=\"reference\"><a href=\"#cite_note-MoussouniClean13-1\">[1]<\/a><\/sup> Afterwards, we attempt to introduce greater trust in the data and analyses by providing recommendations to mitigate or account for these errors and vulnerabilities and point to approaches used by other internet databases. Finally, we summarize our recommendations in the conclusions section.\n<\/p><p>This article focuses on databases which contain public and freely available data. We recognize that other biological databases exist which contain private, sensitive, or otherwise valuable data (e.g., human genomes). While unauthorized disclosure is not a formal concern in public, non-human databases, safeguarding against intentional or unintentional erroneous content is. Some approaches have been proposed to protect unauthorized disclosure<sup id=\"rdp-ebb-cite_ref-KimPrivate15_2-0\" class=\"reference\"><a href=\"#cite_note-KimPrivate15-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MandalData18_3-0\" class=\"reference\"><a href=\"#cite_note-MandalData18-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OzercanRealiz18_4-0\" class=\"reference\"><a href=\"#cite_note-OzercanRealiz18-4\">[4]<\/a><\/sup> and, while we don't survey these approaches in this perspective, we note that the public database community may benefit from these ideas as well.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background:_Problems_with_public_biological_databases\">Background: Problems with public biological databases<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Data_integrity\">Data integrity<\/span><\/h3>\n<p>An important goal for <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> is the continuous improvement of biological databases. Given the rapid nature of this improvement and the rate of data production though, the content of these repositories is not without error. For example, the problem of contaminated sequences has been recognized for nearly two decades, with evidence stating that bacteria and human error are the two most common sources of contamination.<sup id=\"rdp-ebb-cite_ref-MerchantUnexp14_5-0\" class=\"reference\"><a href=\"#cite_note-MerchantUnexp14-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-StrongMicro14_6-0\" class=\"reference\"><a href=\"#cite_note-StrongMicro14-6\">[6]<\/a><\/sup> Ancient DNA is also particularly affected by human contamination.<sup id=\"rdp-ebb-cite_ref-PilliMonit13_7-0\" class=\"reference\"><a href=\"#cite_note-PilliMonit13-7\">[7]<\/a><\/sup> These contaminants are frequently introduced during experiments<sup id=\"rdp-ebb-cite_ref-MerchantUnexp14_5-1\" class=\"reference\"><a href=\"#cite_note-MerchantUnexp14-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BallenghienPatterns17_8-0\" class=\"reference\"><a href=\"#cite_note-BallenghienPatterns17-8\">[8]<\/a><\/sup> from natural associations and insufficient purification.<sup id=\"rdp-ebb-cite_ref-SimionALarge17_9-0\" class=\"reference\"><a href=\"#cite_note-SimionALarge17-9\">[9]<\/a><\/sup> In the past few years, additional reports have highlighted cases of DNA contamination in published genome data<sup id=\"rdp-ebb-cite_ref-WittAnAss09_10-0\" class=\"reference\"><a href=\"#cite_note-WittAnAss09-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LongoAbund11_11-0\" class=\"reference\"><a href=\"#cite_note-LongoAbund11-11\">[11]<\/a><\/sup>, suggesting that DNA contamination may be more widespread than previously thought. We recognize that errors and omissions can occur in open databases both at the sequence and at the <a href=\"https:\/\/www.limswiki.org\/index.php\/Metadata\" title=\"Metadata\" class=\"wiki-link\" data-key=\"f872d4d6272811392bafe802f3edf2d8\">metadata<\/a> levels, but for this article we mainly focus on sequence and taxonomic data concerns for the purposes of illustrating some of the many <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_integrity\" title=\"Data integrity\" class=\"wiki-link\" data-key=\"382a9bb77ee3e36bb3b37c79ed813167\">data integrity<\/a> challenges possible.\n<\/p><p>In addition to contamination, two high-profile examples of sequence errors include the reassembly of a misassembled <i>Francisella tularensis<\/i> genome<sup id=\"rdp-ebb-cite_ref-PuiuReass08_12-0\" class=\"reference\"><a href=\"#cite_note-PuiuReass08-12\">[12]<\/a><\/sup> and the identification of single nucleotide errors in a reference <i>Tobacco mosaic virus<\/i> (TMV) genome.<sup id=\"rdp-ebb-cite_ref-CooperProof14_13-0\" class=\"reference\"><a href=\"#cite_note-CooperProof14-13\">[13]<\/a><\/sup> Without a way to flag or remove the erroneous entries, future researchers are left to continually rediscover them. The errors in the reference TMV sequence are particularly disturbing. The taxonomic assignment corresponds to a pathogenic strain, but due to two erroneous single nucleotide polymorphisms (SNPs), virions synthesized from the published reference sequence are atypically not infectious. Overlooked contamination in reference genomes can thereby lead to wrong or confusing results and may have major detrimental effects on biological conclusions.<sup id=\"rdp-ebb-cite_ref-PhilippeResolv11_14-0\" class=\"reference\"><a href=\"#cite_note-PhilippeResolv11-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Laurin-LemayOrigin12_15-0\" class=\"reference\"><a href=\"#cite_note-Laurin-LemayOrigin12-15\">[15]<\/a><\/sup> While resequencing could be used to identify and correct sequence errors, it is only possible when the original source material is available. For the given example of single nucleotide errors in the TMV genome, the biological sample (sequenced in 1982) no longer exists. In addition to missing samples, samples of high consequence human and agricultural pathogens may not be available for resequencing.\n<\/p><p>Database integrity considerations for proteomics are generally similar to those for genomics because databases of protein sequences are derived from genome sequencing, via genome annotation and <i>in silico<\/i> translation. A sequence database error is unlikely to result in spurious detection of a protein that is present in the <a href=\"https:\/\/www.limswiki.org\/index.php\/Sample_(material)\" title=\"Sample (material)\" class=\"wiki-link\" data-key=\"7f8cd41a077a88d02370c02a3ba3d9d6\">sample<\/a> (false positive), but it could easily lead to a failure to detect a protein that is present (false negative). This is particularly concerning for discovery of accurate peptide signatures for use in targeted assays, a rapidly growing area of research.\n<\/p><p>In this section we discussed the issue of errors in genomic and proteomic databases and their impacts for research and application. Sources of these errors may include, among others, entry errors derived from data transfer, original errors derived from source data, and metadata errors (typically provenance-related) derived from the analysis pipeline. Original errors can arise from sequencing and sample preparation instrumentation chemistry, hardware, and software. Metadata errors can arise from bioinformatics software and faulty human interpretation. Each of these errors may be considered noise or the result of some other unintentional cause, but the key problem to note is that each element of the analytical process introduces some level of artifact when creating the analytical product, i.e., what is defined as a peak or a spot, what is the gene scaffold, what is the closed genome, etc. Any difference in process would therefore by its nature have some impact on the final genome. Our goal here is to start drawing connections between these process elements and genome anomalies.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Vulnerabilities_and_intentional_tampering\">Vulnerabilities and intentional tampering<\/span><\/h3>\n<p>In contrast to the data integrity issues discussed in the prior section, errors may also be intentionally introduced into a biological database. For example, consider the hypothetical scenario discussed by Peccoud <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-PeccoudCyber18_16-0\" class=\"reference\"><a href=\"#cite_note-PeccoudCyber18-16\">[16]<\/a><\/sup> whereby a graduate student reads an article and subsequently requests the plasmids described, but receives a faulty sample. It may be that the published sequences were fabricated, or that the source <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> unwittingly sent faulty plasmids. One could also imagine a scenario where an intentionally mislabeled or harmful sequence is submitted to an open database that could later be unknowingly synthesized in a research setting or, more seriously, in a production capacity. Furthermore, depending on how sequences could be submitted to the database, the adversary may be able to keep the pathogenic sequence from being detected by certain anomaly detection heuristics.\n<\/p><p>Individuals may also exploit the vulnerabilities inherent in the database as a cybersystem, leading to errors introduced after publication of data despite manipulation and deletion controls. As with any database, biological databases can be compromised, enabling data integrity issues related to insertion, manipulation, exfiltration, and deletion of data, as well as providing a platform for privilege escalation, unauthorized surveillance, or distribution of malware. Ultimately, the effects of the operating environment and the tools used to deliver databases will inform the most appropriate threat model.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Approaches_for_improving_biological_databases\">Approaches for improving biological databases<\/span><\/h2>\n<p>In 2000, a workshop titled <i>Bioinformatics: Converting Data to Knowledge<\/i><sup id=\"rdp-ebb-cite_ref-NRCBio00_17-0\" class=\"reference\"><a href=\"#cite_note-NRCBio00-17\">[17]<\/a><\/sup> tackled the question of biological database integrity as one of its focus areas. At that time, suggested solutions included building organism-type (e.g., eukaryote) specific grammar-based tools, enabling database self-validation through specialized ontologies, advocating for quality control in laboratories to minimize likelihood of errors, and authorizing only trained curators and annotators to enter data. They also recommended that data provenance be maintained so that the data history and evolution can be understood over time. These approaches broadly fall into two categories: ensuring integrity before or during data entry and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_analysis\" title=\"Data analysis\" class=\"wiki-link\" data-key=\"545c95e40ca67c9e63cd0a16042a5bd1\">analyzing data<\/a> already in a database. Nearly 20 years later, we still emphasize the importance of <a href=\"https:\/\/www.limswiki.org\/index.php\/Quality_control\" title=\"Quality control\" class=\"wiki-link\" data-key=\"1e0e0c2eb3e45aff02f5d61799821f0f\">quality control<\/a> in laboratories and standardized data entry procedures, but it is clear that errors continue to make their way into databases for a variety of reasons. In this section, we highlight several categories of existing methods to detect data integrity issues in biological databases and outline the strengths and weaknesses of each. We also provide recommendations for improving biological database security.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Automated_approaches_for_detecting_anomalies\">Automated approaches for detecting anomalies<\/span><\/h3>\n<p>Some biological databases take the manual curation approach, such as the SwissProt subset of the UniProt (Universal Protein Resource Database). This effort requires significant resources to maintain, consisting of three principal investigators, a large staff, and external advisory board.<sup id=\"rdp-ebb-cite_ref-PundirUniProt17_18-0\" class=\"reference\"><a href=\"#cite_note-PundirUniProt17-18\">[18]<\/a><\/sup> Given the complexity and exponential growth of biological data, automatic methods are needed.\n<\/p><p>Some tools have been developed to assess the technical quality of genome assemblies (e.g., QUAST<sup id=\"rdp-ebb-cite_ref-GurevichQUAST13_19-0\" class=\"reference\"><a href=\"#cite_note-GurevichQUAST13-19\">[19]<\/a><\/sup>), their completeness in terms of gene content (e.g., BUSCO<sup id=\"rdp-ebb-cite_ref-Sim.C3.A3oBUSCO15_20-0\" class=\"reference\"><a href=\"#cite_note-Sim.C3.A3oBUSCO15-20\">[20]<\/a><\/sup>, ProDeGe<sup id=\"rdp-ebb-cite_ref-TennessenProDeGe16_21-0\" class=\"reference\"><a href=\"#cite_note-TennessenProDeGe16-21\">[21]<\/a><\/sup>) and even their contamination level (e.g., acdc<sup id=\"rdp-ebb-cite_ref-LuxACDC16_22-0\" class=\"reference\"><a href=\"#cite_note-LuxACDC16-22\">[22]<\/a><\/sup>, CheckM<sup id=\"rdp-ebb-cite_ref-ParksCheckM15_23-0\" class=\"reference\"><a href=\"#cite_note-ParksCheckM15-23\">[23]<\/a><\/sup>). Currently there are several analysis pipelines and search methodologies to detect potentially contaminated sequences in the published and assembled genome, such as Taxoblast<sup id=\"rdp-ebb-cite_ref-DittamiDetect17_24-0\" class=\"reference\"><a href=\"#cite_note-DittamiDetect17-24\">[24]<\/a><\/sup>, GenomePeek<sup id=\"rdp-ebb-cite_ref-McNairGenome15_25-0\" class=\"reference\"><a href=\"#cite_note-McNairGenome15-25\">[25]<\/a><\/sup>, homology search<sup id=\"rdp-ebb-cite_ref-RyukovHuman16_26-0\" class=\"reference\"><a href=\"#cite_note-RyukovHuman16-26\">[26]<\/a><\/sup>, and a multi-step cleaning process followed by a consensus of rankings.<sup id=\"rdp-ebb-cite_ref-CornetConcensus18_27-0\" class=\"reference\"><a href=\"#cite_note-CornetConcensus18-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LuRemoving18_28-0\" class=\"reference\"><a href=\"#cite_note-LuRemoving18-28\">[28]<\/a><\/sup> All these tools and methods require human review or use of additional tools to distinguish true positive from true negative and are therefore not feasible at scale.\n<\/p><p>Another database quality issue is the automated identification of taxonomically anomalous, questionable, or erroneous GenBank taxonomic assignments. Automated error identification of taxonomic assignments now draws on methods such as anomaly detection, classification, and prediction techniques. These methods have proved impactful in areas like computer vision<sup id=\"rdp-ebb-cite_ref-KrizhevskyImageNet17_29-0\" class=\"reference\"><a href=\"#cite_note-KrizhevskyImageNet17-29\">[29]<\/a><\/sup> and natural language processing.<sup id=\"rdp-ebb-cite_ref-SutskeverSeq14_30-0\" class=\"reference\"><a href=\"#cite_note-SutskeverSeq14-30\">[30]<\/a><\/sup> They have also been adopted by bioinformatics and computational biology.<sup id=\"rdp-ebb-cite_ref-Larra.C3.B1agaMach06_31-0\" class=\"reference\"><a href=\"#cite_note-Larra.C3.B1agaMach06-31\">[31]<\/a><\/sup> Much of the work in applying machine learning to biological data is for classification and prediction of metadata, e.g., gene or taxonomy prediction in genomics, and structure and function prediction in proteomics. Verification of sequence metadata contained in a database is then performed by comparing with the predicted metadata from the sequence.\n<\/p><p>Sequence-based methods to detect taxonomically misclassified bacterial genome sequences tend to be based either on distance measures between pairs of sequences or on consistency with a reference 16S rRNA phylogeny. Common distance metrics include the average nucleotide identity (ANI), digital DNA-DNA hybridization (dDDH), multi-locus sequence analysis (MLSA), k-mer overlap (summarized by Federhen <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-FederhenMeeting16_32-0\" class=\"reference\"><a href=\"#cite_note-FederhenMeeting16-32\">[32]<\/a><\/sup>), and information theoretic distances.<sup id=\"rdp-ebb-cite_ref-LiTheSim04_33-0\" class=\"reference\"><a href=\"#cite_note-LiTheSim04-33\">[33]<\/a><\/sup> Given a genome distance, taxonomic misclassifications have been discovered by identifying outlier genomes that exceed a manually determined distance threshold to trusted reference genomes.<sup id=\"rdp-ebb-cite_ref-GorisDNA07_34-0\" class=\"reference\"><a href=\"#cite_note-GorisDNA07-34\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ColstonBio14_35-0\" class=\"reference\"><a href=\"#cite_note-ColstonBio14-35\">[35]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FiguerasTax14_36-0\" class=\"reference\"><a href=\"#cite_note-FiguerasTax14-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KimTowards14_37-0\" class=\"reference\"><a href=\"#cite_note-KimTowards14-37\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Beaz-HidalgoStrat15_38-0\" class=\"reference\"><a href=\"#cite_note-Beaz-HidalgoStrat15-38\">[38]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FederhenMeeting16_32-1\" class=\"reference\"><a href=\"#cite_note-FederhenMeeting16-32\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TanizawaDFAST16_39-0\" class=\"reference\"><a href=\"#cite_note-TanizawaDFAST16-39\">[39]<\/a><\/sup> The need for reference genomes is problematic, since approximately 20 percent of the bacterial genome sequences in GenBank currently (as of August, 2017) do not have a reference (or \u201ctype\u201d) genome available (NCBI).<sup id=\"rdp-ebb-cite_ref-NCBIBacterial_40-0\" class=\"reference\"><a href=\"#cite_note-NCBIBacterial-40\">[40]<\/a><\/sup> The lack of bacterial genomes with a \u201ctype\u201d designation is not due to the cost of sequencing but rather the need to satisfy a specific set of formal requirements<sup id=\"rdp-ebb-cite_ref-FederhenType15_41-0\" class=\"reference\"><a href=\"#cite_note-FederhenType15-41\">[41]<\/a><\/sup>, which includes submitting culturable isolates to more than one culture collection. This poses a significant challenge for unculturable bacteria.\n<\/p><p>Distinct from these pairwise distance-based methods, a recent method for identifying taxonomically mislabeled sequences<sup id=\"rdp-ebb-cite_ref-KozlovPhylo16_42-0\" class=\"reference\"><a href=\"#cite_note-KozlovPhylo16-42\">[42]<\/a><\/sup> uses consistency between a given set of taxonomic labels and a phylogenetic tree computed from a multiple sequence alignment of 16S rRNA sequences. This approach uses a single model of evolution to identify sequences whose taxonomic placement is most likely incorrect. However, there are multiple, competing methods for assigning bacterial taxonomy and, in particular, multiple sequence alignment of 16S rRNA can fail to resolve closely related species.<sup id=\"rdp-ebb-cite_ref-RichterShift09_43-0\" class=\"reference\"><a href=\"#cite_note-RichterShift09-43\">[43]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-K.C3.A4mpferProkary12_44-0\" class=\"reference\"><a href=\"#cite_note-K.C3.A4mpferProkary12-44\">[44]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LarsenBench14_45-0\" class=\"reference\"><a href=\"#cite_note-LarsenBench14-45\">[45]<\/a><\/sup>\n<\/p><p>Machine learning has been applied to understand the sequences themselves. For example, the tools DeepBind<sup id=\"rdp-ebb-cite_ref-AlipanahiPredict15_46-0\" class=\"reference\"><a href=\"#cite_note-AlipanahiPredict15-46\">[46]<\/a><\/sup> and DeepSEA<sup id=\"rdp-ebb-cite_ref-ZhouPredict15_47-0\" class=\"reference\"><a href=\"#cite_note-ZhouPredict15-47\">[47]<\/a><\/sup> take sequences as input and learn how variations in the sequences can predict function. The successes of these tools coupled with recent research on sequence anomaly detection using long short-term memory (LSTM) recurrent neural networks (RNNs) in cybersecurity<sup id=\"rdp-ebb-cite_ref-BrownRecurrent18_48-0\" class=\"reference\"><a href=\"#cite_note-BrownRecurrent18-48\">[48]<\/a><\/sup> could enable a new technique for biological sequence anomaly detection. Finally, if available, machine learning could potentially be applied to data concerning the sequence sources or data submitters themselves to evaluate quality and trustworthiness. However, that discussion is beyond the scope of this article.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Protections_against_intentional_errors\">Protections against intentional errors<\/span><\/h3>\n<p>If a trusted method does not exist to ensure the continued quality and revision of content in biological databases, those who use the data should be aware of this risk and account for it in their analysis appropriately. In what follows, we outline previous efforts to develop analytics to detect and mitigate the impact of deliberately introduced database errors, both known and unknown.\n<\/p><p>Any machine learning analytic is necessarily a product of the data it observes. In an open data environment, an adversary can directly control any subsequent analysis by changing the data to change an algorithm's underlying model.<sup id=\"rdp-ebb-cite_ref-GoodfellowGen14_49-0\" class=\"reference\"><a href=\"#cite_note-GoodfellowGen14-49\">[49]<\/a><\/sup> The focus of a \u201ccounter adversarial\u201d approach to data analytics is to harden machine learning methods against the effects of inputs that are designed to mislead supervised<sup id=\"rdp-ebb-cite_ref-DalviAdvers04_50-0\" class=\"reference\"><a href=\"#cite_note-DalviAdvers04-50\">[50]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Kantarc.C4.B1o.C4.9FluClass11_51-0\" class=\"reference\"><a href=\"#cite_note-Kantarc.C4.B1o.C4.9FluClass11-51\">[51]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BiggioEvasion13_52-0\" class=\"reference\"><a href=\"#cite_note-BiggioEvasion13-52\">[52]<\/a><\/sup> and unsupervised<sup id=\"rdp-ebb-cite_ref-DutrisacHiding08_53-0\" class=\"reference\"><a href=\"#cite_note-DutrisacHiding08-53\">[53]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BiggioIsData13_54-0\" class=\"reference\"><a href=\"#cite_note-BiggioIsData13-54\">[54]<\/a><\/sup> algorithms. It has been shown that there exist label tampering attacks which significantly decrease the accuracy of a classifier, while being nearly undetectable by standard cross validation tests.<sup id=\"rdp-ebb-cite_ref-KegelmeyerCounter15_55-0\" class=\"reference\"><a href=\"#cite_note-KegelmeyerCounter15-55\">[55]<\/a><\/sup> In other words, the defender does not know the performance of the classifier has been corrupted. To protect against label tampering, an \u201censembles of outlier measures\u201d (EOM) method has been proposed to identify label tampering. The approach relies on a set of attributes that capture the \u201coutlierness\u201d of a sample to predict whether a sample has been tampered with. Tampered samples can then be remediated by changing the sample class label. In the context of a biological database, these labels may be metadata attributes associated with an entry. In the unsupervised machine learning scenario, an adversary may try to subvert a clustering algorithm by, for example, heuristically inserting data points to arbitrarily poison (i.e., merge)<sup id=\"rdp-ebb-cite_ref-BiggioIsData13_54-1\" class=\"reference\"><a href=\"#cite_note-BiggioIsData13-54\">[54]<\/a><\/sup> clusters. In the context of a genomics database, poisoning of clusters would significantly reduce the ability to detect anomalous genomic sequences. Kegelmeyer <i>et al.<\/i> demonstrate that their remediation methodology based on an EOM applies equally well in the unsupervised context.<sup id=\"rdp-ebb-cite_ref-KegelmeyerCounter15_55-1\" class=\"reference\"><a href=\"#cite_note-KegelmeyerCounter15-55\">[55]<\/a><\/sup>\n<\/p><p>As vulnerable cyber systems, best cyber practices can also be leveraged to protect biological databases. However, in the context of intentional manipulation of biological databases, special consideration must be given to the ability of these databases to enable production of dangerous biological material. The International Gene Synthesis Consortium (IGSC), for instance, provides two principal protections against the manufacture of malicious genetic material\u2014known as the Harmonized Screening Protocol.<sup id=\"rdp-ebb-cite_ref-IGSCHarm17_56-0\" class=\"reference\"><a href=\"#cite_note-IGSCHarm17-56\">[56]<\/a><\/sup> The first is a customer screening, and the second is a screening of DNA sequences against the Regulated Pathogen Database (RPD). This database is built from data on the U.S. Select Agent List, the Australian Group List, and other national lists of regulated pathogens. Members of the IGSC agree to translate each synthetic gene into an amino acid sequence and test for homology. These are then accepted, reviewed, or rejected. The RPD is updated annually and provided to members.\n<\/p><p>The Harmonized Screening Protocol requires at least two difficult processes: sharing the database and updating the database. Sharing the database requires the maintenance of authentication. Providers and users are part of a shared environment where they need to trust that everyone has an authentic and up-to-date version of the database. Updating the database requires maintenance to avoid \u201calert fatigue\u201d from false positives and the dangerous potential case of false negatives resulting in malicious manufacture. Maintaining the security of this requires an environment of authentication and active database inspection and curation. For the former, there may be opportunities to incorporate advanced encryption and authentication algorithms being considered in the cyber domain such as blockchain. However, significant computational resource costs must be contended with.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Preliminary_conclusions\">Preliminary conclusions<\/span><\/h2>\n<p>This survey of concerns with biological databases and methods for ensuring database integrity is certainly not exhaustive but represents broad capabilities within data science and cybersecurity today that have shown promise either within computational biology already, or in tackling similar problems in other domains. A goal of the authors is to illuminate these concerns for a wide audience in the context of the historical lessons learned in cyberspace. In the early days of the internet, the emphasis was on functionality and enabling the actions of largely well-intentioned communities of users. This functionality pervaded every element of our critical infrastructure. However, the same fabric that supports this infrastructure also represents a significant risk. Mitigating this risk after the wide penetration of open functionality is much more difficult than it might have been if the internet had been created with integrity and security in mind. As biological data becomes a bedrock critical infrastructure for the entire bioeconomy\u2014and follows the same exponential trends of size, pervasiveness, and importance as the internet\u2014we have a unique opportunity to ensure that this capability mitigates current and future risks from a worldwide set of actors. This paper calls out several existing research areas that can be leveraged to protect against accidental and intentional modifications and misuse of public biological databases.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The US Department of Energy (DOE) Laboratories are managed by contracted private entities as Federally Funded Research and Development Companies, and as such each laboratory is operated on behalf of DOE, independently from the operating company. Los Alamos National Laboratory is operated for DOE's National Nuclear Security Administration by Triad National Security, LLC under Contract 89233218CNA000001. Pacific Northwest National Laboratory is operated for DOE by Battelle Memorial Institute under contract DE-AC05-76RL01830. Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for DOE's National Nuclear Security Administration under contract DE-NA0003525.\n<\/p><p>Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>JC, JG, CH, EM, CO, EP, CT, and GX wrote sections of the manuscript. JC, JG, CH, EM, CJ, CO, KO, EP, KT, CT, and GX contributed to the formulation of the perspectives therein. NG, KT, and MW worked on document conception and assembly. All authors contributed to manuscript revision and approved of the submitted content.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research was supported in part by the Department of Homeland Security Science and Technology Directorate, award HSHQPM-14-X-00069.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-MoussouniClean13-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoussouniClean13_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Moussouni, F.; Berti\u2010\u00c9quille, L. (2014). \"Cleaning, Integrating, and Warehousing Genomic Data From Biomedical Resources\". In Elloumi, M.; Zomaya, A.Y.. <i>Biological Knowledge Discovery Handbook<\/i>. John Wiley & Sons. pp. 35\u201358. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2F9781118617151.ch02\" target=\"_blank\">10.1002\/9781118617151.ch02<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781118617151.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cleaning%2C+Integrating%2C+and+Warehousing+Genomic+Data+From+Biomedical+Resources&rft.atitle=Biological+Knowledge+Discovery+Handbook&rft.aulast=Moussouni%2C+F.%3B+Berti%E2%80%90%C3%89quille%2C+L.&rft.au=Moussouni%2C+F.%3B+Berti%E2%80%90%C3%89quille%2C+L.&rft.date=2014&rft.pages=pp.%26nbsp%3B35%E2%80%9358&rft.pub=John+Wiley+%26+Sons&rft_id=info:doi\/10.1002%2F9781118617151.ch02&rft.isbn=9781118617151&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimPrivate15-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KimPrivate15_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, M.; Lauter, K. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699052\" target=\"_blank\">\"Private genome analysis through homomorphic encryption\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>15<\/b> (Suppl 5): S3. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-15-S5-S3\" target=\"_blank\">10.1186\/1472-6947-15-S5-S3<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4699052\/\" target=\"_blank\">PMC4699052<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26733152\" target=\"_blank\">26733152<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699052\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4699052<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Private+genome+analysis+through+homomorphic+encryption&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Kim%2C+M.%3B+Lauter%2C+K.&rft.au=Kim%2C+M.%3B+Lauter%2C+K.&rft.date=2015&rft.volume=15&rft.issue=Suppl+5&rft.pages=S3&rft_id=info:doi\/10.1186%2F1472-6947-15-S5-S3&rft_id=info:pmc\/PMC4699052&rft_id=info:pmid\/26733152&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4699052&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MandalData18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MandalData18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Mandal, A.; Mitchell, J.C.; Montgomery, H.; Roy, A. (2018). \"Data oblivious genome variants search on Intel SGX\". In Garcia-Alfaro, J.; Herrera-Joancomart\u00ed, J.; Livraga, G.; Rios, R.. <i>Data Privacy Management, Cryptocurrencies and Blockchain Technology<\/i>. Lecture Notes in Computer Science. Springer International Publishing. pp. 21. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-030-00305-0_21\" target=\"_blank\">10.1007\/978-3-030-00305-0_21<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783030003050.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+oblivious+genome+variants+search+on+Intel+SGX&rft.atitle=Data+Privacy+Management%2C+Cryptocurrencies+and+Blockchain+Technology&rft.aulast=Mandal%2C+A.%3B+Mitchell%2C+J.C.%3B+Montgomery%2C+H.%3B+Roy%2C+A.&rft.au=Mandal%2C+A.%3B+Mitchell%2C+J.C.%3B+Montgomery%2C+H.%3B+Roy%2C+A.&rft.date=2018&rft.series=Lecture+Notes+in+Computer+Science&rft.pages=pp.%26nbsp%3B21&rft.pub=Springer+International+Publishing&rft_id=info:doi\/10.1007%2F978-3-030-00305-0_21&rft.isbn=9783030003050&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OzercanRealiz18-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OzercanRealiz18_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ozercan, H.I.; Ileri, A.M.; Ayday, E.; Alkan, C. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6120626\" target=\"_blank\">\"Realizing the potential of blockchain technologies in genomics\"<\/a>. <i>Gemone Research<\/i> <b>28<\/b> (9): 1255\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.207464.116\" target=\"_blank\">10.1101\/gr.207464.116<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6120626\/\" target=\"_blank\">PMC6120626<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30076130\" target=\"_blank\">30076130<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6120626\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6120626<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Realizing+the+potential+of+blockchain+technologies+in+genomics&rft.jtitle=Gemone+Research&rft.aulast=Ozercan%2C+H.I.%3B+Ileri%2C+A.M.%3B+Ayday%2C+E.%3B+Alkan%2C+C.&rft.au=Ozercan%2C+H.I.%3B+Ileri%2C+A.M.%3B+Ayday%2C+E.%3B+Alkan%2C+C.&rft.date=2018&rft.volume=28&rft.issue=9&rft.pages=1255%E2%80%9363&rft_id=info:doi\/10.1101%2Fgr.207464.116&rft_id=info:pmc\/PMC6120626&rft_id=info:pmid\/30076130&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6120626&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MerchantUnexp14-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MerchantUnexp14_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-MerchantUnexp14_5-1\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Merchant, S.; Wood, D.E.; Salzberg, S.L. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4243333\" target=\"_blank\">\"Unexpected cross-species contamination in genome sequencing projects\"<\/a>. <i>PeerJ<\/i> <b>2<\/b>: e675. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717%2Fpeerj.675\" target=\"_blank\">10.7717\/peerj.675<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4243333\/\" target=\"_blank\">PMC4243333<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25426337\" target=\"_blank\">25426337<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4243333\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4243333<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Unexpected+cross-species+contamination+in+genome+sequencing+projects&rft.jtitle=PeerJ&rft.aulast=Merchant%2C+S.%3B+Wood%2C+D.E.%3B+Salzberg%2C+S.L.&rft.au=Merchant%2C+S.%3B+Wood%2C+D.E.%3B+Salzberg%2C+S.L.&rft.date=2014&rft.volume=2&rft.pages=e675&rft_id=info:doi\/10.7717%2Fpeerj.675&rft_id=info:pmc\/PMC4243333&rft_id=info:pmid\/25426337&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4243333&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StrongMicro14-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StrongMicro14_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Strong, M.J.; Xu, G.; Morici, L. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4239086\" target=\"_blank\">\"Microbial contamination in next generation sequencing: implications for sequence-based analysis of clinical samples\"<\/a>. <i>PLoS Pathogens<\/i> <b>10<\/b> (11): e1004437. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.ppat.1004437\" target=\"_blank\">10.1371\/journal.ppat.1004437<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4239086\/\" target=\"_blank\">PMC4239086<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25412476\" target=\"_blank\">25412476<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4239086\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4239086<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Microbial+contamination+in+next+generation+sequencing%3A+implications+for+sequence-based+analysis+of+clinical+samples&rft.jtitle=PLoS+Pathogens&rft.aulast=Strong%2C+M.J.%3B+Xu%2C+G.%3B+Morici%2C+L.+et+al.&rft.au=Strong%2C+M.J.%3B+Xu%2C+G.%3B+Morici%2C+L.+et+al.&rft.date=2014&rft.volume=10&rft.issue=11&rft.pages=e1004437&rft_id=info:doi\/10.1371%2Fjournal.ppat.1004437&rft_id=info:pmc\/PMC4239086&rft_id=info:pmid\/25412476&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4239086&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PilliMonit13-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PilliMonit13_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pilli, E.; Modi, A.; Serpico, C. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3556025\" target=\"_blank\">\"Monitoring DNA contamination in handled vs. directly excavated ancient human skeletal remains\"<\/a>. <i>PLoS One<\/i> <b>8<\/b> (1): e52524. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0052524\" target=\"_blank\">10.1371\/journal.pone.0052524<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3556025\/\" target=\"_blank\">PMC3556025<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23372650\" target=\"_blank\">23372650<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3556025\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3556025<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Monitoring+DNA+contamination+in+handled+vs.+directly+excavated+ancient+human+skeletal+remains&rft.jtitle=PLoS+One&rft.aulast=Pilli%2C+E.%3B+Modi%2C+A.%3B+Serpico%2C+C.+et+al.&rft.au=Pilli%2C+E.%3B+Modi%2C+A.%3B+Serpico%2C+C.+et+al.&rft.date=2013&rft.volume=8&rft.issue=1&rft.pages=e52524&rft_id=info:doi\/10.1371%2Fjournal.pone.0052524&rft_id=info:pmc\/PMC3556025&rft_id=info:pmid\/23372650&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3556025&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BallenghienPatterns17-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BallenghienPatterns17_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ballenghien, M.; Faivre, N;. Galtier, N. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5370491\" target=\"_blank\">\"Patterns of cross-contamination in a multispecies population genomic project: Detection, quantification, impact, and solutions\"<\/a>. <i>BMC Biology<\/i> <b>15<\/b> (1): 25. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12915-017-0366-6\" target=\"_blank\">10.1186\/s12915-017-0366-6<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5370491\/\" target=\"_blank\">PMC5370491<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28356154\" target=\"_blank\">28356154<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5370491\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5370491<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patterns+of+cross-contamination+in+a+multispecies+population+genomic+project%3A+Detection%2C+quantification%2C+impact%2C+and+solutions&rft.jtitle=BMC+Biology&rft.aulast=Ballenghien%2C+M.%3B+Faivre%2C+N%3B.+Galtier%2C+N.&rft.au=Ballenghien%2C+M.%3B+Faivre%2C+N%3B.+Galtier%2C+N.&rft.date=2017&rft.volume=15&rft.issue=1&rft.pages=25&rft_id=info:doi\/10.1186%2Fs12915-017-0366-6&rft_id=info:pmc\/PMC5370491&rft_id=info:pmid\/28356154&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5370491&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SimionALarge17-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SimionALarge17_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Simion, P.; Philippe, H.; Baurain, D. et al. (2017). \"A Large and Consistent Phylogenomic Dataset Supports Sponges as the Sister Group to All Other Animals\". <i>Current Biology<\/i> <b>27<\/b> (7): 958-967. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cub.2017.02.031\" target=\"_blank\">10.1016\/j.cub.2017.02.031<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28318975\" target=\"_blank\">28318975<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Large+and+Consistent+Phylogenomic+Dataset+Supports+Sponges+as+the+Sister+Group+to+All+Other+Animals&rft.jtitle=Current+Biology&rft.aulast=Simion%2C+P.%3B+Philippe%2C+H.%3B+Baurain%2C+D.+et+al.&rft.au=Simion%2C+P.%3B+Philippe%2C+H.%3B+Baurain%2C+D.+et+al.&rft.date=2017&rft.volume=27&rft.issue=7&rft.pages=958-967&rft_id=info:doi\/10.1016%2Fj.cub.2017.02.031&rft_id=info:pmid\/28318975&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WittAnAss09-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WittAnAss09_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Witt, N.; Rodger, G.; Vandesompele, J. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2777341\" target=\"_blank\">\"An assessment of air as a source of DNA contamination encountered when performing PCR\"<\/a>. <i>Journal of Biomolecular Techniques<\/i> <b>20<\/b> (5): 236\u201340. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2777341\/\" target=\"_blank\">PMC2777341<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19949694\" target=\"_blank\">19949694<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2777341\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2777341<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+assessment+of+air+as+a+source+of+DNA+contamination+encountered+when+performing+PCR&rft.jtitle=Journal+of+Biomolecular+Techniques&rft.aulast=Witt%2C+N.%3B+Rodger%2C+G.%3B+Vandesompele%2C+J.+et+al.&rft.au=Witt%2C+N.%3B+Rodger%2C+G.%3B+Vandesompele%2C+J.+et+al.&rft.date=2009&rft.volume=20&rft.issue=5&rft.pages=236%E2%80%9340&rft_id=info:pmc\/PMC2777341&rft_id=info:pmid\/19949694&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2777341&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LongoAbund11-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LongoAbund11_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Longo, M.S.; O'Neill, M.J.; O'Neill, R.J. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040168\" target=\"_blank\">\"Abundant human DNA contamination identified in non-primate genome databases\"<\/a>. <i>PLoS One<\/i> <b>6<\/b> (2): e16410. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0016410\" target=\"_blank\">10.1371\/journal.pone.0016410<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3040168\/\" target=\"_blank\">PMC3040168<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21358816\" target=\"_blank\">21358816<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040168\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040168<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Abundant+human+DNA+contamination+identified+in+non-primate+genome+databases&rft.jtitle=PLoS+One&rft.aulast=Longo%2C+M.S.%3B+O%27Neill%2C+M.J.%3B+O%27Neill%2C+R.J.&rft.au=Longo%2C+M.S.%3B+O%27Neill%2C+M.J.%3B+O%27Neill%2C+R.J.&rft.date=2011&rft.volume=6&rft.issue=2&rft.pages=e16410&rft_id=info:doi\/10.1371%2Fjournal.pone.0016410&rft_id=info:pmc\/PMC3040168&rft_id=info:pmid\/21358816&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3040168&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PuiuReass08-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PuiuReass08_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Puiu, D.; Salzberg, S.L. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2561293\" target=\"_blank\">\"Re-assembly of the genome of <i>Francisella tularensis<\/i> subsp. holarctica OSU18\"<\/a>. <i>PLoS One<\/i> <b>3<\/b> (10): e3427. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0003427\" target=\"_blank\">10.1371\/journal.pone.0003427<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2561293\/\" target=\"_blank\">PMC2561293<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18927608\" target=\"_blank\">18927608<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2561293\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2561293<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Re-assembly+of+the+genome+of+%27%27Francisella+tularensis%27%27+subsp.+holarctica+OSU18&rft.jtitle=PLoS+One&rft.aulast=Puiu%2C+D.%3B+Salzberg%2C+S.L.&rft.au=Puiu%2C+D.%3B+Salzberg%2C+S.L.&rft.date=2008&rft.volume=3&rft.issue=10&rft.pages=e3427&rft_id=info:doi\/10.1371%2Fjournal.pone.0003427&rft_id=info:pmc\/PMC2561293&rft_id=info:pmid\/18927608&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2561293&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CooperProof14-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CooperProof14_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cooper, P. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4072989\" target=\"_blank\">\"Proof by synthesis of <i>Tobacco mosaic virus<\/i>\"<\/a>. <i>Genome Biology<\/i> <b>15<\/b> (5): R67. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2014-15-5-r67\" target=\"_blank\">10.1186\/gb-2014-15-5-r67<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4072989\/\" target=\"_blank\">PMC4072989<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24887356\" target=\"_blank\">24887356<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4072989\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4072989<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Proof+by+synthesis+of+%27%27Tobacco+mosaic+virus%27%27&rft.jtitle=Genome+Biology&rft.aulast=Cooper%2C+P.&rft.au=Cooper%2C+P.&rft.date=2014&rft.volume=15&rft.issue=5&rft.pages=R67&rft_id=info:doi\/10.1186%2Fgb-2014-15-5-r67&rft_id=info:pmc\/PMC4072989&rft_id=info:pmid\/24887356&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4072989&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PhilippeResolv11-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PhilippeResolv11_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Philippe, H;. Brinkmann, H.; Lavrov, D.V. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3057953\" target=\"_blank\">\"Resolving difficult phylogenetic questions: Why more sequences are not enough\"<\/a>. <i>PLoS Biology<\/i> <b>9<\/b> (3): e1000602. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pbio.1000602\" target=\"_blank\">10.1371\/journal.pbio.1000602<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3057953\/\" target=\"_blank\">PMC3057953<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21423652\" target=\"_blank\">21423652<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3057953\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3057953<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Resolving+difficult+phylogenetic+questions%3A+Why+more+sequences+are+not+enough&rft.jtitle=PLoS+Biology&rft.aulast=Philippe%2C+H%3B.+Brinkmann%2C+H.%3B+Lavrov%2C+D.V.+et+al.&rft.au=Philippe%2C+H%3B.+Brinkmann%2C+H.%3B+Lavrov%2C+D.V.+et+al.&rft.date=2011&rft.volume=9&rft.issue=3&rft.pages=e1000602&rft_id=info:doi\/10.1371%2Fjournal.pbio.1000602&rft_id=info:pmc\/PMC3057953&rft_id=info:pmid\/21423652&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3057953&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Laurin-LemayOrigin12-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Laurin-LemayOrigin12_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Laurin-Lemay, S.; Brinkmann, H.; Philippe, H. (2012). \"Resolving difficult phylogenetic questions: Why more sequences are not enough\". <i>PLoS Biology<\/i> <b>22<\/b> (15): R593\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cub.2012.06.013\" target=\"_blank\">10.1016\/j.cub.2012.06.013<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22877776\" target=\"_blank\">22877776<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Resolving+difficult+phylogenetic+questions%3A+Why+more+sequences+are+not+enough&rft.jtitle=PLoS+Biology&rft.aulast=Laurin-Lemay%2C+S.%3B+Brinkmann%2C+H.%3B+Philippe%2C+H.&rft.au=Laurin-Lemay%2C+S.%3B+Brinkmann%2C+H.%3B+Philippe%2C+H.&rft.date=2012&rft.volume=22&rft.issue=15&rft.pages=R593%E2%80%934&rft_id=info:doi\/10.1016%2Fj.cub.2012.06.013&rft_id=info:pmid\/22877776&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PeccoudCyber18-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PeccoudCyber18_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). \"Cyberbiosecurity: From Naive Trust to Risk Awareness\". <i>Trends in Biotechnology<\/i> <b>36<\/b> (1): 4\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.tibtech.2017.10.012\" target=\"_blank\">10.1016\/j.tibtech.2017.10.012<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29224719\" target=\"_blank\">29224719<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cyberbiosecurity%3A+From+Naive+Trust+to+Risk+Awareness&rft.jtitle=Trends+in+Biotechnology&rft.aulast=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.au=Peccoud%2C+J.%3B+Gallegos%2C+J.E.%3B+Murch%2C+R.+et+al.&rft.date=2018&rft.volume=36&rft.issue=1&rft.pages=4%E2%80%937&rft_id=info:doi\/10.1016%2Fj.tibtech.2017.10.012&rft_id=info:pmid\/29224719&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NRCBio00-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NRCBio00_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Research Council (2000). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nap.edu\/catalog\/9990\/bioinformatics-converting-data-to-knowledge-workshop-summary\" target=\"_blank\">\"Bioinformatics: Converting Data to Knowledge\"<\/a>. National Academies Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.17226%2F9990\" target=\"_blank\">10.17226\/9990<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nap.edu\/catalog\/9990\/bioinformatics-converting-data-to-knowledge-workshop-summary\" target=\"_blank\">https:\/\/www.nap.edu\/catalog\/9990\/bioinformatics-converting-data-to-knowledge-workshop-summary<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bioinformatics%3A+Converting+Data+to+Knowledge&rft.atitle=&rft.aulast=National+Research+Council&rft.au=National+Research+Council&rft.date=2000&rft.pub=National+Academies+Press&rft_id=info:doi\/10.17226%2F9990&rft_id=https%3A%2F%2Fwww.nap.edu%2Fcatalog%2F9990%2Fbioinformatics-converting-data-to-knowledge-workshop-summary&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PundirUniProt17-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PundirUniProt17_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pundir, S;. Martin, M.J.; O'Donovan, C. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5565770\" target=\"_blank\">\"UniProt Protein Knowledgebase\"<\/a>. <i>Methods in Molecular Biology<\/i> <b>1558<\/b>: 41\u201355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-1-4939-6783-4_2\" target=\"_blank\">10.1007\/978-1-4939-6783-4_2<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5565770\/\" target=\"_blank\">PMC5565770<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28150232\" target=\"_blank\">28150232<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5565770\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5565770<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=UniProt+Protein+Knowledgebase&rft.jtitle=Methods+in+Molecular+Biology&rft.aulast=Pundir%2C+S%3B.+Martin%2C+M.J.%3B+O%27Donovan%2C+C.&rft.au=Pundir%2C+S%3B.+Martin%2C+M.J.%3B+O%27Donovan%2C+C.&rft.date=2017&rft.volume=1558&rft.pages=41%E2%80%9355&rft_id=info:doi\/10.1007%2F978-1-4939-6783-4_2&rft_id=info:pmc\/PMC5565770&rft_id=info:pmid\/28150232&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5565770&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GurevichQUAST13-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GurevichQUAST13_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gurevich, A.; Saveliev, V.; Vyahhi, N.; Tesler, G. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3624806\" target=\"_blank\">\"QUAST: Quality assessment tool for genome assemblies\"<\/a>. <i>Bioinformatics<\/i> <b>29<\/b> (8): 1072\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt086\" target=\"_blank\">10.1093\/bioinformatics\/btt086<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3624806\/\" target=\"_blank\">PMC3624806<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23422339\" target=\"_blank\">23422339<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3624806\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3624806<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=QUAST%3A+Quality+assessment+tool+for+genome+assemblies&rft.jtitle=Bioinformatics&rft.aulast=Gurevich%2C+A.%3B+Saveliev%2C+V.%3B+Vyahhi%2C+N.%3B+Tesler%2C+G.&rft.au=Gurevich%2C+A.%3B+Saveliev%2C+V.%3B+Vyahhi%2C+N.%3B+Tesler%2C+G.&rft.date=2013&rft.volume=29&rft.issue=8&rft.pages=1072%E2%80%935&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt086&rft_id=info:pmc\/PMC3624806&rft_id=info:pmid\/23422339&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3624806&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Sim.C3.A3oBUSCO15-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Sim.C3.A3oBUSCO15_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sim\u00e3o, F.A.; Waterhouse, R.M.; Ioannidis, P. et al. (2015). \"BUSCO: Assessing genome assembly and annotation completeness with single-copy orthologs\". <i>Bioinformatics<\/i> <b>31<\/b> (19): 3210\u20132. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtv351\" target=\"_blank\">10.1093\/bioinformatics\/btv351<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26059717\" target=\"_blank\">26059717<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BUSCO%3A+Assessing+genome+assembly+and+annotation+completeness+with+single-copy+orthologs&rft.jtitle=Bioinformatics&rft.aulast=Sim%C3%A3o%2C+F.A.%3B+Waterhouse%2C+R.M.%3B+Ioannidis%2C+P.+et+al.&rft.au=Sim%C3%A3o%2C+F.A.%3B+Waterhouse%2C+R.M.%3B+Ioannidis%2C+P.+et+al.&rft.date=2015&rft.volume=31&rft.issue=19&rft.pages=3210%E2%80%932&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtv351&rft_id=info:pmid\/26059717&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TennessenProDeGe16-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TennessenProDeGe16_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tennessen, K.; Andersen, E.; Clingenpeel, S. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4681846\" target=\"_blank\">\"ProDeGe: A computational protocol for fully automated decontamination of genomes\"<\/a>. <i>The ISME Journal<\/i> <b>10<\/b> (1): 269\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fismej.2015.100\" target=\"_blank\">10.1038\/ismej.2015.100<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4681846\/\" target=\"_blank\">PMC4681846<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26057843\" target=\"_blank\">26057843<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4681846\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4681846<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ProDeGe%3A+A+computational+protocol+for+fully+automated+decontamination+of+genomes&rft.jtitle=The+ISME+Journal&rft.aulast=Tennessen%2C+K.%3B+Andersen%2C+E.%3B+Clingenpeel%2C+S.+et+al.&rft.au=Tennessen%2C+K.%3B+Andersen%2C+E.%3B+Clingenpeel%2C+S.+et+al.&rft.date=2016&rft.volume=10&rft.issue=1&rft.pages=269%E2%80%9372&rft_id=info:doi\/10.1038%2Fismej.2015.100&rft_id=info:pmc\/PMC4681846&rft_id=info:pmid\/26057843&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4681846&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LuxACDC16-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LuxACDC16_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lux, M.; Kr\u00fcger, J.; Rinke, C. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5168860\" target=\"_blank\">\"acdc - Automated Contamination Detection and Confidence estimation for single-cell genome data\"<\/a>. <i>BMC Bioinformatics<\/i> <b>17<\/b> (1): 543. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs12859-016-1397-7\" target=\"_blank\">10.1186\/s12859-016-1397-7<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5168860\/\" target=\"_blank\">PMC5168860<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27998267\" target=\"_blank\">27998267<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5168860\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5168860<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=acdc+-+Automated+Contamination+Detection+and+Confidence+estimation+for+single-cell+genome+data&rft.jtitle=BMC+Bioinformatics&rft.aulast=Lux%2C+M.%3B+Kr%C3%BCger%2C+J.%3B+Rinke%2C+C.++et+al.&rft.au=Lux%2C+M.%3B+Kr%C3%BCger%2C+J.%3B+Rinke%2C+C.++et+al.&rft.date=2016&rft.volume=17&rft.issue=1&rft.pages=543&rft_id=info:doi\/10.1186%2Fs12859-016-1397-7&rft_id=info:pmc\/PMC5168860&rft_id=info:pmid\/27998267&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5168860&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ParksCheckM15-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ParksCheckM15_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Parks, D.H.; Imelfort, M.; Skennerton, C.T. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4484387\" target=\"_blank\">\"CheckM: Assessing the quality of microbial genomes recovered from isolates, single cells, and metagenomes\"<\/a>. <i>Genome Research<\/i> <b>25<\/b> (7): 1043\u201355. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.186072.114\" target=\"_blank\">10.1101\/gr.186072.114<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4484387\/\" target=\"_blank\">PMC4484387<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25977477\" target=\"_blank\">25977477<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4484387\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4484387<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=CheckM%3A+Assessing+the+quality+of+microbial+genomes+recovered+from+isolates%2C+single+cells%2C+and+metagenomes&rft.jtitle=Genome+Research&rft.aulast=Parks%2C+D.H.%3B+Imelfort%2C+M.%3B+Skennerton%2C+C.T.+et+al.&rft.au=Parks%2C+D.H.%3B+Imelfort%2C+M.%3B+Skennerton%2C+C.T.+et+al.&rft.date=2015&rft.volume=25&rft.issue=7&rft.pages=1043%E2%80%9355&rft_id=info:doi\/10.1101%2Fgr.186072.114&rft_id=info:pmc\/PMC4484387&rft_id=info:pmid\/25977477&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4484387&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DittamiDetect17-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DittamiDetect17_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dittami, S.M.; Corre, E. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5695246\" target=\"_blank\">\"Detection of bacterial contaminants and hybrid sequences in the genome of the kelp <i>Saccharina japonica<\/i> using Taxoblast\"<\/a>. <i>PeerJ<\/i> <b>5<\/b>: e4073. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717%2Fpeerj.4073\" target=\"_blank\">10.7717\/peerj.4073<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5695246\/\" target=\"_blank\">PMC5695246<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29158994\" target=\"_blank\">29158994<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5695246\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5695246<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Detection+of+bacterial+contaminants+and+hybrid+sequences+in+the+genome+of+the+kelp+%27%27Saccharina+japonica%27%27+using+Taxoblast&rft.jtitle=PeerJ&rft.aulast=Dittami%2C+S.M.%3B+Corre%2C+E.&rft.au=Dittami%2C+S.M.%3B+Corre%2C+E.&rft.date=2017&rft.volume=5&rft.pages=e4073&rft_id=info:doi\/10.7717%2Fpeerj.4073&rft_id=info:pmc\/PMC5695246&rft_id=info:pmid\/29158994&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5695246&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McNairGenome15-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McNairGenome15_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McNair, K.; Edwards, R.A. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4476108\" target=\"_blank\">\"GenomePeek: An online tool for prokaryotic genome and metagenome analysis\"<\/a>. <i>PeerJ<\/i> <b>3<\/b>: e1025. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717%2Fpeerj.1025\" target=\"_blank\">10.7717\/peerj.1025<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4476108\/\" target=\"_blank\">PMC4476108<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26157610\" target=\"_blank\">26157610<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4476108\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4476108<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GenomePeek%3A+An+online+tool+for+prokaryotic+genome+and+metagenome+analysis&rft.jtitle=PeerJ&rft.aulast=McNair%2C+K.%3B+Edwards%2C+R.A.&rft.au=McNair%2C+K.%3B+Edwards%2C+R.A.&rft.date=2015&rft.volume=3&rft.pages=e1025&rft_id=info:doi\/10.7717%2Fpeerj.1025&rft_id=info:pmc\/PMC4476108&rft_id=info:pmid\/26157610&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4476108&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RyukovHuman16-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RyukovHuman16_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kryukov, K.; Imanishi, T. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5017631\" target=\"_blank\">\"Human Contamination in Public Genome Assemblies\"<\/a>. <i>PLoS One<\/i> <b>11<\/b> (9): e0162424. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0162424\" target=\"_blank\">10.1371\/journal.pone.0162424<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5017631\/\" target=\"_blank\">PMC5017631<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27611326\" target=\"_blank\">27611326<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5017631\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5017631<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Human+Contamination+in+Public+Genome+Assemblies&rft.jtitle=PLoS+One&rft.aulast=Kryukov%2C+K.%3B+Imanishi%2C+T.&rft.au=Kryukov%2C+K.%3B+Imanishi%2C+T.&rft.date=2016&rft.volume=11&rft.issue=9&rft.pages=e0162424&rft_id=info:doi\/10.1371%2Fjournal.pone.0162424&rft_id=info:pmc\/PMC5017631&rft_id=info:pmid\/27611326&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5017631&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CornetConcensus18-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CornetConcensus18_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cornet, L.; Meunier, L.; Van Vlierberghe, M. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6059444\" target=\"_blank\">\"Consensus assessment of the contamination level of publicly available cyanobacterial genomes\"<\/a>. <i>PLoS One<\/i> <b>13<\/b> (7): e0200323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0200323\" target=\"_blank\">10.1371\/journal.pone.0200323<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6059444\/\" target=\"_blank\">PMC6059444<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30044797\" target=\"_blank\">30044797<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6059444\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6059444<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Consensus+assessment+of+the+contamination+level+of+publicly+available+cyanobacterial+genomes&rft.jtitle=PLoS+One&rft.aulast=Cornet%2C+L.%3B+Meunier%2C+L.%3B+Van+Vlierberghe%2C+M.+et+al.&rft.au=Cornet%2C+L.%3B+Meunier%2C+L.%3B+Van+Vlierberghe%2C+M.+et+al.&rft.date=2018&rft.volume=13&rft.issue=7&rft.pages=e0200323&rft_id=info:doi\/10.1371%2Fjournal.pone.0200323&rft_id=info:pmc\/PMC6059444&rft_id=info:pmid\/30044797&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6059444&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LuRemoving18-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LuRemoving18_28-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lu, J.; Salzberg, S.L. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6034898\" target=\"_blank\">\"Removing contaminants from databases of draft genomes\"<\/a>. <i>PLoS Computational Biology<\/i> <b>14<\/b> (6): e1006277. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1006277\" target=\"_blank\">10.1371\/journal.pcbi.1006277<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6034898\/\" target=\"_blank\">PMC6034898<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29939994\" target=\"_blank\">29939994<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6034898\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6034898<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Removing+contaminants+from+databases+of+draft+genomes&rft.jtitle=PLoS+Computational+Biology&rft.aulast=Lu%2C+J.%3B+Salzberg%2C+S.L.&rft.au=Lu%2C+J.%3B+Salzberg%2C+S.L.&rft.date=2018&rft.volume=14&rft.issue=6&rft.pages=e1006277&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1006277&rft_id=info:pmc\/PMC6034898&rft_id=info:pmid\/29939994&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6034898&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrizhevskyImageNet17-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrizhevskyImageNet17_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krizhevsky, A.; Sutskever, U.; Hinton, G.E. (2017). \"ImageNet classification with deep convolutional neural networks\". <i>Communications of the ACM<\/i> <b>60<\/b> (6): 84\u201390. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3065386\" target=\"_blank\">10.1145\/3065386<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ImageNet+classification+with+deep+convolutional+neural+networks&rft.jtitle=Communications+of+the+ACM&rft.aulast=Krizhevsky%2C+A.%3B+Sutskever%2C+U.%3B+Hinton%2C+G.E.&rft.au=Krizhevsky%2C+A.%3B+Sutskever%2C+U.%3B+Hinton%2C+G.E.&rft.date=2017&rft.volume=60&rft.issue=6&rft.pages=84%E2%80%9390&rft_id=info:doi\/10.1145%2F3065386&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SutskeverSeq14-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SutskeverSeq14_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sutskever, U.; Vinyals, O.; Le, Q.V. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/papers.nips.cc\/paper\/5346-sequence-to-sequence-learning-with-neural\" target=\"_blank\">\"Sequence to Sequence Learning with Neural Networks\"<\/a>. <i>Proceedings from Advances in Neural Information Processing Systems 2014<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/papers.nips.cc\/paper\/5346-sequence-to-sequence-learning-with-neural\" target=\"_blank\">http:\/\/papers.nips.cc\/paper\/5346-sequence-to-sequence-learning-with-neural<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sequence+to+Sequence+Learning+with+Neural+Networks&rft.jtitle=Proceedings+from+Advances+in+Neural+Information+Processing+Systems+2014&rft.aulast=Sutskever%2C+U.%3B+Vinyals%2C+O.%3B+Le%2C+Q.V.&rft.au=Sutskever%2C+U.%3B+Vinyals%2C+O.%3B+Le%2C+Q.V.&rft.date=2014&rft_id=http%3A%2F%2Fpapers.nips.cc%2Fpaper%2F5346-sequence-to-sequence-learning-with-neural&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Larra.C3.B1agaMach06-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Larra.C3.B1agaMach06_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Larra\u00f1aga, P.; Calvo, B.; Santana, R. et al. (2006). \"Machine learning in bioinformatics\". <i>Briefings in Bioinformatics<\/i> <b>7<\/b> (1): 86\u2013112. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbk007\" target=\"_blank\">10.1093\/bib\/bbk007<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16761367\" target=\"_blank\">16761367<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Machine+learning+in+bioinformatics&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Larra%C3%B1aga%2C+P.%3B+Calvo%2C+B.%3B+Santana%2C+R.+et+al.&rft.au=Larra%C3%B1aga%2C+P.%3B+Calvo%2C+B.%3B+Santana%2C+R.+et+al.&rft.date=2006&rft.volume=7&rft.issue=1&rft.pages=86%E2%80%93112&rft_id=info:doi\/10.1093%2Fbib%2Fbbk007&rft_id=info:pmid\/16761367&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FederhenMeeting16-32\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FederhenMeeting16_32-0\">32.0<\/a><\/sup> <sup><a href=\"#cite_ref-FederhenMeeting16_32-1\">32.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Federhen, S.; Rossello-Mora, R.; Klenk, H.-P. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4748488\" target=\"_blank\">\"Meeting report: GenBank microbial genomic taxonomy workshop (12\u201313 May, 2015)\"<\/a>. <i>Standards in Genomic Sciences<\/i> <b>11<\/b>: 15. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs40793-016-0134-1\" target=\"_blank\">10.1186\/s40793-016-0134-1<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4748488\/\" target=\"_blank\">PMC4748488<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4748488\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4748488<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Meeting+report%3A+GenBank+microbial+genomic+taxonomy+workshop+%2812%E2%80%9313+May%2C+2015%29&rft.jtitle=Standards+in+Genomic+Sciences&rft.aulast=Federhen%2C+S.%3B+Rossello-Mora%2C+R.%3B+Klenk%2C+H.-P.+et+al.&rft.au=Federhen%2C+S.%3B+Rossello-Mora%2C+R.%3B+Klenk%2C+H.-P.+et+al.&rft.date=2016&rft.volume=11&rft.pages=15&rft_id=info:doi\/10.1186%2Fs40793-016-0134-1&rft_id=info:pmc\/PMC4748488&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4748488&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiTheSim04-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiTheSim04_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, M.; Chen, X.; Li, X. et al. (2004). \"The Similarity Metric\". <i>IEEE Transactions on Information Theory<\/i> <b>50<\/b> (12): 3250\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTIT.2004.838101\" target=\"_blank\">10.1109\/TIT.2004.838101<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Similarity+Metric&rft.jtitle=IEEE+Transactions+on+Information+Theory&rft.aulast=Li%2C+M.%3B+Chen%2C+X.%3B+Li%2C+X.+et+al.&rft.au=Li%2C+M.%3B+Chen%2C+X.%3B+Li%2C+X.+et+al.&rft.date=2004&rft.volume=50&rft.issue=12&rft.pages=3250%E2%80%9364&rft_id=info:doi\/10.1109%2FTIT.2004.838101&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GorisDNA07-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GorisDNA07_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goris, J.; Konstantinudis, K.T.; Klappenbach, J.A. et al. (2007). \"DNA-DNA hybridization values and their relationship to whole-genome sequence similarities\". <i>International Journal of Systematic and Evolutionary Microbiology<\/i> <b>57<\/b> (Pt 1): 81\u201391. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1099%2Fijs.0.64483-0\" target=\"_blank\">10.1099\/ijs.0.64483-0<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17220447\" target=\"_blank\">17220447<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DNA-DNA+hybridization+values+and+their+relationship+to+whole-genome+sequence+similarities&rft.jtitle=International+Journal+of+Systematic+and+Evolutionary+Microbiology&rft.aulast=Goris%2C+J.%3B+Konstantinudis%2C+K.T.%3B+Klappenbach%2C+J.A.+et+al.&rft.au=Goris%2C+J.%3B+Konstantinudis%2C+K.T.%3B+Klappenbach%2C+J.A.+et+al.&rft.date=2007&rft.volume=57&rft.issue=Pt+1&rft.pages=81%E2%80%9391&rft_id=info:doi\/10.1099%2Fijs.0.64483-0&rft_id=info:pmid\/17220447&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ColstonBio14-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ColstonBio14_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Colston, S.M.; Fullmer, M.S.; Beka, L. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4251997\" target=\"_blank\">\"Bioinformatic genome comparisons for taxonomic and phylogenetic assignments using Aeromonas as a test case\"<\/a>. <i>mBio<\/i> <b>5<\/b> (6): e02136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1128%2FmBio.02136-14\" target=\"_blank\">10.1128\/mBio.02136-14<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4251997\/\" target=\"_blank\">PMC4251997<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25406383\" target=\"_blank\">25406383<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4251997\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4251997<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatic+genome+comparisons+for+taxonomic+and+phylogenetic+assignments+using+Aeromonas+as+a+test+case&rft.jtitle=mBio&rft.aulast=Colston%2C+S.M.%3B+Fullmer%2C+M.S.%3B+Beka%2C+L.+et+al.&rft.au=Colston%2C+S.M.%3B+Fullmer%2C+M.S.%3B+Beka%2C+L.+et+al.&rft.date=2014&rft.volume=5&rft.issue=6&rft.pages=e02136&rft_id=info:doi\/10.1128%2FmBio.02136-14&rft_id=info:pmc\/PMC4251997&rft_id=info:pmid\/25406383&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4251997&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FiguerasTax14-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FiguerasTax14_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Figueras, M.J.; Beaz-Hidalgo, R.; Hossain, M.J.; Liles, M.R. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4256179\" target=\"_blank\">\"Taxonomic affiliation of new genomes should be verified using average nucleotide identity and multilocus phylogenetic analysis\"<\/a>. <i>Genome Announcements<\/i> <b>2<\/b> (6): e00927-14. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1128%2FgenomeA.00927-14\" target=\"_blank\">10.1128\/genomeA.00927-14<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4256179\/\" target=\"_blank\">PMC4256179<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25477398\" target=\"_blank\">25477398<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4256179\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4256179<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Taxonomic+affiliation+of+new+genomes+should+be+verified+using+average+nucleotide+identity+and+multilocus+phylogenetic+analysis&rft.jtitle=Genome+Announcements&rft.aulast=Figueras%2C+M.J.%3B+Beaz-Hidalgo%2C+R.%3B+Hossain%2C+M.J.%3B+Liles%2C+M.R.&rft.au=Figueras%2C+M.J.%3B+Beaz-Hidalgo%2C+R.%3B+Hossain%2C+M.J.%3B+Liles%2C+M.R.&rft.date=2014&rft.volume=2&rft.issue=6&rft.pages=e00927-14&rft_id=info:doi\/10.1128%2FgenomeA.00927-14&rft_id=info:pmc\/PMC4256179&rft_id=info:pmid\/25477398&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4256179&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimTowards14-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KimTowards14_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, M.; Oh, H.S.; Park, S.C.; Chun, J. (2014). \"Towards a taxonomic coherence between average nucleotide identity and 16S rRNA gene sequence similarity for species demarcation of prokaryotes\". <i>International Journal of Systematic and Evolutionary Microbiology<\/i> <b>64<\/b> (Pt 2): 346-51. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1099%2Fijs.0.059774-0\" target=\"_blank\">10.1099\/ijs.0.059774-0<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24505072\" target=\"_blank\">24505072<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+a+taxonomic+coherence+between+average+nucleotide+identity+and+16S+rRNA+gene+sequence+similarity+for+species+demarcation+of+prokaryotes&rft.jtitle=International+Journal+of+Systematic+and+Evolutionary+Microbiology&rft.aulast=Kim%2C+M.%3B+Oh%2C+H.S.%3B+Park%2C+S.C.%3B+Chun%2C+J.&rft.au=Kim%2C+M.%3B+Oh%2C+H.S.%3B+Park%2C+S.C.%3B+Chun%2C+J.&rft.date=2014&rft.volume=64&rft.issue=Pt+2&rft.pages=346-51&rft_id=info:doi\/10.1099%2Fijs.0.059774-0&rft_id=info:pmid\/24505072&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Beaz-HidalgoStrat15-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Beaz-HidalgoStrat15_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Beaz-Hidalgo, R;. Hossain, M.J.; Liles, M.R.; Figueras, M.J. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4301921\" target=\"_blank\">\"Strategies to avoid wrongly labelled genomes using as example the detected wrong taxonomic affiliation for aeromonas genomes in the GenBank database\"<\/a>. <i>PLoS One<\/i> <b>10<\/b> (1): e0115813. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0115813\" target=\"_blank\">10.1371\/journal.pone.0115813<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4301921\/\" target=\"_blank\">PMC4301921<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25607802\" target=\"_blank\">25607802<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4301921\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4301921<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Strategies+to+avoid+wrongly+labelled+genomes+using+as+example+the+detected+wrong+taxonomic+affiliation+for+aeromonas+genomes+in+the+GenBank+database&rft.jtitle=PLoS+One&rft.aulast=Beaz-Hidalgo%2C+R%3B.+Hossain%2C+M.J.%3B+Liles%2C+M.R.%3B+Figueras%2C+M.J.&rft.au=Beaz-Hidalgo%2C+R%3B.+Hossain%2C+M.J.%3B+Liles%2C+M.R.%3B+Figueras%2C+M.J.&rft.date=2015&rft.volume=10&rft.issue=1&rft.pages=e0115813&rft_id=info:doi\/10.1371%2Fjournal.pone.0115813&rft_id=info:pmc\/PMC4301921&rft_id=info:pmid\/25607802&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4301921&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TanizawaDFAST16-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TanizawaDFAST16_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tanizawa, Y.; Fujisawa, T.; Kaminuma, E. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5107635\" target=\"_blank\">\"DFAST and DAGA: web-based integrated genome annotation tools and resources\"<\/a>. <i>Bioscience of Microbiota, Food and Health<\/i> <b>35<\/b> (4): 173-184. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.12938%2Fbmfh.16-003\" target=\"_blank\">10.12938\/bmfh.16-003<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5107635\/\" target=\"_blank\">PMC5107635<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27867804\" target=\"_blank\">27867804<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5107635\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5107635<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DFAST+and+DAGA%3A+web-based+integrated+genome+annotation+tools+and+resources&rft.jtitle=Bioscience+of+Microbiota%2C+Food+and+Health&rft.aulast=Tanizawa%2C+Y.%3B+Fujisawa%2C+T.%3B+Kaminuma%2C+E.+et+al.&rft.au=Tanizawa%2C+Y.%3B+Fujisawa%2C+T.%3B+Kaminuma%2C+E.+et+al.&rft.date=2016&rft.volume=35&rft.issue=4&rft.pages=173-184&rft_id=info:doi\/10.12938%2Fbmfh.16-003&rft_id=info:pmc\/PMC5107635&rft_id=info:pmid\/27867804&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5107635&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NCBIBacterial-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NCBIBacterial_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"ftp:\/\/ftp.ncbi.nlm.nih.gov\/genomes\/ASSEMBLY_REPORTS\/ANI_report_bacteria.txt\" target=\"_blank\">\"Bacterial ANI Report\"<\/a>. NCBI<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"ftp:\/\/ftp.ncbi.nlm.nih.gov\/genomes\/ASSEMBLY_REPORTS\/ANI_report_bacteria.txt\" target=\"_blank\">ftp:\/\/ftp.ncbi.nlm.nih.gov\/genomes\/ASSEMBLY_REPORTS\/ANI_report_bacteria.txt<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bacterial+ANI+Report&rft.atitle=&rft.pub=NCBI&rft_id=ftp%3A%2F%2Fftp.ncbi.nlm.nih.gov%2Fgenomes%2FASSEMBLY_REPORTS%2FANI_report_bacteria.txt&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FederhenType15-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FederhenType15_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Federhen, S. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383940\" target=\"_blank\">\"Type material in the NCBI Taxonomy Database\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (DB1): D1086\u201398. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1127\" target=\"_blank\">10.1093\/nar\/gku1127<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383940\/\" target=\"_blank\">PMC4383940<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25398905\" target=\"_blank\">25398905<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383940\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383940<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Type+material+in+the+NCBI+Taxonomy+Database&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Federhen%2C+S.&rft.au=Federhen%2C+S.&rft.date=2015&rft.volume=43&rft.issue=DB1&rft.pages=D1086%E2%80%9398&rft_id=info:doi\/10.1093%2Fnar%2Fgku1127&rft_id=info:pmc\/PMC4383940&rft_id=info:pmid\/25398905&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383940&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KozlovPhylo16-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KozlovPhylo16_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kozlov, A.M.; Zhang, J.; Yilmaz, P. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4914121\" target=\"_blank\">\"Phylogeny-aware identification and correction of taxonomically mislabeled sequences\"<\/a>. <i>Nucleic Acids Research<\/i> <b>44<\/b> (11): 5022-33. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkw396\" target=\"_blank\">10.1093\/nar\/gkw396<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4914121\/\" target=\"_blank\">PMC4914121<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27166378\" target=\"_blank\">27166378<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4914121\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4914121<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Phylogeny-aware+identification+and+correction+of+taxonomically+mislabeled+sequences&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Kozlov%2C+A.M.%3B+Zhang%2C+J.%3B+Yilmaz%2C+P.+et+al.&rft.au=Kozlov%2C+A.M.%3B+Zhang%2C+J.%3B+Yilmaz%2C+P.+et+al.&rft.date=2016&rft.volume=44&rft.issue=11&rft.pages=5022-33&rft_id=info:doi\/10.1093%2Fnar%2Fgkw396&rft_id=info:pmc\/PMC4914121&rft_id=info:pmid\/27166378&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4914121&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RichterShift09-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RichterShift09_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Richter, M.; Rossell\u00f3-M\u00f3ra, R. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2776425\" target=\"_blank\">\"Shifting the genomic gold standard for the prokaryotic species definition\"<\/a>. <i>Proceedings of the National Academy of Sciences of the United States of America<\/i> <b>106<\/b> (45): 19126-31. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1073%2Fpnas.0906412106\" target=\"_blank\">10.1073\/pnas.0906412106<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2776425\/\" target=\"_blank\">PMC2776425<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19855009\" target=\"_blank\">19855009<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2776425\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2776425<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Shifting+the+genomic+gold+standard+for+the+prokaryotic+species+definition&rft.jtitle=Proceedings+of+the+National+Academy+of+Sciences+of+the+United+States+of+America&rft.aulast=Richter%2C+M.%3B+Rossell%C3%B3-M%C3%B3ra%2C+R.&rft.au=Richter%2C+M.%3B+Rossell%C3%B3-M%C3%B3ra%2C+R.&rft.date=2009&rft.volume=106&rft.issue=45&rft.pages=19126-31&rft_id=info:doi\/10.1073%2Fpnas.0906412106&rft_id=info:pmc\/PMC2776425&rft_id=info:pmid\/19855009&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2776425&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-K.C3.A4mpferProkary12-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-K.C3.A4mpferProkary12_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">K\u00e4mpfer, P.; Glaeser, S.P. (2012). \"Prokaryotic taxonomy in the sequencing era--the polyphasic approach revisited\". <i>Environmental Microbiology<\/i> <b>14<\/b> (2): 291-317. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1462-2920.2011.02615.x\" target=\"_blank\">10.1111\/j.1462-2920.2011.02615.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22040009\" target=\"_blank\">22040009<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Prokaryotic+taxonomy+in+the+sequencing+era--the+polyphasic+approach+revisited&rft.jtitle=Environmental+Microbiology&rft.aulast=K%C3%A4mpfer%2C+P.%3B+Glaeser%2C+S.P.&rft.au=K%C3%A4mpfer%2C+P.%3B+Glaeser%2C+S.P.&rft.date=2012&rft.volume=14&rft.issue=2&rft.pages=291-317&rft_id=info:doi\/10.1111%2Fj.1462-2920.2011.02615.x&rft_id=info:pmid\/22040009&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LarsenBench14-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LarsenBench14_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Larsen, M.V.; Cosentino, S.; Lukjancenko, O. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3993634\" target=\"_blank\">\"Benchmarking of methods for genomic taxonomy\"<\/a>. <i>Journal of Clinical Microbiology<\/i> <b>52<\/b> (5): 1529-39. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1128%2FJCM.02981-13\" target=\"_blank\">10.1128\/JCM.02981-13<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3993634\/\" target=\"_blank\">PMC3993634<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24574292\" target=\"_blank\">24574292<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3993634\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3993634<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Benchmarking+of+methods+for+genomic+taxonomy&rft.jtitle=Journal+of+Clinical+Microbiology&rft.aulast=Larsen%2C+M.V.%3B+Cosentino%2C+S.%3B+Lukjancenko%2C+O.+et+al.&rft.au=Larsen%2C+M.V.%3B+Cosentino%2C+S.%3B+Lukjancenko%2C+O.+et+al.&rft.date=2014&rft.volume=52&rft.issue=5&rft.pages=1529-39&rft_id=info:doi\/10.1128%2FJCM.02981-13&rft_id=info:pmc\/PMC3993634&rft_id=info:pmid\/24574292&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3993634&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlipanahiPredict15-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlipanahiPredict15_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alipanahi, B.; Delong, A.; Weirauch, M.T.; Frey, B.J. (2015). \"Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning\". <i>Nature Biotechnology<\/i> <b>33<\/b> (8): 831\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt.3300\" target=\"_blank\">10.1038\/nbt.3300<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26213851\" target=\"_blank\">26213851<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Predicting+the+sequence+specificities+of+DNA-+and+RNA-binding+proteins+by+deep+learning&rft.jtitle=Nature+Biotechnology&rft.aulast=Alipanahi%2C+B.%3B+Delong%2C+A.%3B+Weirauch%2C+M.T.%3B+Frey%2C+B.J.&rft.au=Alipanahi%2C+B.%3B+Delong%2C+A.%3B+Weirauch%2C+M.T.%3B+Frey%2C+B.J.&rft.date=2015&rft.volume=33&rft.issue=8&rft.pages=831%E2%80%938&rft_id=info:doi\/10.1038%2Fnbt.3300&rft_id=info:pmid\/26213851&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhouPredict15-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhouPredict15_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhou, J.; Troyanskaya, O.G. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4768299\" target=\"_blank\">\"Predicting effects of noncoding variants with deep learning-based sequence model\"<\/a>. <i>Nature Methods<\/i> <b>12<\/b> (10): 931\u20134. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth.3547\" target=\"_blank\">10.1038\/nmeth.3547<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4768299\/\" target=\"_blank\">PMC4768299<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26301843\" target=\"_blank\">26301843<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4768299\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4768299<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Predicting+effects+of+noncoding+variants+with+deep+learning-based+sequence+model&rft.jtitle=Nature+Methods&rft.aulast=Zhou%2C+J.%3B+Troyanskaya%2C+O.G.&rft.au=Zhou%2C+J.%3B+Troyanskaya%2C+O.G.&rft.date=2015&rft.volume=12&rft.issue=10&rft.pages=931%E2%80%934&rft_id=info:doi\/10.1038%2Fnmeth.3547&rft_id=info:pmc\/PMC4768299&rft_id=info:pmid\/26301843&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4768299&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrownRecurrent18-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrownRecurrent18_48-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brown, A.; Tuor, A.; Hutchinson, B.; Nichols, N. (2018). \"Recurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detection\". <i>Proceedings of the First Workshop on Machine Learning for Computing Systems<\/i>: 1. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F3217871.3217872\" target=\"_blank\">10.1145\/3217871.3217872<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Recurrent+Neural+Network+Attention+Mechanisms+for+Interpretable+System+Log+Anomaly+Detection&rft.jtitle=Proceedings+of+the+First+Workshop+on+Machine+Learning+for+Computing+Systems&rft.aulast=Brown%2C+A.%3B+Tuor%2C+A.%3B+Hutchinson%2C+B.%3B+Nichols%2C+N.&rft.au=Brown%2C+A.%3B+Tuor%2C+A.%3B+Hutchinson%2C+B.%3B+Nichols%2C+N.&rft.date=2018&rft.pages=1&rft_id=info:doi\/10.1145%2F3217871.3217872&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoodfellowGen14-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoodfellowGen14_49-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goodfellow, I.; Pouget-Abadie, J.; Mirza, M. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/papers.nips.cc\/paper\/5423-generative-adversarial-nets\" target=\"_blank\">\"Generative Adversarial Nets\"<\/a>. <i>Proceedings from Advances in Neural Information Processing Systems 2014<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/papers.nips.cc\/paper\/5423-generative-adversarial-nets\" target=\"_blank\">http:\/\/papers.nips.cc\/paper\/5423-generative-adversarial-nets<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Generative+Adversarial+Nets&rft.jtitle=Proceedings+from+Advances+in+Neural+Information+Processing+Systems+2014&rft.aulast=Goodfellow%2C+I.%3B+Pouget-Abadie%2C+J.%3B+Mirza%2C+M.+et+al.&rft.au=Goodfellow%2C+I.%3B+Pouget-Abadie%2C+J.%3B+Mirza%2C+M.+et+al.&rft.date=2014&rft_id=http%3A%2F%2Fpapers.nips.cc%2Fpaper%2F5423-generative-adversarial-nets&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DalviAdvers04-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DalviAdvers04_50-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dalvi, N.; Domingos, P.; Mausam, S. et al. (2004). \"Adversarial classification\". <i>Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining<\/i>: 99\u2013108. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1014052.1014066\" target=\"_blank\">10.1145\/1014052.1014066<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adversarial+classification&rft.jtitle=Proceedings+of+the+Tenth+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mining&rft.aulast=Dalvi%2C+N.%3B+Domingos%2C+P.%3B+Mausam%2C+S.+et+al.&rft.au=Dalvi%2C+N.%3B+Domingos%2C+P.%3B+Mausam%2C+S.+et+al.&rft.date=2004&rft.pages=99%E2%80%93108&rft_id=info:doi\/10.1145%2F1014052.1014066&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Kantarc.C4.B1o.C4.9FluClass11-51\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Kantarc.C4.B1o.C4.9FluClass11_51-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kantarc\u0131o\u011flu, M.; Xi, B.; Clifton, C. (2011). \"Classifier evaluation and attribute selection against active adversaries\". <i>Data Mining and Knowledge Discovery<\/i> <b>22<\/b> (1\u20132): 291\u2013335. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10618-010-0197-3\" target=\"_blank\">10.1007\/s10618-010-0197-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Classifier+evaluation+and+attribute+selection+against+active+adversaries&rft.jtitle=Data+Mining+and+Knowledge+Discovery&rft.aulast=Kantarc%C4%B1o%C4%9Flu%2C+M.%3B+Xi%2C+B.%3B+Clifton%2C+C.&rft.au=Kantarc%C4%B1o%C4%9Flu%2C+M.%3B+Xi%2C+B.%3B+Clifton%2C+C.&rft.date=2011&rft.volume=22&rft.issue=1%E2%80%932&rft.pages=291%E2%80%93335&rft_id=info:doi\/10.1007%2Fs10618-010-0197-3&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiggioEvasion13-52\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BiggioEvasion13_52-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Biggio, B.; Corona, I.; Maiorca, D. et al. (2013). \"Evasion Attacks against Machine Learning at Test Time\". In Blockeel, H.; Kersting, K.; Nijssen, S.; \u017delezn\u00fd, F.. <i>Machine Learning and Knowledge Discovery in Databases<\/i>. Lecture Notes in Computer Science. Springer. pp. 387\u2013402. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-40994-3_25\" target=\"_blank\">10.1007\/978-3-642-40994-3_25<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783642409943.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Evasion+Attacks+against+Machine+Learning+at+Test+Time&rft.atitle=Machine+Learning+and+Knowledge+Discovery+in+Databases&rft.aulast=Biggio%2C+B.%3B+Corona%2C+I.%3B+Maiorca%2C+D.+et+al.&rft.au=Biggio%2C+B.%3B+Corona%2C+I.%3B+Maiorca%2C+D.+et+al.&rft.date=2013&rft.series=Lecture+Notes+in+Computer+Science&rft.pages=pp.%26nbsp%3B387%E2%80%93402&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-642-40994-3_25&rft.isbn=9783642409943&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DutrisacHiding08-53\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DutrisacHiding08_53-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dutrisac, J.G.; Skillicorn, D.B. (2008). \"Hiding clusters in adversarial settings\". <i>Proceedings from the 2008 IEEE International Conference on Intelligence and Security Informatics<\/i>: 185\u201387. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FISI.2008.4565051\" target=\"_blank\">10.1109\/ISI.2008.4565051<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hiding+clusters+in+adversarial+settings&rft.jtitle=Proceedings+from+the+2008+IEEE+International+Conference+on+Intelligence+and+Security+Informatics&rft.aulast=Dutrisac%2C+J.G.%3B+Skillicorn%2C+D.B.&rft.au=Dutrisac%2C+J.G.%3B+Skillicorn%2C+D.B.&rft.date=2008&rft.pages=185%E2%80%9387&rft_id=info:doi\/10.1109%2FISI.2008.4565051&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiggioIsData13-54\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BiggioIsData13_54-0\">54.0<\/a><\/sup> <sup><a href=\"#cite_ref-BiggioIsData13_54-1\">54.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Biggio, B.; Pillai, I.; Bul\u00f2, S.R. et al. (2013). \"Is data clustering in adversarial settings secure?\". <i>Proceedings of the 2013 ACM Workshop on Artificial Intelligence and Security<\/i>: 87\u201398. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2517312.2517321\" target=\"_blank\">10.1145\/2517312.2517321<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Is+data+clustering+in+adversarial+settings+secure%3F&rft.jtitle=Proceedings+of+the+2013+ACM+Workshop+on+Artificial+Intelligence+and+Security&rft.aulast=Biggio%2C+B.%3B+Pillai%2C+I.%3B+Bul%C3%B2%2C+S.R.+et+al.&rft.au=Biggio%2C+B.%3B+Pillai%2C+I.%3B+Bul%C3%B2%2C+S.R.+et+al.&rft.date=2013&rft.pages=87%E2%80%9398&rft_id=info:doi\/10.1145%2F2517312.2517321&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KegelmeyerCounter15-55\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KegelmeyerCounter15_55-0\">55.0<\/a><\/sup> <sup><a href=\"#cite_ref-KegelmeyerCounter15_55-1\">55.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Kegelmeyer, P.; Shead, T.M.; Crussell, J. et al. (08 May 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sandia.gov\/~wpk\/pubs\/publications\/cada-full-uur.pdf\" target=\"_blank\">\"Counter Adversarial Data Analytics\"<\/a> (PDF). <i>Sandia Report SAND2015-3711<\/i>. Sandia National Laboratory<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sandia.gov\/~wpk\/pubs\/publications\/cada-full-uur.pdf\" target=\"_blank\">https:\/\/www.sandia.gov\/~wpk\/pubs\/publications\/cada-full-uur.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Counter+Adversarial+Data+Analytics&rft.atitle=Sandia+Report+SAND2015-3711&rft.aulast=Kegelmeyer%2C+P.%3B+Shead%2C+T.M.%3B+Crussell%2C+J.+et+al.&rft.au=Kegelmeyer%2C+P.%3B+Shead%2C+T.M.%3B+Crussell%2C+J.+et+al.&rft.date=08+May+2015&rft.pub=Sandia+National+Laboratory&rft_id=https%3A%2F%2Fwww.sandia.gov%2F%7Ewpk%2Fpubs%2Fpublications%2Fcada-full-uur.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IGSCHarm17-56\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IGSCHarm17_56-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">International Gene Synthesis Consortium (19 November 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/genesynthesisconsortium.org\/wp-content\/uploads\/IGSCHarmonizedProtocol11-21-17.pdf\" target=\"_blank\">\"Harmonized Screening Protocol v2.0\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/genesynthesisconsortium.org\/wp-content\/uploads\/IGSCHarmonizedProtocol11-21-17.pdf\" target=\"_blank\">https:\/\/genesynthesisconsortium.org\/wp-content\/uploads\/IGSCHarmonizedProtocol11-21-17.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Harmonized+Screening+Protocol+v2.0&rft.atitle=&rft.aulast=International+Gene+Synthesis+Consortium&rft.au=International+Gene+Synthesis+Consortium&rft.date=19+November+2017&rft_id=https%3A%2F%2Fgenesynthesisconsortium.org%2Fwp-content%2Fuploads%2FIGSCHarmonizedProtocol11-21-17.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The footnote in the original material were turned into an inline references for this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165452\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.352 seconds\nReal time usage: 1.381 seconds\nPreprocessor visited node count: 45214\/1000000\nPreprocessor generated node count: 42110\/1000000\nPost\u2010expand include size: 408396\/2097152 bytes\nTemplate argument size: 133699\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 1356.379 1 - -total\n 91.09% 1235.473 1 - Template:Reflist\n 81.42% 1104.337 56 - Template:Citation\/core\n 73.78% 1000.671 49 - Template:Cite_journal\n 11.85% 160.702 123 - Template:Citation\/identifier\n 6.65% 90.152 3 - Template:Cite_book\n 4.99% 67.666 4 - Template:Cite_web\n 4.89% 66.316 1 - Template:Infobox_journal_article\n 4.75% 64.389 60 - Template:Citation\/make_link\n 4.66% 63.258 1 - Template:Infobox\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11074-0!*!0!!en!*!* and timestamp 20190701165450 and revision id 35900\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure\">https:\/\/www.limswiki.org\/index.php\/Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","484090b9c88bd26fd5f834f6ab7cbd8a_images":[],"484090b9c88bd26fd5f834f6ab7cbd8a_timestamp":1562000090,"74131eedf92ecea39da4d1e4c07690b3_type":"article","74131eedf92ecea39da4d1e4c07690b3_title":"National and transnational security implications of asymmetric access to and use of biological data (Berger and Schneck 2019)","74131eedf92ecea39da4d1e4c07690b3_url":"https:\/\/www.limswiki.org\/index.php\/Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data","74131eedf92ecea39da4d1e4c07690b3_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:National and transnational security implications of asymmetric access to and use of biological data\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nNational and transnational security implications of asymmetric access to and use of biological dataJournal\n \nFrontiers in Bioengineering and BiotechnologyAuthor(s)\n \nBerger, Kavita M.; Schneck, Phyllis A.Author affiliation(s)\n \nGryphon Scientific, LLC; Promontory Financial Group, an IBM CompanyPrimary contact\n \nEmail: kberger at gryphonscientific dot comEditors\n \nMurch, Randall S.Year published\n \n2019Volume and issue\n \n7Page(s)\n \n21DOI\n \n10.3389\/fbioe.2019.00021ISSN\n \n2296-4185Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00021\/fullDownload\n \nhttps:\/\/www.frontiersin.org\/articles\/10.3389\/fbioe.2019.00021\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Current approaches to protecting data \n4 Vulnerability of biotechnology data \n5 Possible prevention and mitigation approaches \n\n5.1 Collaboration \n5.2 End-to-end risk assessment \n5.3 Data-specific risk mitigation \n5.4 Application of the NIST Cybersecurity Framework \n\n\n6 Acknowledgements \n\n6.1 Author contributions \n6.2 Disclaimer \n6.3 Conflict of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nBiology and biotechnology have changed dramatically during the past 20 years, in part because of increases in computational capabilities and use of engineering principles to study biology. The advances in supercomputing, data storage capacity, and cloud platforms enable scientists throughout the world to generate, analyze, share, and store vast amounts of data, some of which are biological and much of which may be used to understand the human condition, agricultural systems, evolution, and environmental ecosystems. These advances and applications have enabled: (1) the emergence of data science, which involves the development of new algorithms to analyze and visualize data; and (2) the use of engineering approaches to manipulate or create new biological organisms that have specific functions, such as production of industrial chemical precursors and development of environmental bio-based sensors. Several biological sciences fields harness the capabilities of computer, data, and engineering sciences, including synthetic biology, precision medicine, precision agriculture, and systems biology. These advances and applications are not limited to one country. This capability has economic and physical consequences but is vulnerable to unauthorized intervention. Healthcare and genomic information of patients, information about pharmaceutical and biotechnology products in development, and results of scientific research have been stolen by state and non-state actors through infiltration of databases and computer systems containing this information. Countries have developed their own policies for governing data generation, access, and sharing with foreign entities, resulting in asymmetry of data sharing. This paper describes security implications of asymmetric access to and use of biological data.\nKeywords: biotechnology, cybersecurity, information security, data vulnerability, biological data, biosecurity, data access, data protection\n\nIntroduction \nAdvances in computer science, engineering, and data science have changed research, development, and application of biology and biotechnology in the United States and internationally. Examples of changes include: (a) increased reliance on internet connectivity for research and laboratory operations[1][2][3]; (b) increased use of automation in life-science laboratories[4]; (c) application of the \u201cdesign-build-test\u201d paradigm to create new biological organisms[5][6]; (d) increased generation, analyses, and computational modeling of information about biological systems, cells, and molecules[7][8]; (e) treatment of organisms and DNA as materials rather than phenomena to study[9][10][11]; and (f) new funders such as venture capital, crowdfunding platforms, and foreign companies and governments.[12][13][14] These changes have transformed the scientific, agricultural, and health communities' ability to understand and manipulate the world around them. In addition, the changes have enabled an influx of new practitioners and problem-solvers into biology, providing opportunities for education and research all over the world.\nBiotechnology harnesses the capabilities of computer, data, and engineering sciences to establish and advance new fields such as synthetic biology, precision medicine, precision agriculture, and systems biology. Cloud-based platforms and open-source, easy-to-use software enable scientists from anywhere in the world to use advanced data analytics in their studies. The software and hardware emerging from these fields improve our collective understanding of molecular and systems-level genetics, new drug therapies for longer and better quality of life, and design of novel and\/or unnatural organisms. Critical to these pursuits is the sharing of research results and underlying data, without which societal decision-making about human, animal, plant, and environmental health cannot be realized fully. However, during the past two decades, concerns about data sharing have been raised, resulting in the issuance of international, regional, and national-level policies governing access to different types of data, including biological data. In addition, the platforms through which data are stored, transported, and analyzed may be vulnerable to unauthorized acquisition of information by malicious actors, which could lead to significant economic and physical harms to the health, safety, and security of a population. Although not considered \u201cdual use life sciences research of concern,\u201d[15][16] the potential for both benefit and risk to humanity meets the spirit of the dual use concept.[17] Given the significant benefits afforded by data sharing and analysis, this paper highlights current data protection policies, potential risks of data exploitation by malicious actors, and potential strategies to mitigate those risks and promote rapid recovery in biotechnology fields that are breached.\nThe interconnectedness between the digital and biological worlds can be exploited by state actors, malicious nonstate actors, and hackers through a variety of means, resulting in harmful consequences from potential theft of information, promulgation of incorrect information, and\/or disruption of activities.[18][19][20] For example, theft of proprietary information from a pharmaceutical or biotechnology company may reveal trade secrets and allow competitors to develop superior products and\/or bring existing products to market more quickly[21], stifling innovation in the global commercial market and allowing adversaries to create harmful, untested therapies. Another example is theft of hundreds of millions of electronic healthcare records, the uses of which are not clear.[22][23][24][25][26] Although unauthorized access to protected data may be aided by technical vulnerabilities in networked computer systems, poor security practices, insider threats in academia, industry, and health facilities, and legal business dealings also can enable adversary access to such data.[27][28][29][30] For examples, more than half of all data breaches at healthcare facilities are caused by healthcare personnel errors, a quarter of which resulted in unauthorized access to or disclosure of patient records through sharing of unencrypted information, sending information to the wrong patients, and accessing the data without authorization.[31][32] In addition, the Federal Bureau of Investigation (FBI) has raised national security concerns about foreign access to genomic data of U.S. citizens through legitimate scientific collaboration, funding of scientific research, investment in genomic sequencing companies (e.g., China-based WuXi Healthcare Ventures investment in the U.S.-based 23andMe[33][34]), and purchase of companies (e.g., Complete Genomics).[35][36] As vulnerabilities are created through scientific advances, such as the use of machine learning algorithms to trick fingerprint authentication systems, new risks are identified.[37][38] Some of these concerns have resulted in the passage of the 2018 Foreign Investment Risk Review Modernization Act, which has initiated reform of the U.S. Government process for evaluating foreign investment in U.S. entities and export control of emerging technologies.[28][39] Yet, these policy activities largely are reactive, rather than proactive.\n\nCurrent approaches to protecting data \nPreventing accidental and deliberate risks typically involves the use of cyber and information security systems that include technological and behavioral solutions. Protection of laboratory control systems, computer networks, and databases often involves the use of technological solutions. However, some risks are addressed better through training of personnel to recognize and report phishing attempts, ensure sensitive information is encrypted, and prevent unauthorized individuals from gaining access to sensitive data, databases, and computer networks. To enhance security, policies for promulgating these practices for specific materials and information have been issued. For example, the U.S. Biological Select Agents and Toxins Regulations include guidance for network security to prevent failure of laboratories, equipment, and access controls to facilities and data.[40] In addition, the U.S. has policies for protecting individual privacy, several of which were described in a 2014 report sponsored by the White House.[41] However, error, carelessness, or negligence by personnel can counteract the benefits afforded by security measures and may lead to devastating consequences if biological data and materials are involved.\nAlthough policies for protecting biological data from cyberattack are limited, policies that govern data access and sharing are prevalent. These top-down, data access policies intend to protect individual rights and\/or prevent sharing or distribution of data, including biological data. Examples of recent policies include: (a) the 2018 update of the European Union General Data Protection Regulation[42], which strengthened the European Union's rules for protecting personal data of individuals, in part by giving its citizens \u201cmore control over their personal data\u201d; (b) the 2018 Chinese Personal Information Security Specification, which is one system under the Chinese Cybersecurity law, involves the \u201ccollection, storage, use, sharing, transfer, and disclosure of personal information,\u201d and enables companies operating in China to access data to \u201cnot hamper the development of fields like AI\u201d[43]; (c) the 2018 General Data Protection Law in Brazil, which provides a framework for the use of personal data in Brazil[44]; and (d) the U.S. Health Insurance Portability and Accountability Act of 1996 (HIPAA), which promotes the protection of privacy and security of patient health information in the United States.[45] At the same time, the U.S. has issued policies governing data generation, access, and sharing to promote information-sharing and transparency of government-sponsored research.[46] Internationally, the Nagoya Protocol of the Convention on Biodiversity[47] promotes governance on access to and fair, equitable sharing of the benefits from the use of non-human biological data. However, questions exist about whether the Nagoya Protocol focuses more on biological samples that provide genetic information or the genetic information itself, which ultimately affects national-level efforts for codifying the international agreement.[48] Despite these activities, protection of some data, such as personal health data, may not extend beyond a country's borders and may apply only to data collected by certain entities. Furthermore, data protection polices do not extend to information that already has been stolen. Taken together, these national, regional, and international level policies for data protection may not prevent the inappropriate or unauthorized acquisition of data to different actors, the consequences of which are unclear for biotechnology data.\n\nVulnerability of biotechnology data \nThe primary challenges in identifying, assessing, and mitigating security vulnerabilities of biotechnology data are understanding: (a) how the data may be exploited by adversaries and what consequences result from this exploitation; and (2) what potential negative effects may arise from digitization of biotechnology and advanced computation of biological data.[2] The term \u201cbiotechnology\u201d refers to the exploitation of biological processes for industrial and scientific purposes, and includes genetic manipulation of microbes, plants, animals, human cells, nucleic acids (the building blocks of genomes), and proteins (the functional units in cells). This definition is expanded further to include generation, incorporation, and use of digital forms of biological data. These biological data may be available online through databases, such as the U.S. National Center for Biotechnology Information's GenBank[49], or generated in a laboratory and stored, shared, and\/or analyzed locally or remotely (via online and\/or cloud-based software). By attempting to answer the questions posed above, specific risks associated with the legal and illegal acquisition of biological data may be identified and mitigated.\nAlthough extraordinary advances in computing power are enabling unprecedented scientific discoveries, its application to biology and healthcare is increasing without effective protection from the risks of adversary acquisition or accidental misuse of information. Scientific data that is generated in basic and applied research laboratories in academia, non-profit research organizations, service providers, and some industry research facilities may be considered fundamental research destined for publication and public benefit. These data are not necessarily sensitive, but they do represent the results of significant investment by governments, industry, investors, and philanthropic organizations. Therefore, theft or large-scale acquisition of these data may have adverse economic consequences to the organization, field, or nation, especially if acquisition was directed by adversarial nation-states to gain competitive advantage in a given sector.[50] As previously described, databases that store sensitive and\/or non-sensitive biological data have been infiltrated by external actors and accessed by unauthorized individuals. Although measures to protect data have been implemented in several institutions, cyber and information security policies, practices, and compliance vary across biotechnology sectors, location, and organization type (e.g., academia, industry). Although implementation of cyber, information, and data security in biological facilities can help to minimize the potential for deliberate or accidental release of protected biological data, these measures are insufficient on their own.[51]\nFurthermore, the increasing size and volume of the datasets, and the complexity of analytic technologies has led many scientists to rely on cloud-based platforms to store, transfer, and analyze data. These platforms and technologies, including online analysis software and applications, often do not prevent unauthorized access to data or ensure software fidelity. Although mitigating specific vulnerabilities may be possible on an individual platform or technology level, implementing protections across the various data generation, analysis, transfer, and storage platforms currently in use in academia, industry, government laboratories, and healthcare facilities is challenging. Countering these risks requires the identification of consequences that are of particular concern to public safety and national security, evaluation of vulnerabilities that may enable the realization of these consequences, and identification of measures to address these vulnerabilities.\n\nPossible prevention and mitigation approaches \nModern cyber and information security reflects the risks experienced as the internet has grown and diversified, and as the capabilities for and speed of storing, processing, and transporting information have increased exponentially.[52] The internet was built without a priority on the protection of data whether \u201cat rest\u201d (i.e., stored data) or \u201cin motion\u201d (i.e., data in transit).[53][54] Current strategies for addressing cyber risks focus on remediation through regulation, organizational support, and actions taken by data owners and consumers in the form of encryption technologies, access control measures, awareness-raising campaigns, risk assessment, blocking, limiting publication of sensitive information, and other similar practices. The challenge is understanding how these measures are to be applied to biotechnology data, how to balance the cost of implementation with the consequences if left unprotected, and what vulnerabilities cannot be mitigated using commercial products.\nOften the entities that assess their cyber vulnerabilities and invest in cyber and information security measures are compelled to do so because of regulation and fiscal responsibility.[55] However, unlike financial information, biotechnology data is regulated in some countries, but not others. For example, China issued a recent policy requiring a domestic collaborator and Ministry-level approval for research involving genomic data of Chinese citizens and\/or biological samples obtained in China to prevent exploitation of these data and samples.[56] This and similar policies raise questions about their intended and unintended effects to nations, to the scientific community, and to international security mainly because the policies that may benefit one country could harm another. These harms may reveal new types of risks associated with the acquisition and use of data to manipulate biological systems. These risks may be perpetrated by different actors; affect sector and country economies, commercial biotechnology, and pharmaceutical markets domestically and internationally; and alter global strategic power dynamics.\nThe risks associated with biotechnology data do not conform to traditional biosecurity concerns, which focus primarily on risks to human health or the food and agriculture economy. These risks involve multiple domains, sectors, and nations resulting in outcomes such as shifting of balance of power of nations at the international level, which could have downstream effects on areas that overlap with biosecurity interests (e.g., biosafety and biosecurity, biothreat reduction, and global health security). Strategies for bridging the biological, cyber, information, and data security include: (a) collaboration between the biological and cybersecurity communities; (b) end-to-end risk assessments; (c) data-specific risk and vulnerability assessments; and (d) application of the NIST Cybersecurity Framework for protecting biological data. \nThese suggested strategies (detailed below) describe various approaches toward protecting biological data from unauthorized acquisition and use, enhancing efforts to preserve data integrity and provenance, and enabling future benefit of biotechnological advances.\n\nCollaboration \nFormal collaboration between the biotechnology and biological, information, data, and cyber security communities would enhance efforts toward identification of risks and vulnerabilities associated with data management, provenance, and integrity, and risk mitigation strategies. Technologies are readily available to protect data, but their use must be harmonized worldwide, because protecting data in one database is ineffective if another database remains vulnerable to external threats. Furthermore, organizations may evade regulatory requirements and industry standards in protecting data because of perceived lack of cost savings for implementing cybersecurity measures or lack of awareness of the risks, which could lead to investor, intruder, or adversary access to sensitive information that may be stored in databases or transferred between computers. These vulnerabilities may be exacerbated by limitations of national laws to other sovereign states, and differences in interpretation of the types of data included in the scope of existing laws. \nThe takeaway: Given these potential vulnerabilities, the cybersecurity and biotechnology communities must engage to create best practices and processes to protect data and mitigate risk while reaping the benefits of computing technology applications to biotechnology.\n\nEnd-to-end risk assessment \nEnd-to-end assessments of the data storage, processing, and transport pipeline can identify outstanding vulnerabilities and technical gaps that may be addressed with currently available cyber, information, and data security solutions. This process would enable identification of gaps for which these measures are insufficient and of institutions that are responsible for implementing controls. Without this type of assessment, vulnerabilities may exist along the pipeline without its users' knowledge. A lack of rigorous analysis makes biological data vulnerable to acquisition or alteration by witting adversaries, potentially resulting in theft of intellectual property for commercial gain, foreign government acquisition of genomic data from large portions of a population for undefined purpose or compromise of software and data integrity. At least one country promotes acquisition of data though legitimate commercial practices (e.g., providing sequencing services to customers; partnering with academia, independent research institutions, and universities; and foreign investment), talent promotion programs[57][58], and theft of data.[59][60][61][62] The FBI has expressed concerns about the theft of U.S. genomics and health information through cyberattacks and foreign investment in the U.S. biotechnology industry.[63] The FBI argues that acquisition of this information can give adversaries an unfair advantage in the international pharmaceutical or biotechnology marketplace. Others have expressed concern about questionable use of genetic information that countries obtain from their own citizens or from other countries' citizens.[64][65][66]\nThe takeway: These risks could be addressed by conducting an end-to-end risk assessment of the software and equipment involved in the data pipeline within individual organizations, between organizations, and across countries.\n\nData-specific risk mitigation \nDefining the consequences of greatest concern to national security is an initial step toward assessing the risks and vulnerabilities of the information itself and data-specific risk mitigation strategies. Evaluating these risks enables the identification of content-specific approaches for detecting and countering exploitation of vulnerabilities by insider and external actors. Without these assessments, only generic cyber and information security measures will be implemented. However, these measures are insufficient to counter adversaries who are intent on acquiring data through a variety of technical, social engineering, or other means. Given this reality, rapid detection and resilience (i.e., rapid recovery after a breach) are critical for reaping the benefits and minimizing the vulnerabilities of advanced electronic computation and mass connectivity. In 2014, the White House explored technology needs for protecting the security and privacy of exposed data, including healthcare data.[67][68] But, these studies did not define consequences of concern related to the unauthorized acquisition of vast amounts of biological data, effectively limiting the identification of data-specific or process-specific prevention measures. \nThe takeaway: Therefore, risk assessments of specific types of data are equally as important to conduct as analyses of vulnerabilities of laboratory control systems, data management platforms, and computer networks.\n\nApplication of the NIST Cybersecurity Framework \nApplication of the National Institute of Standards and Technology (NIST) Cybersecurity Framework to all systems of storage, processing and transport of biological data would help explore where, how, and by whom data is processed with the goal of protecting valuable scientific and health information.[69] The NIST framework involves a collaboration of private sector and government cybersecurity experts that seek to apply the five principles of data protection (i.e., identify, protect, detect, respond, and recover) to systems, including those on which biological data are generated, processed and transported. The framework could augment existing or newly-implemented efforts of vulnerability detection and mitigation, thus decreasing unauthorized exposure of sensitive data. The NIST framework is a widely accepted paradigm for cyber risk management and best practices.[70][71][72] In the U.S., this framework has been used in regulatory dialogues to demonstrate rigor toward cybersecurity in sectors for which such requirements are not well-documented in law.\nThe takeaway: Application of the NIST framework to biotechnology can enhance data protection and a focus on rapid detection of nefarious activity and resiliency after an attack.\n\nAcknowledgements \nAuthor contributions \nKB and PS contributed equally to this manuscript. The concepts, conclusions, and recommendations were generated jointly by the authors and built on their respective expertise in the biological sciences and biosecurity, and computer science and cybersecurity.\n\nDisclaimer \nThe views and conclusions contained herein are those of the authors and should not be interpreted as representing the views and conclusions or official policies and endorsements, either expressed or implied, of Griffin Scientific, Promontory Financial Group, or the U.S. Government.\n\nConflict of interest \nKB was employed by Gryphon Scientific. PS was employed by Promontory Financial Group, which is an IBM Company.\nThe authors declare that the paper was written in the absence of any commercial or financial relationships that would constitute a conflict of interest.\n\nReferences \n\n\n\u2191 Accenture (2015). \"The Future of Applications in Life Sciences\" (PDF). Accenture. https:\/\/www.accenture.com\/_acnmedia\/Accenture\/Conversion-Assets\/DotCom\/Documents\/Global\/PDF\/Dualpub_20\/Accenture-15-1429U-FutureOfApps-LSCS-v5-web.pdf .   \n\n\u2191 2.0 2.1 Bajema, N.E.; DiEuliis, D.; Lutes, C.; Lim, Y.-B. (2018). \"The digitization of biology: Understanding the new risks and implications for governance\". Emergence & Convergence: 3. https:\/\/wmdcenter.ndu.edu\/Media\/News\/Article\/1569559\/the-digitization-of-biology-understanding-the-new-risks-and-implications-for-go\/ .   \n\n\u2191 Olena, A. (01 June 2018). \"Bringing the Internet of Things into the Lab\". The Scientist. https:\/\/www.the-scientist.com\/bio-business\/bringing-the-internet-of-things-into-the-lab-64265 .   \n\n\u2191 Chapman, T. (2003). \"Lab automation and robotics: Automation on the move\". Nature 421 (6923): 661, 663, 665\u20136. doi:10.1038\/421661a. PMID 12571603.   \n\n\u2191 Agapakis, C.M. (2014). \"Designing synthetic biology\". ACS Synthetic Biology 3 (3): 121\u20138. doi:10.1021\/sb4001068. PMID 24156739.   \n\n\u2191 Carbonell, P.; Jervis, A.J.; Robinson, C.J. et al. (2018). \"An automated Design-Build-Test-Learn pipeline for enhanced microbial production of fine chemicals\". Communications Biology 1: 66. doi:10.1038\/s42003-018-0076-9. PMC PMC6123781. PMID 30271948. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6123781 .   \n\n\u2191 Thurow, K.; G\u00f6de, B.; Dingerdissen, U. Stoll, N. (2004). \"Laboratory Information Management Systems for Life Science Applications\". Organic Process Researh & Development 8 (6): 970\u2013982. doi:10.1021\/op040017s.   \n\n\u2191 Walpole, J.; Papin, J.A.; Peirce, S.M. (2013). \"Multiscale computational models of complex biological systems\". Annual Review of Biomedical Engineering 15: 137\u201354. doi:10.1146\/annurev-bioeng-071811-150104. PMC PMC3970111. PMID 23642247. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3970111 .   \n\n\u2191 Service, R.F. (02 March 2017). \"DNA could store all of the world's data in one room\". Science. doi:10.1126\/science.aal0852. https:\/\/www.sciencemag.org\/news\/2017\/03\/dna-could-store-all-worlds-data-one-room .   \n\n\u2191 Anderson, L.A.; Islam, M.A.; Prather, K.L.J. (2018). \"Synthetic biology strategies for improving microbial synthesis of \"green\" biopolymers\". Journal of Biological Chemistry 293 (14): 5053-5061. doi:10.1074\/jbc.TM117.000368. PMC PMC5892568. PMID 29339554. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5892568 .   \n\n\u2191 Patel, P. (20 February 2018). \"DNA Data Storage Gets Random Access\". IEEE Spectrum. https:\/\/spectrum.ieee.org\/the-human-os\/biomedical\/devices\/dna-data-storage-gets-random-access .   \n\n\u2191 von Krogh, G.; Battistini, B.; Pachidou, F.; Baschera, P. (2012). \"The changing face of corporate venturing in biotechnology\". Nature Biotechnology 30 (10): 911\u20135. doi:10.1038\/nbt.2383. PMID 23051802.   \n\n\u2191 Cha, A.E. (18 January 2015). \"Crowdfunding propels scientific research\". The Washington Post. https:\/\/www.washingtonpost.com\/national\/health-science\/crowdfunding-propels-scientific-research\/2015\/01\/18\/c1937690-9758-11e4-8005-1924ede3e54a_story.html?utm_term=.734eb498edb5 .   \n\n\u2191 Mervis, J. (09 March 2017). \"Data check: U.S. government share of basic research funding falls below 50%\". Science. doi:10.1126\/science.aal0890. https:\/\/www.sciencemag.org\/news\/2017\/03\/data-check-us-government-share-basic-research-funding-falls-below-50 .   \n\n\u2191 U.S. Government (March 2012). \"United States Government Policy for Oversight of Life Sciences Dual Use Research of Concern\" (PDF). http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/us-policy-durc-032812.pdf .   \n\n\u2191 U.S. Government (September 2014). \"United States Government Policy for Institutional Oversight of Life Sciences Dual Use Research of Concern\" (PDF). http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/durc-policy.pdf .   \n\n\u2191 National Research Council (2004). Biotechnology Research in an Age of Terrorism. National Academies Press. doi:10.17226\/10827. ISBN 9780309166874. https:\/\/www.nap.edu\/catalog\/10827\/biotechnology-research-in-an-age-of-terrorism .   \n\n\u2191 Lord, R.; Forbes Technology Council (15 December 2017). \"The Real Threat Of Identity Theft Is In Your Medical Records, Not Credit Cards\". Forbes. https:\/\/www.forbes.com\/sites\/forbestechcouncil\/2017\/12\/15\/the-real-threat-of-identity-theft-is-in-your-medical-records-not-credit-cards\/#445711491b59 .   \n\n\u2191 Souza, C. (10 December 2018). \"Lessons for Pharma from the Merck Cyber Attack\". PharmExec.com 38 (12). http:\/\/www.pharmexec.com\/lessons-pharma-merck-cyber-attack . Retrieved 21 January 2019 .   \n\n\u2191 Ward, A. (11 December 2018). \"SIS's Use of Social Media Still Poses a Threat to Stability in the Middle East and Africa\". The RAND Blog. https:\/\/www.rand.org\/blog\/2018\/12\/isiss-use-of-social-media-still-poses-a-threat-to-stability.html . Retrieved 21 January 2019 .   \n\n\u2191 Friedman, A.A. (25 September 2013). \"Cyber Theft of Competitive Data: Asking the Right Questions\". Brookings. The Brookings Institution. https:\/\/www.brookings.edu\/research\/cyber-theft-of-competitive-data-asking-the-right-questions\/ .   \n\n\u2191 Bogle, A. (07 June 2018). \"Healthcare data a growing target for hackers, cybersecurity experts warn\". ABC.net.au. https:\/\/www.abc.net.au\/news\/science\/2018-04-18\/healthcare-target-for-hackers-experts-warn\/9663304 . Retrieved 23 November 2018 .   \n\n\u2191 Cohen, J. (23 March 2018). \"Massive cyberhack by Iran allegedly stole research from 320 universities, governments, and companies\". Science. doi:10.1126\/science.aat6849. https:\/\/www.sciencemag.org\/news\/2018\/03\/massive-cyber-hack-iran-allegedly-stole-research-320-universities-governments-and .   \n\n\u2191 Healthcare IT News Staff (2018). \"The biggest healthcare data breaches of 2018 (so far)\". Healthcare IT News. https:\/\/www.healthcareitnews.com\/projects\/biggest-healthcare-data-breaches-2018-so-far . Retrieved 23 November 2018 .   \n\n\u2191 Huang, E.; Steger, I. (29 October 2018). \"China Is Secretly Enrolling Military Scientists in Western Universities\". Defense One. https:\/\/www.defenseone.com\/threats\/2018\/10\/china-secretly-enrolling-military-scientists-western-universities\/152383\/ . Retrieved 23 November 2018 .   \n\n\u2191 Keown, A. (18 September 2018). \"Second Scientist Pleads Guilty to Stealing GlaxoSmithKline Trade Secrets\". BioSpace. https:\/\/www.biospace.com\/article\/-jc1n-second-scientist-pleads-guilty-to-stealing-glaxosmithkline-trade-secrets\/ . Retrieved 23 November 2018 .   \n\n\u2191 Lynch, D.J. (2017). \"Biotechnology: the US-China Dispute over Genentic Data\". Financial Times. https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe . Retrieved 23 November 2018 .   \n\n\u2191 28.0 28.1 Rappeport, A. (10 October 2018). \"In New Slap at China, U.S. Expands Power to Block Foreign Investments\". The New York Times. https:\/\/www.nytimes.com\/2018\/10\/10\/business\/us-china-investment-cfius.html . Retrieved 23 November 2018 .   \n\n\u2191 Bloomberg News (19 April 2018). \"Chinese funds pour US$1.4b into US biotechnology firms in the first three months of the year\". South China Morning Post. https:\/\/www.scmp.com\/business\/global-economy\/article\/2142351\/chinese-funds-pour-us14b-us-biotechnology-firms-first-three . Retrieved 23 November 2018 .   \n\n\u2191 Respaut, R.; Zhu, J. (23 September 2018). \"As China builds biotech sector, cash floods U.S. startups\". Reuters. https:\/\/www.reuters.com\/article\/us-biotech-china-investment\/as-china-builds-biotech-sector-cash-floods-u-s-startups-idUSKCN1M400G . Retrieved 23 November 2018 .   \n\n\u2191 Bai, G.; Jiang, J.X.; Flasher, R. (2017). \"Hospital risk of data breaches\". JAMA Internal Medicine 1777 (6): 878-880. doi:10.1001\/jamainternmed.2017.0336. PMC PMC5818824. PMID 28384777. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5818824 .   \n\n\u2191 Michigan State University (19 November 2018). \"Healthcare providers -- not hackers -- leak more of your data\". EurekAlert!. https:\/\/eurekalert.org\/pub_releases\/2018-11\/msu-hp-111618.php . Retrieved 23 November 2019 .   \n\n\u2191 BioSpace (21 October 2015). \"WuXi Healthcare Invests In US Genomics Testmaker 23andMe\". BioSpace. https:\/\/www.biospace.com\/article\/releases\/-b-wuxi-healthcare-b-invests-in-us-genomics-testmaker-23andme-\/ .   \n\n\u2191 Mui, Y.Q. (30 December 2016). \"China\u2019s $9 billion effort to beat the U.S. in genetic testing\". The Washington Post. https:\/\/www.washingtonpost.com\/news\/wonk\/wp\/2016\/12\/30\/chinas-9-billion-effort-to-beat-the-u-s-in-genetic-testing\/?noredirect=on&utm_term=.8586cdbf28b8 .   \n\n\u2191 Baker, M. (2012). \"China buys U.S. sequencing firm\". Nature 489 (7417): 485\u20136. doi:10.1038\/489485a. PMID 23018943.   \n\n\u2191 Genome Web Staff Reporter (17 September 2012). \"Complete Genomics, BGI Agree to $117.6M Merger\". Genome Web. https:\/\/www.genomeweb.com\/clinical-sequencing\/complete-genomics-bgi-agree-1176m-merger#.XEqIOFxKiUl . Retrieved 24 January 2019 .   \n\n\u2191 Bontrager, P.; Roy, A.; Togelius, J. et al. (18 October 2018). \"DeepMasterPrints: Generating MasterPrints for Dictionary Attacks via Latent Variable Evolution\". arXiv.org. https:\/\/arxiv.org\/abs\/1705.07386 .   \n\n\u2191 NYU Tandon School of Engineering (20 November 2018). \"Machine Learning Masters the Fingerprint to Fool Biometric Systems\". PR Newswire. https:\/\/www.prnewswire.com\/news-releases\/machine-learning-masters-the-fingerprint-to-fool-biometric-systems-300753375.html .   \n\n\u2191 U.S. Congress (2018). \"S. 2098 (115th): Foreign Investment Risk Review Modernization Act of 2018\". govtrack. https:\/\/www.govtrack.us\/congress\/bills\/115\/s2098 .   \n\n\u2191 CDC, USDA (2017). \"Information Systems Security Control Guidance\". Federal Select Agent Program. https:\/\/www.selectagents.gov\/isg-intro.html .   \n\n\u2191 Big Data and Privacy Working Group (February 2015). \"Big Data: Seizing Opportunities, Preserving Values\" (PDF). U.S. Government. https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/20150204_Big_Data_Seizing_Opportunities_Preserving_Values_Memo.pdf .   \n\n\u2191 European Commission (2018). \"2018 reform of EU data protection rules\". https:\/\/ec.europa.eu\/commission\/priorities\/justice-and-fundamental-rights\/data-protection\/2018-reform-eu-data-protection-rules_en .   \n\n\u2191 Sacks, S. (09 March 2018). \"China\u2019s Emerging Data Privacy System and GDPR\". Center for Strategic & International Studies. https:\/\/www.csis.org\/analysis\/chinas-emerging-data-privacy-system-and-gdpr .   \n\n\u2191 Soares, E. (28 August 2018). \"Brazil: Personal Data Protection Law Enacted\". Global Legal Monitor. https:\/\/www.loc.gov\/law\/foreign-news\/article\/brazil-personal-data-protection-law-enacted\/ .   \n\n\u2191 U.S. Department of Health and Human Services (26 July 2013). \"Summary of the HIPAA Security Rule\". https:\/\/www.hhs.gov\/hipaa\/for-professionals\/security\/laws-regulations\/index.html .   \n\n\u2191 Van Noorden, R. (22 February 2013). \"White House announces new US open-access policy\". Nature NewsBlog. http:\/\/blogs.nature.com\/news\/2013\/02\/us-white-house-announces-open-access-policy.html . Retrieved 23 November 2018 .   \n\n\u2191 United Nations Environment Programme. \"About the Nagoya Protocol\". Convention on Biological Diversity. https:\/\/www.cbd.int\/abs\/about\/ .   \n\n\u2191 dos S. Ribeiro, C.; Koopmans, M.P.; Haringhuizen, G.B. (26 October 2018). \"Threats to timely sharing of pathogen sequence data\". Science. doi:10.1126\/science.aau5229. https:\/\/science.sciencemag.org\/content\/362\/6413\/404 .   \n\n\u2191 NCBI (2013). \"GenBank Overview\". https:\/\/www.ncbi.nlm.nih.gov\/genbank\/ .   \n\n\u2191 The Commission on the Theft of American Intellectual Property (May 2013). \"The IP Commission Report\" (PDF). National Bureau of Asian Research. http:\/\/www.ipcommission.org\/report\/IP_Commission_Report_052213.pdf .   \n\n\u2191 Press, G. (03 December 2018). \"60 Cybersecurity Predictions for 2019\". Forbes. https:\/\/www.forbes.com\/sites\/gilpress\/2018\/12\/03\/60-cybersecurity-predictions-for-2019\/#759f17fc4352 .   \n\n\u2191 Denning, P.J.; Lewis, T.G. (2017). \"Exponential laws of computing growth\". Communications of the ACM 60 (1): 54\u201365. doi:10.1145\/2976758.   \n\n\u2191 Dauch, K.; Nestler, R. (2009). \"Information Assurance Using a Defense In-Depth Strategy\". 2009 Cybersecurity Applications & Technology Conference for Homeland Security: 267\u201372. doi:10.1109\/CATCH.2009.25.   \n\n\u2191 inap_admin (20 June 2013). \"Data in motion vs. data at rest\". ThinkIT blog. INAP. https:\/\/www.inap.com\/blog\/data-in-motion-vs-data-at-rest\/ . Retrieved 24 January 2019 .   \n\n\u2191 McDonald, K. (10 April 2017). \"Private sector's national cybersecurity strategy contributions lacking\". TechTarget SearchCompliance. https:\/\/searchcompliance.techtarget.com\/opinion\/Private-sectors-national-cybersecurity-strategy-contributions-lacking . Retrieved 24 January 2019 .   \n\n\u2191 Tuzman, K.T. (11 October 2018). \"Border security for China\u2019s genomes\". Biocentury. https:\/\/www.biocentury.com\/bc-innovations\/strategy\/2018-10-11\/balancing-protection-and-translation-china%E2%80%99s-genomic-data-troves .   \n\n\u2191 Capaccio, A. (21 June 2018). \"U.S. Faces \u2018Unprecedented Threat\u2019 From China on Tech Takeover\". Bloomberg. https:\/\/www.bloomberg.com\/news\/articles\/2018-06-22\/china-s-thousand-talents-called-key-in-seizing-u-s-expertise . Retrieved 23 November 2018 .   \n\n\u2191 Jia, H. (17 January 2018). \"China\u2019s plan to recruit talented researchers\". Naturejobs Career Guide: China. doi:10.1038\/d41586-018-00538-z. https:\/\/www.nature.com\/articles\/d41586-018-00538-z .   \n\n\u2191 Riley, M.; Walcott, J. (05 June 2015). \"China\u2019s Hack of U.S. Data Tied to Health-Care Record Thefts\". Bloomberg. https:\/\/www.bloomberg.com\/news\/articles\/2015-06-05\/u-s-government-data-breach-tied-to-theft-of-health-care-records .   \n\n\u2191 Dilanian, K. (09 October 2018). \"China's hackers are stealing secrets from U.S. firms again, experts say\". NBC News. https:\/\/www.nbcnews.com\/news\/china\/china-s-hackers-are-stealing-secrets-u-s-firms-again-n917836 . Retrieved 29 January 2019 .   \n\n\u2191 Kaiser, J.; Malakoff, D. (27 August 2018). \"NIH investigating whether U.S. scientists are sharing ideas with foreign governments\". Science. doi:10.1126\/science.aav2343. https:\/\/www.sciencemag.org\/news\/2018\/08\/nih-investigating-whether-us-scientists-are-sharing-ideas-foreign-governments .   \n\n\u2191 Wilber, D.Q.. \"Chinese hackers charged with stealing data from Navy, JPL and U.S. companies\". Los Angeles Times. https:\/\/www.latimes.com\/politics\/la-na-pol-chinese-espionage-indictment-20181220-story.html .   \n\n\u2191 You, E.H. (10 March 2017). \"Safeguarding the Bioeconomy: U.S. Opportunities and Challenges\" (PDF). https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf .   \n\n\u2191 Human Rights Watch (13 December 2017). \"China: Minority Region Collects Data from Millions\". Human Rights Watch News. https:\/\/www.hrw.org\/news\/2017\/12\/13\/china-minority-region-collects-dna-millions# .   \n\n\u2191 Lynch, D.J. (2017). \"Biotechnology: the US-China Dispute over Genentic Data\". Financial Times. https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe . Retrieved 23 November 2018 .   \n\n\u2191 Pauwels, E.; Vidyarthi, A. (19 November 2017). \"Who Will Own The Secrets In Our Genes? A U.S. \u2013 China Race in Artificial Intelligence and Genomics\". Science and Technology Innovation Program. Wilson Center. https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics .   \n\n\u2191 Big Data and Privacy Working Group (May 2014). \"Big Data: Seizing Opportunities, Preserving Values\" (PDF). U.S. Government. https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/big_data_privacy_report_may_1_2014.pdf .   \n\n\u2191 President's Council of Advisors on Science and Technology (May 2014). \"Big Data and Privacy: A Technological Perspective\" (PDF). U.S. Government. https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/microsites\/ostp\/PCAST\/pcast_big_data_and_privacy_-_may_2014.pdf .   \n\n\u2191 National Institute of Standards and Technology (2018). \"Cybersecurity Framework\". https:\/\/www.nist.gov\/cyberframework .   \n\n\u2191 Department of Homeland Security (22 August 2018). \"Using the Cybersecurity Framework\". https:\/\/www.dhs.gov\/using-cybersecurity-framework .   \n\n\u2191 Lohrmann, D. (20 May 2018). \"Why You Need the Cybersecurity Framework\". Government Technology. https:\/\/www.govtech.com\/blogs\/lohrmann-on-cybersecurity\/why-you-need-the-cybersecurity-framework.html .   \n\n\u2191 Roncevich, T. (14 June 2018). \"Healthcare IT Security Best Practices: Adopting NIST's Cybersecurity Framework\". Cyberguard Compliance Blog. https:\/\/info.cgcompliance.com\/blog\/healthcare-it-security-best-practices-adopting-nists-cybersecurity-framework . Retrieved 24 January 2019 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The two footnotes in the original material were turned into inline references for this version.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\">https:\/\/www.limswiki.org\/index.php\/Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on cybersecurityLIMSwiki journal articles on data management and sharing\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 18 June 2019, at 18:25.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 135 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","74131eedf92ecea39da4d1e4c07690b3_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:National and transnational security implications of asymmetric access to and use of biological data<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Biology and <a href=\"https:\/\/www.limswiki.org\/index.php\/Biotechnology\" title=\"Biotechnology\" class=\"wiki-link\" data-key=\"115005039d4cf0b4ef55ec14dc6d66da\">biotechnology<\/a> have changed dramatically during the past 20 years, in part because of increases in computational capabilities and use of engineering principles to study biology. The advances in supercomputing, data storage capacity, and <a href=\"https:\/\/www.limswiki.org\/index.php\/Cloud_computing\" title=\"Cloud computing\" class=\"wiki-link\" data-key=\"fcfe5882eaa018d920cedb88398b604f\">cloud platforms<\/a> enable scientists throughout the world to generate, analyze, share, and store vast amounts of data, some of which are biological and much of which may be used to understand the human condition, agricultural systems, evolution, and environmental ecosystems. These advances and applications have enabled: (1) the emergence of data science, which involves the development of new algorithms to analyze and <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_visualization\" title=\"Data visualization\" class=\"wiki-link\" data-key=\"4a3b86cba74bc7bb7471aa3fc2fcccc3\">visualize data<\/a>; and (2) the use of engineering approaches to manipulate or create new biological organisms that have specific functions, such as production of industrial chemical precursors and development of environmental bio-based sensors. Several biological sciences fields harness the capabilities of computer, data, and engineering sciences, including synthetic biology, precision medicine, precision agriculture, and systems biology. These advances and applications are not limited to one country. This capability has economic and physical consequences but is vulnerable to unauthorized intervention. Healthcare and <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomic<\/a> information of patients, <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> about pharmaceutical and biotechnology products in development, and results of scientific <a href=\"https:\/\/www.limswiki.org\/index.php\/Research\" title=\"Research\" class=\"wiki-link\" data-key=\"409634fd90113f119362927fe222f549\">research<\/a> have been stolen by state and non-state actors through infiltration of databases and computer systems containing this information. Countries have developed their own policies for governing data generation, access, and sharing with foreign entities, resulting in asymmetry of data sharing. This paper describes security implications of asymmetric access to and use of biological data.\n<\/p><p><b>Keywords<\/b>: biotechnology, cybersecurity, information security, data vulnerability, biological data, biosecurity, data access, data protection\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Advances in computer science, engineering, and data science have changed research, development, and application of biology and biotechnology in the United States and internationally. Examples of changes include: (a) increased reliance on internet connectivity for research and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> operations<sup id=\"rdp-ebb-cite_ref-1\" class=\"reference\"><a href=\"#cite_note-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BajemaTheDig18_2-0\" class=\"reference\"><a href=\"#cite_note-BajemaTheDig18-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OlenaBringing18_3-0\" class=\"reference\"><a href=\"#cite_note-OlenaBringing18-3\">[3]<\/a><\/sup>; (b) increased use of automation in life-science laboratories<sup id=\"rdp-ebb-cite_ref-ChapmanLab03_4-0\" class=\"reference\"><a href=\"#cite_note-ChapmanLab03-4\">[4]<\/a><\/sup>; (c) application of the \u201cdesign-build-test\u201d paradigm to create new biological organisms<sup id=\"rdp-ebb-cite_ref-AgapakisDesign14_5-0\" class=\"reference\"><a href=\"#cite_note-AgapakisDesign14-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CarbonellAnAuto18_6-0\" class=\"reference\"><a href=\"#cite_note-CarbonellAnAuto18-6\">[6]<\/a><\/sup>; (d) increased generation, analyses, and computational modeling of information about biological systems, cells, and molecules<sup id=\"rdp-ebb-cite_ref-ThurowLab04_7-0\" class=\"reference\"><a href=\"#cite_note-ThurowLab04-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WalpoleMulti13_8-0\" class=\"reference\"><a href=\"#cite_note-WalpoleMulti13-8\">[8]<\/a><\/sup>; (e) treatment of organisms and DNA as materials rather than phenomena to study<sup id=\"rdp-ebb-cite_ref-ServiceDNA17_9-0\" class=\"reference\"><a href=\"#cite_note-ServiceDNA17-9\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AndersonSynth18_10-0\" class=\"reference\"><a href=\"#cite_note-AndersonSynth18-10\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PatelDNA18_11-0\" class=\"reference\"><a href=\"#cite_note-PatelDNA18-11\">[11]<\/a><\/sup>; and (f) new funders such as venture capital, crowdfunding platforms, and foreign companies and governments.<sup id=\"rdp-ebb-cite_ref-vonKroghTheChang12_12-0\" class=\"reference\"><a href=\"#cite_note-vonKroghTheChang12-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChaCrowd15_13-0\" class=\"reference\"><a href=\"#cite_note-ChaCrowd15-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MervisData17_14-0\" class=\"reference\"><a href=\"#cite_note-MervisData17-14\">[14]<\/a><\/sup> These changes have transformed the scientific, agricultural, and health communities' ability to understand and manipulate the world around them. In addition, the changes have enabled an influx of new practitioners and problem-solvers into biology, providing opportunities for education and research all over the world.\n<\/p><p>Biotechnology harnesses the capabilities of computer, data, and engineering sciences to establish and advance new fields such as synthetic biology, precision medicine, precision agriculture, and systems biology. Cloud-based platforms and open-source, easy-to-use software enable scientists from anywhere in the world to use advanced data analytics in their studies. The software and hardware emerging from these fields improve our collective understanding of molecular and systems-level genetics, new drug therapies for longer and better quality of life, and design of novel and\/or unnatural organisms. Critical to these pursuits is the sharing of research results and underlying data, without which societal decision-making about human, animal, plant, and environmental health cannot be realized fully. However, during the past two decades, concerns about data sharing have been raised, resulting in the issuance of international, regional, and national-level policies governing access to different types of data, including biological data. In addition, the platforms through which data are stored, transported, and analyzed may be vulnerable to unauthorized acquisition of information by malicious actors, which could lead to significant economic and physical harms to the health, safety, and security of a population. Although not considered \u201cdual use life sciences research of concern,\u201d<sup id=\"rdp-ebb-cite_ref-USGovLifeSciences12_15-0\" class=\"reference\"><a href=\"#cite_note-USGovLifeSciences12-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-USGovPolicy14_16-0\" class=\"reference\"><a href=\"#cite_note-USGovPolicy14-16\">[16]<\/a><\/sup> the potential for both benefit and risk to humanity meets the spirit of the dual use concept.<sup id=\"rdp-ebb-cite_ref-NRCBio04_17-0\" class=\"reference\"><a href=\"#cite_note-NRCBio04-17\">[17]<\/a><\/sup> Given the significant benefits afforded by data sharing and analysis, this paper highlights current data protection policies, potential risks of data exploitation by malicious actors, and potential strategies to mitigate those risks and promote rapid recovery in biotechnology fields that are breached.\n<\/p><p>The interconnectedness between the digital and biological worlds can be exploited by state actors, malicious nonstate actors, and hackers through a variety of means, resulting in harmful consequences from potential theft of information, promulgation of incorrect information, and\/or disruption of activities.<sup id=\"rdp-ebb-cite_ref-LordTheReal17_18-0\" class=\"reference\"><a href=\"#cite_note-LordTheReal17-18\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SouzaLessons18_19-0\" class=\"reference\"><a href=\"#cite_note-SouzaLessons18-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WardISIS18_20-0\" class=\"reference\"><a href=\"#cite_note-WardISIS18-20\">[20]<\/a><\/sup> For example, theft of proprietary information from a pharmaceutical or biotechnology company may reveal trade secrets and allow competitors to develop superior products and\/or bring existing products to market more quickly<sup id=\"rdp-ebb-cite_ref-FriedmanCyber13_21-0\" class=\"reference\"><a href=\"#cite_note-FriedmanCyber13-21\">[21]<\/a><\/sup>, stifling innovation in the global commercial market and allowing adversaries to create harmful, untested therapies. Another example is theft of hundreds of millions of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic healthcare records<\/a>, the uses of which are not clear.<sup id=\"rdp-ebb-cite_ref-BogleHeath18_22-0\" class=\"reference\"><a href=\"#cite_note-BogleHeath18-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CohenMassive18_23-0\" class=\"reference\"><a href=\"#cite_note-CohenMassive18-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HITTheBig18_24-0\" class=\"reference\"><a href=\"#cite_note-HITTheBig18-24\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HuangChina18_25-0\" class=\"reference\"><a href=\"#cite_note-HuangChina18-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KeownSecond18_26-0\" class=\"reference\"><a href=\"#cite_note-KeownSecond18-26\">[26]<\/a><\/sup> Although unauthorized access to protected data may be aided by technical vulnerabilities in networked computer systems, poor security practices, insider threats in academia, industry, and health facilities, and legal business dealings also can enable adversary access to such data.<sup id=\"rdp-ebb-cite_ref-LynchBio17_27-0\" class=\"reference\"><a href=\"#cite_note-LynchBio17-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RappeportInNew18_28-0\" class=\"reference\"><a href=\"#cite_note-RappeportInNew18-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BloombergChinese18_29-0\" class=\"reference\"><a href=\"#cite_note-BloombergChinese18-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RespautAsChina18_30-0\" class=\"reference\"><a href=\"#cite_note-RespautAsChina18-30\">[30]<\/a><\/sup> For examples, more than half of all data breaches at healthcare facilities are caused by healthcare personnel errors, a quarter of which resulted in unauthorized access to or disclosure of patient records through sharing of unencrypted information, sending information to the wrong patients, and accessing the data without authorization.<sup id=\"rdp-ebb-cite_ref-BaiHospital17_31-0\" class=\"reference\"><a href=\"#cite_note-BaiHospital17-31\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MSUHealthcare18_32-0\" class=\"reference\"><a href=\"#cite_note-MSUHealthcare18-32\">[32]<\/a><\/sup> In addition, the Federal Bureau of Investigation (FBI) has raised national security concerns about foreign access to genomic data of U.S. citizens through legitimate scientific collaboration, funding of scientific research, investment in genomic sequencing companies (e.g., China-based WuXi Healthcare Ventures investment in the U.S.-based 23andMe<sup id=\"rdp-ebb-cite_ref-BSWuXi15_33-0\" class=\"reference\"><a href=\"#cite_note-BSWuXi15-33\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MuiChina16_34-0\" class=\"reference\"><a href=\"#cite_note-MuiChina16-34\">[34]<\/a><\/sup>), and purchase of companies (e.g., Complete Genomics).<sup id=\"rdp-ebb-cite_ref-BakerChina12_35-0\" class=\"reference\"><a href=\"#cite_note-BakerChina12-35\">[35]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GWComplete12_36-0\" class=\"reference\"><a href=\"#cite_note-GWComplete12-36\">[36]<\/a><\/sup> As vulnerabilities are created through scientific advances, such as the use of machine learning algorithms to trick fingerprint authentication systems, new risks are identified.<sup id=\"rdp-ebb-cite_ref-BontragerDeep18_37-0\" class=\"reference\"><a href=\"#cite_note-BontragerDeep18-37\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NYUTandonMachine18_38-0\" class=\"reference\"><a href=\"#cite_note-NYUTandonMachine18-38\">[38]<\/a><\/sup> Some of these concerns have resulted in the passage of the 2018 Foreign Investment Risk Review Modernization Act, which has initiated reform of the U.S. Government process for evaluating foreign investment in U.S. entities and export control of emerging technologies.<sup id=\"rdp-ebb-cite_ref-RappeportInNew18_28-1\" class=\"reference\"><a href=\"#cite_note-RappeportInNew18-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-USCongressForeign18_39-0\" class=\"reference\"><a href=\"#cite_note-USCongressForeign18-39\">[39]<\/a><\/sup> Yet, these policy activities largely are reactive, rather than proactive.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Current_approaches_to_protecting_data\">Current approaches to protecting data<\/span><\/h2>\n<p>Preventing accidental and deliberate risks typically involves the use of cyber and information security systems that include technological and behavioral solutions. Protection of laboratory control systems, computer networks, and databases often involves the use of technological solutions. However, some risks are addressed better through training of personnel to recognize and report phishing attempts, ensure sensitive information is <a href=\"https:\/\/www.limswiki.org\/index.php\/Encryption\" title=\"Encryption\" class=\"wiki-link\" data-key=\"86a503652ed5cc9d8e2b0252a480b5e1\">encrypted<\/a>, and prevent unauthorized individuals from gaining access to sensitive data, databases, and computer networks. To enhance security, policies for promulgating these practices for specific materials and information have been issued. For example, the U.S. Biological Select Agents and Toxins Regulations include guidance for network security to prevent failure of laboratories, equipment, and access controls to facilities and data.<sup id=\"rdp-ebb-cite_ref-CDCInformation17_40-0\" class=\"reference\"><a href=\"#cite_note-CDCInformation17-40\">[40]<\/a><\/sup> In addition, the U.S. has policies for protecting individual privacy, several of which were described in a 2014 report sponsored by the White House.<sup id=\"rdp-ebb-cite_ref-BDPWGBigData15_41-0\" class=\"reference\"><a href=\"#cite_note-BDPWGBigData15-41\">[41]<\/a><\/sup> However, error, carelessness, or negligence by personnel can counteract the benefits afforded by security measures and may lead to devastating consequences if biological data and materials are involved.\n<\/p><p>Although policies for protecting biological data from cyberattack are limited, policies that govern data access and sharing are prevalent. These top-down, data access policies intend to protect individual rights and\/or prevent sharing or distribution of data, including biological data. Examples of recent policies include: (a) the 2018 update of the European Union General Data Protection Regulation<sup id=\"rdp-ebb-cite_ref-EC2018Reform18_42-0\" class=\"reference\"><a href=\"#cite_note-EC2018Reform18-42\">[42]<\/a><\/sup>, which strengthened the European Union's rules for protecting personal data of individuals, in part by giving its citizens \u201cmore control over their personal data\u201d; (b) the 2018 Chinese Personal Information Security Specification, which is one system under the Chinese Cybersecurity law, involves the \u201ccollection, storage, use, sharing, transfer, and disclosure of personal information,\u201d and enables companies operating in China to access data to \u201cnot hamper the development of fields like AI\u201d<sup id=\"rdp-ebb-cite_ref-SacksChinas18_43-0\" class=\"reference\"><a href=\"#cite_note-SacksChinas18-43\">[43]<\/a><\/sup>; (c) the 2018 General Data Protection Law in Brazil, which provides a framework for the use of personal data in Brazil<sup id=\"rdp-ebb-cite_ref-SoaresBrazil18_44-0\" class=\"reference\"><a href=\"#cite_note-SoaresBrazil18-44\">[44]<\/a><\/sup>; and (d) the U.S. <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Insurance_Portability_and_Accountability_Act\" title=\"Health Insurance Portability and Accountability Act\" class=\"wiki-link\" data-key=\"b70673a0117c21576016cb7498867153\">Health Insurance Portability and Accountability Act<\/a> of 1996 (HIPAA), which promotes the protection of privacy and security of patient health information in the United States.<sup id=\"rdp-ebb-cite_ref-HHSSummary13_45-0\" class=\"reference\"><a href=\"#cite_note-HHSSummary13-45\">[45]<\/a><\/sup> At the same time, the U.S. has issued policies governing data generation, access, and sharing to promote information-sharing and transparency of government-sponsored research.<sup id=\"rdp-ebb-cite_ref-VanNoordenWhite13_46-0\" class=\"reference\"><a href=\"#cite_note-VanNoordenWhite13-46\">[46]<\/a><\/sup> Internationally, the Nagoya Protocol of the Convention on Biodiversity<sup id=\"rdp-ebb-cite_ref-CBDAbout_47-0\" class=\"reference\"><a href=\"#cite_note-CBDAbout-47\">[47]<\/a><\/sup> promotes governance on access to and fair, equitable sharing of the benefits from the use of non-human biological data. However, questions exist about whether the Nagoya Protocol focuses more on biological samples that provide genetic information or the genetic information itself, which ultimately affects national-level efforts for codifying the international agreement.<sup id=\"rdp-ebb-cite_ref-dosSRibeiroThreats18_48-0\" class=\"reference\"><a href=\"#cite_note-dosSRibeiroThreats18-48\">[48]<\/a><\/sup> Despite these activities, protection of some data, such as personal health data, may not extend beyond a country's borders and may apply only to data collected by certain entities. Furthermore, data protection polices do not extend to information that already has been stolen. Taken together, these national, regional, and international level policies for data protection may not prevent the inappropriate or unauthorized acquisition of data to different actors, the consequences of which are unclear for biotechnology data.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Vulnerability_of_biotechnology_data\">Vulnerability of biotechnology data<\/span><\/h2>\n<p>The primary challenges in identifying, assessing, and mitigating security vulnerabilities of biotechnology data are understanding: (a) how the data may be exploited by adversaries and what consequences result from this exploitation; and (2) what potential negative effects may arise from digitization of biotechnology and advanced computation of biological data.<sup id=\"rdp-ebb-cite_ref-BajemaTheDig18_2-1\" class=\"reference\"><a href=\"#cite_note-BajemaTheDig18-2\">[2]<\/a><\/sup> The term \u201cbiotechnology\u201d refers to the exploitation of biological processes for industrial and scientific purposes, and includes genetic manipulation of microbes, plants, animals, human cells, nucleic acids (the building blocks of genomes), and proteins (the functional units in cells). This definition is expanded further to include generation, incorporation, and use of digital forms of biological data. These biological data may be available online through databases, such as the U.S. National Center for Biotechnology Information's GenBank<sup id=\"rdp-ebb-cite_ref-NCBIGenBank13_49-0\" class=\"reference\"><a href=\"#cite_note-NCBIGenBank13-49\">[49]<\/a><\/sup>, or generated in a laboratory and stored, shared, and\/or analyzed locally or remotely (via online and\/or cloud-based software). By attempting to answer the questions posed above, specific risks associated with the legal and illegal acquisition of biological data may be identified and mitigated.\n<\/p><p>Although extraordinary advances in computing power are enabling unprecedented scientific discoveries, its application to biology and healthcare is increasing without effective protection from the risks of adversary acquisition or accidental misuse of information. Scientific data that is generated in basic and applied research laboratories in academia, non-profit research organizations, service providers, and some industry research facilities may be considered fundamental research destined for publication and public benefit. These data are not necessarily sensitive, but they do represent the results of significant investment by governments, industry, investors, and philanthropic organizations. Therefore, theft or large-scale acquisition of these data may have adverse economic consequences to the organization, field, or nation, especially if acquisition was directed by adversarial nation-states to gain competitive advantage in a given sector.<sup id=\"rdp-ebb-cite_ref-CTAIPTheIPComm13_50-0\" class=\"reference\"><a href=\"#cite_note-CTAIPTheIPComm13-50\">[50]<\/a><\/sup> As previously described, databases that store sensitive and\/or non-sensitive biological data have been infiltrated by external actors and accessed by unauthorized individuals. Although measures to protect data have been implemented in several institutions, cyber and information security policies, practices, and compliance vary across biotechnology sectors, location, and organization type (e.g., academia, industry). Although implementation of cyber, information, and data security in biological facilities can help to minimize the potential for deliberate or accidental release of protected biological data, these measures are insufficient on their own.<sup id=\"rdp-ebb-cite_ref-Press60_18_51-0\" class=\"reference\"><a href=\"#cite_note-Press60_18-51\">[51]<\/a><\/sup>\n<\/p><p>Furthermore, the increasing size and volume of the datasets, and the complexity of analytic technologies has led many scientists to rely on cloud-based platforms to store, transfer, and analyze data. These platforms and technologies, including online analysis software and applications, often do not prevent unauthorized access to data or ensure software fidelity. Although mitigating specific vulnerabilities may be possible on an individual platform or technology level, implementing protections across the various data generation, analysis, transfer, and storage platforms currently in use in academia, industry, government laboratories, and healthcare facilities is challenging. Countering these risks requires the identification of consequences that are of particular concern to public safety and national security, evaluation of vulnerabilities that may enable the realization of these consequences, and identification of measures to address these vulnerabilities.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Possible_prevention_and_mitigation_approaches\">Possible prevention and mitigation approaches<\/span><\/h2>\n<p>Modern cyber and information security reflects the risks experienced as the internet has grown and diversified, and as the capabilities for and speed of storing, processing, and transporting information have increased exponentially.<sup id=\"rdp-ebb-cite_ref-DenningExpon17_52-0\" class=\"reference\"><a href=\"#cite_note-DenningExpon17-52\">[52]<\/a><\/sup> The internet was built without a priority on the protection of data whether \u201cat rest\u201d (i.e., stored data) or \u201cin motion\u201d (i.e., data in transit).<sup id=\"rdp-ebb-cite_ref-DauchInform09_53-0\" class=\"reference\"><a href=\"#cite_note-DauchInform09-53\">[53]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-inap_adminData13_54-0\" class=\"reference\"><a href=\"#cite_note-inap_adminData13-54\">[54]<\/a><\/sup> Current strategies for addressing cyber risks focus on remediation through regulation, organizational support, and actions taken by data owners and consumers in the form of encryption technologies, access control measures, awareness-raising campaigns, risk assessment, blocking, limiting publication of sensitive information, and other similar practices. The challenge is understanding how these measures are to be applied to biotechnology data, how to balance the cost of implementation with the consequences if left unprotected, and what vulnerabilities cannot be mitigated using commercial products.\n<\/p><p>Often the entities that assess their cyber vulnerabilities and invest in cyber and information security measures are compelled to do so because of regulation and fiscal responsibility.<sup id=\"rdp-ebb-cite_ref-McDonaldPrivate17_55-0\" class=\"reference\"><a href=\"#cite_note-McDonaldPrivate17-55\">[55]<\/a><\/sup> However, unlike financial information, biotechnology data is regulated in some countries, but not others. For example, China issued a recent policy requiring a domestic collaborator and Ministry-level approval for research involving genomic data of Chinese citizens and\/or biological samples obtained in China to prevent exploitation of these data and samples.<sup id=\"rdp-ebb-cite_ref-TuzmanBorder18_56-0\" class=\"reference\"><a href=\"#cite_note-TuzmanBorder18-56\">[56]<\/a><\/sup> This and similar policies raise questions about their intended and unintended effects to nations, to the scientific community, and to international security mainly because the policies that may benefit one country could harm another. These harms may reveal new types of risks associated with the acquisition and use of data to manipulate biological systems. These risks may be perpetrated by different actors; affect sector and country economies, commercial biotechnology, and pharmaceutical markets domestically and internationally; and alter global strategic power dynamics.\n<\/p><p>The risks associated with biotechnology data do not conform to traditional biosecurity concerns, which focus primarily on risks to human health or the food and agriculture economy. These risks involve multiple domains, sectors, and nations resulting in outcomes such as shifting of balance of power of nations at the international level, which could have downstream effects on areas that overlap with biosecurity interests (e.g., biosafety and biosecurity, biothreat reduction, and global health security). Strategies for bridging the biological, cyber, information, and data security include: (a) collaboration between the biological and cybersecurity communities; (b) end-to-end risk assessments; (c) data-specific risk and vulnerability assessments; and (d) application of the NIST Cybersecurity Framework for protecting biological data. \n<\/p><p>These suggested strategies (detailed below) describe various approaches toward protecting biological data from unauthorized acquisition and use, enhancing efforts to preserve data integrity and provenance, and enabling future benefit of biotechnological advances.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Collaboration\">Collaboration<\/span><\/h3>\n<p>Formal collaboration between the biotechnology and biological, information, data, and cyber security communities would enhance efforts toward identification of risks and vulnerabilities associated with data management, provenance, and integrity, and risk mitigation strategies. Technologies are readily available to protect data, but their use must be harmonized worldwide, because protecting data in one database is ineffective if another database remains vulnerable to external threats. Furthermore, organizations may evade regulatory requirements and industry standards in protecting data because of perceived lack of cost savings for implementing cybersecurity measures or lack of awareness of the risks, which could lead to investor, intruder, or adversary access to sensitive information that may be stored in databases or transferred between computers. These vulnerabilities may be exacerbated by limitations of national laws to other sovereign states, and differences in interpretation of the types of data included in the scope of existing laws. \n<\/p><p><b>The takeaway<\/b>: Given these potential vulnerabilities, the cybersecurity and biotechnology communities must engage to create best practices and processes to protect data and mitigate risk while reaping the benefits of computing technology applications to biotechnology.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"End-to-end_risk_assessment\">End-to-end risk assessment<\/span><\/h3>\n<p>End-to-end assessments of the data storage, processing, and transport pipeline can identify outstanding vulnerabilities and technical gaps that may be addressed with currently available cyber, information, and data security solutions. This process would enable identification of gaps for which these measures are insufficient and of institutions that are responsible for implementing controls. Without this type of assessment, vulnerabilities may exist along the pipeline without its users' knowledge. A lack of rigorous analysis makes biological data vulnerable to acquisition or alteration by witting adversaries, potentially resulting in theft of intellectual property for commercial gain, foreign government acquisition of genomic data from large portions of a population for undefined purpose or compromise of software and data integrity. At least one country promotes acquisition of data though legitimate commercial practices (e.g., providing sequencing services to customers; partnering with academia, independent research institutions, and universities; and foreign investment), talent promotion programs<sup id=\"rdp-ebb-cite_ref-CapaccioUSFaces18_57-0\" class=\"reference\"><a href=\"#cite_note-CapaccioUSFaces18-57\">[57]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JiaChina18_58-0\" class=\"reference\"><a href=\"#cite_note-JiaChina18-58\">[58]<\/a><\/sup>, and theft of data.<sup id=\"rdp-ebb-cite_ref-RileyChina15_59-0\" class=\"reference\"><a href=\"#cite_note-RileyChina15-59\">[59]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DilanianChina18_60-0\" class=\"reference\"><a href=\"#cite_note-DilanianChina18-60\">[60]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KaiserNIH18_61-0\" class=\"reference\"><a href=\"#cite_note-KaiserNIH18-61\">[61]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WilberChinese18_62-0\" class=\"reference\"><a href=\"#cite_note-WilberChinese18-62\">[62]<\/a><\/sup> The FBI has expressed concerns about the theft of U.S. genomics and health information through cyberattacks and foreign investment in the U.S. biotechnology industry.<sup id=\"rdp-ebb-cite_ref-YouSafeguarding17_63-0\" class=\"reference\"><a href=\"#cite_note-YouSafeguarding17-63\">[63]<\/a><\/sup> The FBI argues that acquisition of this information can give adversaries an unfair advantage in the international pharmaceutical or biotechnology marketplace. Others have expressed concern about questionable use of genetic information that countries obtain from their own citizens or from other countries' citizens.<sup id=\"rdp-ebb-cite_ref-HRWChina17_64-0\" class=\"reference\"><a href=\"#cite_note-HRWChina17-64\">[64]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LynchBiotech17_65-0\" class=\"reference\"><a href=\"#cite_note-LynchBiotech17-65\">[65]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PauwelsWho17_66-0\" class=\"reference\"><a href=\"#cite_note-PauwelsWho17-66\">[66]<\/a><\/sup>\n<\/p><p><b>The takeway<\/b>: These risks could be addressed by conducting an end-to-end risk assessment of the software and equipment involved in the data pipeline within individual organizations, between organizations, and across countries.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data-specific_risk_mitigation\">Data-specific risk mitigation<\/span><\/h3>\n<p>Defining the consequences of greatest concern to national security is an initial step toward assessing the risks and vulnerabilities of the information itself and data-specific risk mitigation strategies. Evaluating these risks enables the identification of content-specific approaches for detecting and countering exploitation of vulnerabilities by insider and external actors. Without these assessments, only generic cyber and information security measures will be implemented. However, these measures are insufficient to counter adversaries who are intent on acquiring data through a variety of technical, social engineering, or other means. Given this reality, rapid detection and resilience (i.e., rapid recovery after a breach) are critical for reaping the benefits and minimizing the vulnerabilities of advanced electronic computation and mass connectivity. In 2014, the White House explored technology needs for protecting the security and privacy of exposed data, including healthcare data.<sup id=\"rdp-ebb-cite_ref-BDPWGBigData14_67-0\" class=\"reference\"><a href=\"#cite_note-BDPWGBigData14-67\">[67]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PCASTBigData14_68-0\" class=\"reference\"><a href=\"#cite_note-PCASTBigData14-68\">[68]<\/a><\/sup> But, these studies did not define consequences of concern related to the unauthorized acquisition of vast amounts of biological data, effectively limiting the identification of data-specific or process-specific prevention measures. \n<\/p><p><b>The takeaway<\/b>: Therefore, risk assessments of specific types of data are equally as important to conduct as analyses of vulnerabilities of laboratory control systems, data management platforms, and computer networks.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Application_of_the_NIST_Cybersecurity_Framework\">Application of the NIST Cybersecurity Framework<\/span><\/h3>\n<p>Application of the National Institute of Standards and Technology (NIST) Cybersecurity Framework to all systems of storage, processing and transport of biological data would help explore where, how, and by whom data is processed with the goal of protecting valuable scientific and health information.<sup id=\"rdp-ebb-cite_ref-NISTCyber_69-0\" class=\"reference\"><a href=\"#cite_note-NISTCyber-69\">[69]<\/a><\/sup> The NIST framework involves a collaboration of private sector and government cybersecurity experts that seek to apply the five principles of data protection (i.e., identify, protect, detect, respond, and recover) to systems, including those on which biological data are generated, processed and transported. The framework could augment existing or newly-implemented efforts of vulnerability detection and mitigation, thus decreasing unauthorized exposure of sensitive data. The NIST framework is a widely accepted paradigm for cyber risk management and best practices.<sup id=\"rdp-ebb-cite_ref-DHSUsing18_70-0\" class=\"reference\"><a href=\"#cite_note-DHSUsing18-70\">[70]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LohrmannWhy18_71-0\" class=\"reference\"><a href=\"#cite_note-LohrmannWhy18-71\">[71]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RoncevichHealth18_72-0\" class=\"reference\"><a href=\"#cite_note-RoncevichHealth18-72\">[72]<\/a><\/sup> In the U.S., this framework has been used in regulatory dialogues to demonstrate rigor toward cybersecurity in sectors for which such requirements are not well-documented in law.\n<\/p><p><b>The takeaway<\/b>: Application of the NIST framework to biotechnology can enhance data protection and a focus on rapid detection of nefarious activity and resiliency after an attack.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>KB and PS contributed equally to this manuscript. The concepts, conclusions, and recommendations were generated jointly by the authors and built on their respective expertise in the biological sciences and biosecurity, and computer science and cybersecurity.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Disclaimer\">Disclaimer<\/span><\/h3>\n<p>The views and conclusions contained herein are those of the authors and should not be interpreted as representing the views and conclusions or official policies and endorsements, either expressed or implied, of Griffin Scientific, Promontory Financial Group, or the U.S. Government.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflict_of_interest\">Conflict of interest<\/span><\/h3>\n<p>KB was employed by Gryphon Scientific. PS was employed by Promontory Financial Group, which is an IBM Company.\n<\/p><p>The authors declare that the paper was written in the absence of any commercial or financial relationships that would constitute a conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-1\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Accenture (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.accenture.com\/_acnmedia\/Accenture\/Conversion-Assets\/DotCom\/Documents\/Global\/PDF\/Dualpub_20\/Accenture-15-1429U-FutureOfApps-LSCS-v5-web.pdf\" target=\"_blank\">\"The Future of Applications in Life Sciences\"<\/a> (PDF). Accenture<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.accenture.com\/_acnmedia\/Accenture\/Conversion-Assets\/DotCom\/Documents\/Global\/PDF\/Dualpub_20\/Accenture-15-1429U-FutureOfApps-LSCS-v5-web.pdf\" target=\"_blank\">https:\/\/www.accenture.com\/_acnmedia\/Accenture\/Conversion-Assets\/DotCom\/Documents\/Global\/PDF\/Dualpub_20\/Accenture-15-1429U-FutureOfApps-LSCS-v5-web.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Future+of+Applications+in+Life+Sciences&rft.atitle=&rft.aulast=Accenture&rft.au=Accenture&rft.date=2015&rft.pub=Accenture&rft_id=https%3A%2F%2Fwww.accenture.com%2F_acnmedia%2FAccenture%2FConversion-Assets%2FDotCom%2FDocuments%2FGlobal%2FPDF%2FDualpub_20%2FAccenture-15-1429U-FutureOfApps-LSCS-v5-web.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BajemaTheDig18-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BajemaTheDig18_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-BajemaTheDig18_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bajema, N.E.; DiEuliis, D.; Lutes, C.; Lim, Y.-B. (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/wmdcenter.ndu.edu\/Media\/News\/Article\/1569559\/the-digitization-of-biology-understanding-the-new-risks-and-implications-for-go\/\" target=\"_blank\">\"The digitization of biology: Understanding the new risks and implications for governance\"<\/a>. <i>Emergence & Convergence<\/i>: 3<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/wmdcenter.ndu.edu\/Media\/News\/Article\/1569559\/the-digitization-of-biology-understanding-the-new-risks-and-implications-for-go\/\" target=\"_blank\">https:\/\/wmdcenter.ndu.edu\/Media\/News\/Article\/1569559\/the-digitization-of-biology-understanding-the-new-risks-and-implications-for-go\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+digitization+of+biology%3A+Understanding+the+new+risks+and+implications+for+governance&rft.jtitle=Emergence+%26+Convergence&rft.aulast=Bajema%2C+N.E.%3B+DiEuliis%2C+D.%3B+Lutes%2C+C.%3B+Lim%2C+Y.-B.&rft.au=Bajema%2C+N.E.%3B+DiEuliis%2C+D.%3B+Lutes%2C+C.%3B+Lim%2C+Y.-B.&rft.date=2018&rft.pages=3&rft_id=https%3A%2F%2Fwmdcenter.ndu.edu%2FMedia%2FNews%2FArticle%2F1569559%2Fthe-digitization-of-biology-understanding-the-new-risks-and-implications-for-go%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OlenaBringing18-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OlenaBringing18_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Olena, A. (01 June 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.the-scientist.com\/bio-business\/bringing-the-internet-of-things-into-the-lab-64265\" target=\"_blank\">\"Bringing the Internet of Things into the Lab\"<\/a>. <i>The Scientist<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.the-scientist.com\/bio-business\/bringing-the-internet-of-things-into-the-lab-64265\" target=\"_blank\">https:\/\/www.the-scientist.com\/bio-business\/bringing-the-internet-of-things-into-the-lab-64265<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bringing+the+Internet+of+Things+into+the+Lab&rft.atitle=The+Scientist&rft.aulast=Olena%2C+A.&rft.au=Olena%2C+A.&rft.date=01+June+2018&rft_id=https%3A%2F%2Fwww.the-scientist.com%2Fbio-business%2Fbringing-the-internet-of-things-into-the-lab-64265&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChapmanLab03-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChapmanLab03_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chapman, T. (2003). \"Lab automation and robotics: Automation on the move\". <i>Nature<\/i> <b>421<\/b> (6923): 661, 663, 665\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F421661a\" target=\"_blank\">10.1038\/421661a<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12571603\" target=\"_blank\">12571603<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lab+automation+and+robotics%3A+Automation+on+the+move&rft.jtitle=Nature&rft.aulast=Chapman%2C+T.&rft.au=Chapman%2C+T.&rft.date=2003&rft.volume=421&rft.issue=6923&rft.pages=661%2C+663%2C+665%E2%80%936&rft_id=info:doi\/10.1038%2F421661a&rft_id=info:pmid\/12571603&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AgapakisDesign14-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AgapakisDesign14_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Agapakis, C.M. (2014). \"Designing synthetic biology\". <i>ACS Synthetic Biology<\/i> <b>3<\/b> (3): 121\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fsb4001068\" target=\"_blank\">10.1021\/sb4001068<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24156739\" target=\"_blank\">24156739<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Designing+synthetic+biology&rft.jtitle=ACS+Synthetic+Biology&rft.aulast=Agapakis%2C+C.M.&rft.au=Agapakis%2C+C.M.&rft.date=2014&rft.volume=3&rft.issue=3&rft.pages=121%E2%80%938&rft_id=info:doi\/10.1021%2Fsb4001068&rft_id=info:pmid\/24156739&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarbonellAnAuto18-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CarbonellAnAuto18_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Carbonell, P.; Jervis, A.J.; Robinson, C.J. et al. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6123781\" target=\"_blank\">\"An automated Design-Build-Test-Learn pipeline for enhanced microbial production of fine chemicals\"<\/a>. <i>Communications Biology<\/i> <b>1<\/b>: 66. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fs42003-018-0076-9\" target=\"_blank\">10.1038\/s42003-018-0076-9<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC6123781\/\" target=\"_blank\">PMC6123781<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/30271948\" target=\"_blank\">30271948<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6123781\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC6123781<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+automated+Design-Build-Test-Learn+pipeline+for+enhanced+microbial+production+of+fine+chemicals&rft.jtitle=Communications+Biology&rft.aulast=Carbonell%2C+P.%3B+Jervis%2C+A.J.%3B+Robinson%2C+C.J.+et+al.&rft.au=Carbonell%2C+P.%3B+Jervis%2C+A.J.%3B+Robinson%2C+C.J.+et+al.&rft.date=2018&rft.volume=1&rft.pages=66&rft_id=info:doi\/10.1038%2Fs42003-018-0076-9&rft_id=info:pmc\/PMC6123781&rft_id=info:pmid\/30271948&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC6123781&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ThurowLab04-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ThurowLab04_7-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thurow, K.; G\u00f6de, B.; Dingerdissen, U. Stoll, N. (2004). \"Laboratory Information Management Systems for Life Science Applications\". <i>Organic Process Researh & Development<\/i> <b>8<\/b> (6): 970\u2013982. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fop040017s\" target=\"_blank\">10.1021\/op040017s<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Laboratory+Information+Management+Systems+for+Life+Science+Applications&rft.jtitle=Organic+Process+Researh+%26+Development&rft.aulast=Thurow%2C+K.%3B+G%C3%B6de%2C+B.%3B+Dingerdissen%2C+U.+Stoll%2C+N.&rft.au=Thurow%2C+K.%3B+G%C3%B6de%2C+B.%3B+Dingerdissen%2C+U.+Stoll%2C+N.&rft.date=2004&rft.volume=8&rft.issue=6&rft.pages=970%E2%80%93982&rft_id=info:doi\/10.1021%2Fop040017s&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WalpoleMulti13-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WalpoleMulti13_8-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Walpole, J.; Papin, J.A.; Peirce, S.M. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3970111\" target=\"_blank\">\"Multiscale computational models of complex biological systems\"<\/a>. <i>Annual Review of Biomedical Engineering<\/i> <b>15<\/b>: 137\u201354. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1146%2Fannurev-bioeng-071811-150104\" target=\"_blank\">10.1146\/annurev-bioeng-071811-150104<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3970111\/\" target=\"_blank\">PMC3970111<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23642247\" target=\"_blank\">23642247<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3970111\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3970111<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multiscale+computational+models+of+complex+biological+systems&rft.jtitle=Annual+Review+of+Biomedical+Engineering&rft.aulast=Walpole%2C+J.%3B+Papin%2C+J.A.%3B+Peirce%2C+S.M.&rft.au=Walpole%2C+J.%3B+Papin%2C+J.A.%3B+Peirce%2C+S.M.&rft.date=2013&rft.volume=15&rft.pages=137%E2%80%9354&rft_id=info:doi\/10.1146%2Fannurev-bioeng-071811-150104&rft_id=info:pmc\/PMC3970111&rft_id=info:pmid\/23642247&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3970111&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ServiceDNA17-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ServiceDNA17_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Service, R.F. (02 March 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sciencemag.org\/news\/2017\/03\/dna-could-store-all-worlds-data-one-room\" target=\"_blank\">\"DNA could store all of the world's data in one room\"<\/a>. <i>Science<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.aal0852\" target=\"_blank\">10.1126\/science.aal0852<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sciencemag.org\/news\/2017\/03\/dna-could-store-all-worlds-data-one-room\" target=\"_blank\">https:\/\/www.sciencemag.org\/news\/2017\/03\/dna-could-store-all-worlds-data-one-room<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=DNA+could+store+all+of+the+world%27s+data+in+one+room&rft.atitle=Science&rft.aulast=Service%2C+R.F.&rft.au=Service%2C+R.F.&rft.date=02+March+2017&rft_id=info:doi\/10.1126%2Fscience.aal0852&rft_id=https%3A%2F%2Fwww.sciencemag.org%2Fnews%2F2017%2F03%2Fdna-could-store-all-worlds-data-one-room&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AndersonSynth18-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AndersonSynth18_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Anderson, L.A.; Islam, M.A.; Prather, K.L.J. (2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5892568\" target=\"_blank\">\"Synthetic biology strategies for improving microbial synthesis of \"green\" biopolymers\"<\/a>. <i>Journal of Biological Chemistry<\/i> <b>293<\/b> (14): 5053-5061. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1074%2Fjbc.TM117.000368\" target=\"_blank\">10.1074\/jbc.TM117.000368<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5892568\/\" target=\"_blank\">PMC5892568<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29339554\" target=\"_blank\">29339554<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5892568\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5892568<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Synthetic+biology+strategies+for+improving+microbial+synthesis+of+%22green%22+biopolymers&rft.jtitle=Journal+of+Biological+Chemistry&rft.aulast=Anderson%2C+L.A.%3B+Islam%2C+M.A.%3B+Prather%2C+K.L.J.&rft.au=Anderson%2C+L.A.%3B+Islam%2C+M.A.%3B+Prather%2C+K.L.J.&rft.date=2018&rft.volume=293&rft.issue=14&rft.pages=5053-5061&rft_id=info:doi\/10.1074%2Fjbc.TM117.000368&rft_id=info:pmc\/PMC5892568&rft_id=info:pmid\/29339554&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5892568&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PatelDNA18-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PatelDNA18_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Patel, P. (20 February 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/spectrum.ieee.org\/the-human-os\/biomedical\/devices\/dna-data-storage-gets-random-access\" target=\"_blank\">\"DNA Data Storage Gets Random Access\"<\/a>. <i>IEEE Spectrum<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/spectrum.ieee.org\/the-human-os\/biomedical\/devices\/dna-data-storage-gets-random-access\" target=\"_blank\">https:\/\/spectrum.ieee.org\/the-human-os\/biomedical\/devices\/dna-data-storage-gets-random-access<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=DNA+Data+Storage+Gets+Random+Access&rft.atitle=IEEE+Spectrum&rft.aulast=Patel%2C+P.&rft.au=Patel%2C+P.&rft.date=20+February+2018&rft_id=https%3A%2F%2Fspectrum.ieee.org%2Fthe-human-os%2Fbiomedical%2Fdevices%2Fdna-data-storage-gets-random-access&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-vonKroghTheChang12-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-vonKroghTheChang12_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">von Krogh, G.; Battistini, B.; Pachidou, F.; Baschera, P. (2012). \"The changing face of corporate venturing in biotechnology\". <i>Nature Biotechnology<\/i> <b>30<\/b> (10): 911\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt.2383\" target=\"_blank\">10.1038\/nbt.2383<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23051802\" target=\"_blank\">23051802<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+changing+face+of+corporate+venturing+in+biotechnology&rft.jtitle=Nature+Biotechnology&rft.aulast=von+Krogh%2C+G.%3B+Battistini%2C+B.%3B+Pachidou%2C+F.%3B+Baschera%2C+P.&rft.au=von+Krogh%2C+G.%3B+Battistini%2C+B.%3B+Pachidou%2C+F.%3B+Baschera%2C+P.&rft.date=2012&rft.volume=30&rft.issue=10&rft.pages=911%E2%80%935&rft_id=info:doi\/10.1038%2Fnbt.2383&rft_id=info:pmid\/23051802&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChaCrowd15-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChaCrowd15_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Cha, A.E. (18 January 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.washingtonpost.com\/national\/health-science\/crowdfunding-propels-scientific-research\/2015\/01\/18\/c1937690-9758-11e4-8005-1924ede3e54a_story.html?utm_term=.734eb498edb5\" target=\"_blank\">\"Crowdfunding propels scientific research\"<\/a>. <i>The Washington Post<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.washingtonpost.com\/national\/health-science\/crowdfunding-propels-scientific-research\/2015\/01\/18\/c1937690-9758-11e4-8005-1924ede3e54a_story.html?utm_term=.734eb498edb5\" target=\"_blank\">https:\/\/www.washingtonpost.com\/national\/health-science\/crowdfunding-propels-scientific-research\/2015\/01\/18\/c1937690-9758-11e4-8005-1924ede3e54a_story.html?utm_term=.734eb498edb5<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Crowdfunding+propels+scientific+research&rft.atitle=The+Washington+Post&rft.aulast=Cha%2C+A.E.&rft.au=Cha%2C+A.E.&rft.date=18+January+2015&rft_id=https%3A%2F%2Fwww.washingtonpost.com%2Fnational%2Fhealth-science%2Fcrowdfunding-propels-scientific-research%2F2015%2F01%2F18%2Fc1937690-9758-11e4-8005-1924ede3e54a_story.html%3Futm_term%3D.734eb498edb5&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MervisData17-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MervisData17_14-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Mervis, J. (09 March 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sciencemag.org\/news\/2017\/03\/data-check-us-government-share-basic-research-funding-falls-below-50\" target=\"_blank\">\"Data check: U.S. government share of basic research funding falls below 50%\"<\/a>. <i>Science<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.aal0890\" target=\"_blank\">10.1126\/science.aal0890<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sciencemag.org\/news\/2017\/03\/data-check-us-government-share-basic-research-funding-falls-below-50\" target=\"_blank\">https:\/\/www.sciencemag.org\/news\/2017\/03\/data-check-us-government-share-basic-research-funding-falls-below-50<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+check%3A+U.S.+government+share+of+basic+research+funding+falls+below+50%25&rft.atitle=Science&rft.aulast=Mervis%2C+J.&rft.au=Mervis%2C+J.&rft.date=09+March+2017&rft_id=info:doi\/10.1126%2Fscience.aal0890&rft_id=https%3A%2F%2Fwww.sciencemag.org%2Fnews%2F2017%2F03%2Fdata-check-us-government-share-basic-research-funding-falls-below-50&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-USGovLifeSciences12-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-USGovLifeSciences12_15-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">U.S. Government (March 2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/us-policy-durc-032812.pdf\" target=\"_blank\">\"United States Government Policy for Oversight of Life Sciences Dual Use Research of Concern\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/us-policy-durc-032812.pdf\" target=\"_blank\">http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/us-policy-durc-032812.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=United+States+Government+Policy+for+Oversight+of+Life+Sciences+Dual+Use+Research+of+Concern&rft.atitle=&rft.aulast=U.S.+Government&rft.au=U.S.+Government&rft.date=March+2012&rft_id=http%3A%2F%2Fwww.phe.gov%2Fs3%2Fdualuse%2FDocuments%2Fus-policy-durc-032812.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-USGovPolicy14-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-USGovPolicy14_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">U.S. Government (September 2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/durc-policy.pdf\" target=\"_blank\">\"United States Government Policy for Institutional Oversight of Life Sciences Dual Use Research of Concern\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/durc-policy.pdf\" target=\"_blank\">http:\/\/www.phe.gov\/s3\/dualuse\/Documents\/durc-policy.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=United+States+Government+Policy+for+Institutional+Oversight+of+Life+Sciences+Dual+Use+Research+of+Concern&rft.atitle=&rft.aulast=U.S.+Government&rft.au=U.S.+Government&rft.date=September+2014&rft_id=http%3A%2F%2Fwww.phe.gov%2Fs3%2Fdualuse%2FDocuments%2Fdurc-policy.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NRCBio04-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NRCBio04_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">National Research Council (2004). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nap.edu\/catalog\/10827\/biotechnology-research-in-an-age-of-terrorism\" target=\"_blank\"><i>Biotechnology Research in an Age of Terrorism<\/i><\/a>. National Academies Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.17226%2F10827\" target=\"_blank\">10.17226\/10827<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780309166874<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nap.edu\/catalog\/10827\/biotechnology-research-in-an-age-of-terrorism\" target=\"_blank\">https:\/\/www.nap.edu\/catalog\/10827\/biotechnology-research-in-an-age-of-terrorism<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Biotechnology+Research+in+an+Age+of+Terrorism&rft.aulast=National+Research+Council&rft.au=National+Research+Council&rft.date=2004&rft.pub=National+Academies+Press&rft_id=info:doi\/10.17226%2F10827&rft.isbn=9780309166874&rft_id=https%3A%2F%2Fwww.nap.edu%2Fcatalog%2F10827%2Fbiotechnology-research-in-an-age-of-terrorism&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LordTheReal17-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LordTheReal17_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lord, R.; Forbes Technology Council (15 December 2017). <a rel=\"nofollow\" class=\"external text\" href=\"#445711491b59\">\"The Real Threat Of Identity Theft Is In Your Medical Records, Not Credit Cards\"<\/a>. <i>Forbes<\/i><span class=\"printonly\">. <a rel=\"nofollow\" class=\"external free\" href=\"#445711491b59\">https:\/\/www.forbes.com\/sites\/forbestechcouncil\/2017\/12\/15\/the-real-threat-of-identity-theft-is-in-your-medical-records-not-credit-cards\/#445711491b59<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Real+Threat+Of+Identity+Theft+Is+In+Your+Medical+Records%2C+Not+Credit+Cards&rft.atitle=Forbes&rft.aulast=Lord%2C+R.%3B+Forbes+Technology+Council&rft.au=Lord%2C+R.%3B+Forbes+Technology+Council&rft.date=15+December+2017&rft_id=https%3A%2F%2Fwww.forbes.com%2Fsites%2Fforbestechcouncil%2F2017%2F12%2F15%2Fthe-real-threat-of-identity-theft-is-in-your-medical-records-not-credit-cards%2F%23445711491b59&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SouzaLessons18-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SouzaLessons18_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Souza, C. (10 December 2018). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pharmexec.com\/lessons-pharma-merck-cyber-attack\" target=\"_blank\">\"Lessons for Pharma from the Merck Cyber Attack\"<\/a>. <i>PharmExec.com<\/i> <b>38<\/b> (12)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pharmexec.com\/lessons-pharma-merck-cyber-attack\" target=\"_blank\">http:\/\/www.pharmexec.com\/lessons-pharma-merck-cyber-attack<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lessons+for+Pharma+from+the+Merck+Cyber+Attack&rft.jtitle=PharmExec.com&rft.aulast=Souza%2C+C.&rft.au=Souza%2C+C.&rft.date=10+December+2018&rft.volume=38&rft.issue=12&rft_id=http%3A%2F%2Fwww.pharmexec.com%2Flessons-pharma-merck-cyber-attack&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WardISIS18-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WardISIS18_20-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Ward, A. (11 December 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.rand.org\/blog\/2018\/12\/isiss-use-of-social-media-still-poses-a-threat-to-stability.html\" target=\"_blank\">\"SIS's Use of Social Media Still Poses a Threat to Stability in the Middle East and Africa\"<\/a>. <i>The RAND Blog<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.rand.org\/blog\/2018\/12\/isiss-use-of-social-media-still-poses-a-threat-to-stability.html\" target=\"_blank\">https:\/\/www.rand.org\/blog\/2018\/12\/isiss-use-of-social-media-still-poses-a-threat-to-stability.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SIS%27s+Use+of+Social+Media+Still+Poses+a+Threat+to+Stability+in+the+Middle+East+and+Africa&rft.atitle=The+RAND+Blog&rft.aulast=Ward%2C+A.&rft.au=Ward%2C+A.&rft.date=11+December+2018&rft_id=https%3A%2F%2Fwww.rand.org%2Fblog%2F2018%2F12%2Fisiss-use-of-social-media-still-poses-a-threat-to-stability.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FriedmanCyber13-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FriedmanCyber13_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Friedman, A.A. (25 September 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.brookings.edu\/research\/cyber-theft-of-competitive-data-asking-the-right-questions\/\" target=\"_blank\">\"Cyber Theft of Competitive Data: Asking the Right Questions\"<\/a>. <i>Brookings<\/i>. The Brookings Institution<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.brookings.edu\/research\/cyber-theft-of-competitive-data-asking-the-right-questions\/\" target=\"_blank\">https:\/\/www.brookings.edu\/research\/cyber-theft-of-competitive-data-asking-the-right-questions\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cyber+Theft+of+Competitive+Data%3A+Asking+the+Right+Questions&rft.atitle=Brookings&rft.aulast=Friedman%2C+A.A.&rft.au=Friedman%2C+A.A.&rft.date=25+September+2013&rft.pub=The+Brookings+Institution&rft_id=https%3A%2F%2Fwww.brookings.edu%2Fresearch%2Fcyber-theft-of-competitive-data-asking-the-right-questions%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BogleHeath18-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BogleHeath18_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bogle, A. (07 June 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.abc.net.au\/news\/science\/2018-04-18\/healthcare-target-for-hackers-experts-warn\/9663304\" target=\"_blank\">\"Healthcare data a growing target for hackers, cybersecurity experts warn\"<\/a>. <i>ABC.net.au<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.abc.net.au\/news\/science\/2018-04-18\/healthcare-target-for-hackers-experts-warn\/9663304\" target=\"_blank\">https:\/\/www.abc.net.au\/news\/science\/2018-04-18\/healthcare-target-for-hackers-experts-warn\/9663304<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Healthcare+data+a+growing+target+for+hackers%2C+cybersecurity+experts+warn&rft.atitle=ABC.net.au&rft.aulast=Bogle%2C+A.&rft.au=Bogle%2C+A.&rft.date=07+June+2018&rft_id=https%3A%2F%2Fwww.abc.net.au%2Fnews%2Fscience%2F2018-04-18%2Fhealthcare-target-for-hackers-experts-warn%2F9663304&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CohenMassive18-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CohenMassive18_23-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Cohen, J. (23 March 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sciencemag.org\/news\/2018\/03\/massive-cyber-hack-iran-allegedly-stole-research-320-universities-governments-and\" target=\"_blank\">\"Massive cyberhack by Iran allegedly stole research from 320 universities, governments, and companies\"<\/a>. <i>Science<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.aat6849\" target=\"_blank\">10.1126\/science.aat6849<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sciencemag.org\/news\/2018\/03\/massive-cyber-hack-iran-allegedly-stole-research-320-universities-governments-and\" target=\"_blank\">https:\/\/www.sciencemag.org\/news\/2018\/03\/massive-cyber-hack-iran-allegedly-stole-research-320-universities-governments-and<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Massive+cyberhack+by+Iran+allegedly+stole+research+from+320+universities%2C+governments%2C+and+companies&rft.atitle=Science&rft.aulast=Cohen%2C+J.&rft.au=Cohen%2C+J.&rft.date=23+March+2018&rft_id=info:doi\/10.1126%2Fscience.aat6849&rft_id=https%3A%2F%2Fwww.sciencemag.org%2Fnews%2F2018%2F03%2Fmassive-cyber-hack-iran-allegedly-stole-research-320-universities-governments-and&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HITTheBig18-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HITTheBig18_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Healthcare IT News Staff (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.healthcareitnews.com\/projects\/biggest-healthcare-data-breaches-2018-so-far\" target=\"_blank\">\"The biggest healthcare data breaches of 2018 (so far)\"<\/a>. <i>Healthcare IT News<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.healthcareitnews.com\/projects\/biggest-healthcare-data-breaches-2018-so-far\" target=\"_blank\">https:\/\/www.healthcareitnews.com\/projects\/biggest-healthcare-data-breaches-2018-so-far<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+biggest+healthcare+data+breaches+of+2018+%28so+far%29&rft.atitle=Healthcare+IT+News&rft.aulast=Healthcare+IT+News+Staff&rft.au=Healthcare+IT+News+Staff&rft.date=2018&rft_id=https%3A%2F%2Fwww.healthcareitnews.com%2Fprojects%2Fbiggest-healthcare-data-breaches-2018-so-far&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuangChina18-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HuangChina18_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Huang, E.; Steger, I. (29 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.defenseone.com\/threats\/2018\/10\/china-secretly-enrolling-military-scientists-western-universities\/152383\/\" target=\"_blank\">\"China Is Secretly Enrolling Military Scientists in Western Universities\"<\/a>. <i>Defense One<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.defenseone.com\/threats\/2018\/10\/china-secretly-enrolling-military-scientists-western-universities\/152383\/\" target=\"_blank\">https:\/\/www.defenseone.com\/threats\/2018\/10\/china-secretly-enrolling-military-scientists-western-universities\/152383\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=China+Is+Secretly+Enrolling+Military+Scientists+in+Western+Universities&rft.atitle=Defense+One&rft.aulast=Huang%2C+E.%3B+Steger%2C+I.&rft.au=Huang%2C+E.%3B+Steger%2C+I.&rft.date=29+October+2018&rft_id=https%3A%2F%2Fwww.defenseone.com%2Fthreats%2F2018%2F10%2Fchina-secretly-enrolling-military-scientists-western-universities%2F152383%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KeownSecond18-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KeownSecond18_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Keown, A. (18 September 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.biospace.com\/article\/-jc1n-second-scientist-pleads-guilty-to-stealing-glaxosmithkline-trade-secrets\/\" target=\"_blank\">\"Second Scientist Pleads Guilty to Stealing GlaxoSmithKline Trade Secrets\"<\/a>. <i>BioSpace<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.biospace.com\/article\/-jc1n-second-scientist-pleads-guilty-to-stealing-glaxosmithkline-trade-secrets\/\" target=\"_blank\">https:\/\/www.biospace.com\/article\/-jc1n-second-scientist-pleads-guilty-to-stealing-glaxosmithkline-trade-secrets\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Second+Scientist+Pleads+Guilty+to+Stealing+GlaxoSmithKline+Trade+Secrets&rft.atitle=BioSpace&rft.aulast=Keown%2C+A.&rft.au=Keown%2C+A.&rft.date=18+September+2018&rft_id=https%3A%2F%2Fwww.biospace.com%2Farticle%2F-jc1n-second-scientist-pleads-guilty-to-stealing-glaxosmithkline-trade-secrets%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LynchBio17-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LynchBio17_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lynch, D.J. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe\" target=\"_blank\">\"Biotechnology: the US-China Dispute over Genentic Data\"<\/a>. <i>Financial Times<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe\" target=\"_blank\">https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Biotechnology%3A+the+US-China+Dispute+over+Genentic+Data&rft.atitle=Financial+Times&rft.aulast=Lynch%2C+D.J.&rft.au=Lynch%2C+D.J.&rft.date=2017&rft_id=https%3A%2F%2Fwww.ft.com%2Fcontent%2F245a7c60-6880-11e7-9a66-93fb352ba1fe&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RappeportInNew18-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RappeportInNew18_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-RappeportInNew18_28-1\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Rappeport, A. (10 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nytimes.com\/2018\/10\/10\/business\/us-china-investment-cfius.html\" target=\"_blank\">\"In New Slap at China, U.S. Expands Power to Block Foreign Investments\"<\/a>. <i>The New York Times<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nytimes.com\/2018\/10\/10\/business\/us-china-investment-cfius.html\" target=\"_blank\">https:\/\/www.nytimes.com\/2018\/10\/10\/business\/us-china-investment-cfius.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=In+New+Slap+at+China%2C+U.S.+Expands+Power+to+Block+Foreign+Investments&rft.atitle=The+New+York+Times&rft.aulast=Rappeport%2C+A.&rft.au=Rappeport%2C+A.&rft.date=10+October+2018&rft_id=https%3A%2F%2Fwww.nytimes.com%2F2018%2F10%2F10%2Fbusiness%2Fus-china-investment-cfius.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BloombergChinese18-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BloombergChinese18_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bloomberg News (19 April 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.scmp.com\/business\/global-economy\/article\/2142351\/chinese-funds-pour-us14b-us-biotechnology-firms-first-three\" target=\"_blank\">\"Chinese funds pour US$1.4b into US biotechnology firms in the first three months of the year\"<\/a>. <i>South China Morning Post<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.scmp.com\/business\/global-economy\/article\/2142351\/chinese-funds-pour-us14b-us-biotechnology-firms-first-three\" target=\"_blank\">https:\/\/www.scmp.com\/business\/global-economy\/article\/2142351\/chinese-funds-pour-us14b-us-biotechnology-firms-first-three<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chinese+funds+pour+US%241.4b+into+US+biotechnology+firms+in+the+first+three+months+of+the+year&rft.atitle=South+China+Morning+Post&rft.aulast=Bloomberg+News&rft.au=Bloomberg+News&rft.date=19+April+2018&rft_id=https%3A%2F%2Fwww.scmp.com%2Fbusiness%2Fglobal-economy%2Farticle%2F2142351%2Fchinese-funds-pour-us14b-us-biotechnology-firms-first-three&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RespautAsChina18-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RespautAsChina18_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Respaut, R.; Zhu, J. (23 September 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.reuters.com\/article\/us-biotech-china-investment\/as-china-builds-biotech-sector-cash-floods-u-s-startups-idUSKCN1M400G\" target=\"_blank\">\"As China builds biotech sector, cash floods U.S. startups\"<\/a>. <i>Reuters<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.reuters.com\/article\/us-biotech-china-investment\/as-china-builds-biotech-sector-cash-floods-u-s-startups-idUSKCN1M400G\" target=\"_blank\">https:\/\/www.reuters.com\/article\/us-biotech-china-investment\/as-china-builds-biotech-sector-cash-floods-u-s-startups-idUSKCN1M400G<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=As+China+builds+biotech+sector%2C+cash+floods+U.S.+startups&rft.atitle=Reuters&rft.aulast=Respaut%2C+R.%3B+Zhu%2C+J.&rft.au=Respaut%2C+R.%3B+Zhu%2C+J.&rft.date=23+September+2018&rft_id=https%3A%2F%2Fwww.reuters.com%2Farticle%2Fus-biotech-china-investment%2Fas-china-builds-biotech-sector-cash-floods-u-s-startups-idUSKCN1M400G&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BaiHospital17-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BaiHospital17_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bai, G.; Jiang, J.X.; Flasher, R. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5818824\" target=\"_blank\">\"Hospital risk of data breaches\"<\/a>. <i>JAMA Internal Medicine<\/i> <b>1777<\/b> (6): 878-880. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjamainternmed.2017.0336\" target=\"_blank\">10.1001\/jamainternmed.2017.0336<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5818824\/\" target=\"_blank\">PMC5818824<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28384777\" target=\"_blank\">28384777<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5818824\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5818824<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hospital+risk+of+data+breaches&rft.jtitle=JAMA+Internal+Medicine&rft.aulast=Bai%2C+G.%3B+Jiang%2C+J.X.%3B+Flasher%2C+R.&rft.au=Bai%2C+G.%3B+Jiang%2C+J.X.%3B+Flasher%2C+R.&rft.date=2017&rft.volume=1777&rft.issue=6&rft.pages=878-880&rft_id=info:doi\/10.1001%2Fjamainternmed.2017.0336&rft_id=info:pmc\/PMC5818824&rft_id=info:pmid\/28384777&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5818824&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MSUHealthcare18-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MSUHealthcare18_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Michigan State University (19 November 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/eurekalert.org\/pub_releases\/2018-11\/msu-hp-111618.php\" target=\"_blank\">\"Healthcare providers -- not hackers -- leak more of your data\"<\/a>. <i>EurekAlert!<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/eurekalert.org\/pub_releases\/2018-11\/msu-hp-111618.php\" target=\"_blank\">https:\/\/eurekalert.org\/pub_releases\/2018-11\/msu-hp-111618.php<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Healthcare+providers+--+not+hackers+--+leak+more+of+your+data&rft.atitle=EurekAlert%21&rft.aulast=Michigan+State+University&rft.au=Michigan+State+University&rft.date=19+November+2018&rft_id=https%3A%2F%2Feurekalert.org%2Fpub_releases%2F2018-11%2Fmsu-hp-111618.php&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BSWuXi15-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BSWuXi15_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">BioSpace (21 October 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.biospace.com\/article\/releases\/-b-wuxi-healthcare-b-invests-in-us-genomics-testmaker-23andme-\/\" target=\"_blank\">\"WuXi Healthcare Invests In US Genomics Testmaker 23andMe\"<\/a>. <i>BioSpace<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.biospace.com\/article\/releases\/-b-wuxi-healthcare-b-invests-in-us-genomics-testmaker-23andme-\/\" target=\"_blank\">https:\/\/www.biospace.com\/article\/releases\/-b-wuxi-healthcare-b-invests-in-us-genomics-testmaker-23andme-\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=WuXi+Healthcare+Invests+In+US+Genomics+Testmaker+23andMe&rft.atitle=BioSpace&rft.aulast=BioSpace&rft.au=BioSpace&rft.date=21+October+2015&rft_id=https%3A%2F%2Fwww.biospace.com%2Farticle%2Freleases%2F-b-wuxi-healthcare-b-invests-in-us-genomics-testmaker-23andme-%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MuiChina16-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MuiChina16_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Mui, Y.Q. (30 December 2016). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.washingtonpost.com\/news\/wonk\/wp\/2016\/12\/30\/chinas-9-billion-effort-to-beat-the-u-s-in-genetic-testing\/?noredirect=on&utm_term=.8586cdbf28b8\" target=\"_blank\">\"China\u2019s $9 billion effort to beat the U.S. in genetic testing\"<\/a>. <i>The Washington Post<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.washingtonpost.com\/news\/wonk\/wp\/2016\/12\/30\/chinas-9-billion-effort-to-beat-the-u-s-in-genetic-testing\/?noredirect=on&utm_term=.8586cdbf28b8\" target=\"_blank\">https:\/\/www.washingtonpost.com\/news\/wonk\/wp\/2016\/12\/30\/chinas-9-billion-effort-to-beat-the-u-s-in-genetic-testing\/?noredirect=on&utm_term=.8586cdbf28b8<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=China%E2%80%99s+%249+billion+effort+to+beat+the+U.S.+in+genetic+testing&rft.atitle=The+Washington+Post&rft.aulast=Mui%2C+Y.Q.&rft.au=Mui%2C+Y.Q.&rft.date=30+December+2016&rft_id=https%3A%2F%2Fwww.washingtonpost.com%2Fnews%2Fwonk%2Fwp%2F2016%2F12%2F30%2Fchinas-9-billion-effort-to-beat-the-u-s-in-genetic-testing%2F%3Fnoredirect%3Don%26utm_term%3D.8586cdbf28b8&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BakerChina12-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BakerChina12_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Baker, M. (2012). \"China buys U.S. sequencing firm\". <i>Nature<\/i> <b>489<\/b> (7417): 485\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F489485a\" target=\"_blank\">10.1038\/489485a<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23018943\" target=\"_blank\">23018943<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=China+buys+U.S.+sequencing+firm&rft.jtitle=Nature&rft.aulast=Baker%2C+M.&rft.au=Baker%2C+M.&rft.date=2012&rft.volume=489&rft.issue=7417&rft.pages=485%E2%80%936&rft_id=info:doi\/10.1038%2F489485a&rft_id=info:pmid\/23018943&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GWComplete12-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GWComplete12_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Genome Web Staff Reporter (17 September 2012). <a rel=\"nofollow\" class=\"external text\" href=\"#.XEqIOFxKiUl\">\"Complete Genomics, BGI Agree to $117.6M Merger\"<\/a>. <i>Genome Web<\/i><span class=\"printonly\">. <a rel=\"nofollow\" class=\"external free\" href=\"#.XEqIOFxKiUl\">https:\/\/www.genomeweb.com\/clinical-sequencing\/complete-genomics-bgi-agree-1176m-merger#.XEqIOFxKiUl<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 24 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Complete+Genomics%2C+BGI+Agree+to+%24117.6M+Merger&rft.atitle=Genome+Web&rft.aulast=Genome+Web+Staff+Reporter&rft.au=Genome+Web+Staff+Reporter&rft.date=17+September+2012&rft_id=https%3A%2F%2Fwww.genomeweb.com%2Fclinical-sequencing%2Fcomplete-genomics-bgi-agree-1176m-merger%23.XEqIOFxKiUl&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BontragerDeep18-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BontragerDeep18_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bontrager, P.; Roy, A.; Togelius, J. et al. (18 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/arxiv.org\/abs\/1705.07386\" target=\"_blank\">\"DeepMasterPrints: Generating MasterPrints for Dictionary Attacks via Latent Variable Evolution\"<\/a>. <i>arXiv.org<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/arxiv.org\/abs\/1705.07386\" target=\"_blank\">https:\/\/arxiv.org\/abs\/1705.07386<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=DeepMasterPrints%3A+Generating+MasterPrints+for+Dictionary+Attacks+via+Latent+Variable+Evolution&rft.atitle=arXiv.org&rft.aulast=Bontrager%2C+P.%3B+Roy%2C+A.%3B+Togelius%2C+J.+et+al.&rft.au=Bontrager%2C+P.%3B+Roy%2C+A.%3B+Togelius%2C+J.+et+al.&rft.date=18+October+2018&rft_id=https%3A%2F%2Farxiv.org%2Fabs%2F1705.07386&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NYUTandonMachine18-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NYUTandonMachine18_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">NYU Tandon School of Engineering (20 November 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.prnewswire.com\/news-releases\/machine-learning-masters-the-fingerprint-to-fool-biometric-systems-300753375.html\" target=\"_blank\">\"Machine Learning Masters the Fingerprint to Fool Biometric Systems\"<\/a>. <i>PR Newswire<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.prnewswire.com\/news-releases\/machine-learning-masters-the-fingerprint-to-fool-biometric-systems-300753375.html\" target=\"_blank\">https:\/\/www.prnewswire.com\/news-releases\/machine-learning-masters-the-fingerprint-to-fool-biometric-systems-300753375.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Machine+Learning+Masters+the+Fingerprint+to+Fool+Biometric+Systems&rft.atitle=PR+Newswire&rft.aulast=NYU+Tandon+School+of+Engineering&rft.au=NYU+Tandon+School+of+Engineering&rft.date=20+November+2018&rft_id=https%3A%2F%2Fwww.prnewswire.com%2Fnews-releases%2Fmachine-learning-masters-the-fingerprint-to-fool-biometric-systems-300753375.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-USCongressForeign18-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-USCongressForeign18_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">U.S. Congress (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.govtrack.us\/congress\/bills\/115\/s2098\" target=\"_blank\">\"S. 2098 (115<sup>th<\/sup>): Foreign Investment Risk Review Modernization Act of 2018\"<\/a>. <i>govtrack<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.govtrack.us\/congress\/bills\/115\/s2098\" target=\"_blank\">https:\/\/www.govtrack.us\/congress\/bills\/115\/s2098<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=S.+2098+%28115%3Csup%3Eth%3C%2Fsup%3E%29%3A+Foreign+Investment+Risk+Review+Modernization+Act+of+2018&rft.atitle=govtrack&rft.aulast=U.S.+Congress&rft.au=U.S.+Congress&rft.date=2018&rft_id=https%3A%2F%2Fwww.govtrack.us%2Fcongress%2Fbills%2F115%2Fs2098&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CDCInformation17-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CDCInformation17_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">CDC, USDA (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.selectagents.gov\/isg-intro.html\" target=\"_blank\">\"Information Systems Security Control Guidance\"<\/a>. <i>Federal Select Agent Program<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.selectagents.gov\/isg-intro.html\" target=\"_blank\">https:\/\/www.selectagents.gov\/isg-intro.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Information+Systems+Security+Control+Guidance&rft.atitle=Federal+Select+Agent+Program&rft.aulast=CDC%2C+USDA&rft.au=CDC%2C+USDA&rft.date=2017&rft_id=https%3A%2F%2Fwww.selectagents.gov%2Fisg-intro.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BDPWGBigData15-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BDPWGBigData15_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Big Data and Privacy Working Group (February 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/20150204_Big_Data_Seizing_Opportunities_Preserving_Values_Memo.pdf\" target=\"_blank\">\"Big Data: Seizing Opportunities, Preserving Values\"<\/a> (PDF). U.S. Government<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/20150204_Big_Data_Seizing_Opportunities_Preserving_Values_Memo.pdf\" target=\"_blank\">https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/20150204_Big_Data_Seizing_Opportunities_Preserving_Values_Memo.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Big+Data%3A+Seizing+Opportunities%2C+Preserving+Values&rft.atitle=&rft.aulast=Big+Data+and+Privacy+Working+Group&rft.au=Big+Data+and+Privacy+Working+Group&rft.date=February+2015&rft.pub=U.S.+Government&rft_id=https%3A%2F%2Fobamawhitehouse.archives.gov%2Fsites%2Fdefault%2Ffiles%2Fdocs%2F20150204_Big_Data_Seizing_Opportunities_Preserving_Values_Memo.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EC2018Reform18-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EC2018Reform18_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">European Commission (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ec.europa.eu\/commission\/priorities\/justice-and-fundamental-rights\/data-protection\/2018-reform-eu-data-protection-rules_en\" target=\"_blank\">\"2018 reform of EU data protection rules\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ec.europa.eu\/commission\/priorities\/justice-and-fundamental-rights\/data-protection\/2018-reform-eu-data-protection-rules_en\" target=\"_blank\">https:\/\/ec.europa.eu\/commission\/priorities\/justice-and-fundamental-rights\/data-protection\/2018-reform-eu-data-protection-rules_en<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=2018+reform+of+EU+data+protection+rules&rft.atitle=&rft.aulast=European+Commission&rft.au=European+Commission&rft.date=2018&rft_id=https%3A%2F%2Fec.europa.eu%2Fcommission%2Fpriorities%2Fjustice-and-fundamental-rights%2Fdata-protection%2F2018-reform-eu-data-protection-rules_en&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SacksChinas18-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SacksChinas18_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Sacks, S. (09 March 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.csis.org\/analysis\/chinas-emerging-data-privacy-system-and-gdpr\" target=\"_blank\">\"China\u2019s Emerging Data Privacy System and GDPR\"<\/a>. <i>Center for Strategic & International Studies<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.csis.org\/analysis\/chinas-emerging-data-privacy-system-and-gdpr\" target=\"_blank\">https:\/\/www.csis.org\/analysis\/chinas-emerging-data-privacy-system-and-gdpr<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=China%E2%80%99s+Emerging+Data+Privacy+System+and+GDPR&rft.atitle=Center+for+Strategic+%26+International+Studies&rft.aulast=Sacks%2C+S.&rft.au=Sacks%2C+S.&rft.date=09+March+2018&rft_id=https%3A%2F%2Fwww.csis.org%2Fanalysis%2Fchinas-emerging-data-privacy-system-and-gdpr&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SoaresBrazil18-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SoaresBrazil18_44-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Soares, E. (28 August 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.loc.gov\/law\/foreign-news\/article\/brazil-personal-data-protection-law-enacted\/\" target=\"_blank\">\"Brazil: Personal Data Protection Law Enacted\"<\/a>. <i>Global Legal Monitor<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.loc.gov\/law\/foreign-news\/article\/brazil-personal-data-protection-law-enacted\/\" target=\"_blank\">https:\/\/www.loc.gov\/law\/foreign-news\/article\/brazil-personal-data-protection-law-enacted\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Brazil%3A+Personal+Data+Protection+Law+Enacted&rft.atitle=Global+Legal+Monitor&rft.aulast=Soares%2C+E.&rft.au=Soares%2C+E.&rft.date=28+August+2018&rft_id=https%3A%2F%2Fwww.loc.gov%2Flaw%2Fforeign-news%2Farticle%2Fbrazil-personal-data-protection-law-enacted%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HHSSummary13-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HHSSummary13_45-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">U.S. Department of Health and Human Services (26 July 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.hhs.gov\/hipaa\/for-professionals\/security\/laws-regulations\/index.html\" target=\"_blank\">\"Summary of the HIPAA Security Rule\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hhs.gov\/hipaa\/for-professionals\/security\/laws-regulations\/index.html\" target=\"_blank\">https:\/\/www.hhs.gov\/hipaa\/for-professionals\/security\/laws-regulations\/index.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Summary+of+the+HIPAA+Security+Rule&rft.atitle=&rft.aulast=U.S.+Department+of+Health+and+Human+Services&rft.au=U.S.+Department+of+Health+and+Human+Services&rft.date=26+July+2013&rft_id=https%3A%2F%2Fwww.hhs.gov%2Fhipaa%2Ffor-professionals%2Fsecurity%2Flaws-regulations%2Findex.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VanNoordenWhite13-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VanNoordenWhite13_46-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Van Noorden, R. (22 February 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/blogs.nature.com\/news\/2013\/02\/us-white-house-announces-open-access-policy.html\" target=\"_blank\">\"White House announces new US open-access policy\"<\/a>. <i>Nature NewsBlog<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/blogs.nature.com\/news\/2013\/02\/us-white-house-announces-open-access-policy.html\" target=\"_blank\">http:\/\/blogs.nature.com\/news\/2013\/02\/us-white-house-announces-open-access-policy.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=White+House+announces+new+US+open-access+policy&rft.atitle=Nature+NewsBlog&rft.aulast=Van+Noorden%2C+R.&rft.au=Van+Noorden%2C+R.&rft.date=22+February+2013&rft_id=http%3A%2F%2Fblogs.nature.com%2Fnews%2F2013%2F02%2Fus-white-house-announces-open-access-policy.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CBDAbout-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CBDAbout_47-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">United Nations Environment Programme. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cbd.int\/abs\/about\/\" target=\"_blank\">\"About the Nagoya Protocol\"<\/a>. <i>Convention on Biological Diversity<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cbd.int\/abs\/about\/\" target=\"_blank\">https:\/\/www.cbd.int\/abs\/about\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=About+the+Nagoya+Protocol&rft.atitle=Convention+on+Biological+Diversity&rft.aulast=United+Nations+Environment+Programme&rft.au=United+Nations+Environment+Programme&rft_id=https%3A%2F%2Fwww.cbd.int%2Fabs%2Fabout%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-dosSRibeiroThreats18-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-dosSRibeiroThreats18_48-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">dos S. Ribeiro, C.; Koopmans, M.P.; Haringhuizen, G.B. (26 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/science.sciencemag.org\/content\/362\/6413\/404\" target=\"_blank\">\"Threats to timely sharing of pathogen sequence data\"<\/a>. <i>Science<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.aau5229\" target=\"_blank\">10.1126\/science.aau5229<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/science.sciencemag.org\/content\/362\/6413\/404\" target=\"_blank\">https:\/\/science.sciencemag.org\/content\/362\/6413\/404<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Threats+to+timely+sharing+of+pathogen+sequence+data&rft.atitle=Science&rft.aulast=dos+S.+Ribeiro%2C+C.%3B+Koopmans%2C+M.P.%3B+Haringhuizen%2C+G.B.&rft.au=dos+S.+Ribeiro%2C+C.%3B+Koopmans%2C+M.P.%3B+Haringhuizen%2C+G.B.&rft.date=26+October+2018&rft_id=info:doi\/10.1126%2Fscience.aau5229&rft_id=https%3A%2F%2Fscience.sciencemag.org%2Fcontent%2F362%2F6413%2F404&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NCBIGenBank13-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NCBIGenBank13_49-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">NCBI (2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ncbi.nlm.nih.gov\/genbank\/\" target=\"_blank\">\"GenBank Overview\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ncbi.nlm.nih.gov\/genbank\/\" target=\"_blank\">https:\/\/www.ncbi.nlm.nih.gov\/genbank\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GenBank+Overview&rft.atitle=&rft.aulast=NCBI&rft.au=NCBI&rft.date=2013&rft_id=https%3A%2F%2Fwww.ncbi.nlm.nih.gov%2Fgenbank%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CTAIPTheIPComm13-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CTAIPTheIPComm13_50-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">The Commission on the Theft of American Intellectual Property (May 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ipcommission.org\/report\/IP_Commission_Report_052213.pdf\" target=\"_blank\">\"The IP Commission Report\"<\/a> (PDF). National Bureau of Asian Research<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ipcommission.org\/report\/IP_Commission_Report_052213.pdf\" target=\"_blank\">http:\/\/www.ipcommission.org\/report\/IP_Commission_Report_052213.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+IP+Commission+Report&rft.atitle=&rft.aulast=The+Commission+on+the+Theft+of+American+Intellectual+Property&rft.au=The+Commission+on+the+Theft+of+American+Intellectual+Property&rft.date=May+2013&rft.pub=National+Bureau+of+Asian+Research&rft_id=http%3A%2F%2Fwww.ipcommission.org%2Freport%2FIP_Commission_Report_052213.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Press60_18-51\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Press60_18_51-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Press, G. (03 December 2018). <a rel=\"nofollow\" class=\"external text\" href=\"#759f17fc4352\">\"60 Cybersecurity Predictions for 2019\"<\/a>. <i>Forbes<\/i><span class=\"printonly\">. <a rel=\"nofollow\" class=\"external free\" href=\"#759f17fc4352\">https:\/\/www.forbes.com\/sites\/gilpress\/2018\/12\/03\/60-cybersecurity-predictions-for-2019\/#759f17fc4352<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=60+Cybersecurity+Predictions+for+2019&rft.atitle=Forbes&rft.aulast=Press%2C+G.&rft.au=Press%2C+G.&rft.date=03+December+2018&rft_id=https%3A%2F%2Fwww.forbes.com%2Fsites%2Fgilpress%2F2018%2F12%2F03%2F60-cybersecurity-predictions-for-2019%2F%23759f17fc4352&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DenningExpon17-52\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DenningExpon17_52-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Denning, P.J.; Lewis, T.G. (2017). \"Exponential laws of computing growth\". <i>Communications of the ACM<\/i> <b>60<\/b> (1): 54\u201365. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F2976758\" target=\"_blank\">10.1145\/2976758<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exponential+laws+of+computing+growth&rft.jtitle=Communications+of+the+ACM&rft.aulast=Denning%2C+P.J.%3B+Lewis%2C+T.G.&rft.au=Denning%2C+P.J.%3B+Lewis%2C+T.G.&rft.date=2017&rft.volume=60&rft.issue=1&rft.pages=54%E2%80%9365&rft_id=info:doi\/10.1145%2F2976758&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DauchInform09-53\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DauchInform09_53-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dauch, K.; Nestler, R. (2009). \"Information Assurance Using a Defense In-Depth Strategy\". <i>2009 Cybersecurity Applications & Technology Conference for Homeland Security<\/i>: 267\u201372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FCATCH.2009.25\" target=\"_blank\">10.1109\/CATCH.2009.25<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+Assurance+Using+a+Defense+In-Depth+Strategy&rft.jtitle=2009+Cybersecurity+Applications+%26+Technology+Conference+for+Homeland+Security&rft.aulast=Dauch%2C+K.%3B+Nestler%2C+R.&rft.au=Dauch%2C+K.%3B+Nestler%2C+R.&rft.date=2009&rft.pages=267%E2%80%9372&rft_id=info:doi\/10.1109%2FCATCH.2009.25&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-inap_adminData13-54\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-inap_adminData13_54-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">inap_admin (20 June 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.inap.com\/blog\/data-in-motion-vs-data-at-rest\/\" target=\"_blank\">\"Data in motion vs. data at rest\"<\/a>. <i>ThinkIT blog<\/i>. INAP<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.inap.com\/blog\/data-in-motion-vs-data-at-rest\/\" target=\"_blank\">https:\/\/www.inap.com\/blog\/data-in-motion-vs-data-at-rest\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 24 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Data+in+motion+vs.+data+at+rest&rft.atitle=ThinkIT+blog&rft.aulast=inap_admin&rft.au=inap_admin&rft.date=20+June+2013&rft.pub=INAP&rft_id=https%3A%2F%2Fwww.inap.com%2Fblog%2Fdata-in-motion-vs-data-at-rest%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McDonaldPrivate17-55\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McDonaldPrivate17_55-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">McDonald, K. (10 April 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/searchcompliance.techtarget.com\/opinion\/Private-sectors-national-cybersecurity-strategy-contributions-lacking\" target=\"_blank\">\"Private sector's national cybersecurity strategy contributions lacking\"<\/a>. <i>TechTarget SearchCompliance<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/searchcompliance.techtarget.com\/opinion\/Private-sectors-national-cybersecurity-strategy-contributions-lacking\" target=\"_blank\">https:\/\/searchcompliance.techtarget.com\/opinion\/Private-sectors-national-cybersecurity-strategy-contributions-lacking<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 24 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Private+sector%27s+national+cybersecurity+strategy+contributions+lacking&rft.atitle=TechTarget+SearchCompliance&rft.aulast=McDonald%2C+K.&rft.au=McDonald%2C+K.&rft.date=10+April+2017&rft_id=https%3A%2F%2Fsearchcompliance.techtarget.com%2Fopinion%2FPrivate-sectors-national-cybersecurity-strategy-contributions-lacking&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TuzmanBorder18-56\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TuzmanBorder18_56-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Tuzman, K.T. (11 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.biocentury.com\/bc-innovations\/strategy\/2018-10-11\/balancing-protection-and-translation-china%E2%80%99s-genomic-data-troves\" target=\"_blank\">\"Border security for China\u2019s genomes\"<\/a>. <i>Biocentury<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.biocentury.com\/bc-innovations\/strategy\/2018-10-11\/balancing-protection-and-translation-china%E2%80%99s-genomic-data-troves\" target=\"_blank\">https:\/\/www.biocentury.com\/bc-innovations\/strategy\/2018-10-11\/balancing-protection-and-translation-china%E2%80%99s-genomic-data-troves<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Border+security+for+China%E2%80%99s+genomes&rft.atitle=Biocentury&rft.aulast=Tuzman%2C+K.T.&rft.au=Tuzman%2C+K.T.&rft.date=11+October+2018&rft_id=https%3A%2F%2Fwww.biocentury.com%2Fbc-innovations%2Fstrategy%2F2018-10-11%2Fbalancing-protection-and-translation-china%25E2%2580%2599s-genomic-data-troves&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CapaccioUSFaces18-57\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CapaccioUSFaces18_57-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Capaccio, A. (21 June 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bloomberg.com\/news\/articles\/2018-06-22\/china-s-thousand-talents-called-key-in-seizing-u-s-expertise\" target=\"_blank\">\"U.S. Faces \u2018Unprecedented Threat\u2019 From China on Tech Takeover\"<\/a>. <i>Bloomberg<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bloomberg.com\/news\/articles\/2018-06-22\/china-s-thousand-talents-called-key-in-seizing-u-s-expertise\" target=\"_blank\">https:\/\/www.bloomberg.com\/news\/articles\/2018-06-22\/china-s-thousand-talents-called-key-in-seizing-u-s-expertise<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=U.S.+Faces+%E2%80%98Unprecedented+Threat%E2%80%99+From+China+on+Tech+Takeover&rft.atitle=Bloomberg&rft.aulast=Capaccio%2C+A.&rft.au=Capaccio%2C+A.&rft.date=21+June+2018&rft_id=https%3A%2F%2Fwww.bloomberg.com%2Fnews%2Farticles%2F2018-06-22%2Fchina-s-thousand-talents-called-key-in-seizing-u-s-expertise&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JiaChina18-58\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JiaChina18_58-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Jia, H. (17 January 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nature.com\/articles\/d41586-018-00538-z\" target=\"_blank\">\"China\u2019s plan to recruit talented researchers\"<\/a>. <i>Naturejobs Career Guide: China<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fd41586-018-00538-z\" target=\"_blank\">10.1038\/d41586-018-00538-z<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nature.com\/articles\/d41586-018-00538-z\" target=\"_blank\">https:\/\/www.nature.com\/articles\/d41586-018-00538-z<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=China%E2%80%99s+plan+to+recruit+talented+researchers&rft.atitle=Naturejobs+Career+Guide%3A+China&rft.aulast=Jia%2C+H.&rft.au=Jia%2C+H.&rft.date=17+January+2018&rft_id=info:doi\/10.1038%2Fd41586-018-00538-z&rft_id=https%3A%2F%2Fwww.nature.com%2Farticles%2Fd41586-018-00538-z&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RileyChina15-59\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RileyChina15_59-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Riley, M.; Walcott, J. (05 June 2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.bloomberg.com\/news\/articles\/2015-06-05\/u-s-government-data-breach-tied-to-theft-of-health-care-records\" target=\"_blank\">\"China\u2019s Hack of U.S. Data Tied to Health-Care Record Thefts\"<\/a>. <i>Bloomberg<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.bloomberg.com\/news\/articles\/2015-06-05\/u-s-government-data-breach-tied-to-theft-of-health-care-records\" target=\"_blank\">https:\/\/www.bloomberg.com\/news\/articles\/2015-06-05\/u-s-government-data-breach-tied-to-theft-of-health-care-records<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=China%E2%80%99s+Hack+of+U.S.+Data+Tied+to+Health-Care+Record+Thefts&rft.atitle=Bloomberg&rft.aulast=Riley%2C+M.%3B+Walcott%2C+J.&rft.au=Riley%2C+M.%3B+Walcott%2C+J.&rft.date=05+June+2015&rft_id=https%3A%2F%2Fwww.bloomberg.com%2Fnews%2Farticles%2F2015-06-05%2Fu-s-government-data-breach-tied-to-theft-of-health-care-records&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DilanianChina18-60\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DilanianChina18_60-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Dilanian, K. (09 October 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nbcnews.com\/news\/china\/china-s-hackers-are-stealing-secrets-u-s-firms-again-n917836\" target=\"_blank\">\"China's hackers are stealing secrets from U.S. firms again, experts say\"<\/a>. <i>NBC News<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nbcnews.com\/news\/china\/china-s-hackers-are-stealing-secrets-u-s-firms-again-n917836\" target=\"_blank\">https:\/\/www.nbcnews.com\/news\/china\/china-s-hackers-are-stealing-secrets-u-s-firms-again-n917836<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 29 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=China%27s+hackers+are+stealing+secrets+from+U.S.+firms+again%2C+experts+say&rft.atitle=NBC+News&rft.aulast=Dilanian%2C+K.&rft.au=Dilanian%2C+K.&rft.date=09+October+2018&rft_id=https%3A%2F%2Fwww.nbcnews.com%2Fnews%2Fchina%2Fchina-s-hackers-are-stealing-secrets-u-s-firms-again-n917836&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KaiserNIH18-61\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KaiserNIH18_61-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Kaiser, J.; Malakoff, D. (27 August 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.sciencemag.org\/news\/2018\/08\/nih-investigating-whether-us-scientists-are-sharing-ideas-foreign-governments\" target=\"_blank\">\"NIH investigating whether U.S. scientists are sharing ideas with foreign governments\"<\/a>. <i>Science<\/i>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.aav2343\" target=\"_blank\">10.1126\/science.aav2343<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.sciencemag.org\/news\/2018\/08\/nih-investigating-whether-us-scientists-are-sharing-ideas-foreign-governments\" target=\"_blank\">https:\/\/www.sciencemag.org\/news\/2018\/08\/nih-investigating-whether-us-scientists-are-sharing-ideas-foreign-governments<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NIH+investigating+whether+U.S.+scientists+are+sharing+ideas+with+foreign+governments&rft.atitle=Science&rft.aulast=Kaiser%2C+J.%3B+Malakoff%2C+D.&rft.au=Kaiser%2C+J.%3B+Malakoff%2C+D.&rft.date=27+August+2018&rft_id=info:doi\/10.1126%2Fscience.aav2343&rft_id=https%3A%2F%2Fwww.sciencemag.org%2Fnews%2F2018%2F08%2Fnih-investigating-whether-us-scientists-are-sharing-ideas-foreign-governments&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilberChinese18-62\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilberChinese18_62-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wilber, D.Q.. <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.latimes.com\/politics\/la-na-pol-chinese-espionage-indictment-20181220-story.html\" target=\"_blank\">\"Chinese hackers charged with stealing data from Navy, JPL and U.S. companies\"<\/a>. <i>Los Angeles Times<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.latimes.com\/politics\/la-na-pol-chinese-espionage-indictment-20181220-story.html\" target=\"_blank\">https:\/\/www.latimes.com\/politics\/la-na-pol-chinese-espionage-indictment-20181220-story.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chinese+hackers+charged+with+stealing+data+from+Navy%2C+JPL+and+U.S.+companies&rft.atitle=Los+Angeles+Times&rft.aulast=Wilber%2C+D.Q.&rft.au=Wilber%2C+D.Q.&rft_id=https%3A%2F%2Fwww.latimes.com%2Fpolitics%2Fla-na-pol-chinese-espionage-indictment-20181220-story.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YouSafeguarding17-63\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YouSafeguarding17_63-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">You, E.H. (10 March 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf\" target=\"_blank\">\"Safeguarding the Bioeconomy: U.S. Opportunities and Challenges\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf\" target=\"_blank\">https:\/\/www.ehidc.org\/sites\/default\/files\/resources\/files\/Ed_You_Testimony_USCC.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Safeguarding+the+Bioeconomy%3A+U.S.+Opportunities+and+Challenges&rft.atitle=&rft.aulast=You%2C+E.H.&rft.au=You%2C+E.H.&rft.date=10+March+2017&rft_id=https%3A%2F%2Fwww.ehidc.org%2Fsites%2Fdefault%2Ffiles%2Fresources%2Ffiles%2FEd_You_Testimony_USCC.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HRWChina17-64\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HRWChina17_64-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Human Rights Watch (13 December 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.hrw.org\/news\/2017\/12\/13\/china-minority-region-collects-dna-millions\" target=\"_blank\">\"China: Minority Region Collects Data from Millions\"<\/a>. <i>Human Rights Watch News<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hrw.org\/news\/2017\/12\/13\/china-minority-region-collects-dna-millions\" target=\"_blank\">https:\/\/www.hrw.org\/news\/2017\/12\/13\/china-minority-region-collects-dna-millions#<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=China%3A+Minority+Region+Collects+Data+from+Millions&rft.atitle=Human+Rights+Watch+News&rft.aulast=Human+Rights+Watch&rft.au=Human+Rights+Watch&rft.date=13+December+2017&rft_id=https%3A%2F%2Fwww.hrw.org%2Fnews%2F2017%2F12%2F13%2Fchina-minority-region-collects-dna-millions%23&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LynchBiotech17-65\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LynchBiotech17_65-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lynch, D.J. (2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe\" target=\"_blank\">\"Biotechnology: the US-China Dispute over Genentic Data\"<\/a>. <i>Financial Times<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe\" target=\"_blank\">https:\/\/www.ft.com\/content\/245a7c60-6880-11e7-9a66-93fb352ba1fe<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 November 2018<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Biotechnology%3A+the+US-China+Dispute+over+Genentic+Data&rft.atitle=Financial+Times&rft.aulast=Lynch%2C+D.J.&rft.au=Lynch%2C+D.J.&rft.date=2017&rft_id=https%3A%2F%2Fwww.ft.com%2Fcontent%2F245a7c60-6880-11e7-9a66-93fb352ba1fe&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PauwelsWho17-66\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PauwelsWho17_66-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pauwels, E.; Vidyarthi, A. (19 November 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics\" target=\"_blank\">\"Who Will Own The Secrets In Our Genes? A U.S. \u2013 China Race in Artificial Intelligence and Genomics\"<\/a>. <i>Science and Technology Innovation Program<\/i>. Wilson Center<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics\" target=\"_blank\">https:\/\/www.wilsoncenter.org\/publication\/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Who+Will+Own+The+Secrets+In+Our+Genes%3F+A+U.S.+%E2%80%93+China+Race+in+Artificial+Intelligence+and+Genomics&rft.atitle=Science+and+Technology+Innovation+Program&rft.aulast=Pauwels%2C+E.%3B+Vidyarthi%2C+A.&rft.au=Pauwels%2C+E.%3B+Vidyarthi%2C+A.&rft.date=19+November+2017&rft.pub=Wilson+Center&rft_id=https%3A%2F%2Fwww.wilsoncenter.org%2Fpublication%2Fwho-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BDPWGBigData14-67\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BDPWGBigData14_67-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Big Data and Privacy Working Group (May 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/big_data_privacy_report_may_1_2014.pdf\" target=\"_blank\">\"Big Data: Seizing Opportunities, Preserving Values\"<\/a> (PDF). U.S. Government<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/big_data_privacy_report_may_1_2014.pdf\" target=\"_blank\">https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/docs\/big_data_privacy_report_may_1_2014.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Big+Data%3A+Seizing+Opportunities%2C+Preserving+Values&rft.atitle=&rft.aulast=Big+Data+and+Privacy+Working+Group&rft.au=Big+Data+and+Privacy+Working+Group&rft.date=May+2014&rft.pub=U.S.+Government&rft_id=https%3A%2F%2Fobamawhitehouse.archives.gov%2Fsites%2Fdefault%2Ffiles%2Fdocs%2Fbig_data_privacy_report_may_1_2014.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PCASTBigData14-68\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PCASTBigData14_68-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">President's Council of Advisors on Science and Technology (May 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/microsites\/ostp\/PCAST\/pcast_big_data_and_privacy_-_may_2014.pdf\" target=\"_blank\">\"Big Data and Privacy: A Technological Perspective\"<\/a> (PDF). U.S. Government<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/microsites\/ostp\/PCAST\/pcast_big_data_and_privacy_-_may_2014.pdf\" target=\"_blank\">https:\/\/obamawhitehouse.archives.gov\/sites\/default\/files\/microsites\/ostp\/PCAST\/pcast_big_data_and_privacy_-_may_2014.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Big+Data+and+Privacy%3A+A+Technological+Perspective&rft.atitle=&rft.aulast=President%27s+Council+of+Advisors+on+Science+and+Technology&rft.au=President%27s+Council+of+Advisors+on+Science+and+Technology&rft.date=May+2014&rft.pub=U.S.+Government&rft_id=https%3A%2F%2Fobamawhitehouse.archives.gov%2Fsites%2Fdefault%2Ffiles%2Fmicrosites%2Fostp%2FPCAST%2Fpcast_big_data_and_privacy_-_may_2014.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NISTCyber-69\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NISTCyber_69-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">National Institute of Standards and Technology (2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nist.gov\/cyberframework\" target=\"_blank\">\"Cybersecurity Framework\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nist.gov\/cyberframework\" target=\"_blank\">https:\/\/www.nist.gov\/cyberframework<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cybersecurity+Framework&rft.atitle=&rft.aulast=National+Institute+of+Standards+and+Technology&rft.au=National+Institute+of+Standards+and+Technology&rft.date=2018&rft_id=https%3A%2F%2Fwww.nist.gov%2Fcyberframework&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DHSUsing18-70\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DHSUsing18_70-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Department of Homeland Security (22 August 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.dhs.gov\/using-cybersecurity-framework\" target=\"_blank\">\"Using the Cybersecurity Framework\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.dhs.gov\/using-cybersecurity-framework\" target=\"_blank\">https:\/\/www.dhs.gov\/using-cybersecurity-framework<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Using+the+Cybersecurity+Framework&rft.atitle=&rft.aulast=Department+of+Homeland+Security&rft.au=Department+of+Homeland+Security&rft.date=22+August+2018&rft_id=https%3A%2F%2Fwww.dhs.gov%2Fusing-cybersecurity-framework&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LohrmannWhy18-71\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LohrmannWhy18_71-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lohrmann, D. (20 May 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.govtech.com\/blogs\/lohrmann-on-cybersecurity\/why-you-need-the-cybersecurity-framework.html\" target=\"_blank\">\"Why You Need the Cybersecurity Framework\"<\/a>. <i>Government Technology<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.govtech.com\/blogs\/lohrmann-on-cybersecurity\/why-you-need-the-cybersecurity-framework.html\" target=\"_blank\">https:\/\/www.govtech.com\/blogs\/lohrmann-on-cybersecurity\/why-you-need-the-cybersecurity-framework.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Why+You+Need+the+Cybersecurity+Framework&rft.atitle=Government+Technology&rft.aulast=Lohrmann%2C+D.&rft.au=Lohrmann%2C+D.&rft.date=20+May+2018&rft_id=https%3A%2F%2Fwww.govtech.com%2Fblogs%2Flohrmann-on-cybersecurity%2Fwhy-you-need-the-cybersecurity-framework.html&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoncevichHealth18-72\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RoncevichHealth18_72-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Roncevich, T. (14 June 2018). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/info.cgcompliance.com\/blog\/healthcare-it-security-best-practices-adopting-nists-cybersecurity-framework\" target=\"_blank\">\"Healthcare IT Security Best Practices: Adopting NIST's Cybersecurity Framework\"<\/a>. <i>Cyberguard Compliance Blog<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/info.cgcompliance.com\/blog\/healthcare-it-security-best-practices-adopting-nists-cybersecurity-framework\" target=\"_blank\">https:\/\/info.cgcompliance.com\/blog\/healthcare-it-security-best-practices-adopting-nists-cybersecurity-framework<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 24 January 2019<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Healthcare+IT+Security+Best+Practices%3A+Adopting+NIST%27s+Cybersecurity+Framework&rft.atitle=Cyberguard+Compliance+Blog&rft.aulast=Roncevich%2C+T.&rft.au=Roncevich%2C+T.&rft.date=14+June+2018&rft_id=https%3A%2F%2Finfo.cgcompliance.com%2Fblog%2Fhealthcare-it-security-best-practices-adopting-nists-cybersecurity-framework&rfr_id=info:sid\/en.wikipedia.org:Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added. The two footnotes in the original material were turned into inline references for this version.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165450\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.450 seconds\nReal time usage: 1.482 seconds\nPreprocessor visited node count: 48070\/1000000\nPreprocessor generated node count: 42752\/1000000\nPost\u2010expand include size: 367506\/2097152 bytes\nTemplate argument size: 146212\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 1452.813 1 - -total\n 88.65% 1287.900 1 - Template:Reflist\n 78.26% 1136.910 72 - Template:Citation\/core\n 62.88% 913.531 58 - Template:Cite_web\n 17.79% 258.487 13 - Template:Cite_journal\n 6.35% 92.255 1 - Template:Infobox_journal_article\n 6.10% 88.603 1 - Template:Infobox\n 5.64% 81.891 117 - Template:Citation\/make_link\n 3.64% 52.849 80 - Template:Infobox\/row\n 3.19% 46.395 31 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11053-0!*!0!!en!*!* and timestamp 20190701165449 and revision id 35887\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data\">https:\/\/www.limswiki.org\/index.php\/Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","74131eedf92ecea39da4d1e4c07690b3_images":[],"74131eedf92ecea39da4d1e4c07690b3_timestamp":1562000089,"ec56e6ba701339e54fcca672efb020ae_type":"article","ec56e6ba701339e54fcca672efb020ae_title":"Chemometric analysis of cannabinoids: Chemotaxonomy and domestication syndrome (Mudge et al. 2018)","ec56e6ba701339e54fcca672efb020ae_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome","ec56e6ba701339e54fcca672efb020ae_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Chemometric analysis of cannabinoids: Chemotaxonomy and domestication syndrome\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nChemometric analysis of cannabinoids: Chemotaxonomy and domestication syndromeJournal\n \nScientific ReportsAuthor(s)\n \nMudge, E.M.; Murch, S.J.; Brown, P.N.Author affiliation(s)\n \nUniversity of British Columbia, British Columbia Institute of TechnologyPrimary contact\n \nEmail: Send message through journal websiteYear published\n \n2018Volume and issue\n \n8Page(s)\n \n13090DOI\n \n10.1038\/s41598-018-31120-2ISSN\n \n2045-2322Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.nature.com\/articles\/s41598-018-31120-2Download\n \nhttps:\/\/www.nature.com\/articles\/s41598-018-31120-2.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Results \n\n3.1 Targeted metabolomics of cannabinoids \n3.2 Classification of strains \n3.3 Untargeted metabolomics analysis \n3.4 Relationships between known and unknown cannabinoids \n3.5 Putative identifications and pathways \n\n\n4 Discussion \n5 Methods \n\n5.1 Reagents \n5.2 Plant materials \n5.3 Targeted metabolomics of cannabinoids \n5.4 Untargeted metabolomics \n5.5 Data analysis \n\n\n6 Data availability \n7 Acknowledgements \n\n7.1 Author contributions \n7.2 Competiting interests \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nCannabis is an interesting domesticated crop with a long history of cultivation and use. Strains have been selected through informal breeding programs with undisclosed parentage and criteria. The term \u201cstrain\u201d refers to minor morphological differences and grower branding rather than distinct cultivated varieties. We hypothesized that strains sold by different licensed producers are chemotaxonomically indistinguishable and that the commercial practice of identifying strains by the ratio of total \u03949-tetrahydrocannabinol (THC) and cannabidiol (CBD) is insufficient to account for the reported human health outcomes. We used targeted metabolomics to analyze 11 known cannabinoids and an untargeted metabolomics approach to identify 21 unknown cannabinoids. Five clusters of chemotaxonomically indistinguishable strains were identified from the 33 commercial products. Only three of the clusters produce cannabidiolic acid (CBDA) in significant quantities, while the other two clusters redirect metabolic resources toward the tetrahydrocannabinolic acid (THCA) production pathways. Six unknown metabolites were unique to CBD-rich strains and\/or correlated to CBDA, and three unknowns were found only in THC-rich strains. Together, these data indicate the domestication of the Cannabis germplasm has resulted in a loss of the CBDA pathway in some strains and reallocation of resources between CBDA and THCA pathways in others. The impact of domestication is a lack of chemical diversity and loss of biodiversity in modern Cannabis strains.\n\nIntroduction \nCannabis sativa L. (marijuana) is a dioecious, annual plant from Central Asia that has been used medicinally and recreationally for thousands of years.[1] The domestication of Cannabis has included human selection, inbreeding, and cross breeding, as well as natural outcrossing and genome mixing.[1] Strains are not easily delineated by genotype, and only moderate correlations have been observed between C. indica and C. sativa ancestry. In addition, large genetic variance has been observed within identically named strains.[2][3] Standardized, highly controlled programs to breed elite varieties or cultivars by selection of phytochemical profile have been limited.[4][5] It is estimated that there are several hundred or perhaps thousands of strains of cannabis currently being cultivated in legal and illegal markets.[4] It is possible that chemically identical or very closely related plant material is being sold under several different names by different producers, with no clear definition of the concept of a \u201cstrain.\u201d\nCannabis producers market their products based on the amounts of total THC and CBD with the assumption that the overall phytochemical composition of the material can be extrapolated from these values, but there is considerable anecdotal evidence suggesting that strains with similar THC\/CBD content have different effects on human physiology.[6][7] More than 120 different cannabinoids have been described in Cannabis[8][9], with the most interesting phytochemistry found in the glandular trichomes on the flowers of the female inflorescences.[10] THC is the most researched cannabinoid, and there are 10 additional classes of cannabinoids with varying chemical structures.[8] Cannabinoids are synthesized in acidic forms through the condensation of geranyl diphosphate (GPP) and most commonly olivetolic acid, products of the methylerythritol phosphate (MEP) and polyketide pathways.[11][12] There are several other polyketides that can be used in place of olivetolic acid, which contribute to the wide variation within this chemical class.[13][14] Neutral cannabinoids are products of decarboxylation from processing and handling harvested flowers.\nChemometric models are used to evaluate metabolite datasets to delineate relationships and identify potential influences on phytochemical diversity.[15][16][17] These approaches can be classified as targeted analysis, untargeted phytochemical discovery, metabolomic profiling. or fingerprinting.[15] Targeted metabolomics determines differences in known phytochemicals, while the untargeted approaches evaluate unidentified compounds in the phytochemical profiles.[15] Targeted-untargeted approaches combine known metabolites with the untargeted datasets as a hypothesis-generating tool to discover metabolite relationships, clusters, families and biochemical pathways.[15][18] The use of these models and algorithms enables a better understanding of metabolite commonality and diversity within plant species.[19]\nWe hypothesized that the total THC and CBD content is not sufficient to distinguish strains and that a combination of targeted and untargeted chemometric approaches can be used to predict cannabinoid composition and to better understand the impact of informal breeding program and selection on the phytochemical diversity of Cannabis. To investigate these hypotheses, we assembled a collection of Cannabis strains sold by licensed producers in Canada primarily based on total THC\/CBD content, and analyzed the strains for known cannabinoids using a previously validated analytical method[20] to establish clusters of similar plant materials.\nWe then used an untargeted metabolomics approach to identify previously uncharacterized compounds and chemical relationships. We identified five clusters of chemotaxonomically indistinguishable strains within the collection. Our results show that the variation in less abundant cannabinoids between Cannabis strains was not dependent on the total THC and CBD content. These data suggest that the domestication of the Cannabis germplasm has resulted in the loss of the CBDA pathway in some strains and the reallocation of resources between CBDA and THCA pathways in others.\n\nResults \nTargeted metabolomics of cannabinoids \nTwo cannabinoids for which standards were obtained, cannabidivarin (CBDV) and cannabicyclol (CBL), were not detected in any strain. The 11 remaining cannabinoids with available chemical reference standards were identified and quantified. THCA content ranged from 0.76 to 20.71% w\/w, with almost a linear increase in content from the lowest to highest strain with an r2 of 0.97, while CBDA content ranged from <MDL to 18.11% w\/w, with the highest CBDA strains having the lowest THCA contents (Fig. 1). In THC-abundant strains, the CBDA levels were less than 0.15%, while in CBD abundant strains the content was greater than 5%. THC, the decarboxylated form of THCA, was present in strains from <LOQ up to 2% by weight in some strains, while CBD contents ranged from <MDL to 0.8%. CBD was most prevalent in high-CBDA strains. In addition, seven cannabinoids present at lower levels were quantified using individual calibration standards: tetrahydrocannabivarin (THCV), cannabigerol (CBG), cannabinol (CBN), cannabichromene (CBC), cannabidivarinic acid (CBDVA), cannabigerolic acid (CBGA), and \u03948-THC.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 Biosynthetic pathway of cannabinoids originating from olivetolic acid and geranyl pyrophosphate. Graphs describe the cannabinoid contents within the 33 strains obtained arranged from lowest to highest total THC.\n\n\n\nClassification of strains \nWe hypothesized that individual plant breeders selected for Cannabis strains by up-regulating and down-regulating specific enzymes within the biosynthetic pathways, resulting in a redirection of metabolites between THCA and CBDA. Our data analysis identified five clusters of strains that fall within a narrow range of total CBD\/THC values consistent with this hypothesis (Table 1). The branch of the biosynthetic pathway with olivetolic acid and geranyl pyrophosphate as precursors produces CBGA, CBG, CBCA, CBC, THCA, THC, CBDA and CBD (Fig. 1). Strains from all clusters contained measurable amounts of CBGA, CBG, THCA and THC (Fig. 1). Nine strains from the clusters with higher concentrations of THCA (blue and purple) did not contain detectable levels of CBC (Fig. 1). Two of the clusters were not found to contain significant quantities of CBDA and CBD (Fig. 1; blue and purple). One strain was different from all others and had a greater CBDA content and detectable levels of CBGA, CBG, CBC, and CBD with minimal THCA and THC (Fig. 1; red).\n\n\n\n\n\n\n\nTable 1. Strains of Cannabis were clustered into five distinct groups that could be separated by the flow of metabolites through the CBD and THC pathways.\n\n\n\nGroup\n\nColor code\n\nCBD range (% w\/w)\n\nTHC range (% w\/w)\n\n# of strains\n\n\nA\n\nBlue\n\n<MDL\u20130.08\n\n11.3\u201319.1\n\n20\n\n\nB\n\nPurple\n\n<MDL\u20130.02\n\n8.0\u20139.9\n\n3\n\n\nC\n\nOrange\n\n7.1\u20139.7\n\n5.0\u20136.7\n\n6\n\n\nD\n\nGreen\n\n5.3\u20138.8\n\n1.7\u20133.1\n\n3\n\n\nE\n\nRed\n\n16.1\n\n0.7\n\n1\n\n\n\nCompounds produced from the precursors divarinolic acid and geranyl pyrophosphate via cannabigerovarin acid (CBGVA) were also found to differ by strain cluster (Fig. 2). CBGVA appears to be a branch point for allocation of resources in Cannabis between THCV and CBDVA, indicating that the enzyme activity or the resource allocation mechanism for production of THCV was lost in the breeding process of strains clustered in the red, orange, and green groups (Fig. 2).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 Biosynthetic pathway of cannabinoids originating from divarinolic acid and geranyl pyrophosphate. Graphs describe the cannabinoid contents within the 33 strains obtained arranged from lowest to highest total THC.\n\n\n\nUntargeted metabolomics analysis \nIn addition to the 11 cannabinoids that corresponded with authentic standards, 21 peaks were identified in the chromatograms with UV spectra characteristic of cannabinoids. By comparison to THC, the contents were estimated from <MDL up to 0.34% by weight. Two unknown cannabinoids (CMPD-7 and CMPD-11) were detected in all strains, while CMPD-3 and CMPD-20 were each only detected in a single strain.\n\nRelationships between known and unknown cannabinoids \nA principal component analysis (PCA) of the autoscaled cannabinoid data was plotted to show the clustering of the samples in an unsupervised fashion (Fig. 3). In the PCA plot, the first two principal components (PC) captured 36.6% of the variance in the data. Based on the loadings plot, the first PC was most highly influenced by the THCA and CBDA content of the strains, which are negatively correlated. There are two high THC strains (CAN17 and CAN21) and one CBD strain (CAN34) that were separated from the data clustered within the 95% confidence limit of the total data variance. Based on the loadings plot (Fig. 3B), CAN17 and CAN21 may be influenced by a significant number of low abundance cannabinoids including CBGA, CMPD-12, and CMPD-11. CAN34 is likely due to its significantly higher CBDA content relative to the other strains and because it contained less than 1% total THC.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3 Principal Component Analysis (PCA) of cannabinoid profiles classified according to THC\/CBD contents (a) scores plot (b) loadings plot.\n\n\n\nWhile the first two principal components of PCA describe 36% of the variance, there is a remaining 64% of the variance in the cannabinoids not being described with this model. Therefore, additional models were employed to understand the relationships between cannabinoids and to identify additional strain classes based on the content of these 32 different cannabinoids. Multiple linear regression (MLR) analysis showed that 14 cannabinoids were better suited compared to all cannabinoids for predicting THCA content with validation r2 values improving from 0.02 and 0.88, respectively and for predicting CBDA content 14 cannabinoids improved the validation r2 values from 0.49 to 0.95 when compared with using the entire data set.\nPearson correlations were used to determine whether any of the unidentified cannabinoids could be associated with the major cannabinoids THCA, THC, CBDA, and CBD (Table 2). There was no significant correlation of THCA or THC and any of the unknown compounds (Table 2). The CBDA content was positively correlated with CMPD1, CBDVA, CMPD5, CMPD6, CMPD16, and CMPD18 (Table 2). CBD was potentially weakly correlated with CMPD1, CMPD6, and CBDA (Table 2).\n\n\n\n\n\n\n\nTable 2. Pearson correlation coefficients of all cannabinoids relative to the four major cannabinoids (THCA, CBDA, THC and CBD) in addition to UV spectral analysis describing cannabinoids as acidic or neutral.\n\n\n\nCannabinoid\n\nUV Spectrum: Acidic\/Neutral\n\nTHCA\n\nCBDA\n\nTHC\n\nCBD\n\nPutative ID\n\n\nCMPD1\n\nAcidic\n\n-0.71\n\n0.93\n\n-0.39\n\n0.54\n\nCBDA-C1\n\n\nCMPD2\n\nAcidic\n\n0.22\n\n-0.14\n\n0.03\n\n-0.11\n\n\n\n\nCBDVA\n\nAcidic\n\n-0.70\n\n0.93\n\n-0.36\n\n0.49\n\n\n\n\nCMPD3\n\nAcidic\n\n-0.20\n\n0.20\n\n0.15\n\n0.40\n\n\n\n\nCMPD4\n\nAcidic\n\n0.18\n\n-0.19\n\n0.17\n\n-0.20\n\n\n\n\nCMPD5\n\n\n\n-0.36\n\n0.61\n\n-0.26\n\n0.12\n\n\n\n\nCMPD6\n\nAcidic\n\n-0.65\n\n0.84\n\n-0.27\n\n0.58\n\nCBDA-C4\n\n\nCBDA\n\nAcidic\n\n-0.81\n\n1.00\n\n-0.34\n\n0.68\n\n\n\n\nCMPD7\n\nAcidic\n\n0.53\n\n-0.41\n\n0.21\n\n-0.29\n\n\n\n\nCMPD8\n\nAcidic\n\n0.21\n\n-0.26\n\n0.12\n\n0.01\n\n\n\n\nCBGA\n\nAcidic\n\n0.46\n\n-0.18\n\n0.52\n\n-0.17\n\n\n\n\nCMPD9\n\n\n\n-0.29\n\n0.22\n\n0.23\n\n0.30\n\n\n\n\nCMPD10\n\nNeutral\n\n0.16\n\n-0.12\n\n0.35\n\n-0.12\n\n\n\n\nTHCV\n\nNeutral\n\n0.05\n\n0.16\n\n0.15\n\n-0.10\n\n\n\n\nCMPD11\n\nNeutral\n\n0.36\n\n-0.09\n\n0.31\n\n-0.21\n\n\n\n\nCBD\n\nNeutral\n\n\u22120.68\n\n0.68\n\n0.00\n\n1.00\n\n\n\n\nCMPD12\n\nNeutral\n\n0.28\n\n-0.23\n\n0.24\n\n-0.12\n\n\n\n\nCBG\n\nNeutral\n\n0.66\n\n-0.35\n\n0.43\n\n-0.27\n\n\n\n\nCMPD13\n\nNeutral\n\n-0.04\n\n0.02\n\n0.23\n\n0.17\n\n\n\n\nCMPD14\n\nNeutral\n\n-0.19\n\n0.34\n\n0.07\n\n0.20\n\n\n\n\nCMPD15\n\nAcidic\n\n-0.24\n\n0.39\n\n-0.32\n\n0.08\n\nCBDMA\n\n\nCMPD16\n\nNeutral\n\n-0.55\n\n0.68\n\n-0.08\n\n0.44\n\n\n\n\nCMPD17\n\n\n\n0.00\n\n0.19\n\n-0.19\n\n0.07\n\n\n\n\nTHCA\n\nAcidic\n\n1.00\n\n-0.81\n\n0.39\n\n-0.68\n\n\n\n\nCBN\n\nNeutral\n\n-0.26\n\n0.32\n\n-0.05\n\n0.41\n\n\n\n\nCMPD18\n\nNeutral\n\n-0.73\n\n0.91\n\n-0.25\n\n0.72\n\nCBDM\n\n\nTHC\n\nNeutral\n\n0.39\n\n-0.34\n\n1.00\n\n0.00\n\n\n\n\n8-THC\n\nNeutral\n\n0.28\n\n-0.14\n\n-0.13\n\n-0.07\n\n\n\n\nCBC\n\nNeutral\n\n-0.21\n\n0.30\n\n0.63\n\n0.59\n\n\n\n\nCMPD19\n\nNeutral\n\n0.19\n\n-0.08\n\n-0.24\n\n-0.21\n\n\n\n\nCMPD20\n\nAcidic\n\n-0.04\n\n-0.11\n\n0.01\n\n-0.10\n\n\n\n\nCMPD21\n\nNeutral\n\n0.02\n\n0.05\n\n-0.05\n\n0.18\n\n\n\n\n\nPutative identifications and pathways \nTen of the unknowns were found across multiple strains from all of the clusters (Fig. 4). CMPD1 was strongly correlated with CBDA according to Pearson\u2019s correlation (Table 2), and although it was found in many of the strains classified as blue or purple, it was at much higher concentrations in the red, green, and orange clusters (Fig. 5a). Compounds 3, 5, 6, 15, and 18 were found only in the CBD-rich clusters red, green, and orange (Fig. 5b\u2013f). Compounds 2, 12, and 20 were found only in THC-dominant strains (Fig. 6a\u2013c).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4 Unknown cannabinoids determined by untargeted metabolomics analysis to be common to all clusters of strains. (a) CMPD4, (b) CMPD7, (c) CMPD8, (d) CMPD9, (e) CMPD10, (f) CMPD11, (g) CMPD14, (h) CMPD16, (i) CMPD19, (j) CMPD21.\n\n\n\n\n\n\n\n\n\n\n\n\n Fig. 5 Unidentified cannabinoids determined by untargeted metabolomics analysis to be unique to CBD-rich strains. (a) CMPD1, (b) CMPD3, (c) CMPD5, (d) CMPD6, (e) CMPD15, (f) CMPD18.\n\n\n\n\n\n\n\n\n\n\n\n\n Fig. 6 Unidentified cannabinoids determined by untargeted metabolomics analysis to be unique to THC dominant strains. (a) CMPD2, (b) CMPD12, (c) CMPD20.\n\n\n\nDiscussion \nThe long history of human use has made the exact region of origin for Cannabis difficult to establish, though literature supports Northeast Asia.[1][4][21] Breeding of Cannabis cultivars in the hemp industry has focused on morphological improvements through established breeding programs, while marijuana, or drug-type cannabis, has primarily taken place in underground\/clandestine programs through crossing landraces and\/or indica and sativa lineages.[22][23] The major focus of breeding was increasing the yield of THC, although other features were considered, including organoleptics (aroma), morphology, color, and trichome density.[1][4][24] The genetic diversity between marijuana strains is lower in comparison with hemp varieties due to crossing closely related varieties.[2][3] CBDA and THCA synthases are thought to be controlled by two alleles on a single locus (B), crossing of CBDA and THCA dominant strains will produce offspring with intermediate total THC:CBD ratios.[5][25] With the prevalence of propagation through cuttings of mother plants, feminization of seeds, and production of sensimilla, the need for male plants has decreased, resulting in potential loss of genetic and phytochemical diversity.[1] The sativa and indica lineages used to describe Cannabis throughout the industry are based on postulation that sativa strains originated from European hemp cultivars, while indica are from potent, resinous Indian Cannabis[4], but given the use and trade of the plant in ancient times, the exact origin is unknown and these may not be distinct species.[21] Modern strains are considered dominant in either of these two \u201clineages\u201d or hybrids between close relatives. These classifications focus on the pharmacological effects associated with the strains where sativa plants are considered stimulating and indica plants are associated with relaxation and sedation, but this is not a botanical or chemotaxonomical classification.[23] Comparisons of cannabinoid contents of these classifications have shown that the THC content can be identical between these two classification groups.[3][26] Many questions remain: What is a \u201cstrain\u201d? Does a \u201cstrain\u201d represent a phytochemically unique variety? Are \u201cstrains\u201d from different growers actually different? Is there a more appropriate way to classify \u201cstrains\u201d? Are these cultivars, varietals, landraces or even species? What is the impact of domestication on the ecological fitness of the species?\nBreeding closely related plants potentially leads to loss of genetic diversity within the genome.[27] Traits signifying domestication syndrome include phenotypic changes such as increase seed size, loss of shattering, changes in reproduction, changes in secondary metabolites and loss of pest resistance compared with wild ancestors.[27][28] Reviews of Cannabis breeding have summarized domestication in terms of morphology, while focus on secondary metabolism has focused primarily on THC content.[1][4] Recent forensic evaluations of confiscated sensimilla Cannabis in the U.S. has shown dramatic increases in total THC content over the last 30 years, from 6.3% to 11.5%[29][30], but strains with greater than 20% total THC are available in the marketplace. This artificial increase in THCA production has resulted in the loss of CBDA synthase activity in THC dominant strains. Although crossbreeding will result in THC:CBD hybrid offspring, the loss of other biosynthetic pathways is unknown due to the non-rigorous breeding programs focusing primarily on the production of a single metabolite. Our data indicate that these breeding programs have also impacted unknown related metabolites with undetermined function.\nMetabolomics analysis can generate chemotaxonomic classifications of plants in addition to hypothesis-generating insight of data correlations, metabolite identification, and relationships that would not be possible through single metabolite evaluation.[15][31] Using the correlation data and PCA loadings plots, we can hypothesize the putative identity of some of these unknown cannabinoids. For example, CMPD6 had a Pearson correlation coefficient of 0.89 with CBDA and occupies the same space within the PCA loadings plot. The UV spectra with a maximum of 224\u2009nm identifies this compound as an acidic cannabinoid which was only detected in the presence of CBDA. Further evaluation showed that it eluted between CBDVA and CBDA; therefore, is hypothesized to be CBDA-C4 with a butyl side chain on the polyketide (Table 2).[32] Likewise, we putatively identified CBDA-C1, CBDM, and CBDMA among the unknowns separated by our chromatography protocol (Table 2). Due to the presence of THCA synthase in all strains, the correlation cannabinoids produced with this enzyme is less obvious. It was previously reported that low abundance cannabinoids may be regulated by upstream biosynthesis of precursor polyketides.[14] We found fewer unknown cannabinoids in the strains selected for higher THC content. With such strong emphasis on the synthesis of a single metabolite, there is a strong possibility that other biosynthetic pathways have been lost in the process.[27][28]\nSeveral classification systems have been proposed for Cannabis based on a limited number of phenotypic attributes.[1][4][33] The concept of a \u201cstrain\u201d does not reflect the crop domestication, breeding programs, or plant chemistry. The strains available in the Canadian marketplace are closely related, and evaluating single metabolite classes does not provide sufficient information to understand the phytochemical diversity available. The abundance of secondary metabolites within plants does not necessarily correlate with pharmacological significance, and with Cannabis there is the postulated \u201centourage effect\u201d describing the synergistic effects of many metabolites for anecdotal medical efficacy.[7] Domestication of the crop has limited the genetic variability in the crop, and the impact on crop diversity, physiology, and metabolism is not fully understood. Further research is needed to evaluate the low abundance cannabinoids for potential medicinal efficacy and to determine their roles in plant metabolism.\n\nMethods \nReagents \nMethanol, acetonitrile, ammonium formate and formic acid (98%) were HPLC grade. Water was deionized and purified to 18.2\u2009M\u03a9 using a Barnstead Smart2Pure nanopure system (Thermo Scientific). Cannabinoid standards for quantification were purchased from Cerilliant Corp. (Round Rock, TX) for tetrahydrocannabinolic acid (THCA), THC, cannabidiolic acid (CBDA), CBD, cannabigerol (CBG), cannabichromene (CBC), tetrahydrocannabivarin (THCV) and cannabinol (CBN), \u03948-THC, cannabidivarinic acid (CBDVA), cannabidivarin (CBDV), cannabigerolic acid (CBGA) and cannabicyclol (CBL). All standards were provided as 1.0\u2009mg\/mL solutions in either methanol or acetonitrile.\n\nPlant materials \nThirty-three strains of Cannabis were purchased from five licensed producers in Canada under the Access to Cannabis for Medical Purposes Regulations, and laboratory analysis was performed under a Health Canada Controlled Drugs and Substances License. The test samples were provided as whole or milled flowers in five, 10, and 15 gram packages and stored at room temperature until use. Due to the legal restrictions pertaining to the storage of Cannabis strains, submission of voucher specimens to a herbarium were not possible, but given the regulatory framework associated with these plants, their identify has been confirmed as Cannabis sativa L.\n\nTargeted metabolomics of cannabinoids \nThe content of 13 cannabinoids was determined according to a previously validated analytical method.[20] In brief, ground Cannabis flowers (0.200\u2009g) were extracted with 25\u2009mL of 80% methanol in a 50\u2009mL amber centrifuge tube for 15\u2009minutes by sonication at room temperature with vortexing every five\u2009minutes, followed by centrifugation at 4500\u2009g for five\u2009minutes and filtration with a 0.22\u2009\u00b5m PTFE filter. Extracts were diluted to within the calibration range using the extraction solvent and placed in the 4\u00b0C sample holder for same-day analysis. Chromatographic separation was performed on an Agilent 1200 UHPLC with a Kinetex C18 100\u2009mm\u2009\u00d7\u20093.0\u2009mm, 1.8\u2009\u00b5m column (Phenomenex; Torrance, CA) using gradient elution with 10\u2009mM ammonium formate (pH 3.6) and acetonitrile. The autosampler was maintained at 4\u00b0C and detection was at 220\u2009nm. The peak areas for peaks with typical acidic or neutral cannabinoid UV spectra eluting between 2.5 and 14.5\u2009minutes were collected using Chemstation software (Agilent Technologies), and known cannabinoids were identified. Known cannabinoids were quantified in % w\/w against their individual calibration curves using external calibration in Microsoft Excel. The total THC content was determined as the sum of THC and THCA in addition to the total degradation products of THC: CBN and \u03948-THC, adjusted by the molar mass ratios. Total CBD content was determined as the sum of CBDA and CBD adjusted by molar mass ratios.\n\nUntargeted metabolomics \nUnknown cannabinoids were identified and numbered in sequential order as they appeared in the chromatogram. Unknown cannabinoids were quantified as THC equivalents using the THC calibration curves and ordered in sequential order in the chromatogram as CMPD#.\n\nData analysis \nFor multivariate analysis, missing values were replaced with the method detection limit (MDL) divided by two for each assigned cannabinoid. In the cannabinoid profiles, where the MDL has not been determined for unassigned peaks, the missing data was replaced with half of the MDL of THC. Pearson correlation coefficients to determine relationships between metabolites were calculated using the cor script in R. As the concentration of a given metabolite does not necessarily correlate with pharmacological activity, the data were autoscaled by mean centering and scaling to unit variance in order to give each metabolite equal weight prior to multivariate analyses. Principal component analysis (PCA) and multiple linear regression (MLR) analysis were subsequently modeled using Solo\u2009+\u2009MIA (Eigenvector Research).\n\nData availability \nThe datasets generated during and\/or analysed during the current study are available from the corresponding author on reasonable request.\n\nAcknowledgements \nThis research was undertaken, in part, thanks to funding from the Canada Research Chairs program.\n\nAuthor contributions \nE.M., S.M. and P.B. designed the study. E.M. conducted the laboratory analysis. All authors analyzed the data and wrote the manuscript. All authors gave final approval for publication.\n\nCompetiting interests \nThe authors declare no competing interests.\n\nReferences \n\n\n\u2191 1.0 1.1 1.2 1.3 1.4 1.5 1.6 Clarke, R.C.; Merlin, M.D. (2016). \"Cannabis Domestication, Breeding History, Present-day Genetic Diversity, and Future Prospects\". Critical Reviews in Plant Sciences 35 (5\u20136): 293\u2013327. doi:10.1080\/07352689.2016.1267498.   \n\n\u2191 2.0 2.1 Sawler, J.; Stout, J.M.; Fardner, K.M. et al. (2015). \"The Genetic Structure of Marijuana and Hemp\". PLoS One 10 (8): e0133292. doi:10.1371\/journal.pone.0133292. PMC PMC4550350. PMID 26308334. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350 .   \n\n\u2191 3.0 3.1 3.2 Soler, S.; Gramazio, P.; Fig\u00e0s, M.R. et al. (2017). \"Genetic structure of Cannabis sativa var. indica cultivars based on genomic SSR (gSSR) markers: Implications for breeding and germplasm management\". Industrial Crops and Products 104: 171\u201378. doi:10.1016\/j.indcrop.2017.04.043.   \n\n\u2191 4.0 4.1 4.2 4.3 4.4 4.5 4.6 Small, E. (2015). \"Evolution and Classification of Cannabis sativa (Marijuana, Hemp) in Relation to Human Utilization\". The Botanical Review 81 (3): 189\u2013294. doi:10.1007\/s12229-015-9157-3.   \n\n\u2191 5.0 5.1 de Meijer, E. (2014). \"Chapter 5: The Chemical Phenotypes (Chemotypes) of Cannabis\". In Pertwee, R.. Handbook of Cannabis. Oxford Scholarship Online. pp. 89\u2013110. doi:10.1093\/acprof:oso\/9780199662685.003.0005. ISBN 9780199662685.   \n\n\u2191 McPartland, J.M.; Russo, E.B. (2001). \"Cannabis and Cannabis Extracts\". Journal of Cannabis Therapeutics 1 (3\u20134): 103\u201332. doi:10.1300\/J175v01n03_08.   \n\n\u2191 7.0 7.1 Russo, E.B. (2011). \"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\". British Journal of Pharmacology 163 (7): 1344\u201364. doi:10.1111\/j.1476-5381.2011.01238.x. PMC PMC3165946. PMID 21749363. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946 .   \n\n\u2191 8.0 8.1 ElSohly, M.a.; Gul, W. (2014). \"Chapter 1: Constituents of Cannabis sativa\". In Pertwee, R.. Handbook of Cannabis. Oxford Scholarship Online. pp. 3\u201322. doi:10.1093\/acprof:oso\/9780199662685.003.0001. ISBN 9780199662685.   \n\n\u2191 Turner, C.E.; ElSohly, M.A.; Boeren, E.G. (1980). \"Constituents of Cannabis sativa L. XVII. A review of the natural constituents\". Journal of Natural Products 43 (2): 169-234. PMID 6991645.   \n\n\u2191 Turner, J.C.; Hemphill, J.K.; Mahlberg, P.G. (1978). \"Quantitative determination of cannabinoids in individual glandular trichomes of Cannabis sativa L. (Cannabacaea)\". American Journal of Botany 65 (10): 1103\u201306. doi:10.1002\/j.1537-2197.1978.tb06177.x.   \n\n\u2191 Flores-Sanchez, I.J.; Verpoorte, R. (2008). \"Secondary metabolism in cannabis\". Phytochemistry Reviews 7 (3): 615\u2013639. doi:10.1007\/s11101-008-9094-4.   \n\n\u2191 Fellermeier, M.; Eisenreich, W.; Bacher, A. et al. (2001). \"Biosynthesis of cannabinoids. Incorporation experiments with (13)C-labeled glucoses\". European Journal of Biochemistry 268 (6): 1596-604. PMID 11248677.   \n\n\u2191 Degenhardt, F.; Stehle, F.; Kayser, O. (2017). \"Chapter 2: The Biosynthesis of Cannabinoids\". In Preedy, V.R.. Handbook of Cannabis and Related Pathologies. Elsevier. pp. 13\u201323. doi:10.1016\/B978-0-12-800756-3.00002-8. ISBN 9780128007563.   \n\n\u2191 14.0 14.1 Shoyama, Y.; Hirano, H.; Nishioka, I. (1984). \"Biosynthesis of propyl cannabinoid acid and its biosynthetic relationship with pentyl and methyl cannabinoid acids\". Phytochemistry 23 (9): 1909\u201312. doi:10.1016\/S0031-9422(00)84939-0.   \n\n\u2191 15.0 15.1 15.2 15.3 15.4 Turi, C.E.; Finley, J.; Shipley, P.R. et al. (2015). \"Metabolomics for phytochemical discovery: development of statistical approaches using a cranberry model system\". Journal of Natural Products 78 (4): 953-66. doi:10.1021\/np500667z. PMID 25751407.   \n\n\u2191 Worley, B.; Powers, R. (2013). \"Multivariate Analysis in Metabolomics\". Current Metabolomics 1 (1): 92\u2013107. doi:10.2174\/2213235X11301010092. PMC PMC4465187. PMID 26078916. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4465187 .   \n\n\u2191 Hagel, J.M.; Facchini, P.J. (2008). \"Plant metabolomics: analytical platforms and integration with functional genomics\". Phytochemistry Reviews 7 (3): 479\u2013497. doi:10.1007\/s11101-007-9086-9.   \n\n\u2191 Brown, P.N;. Murch, S.J.; Shipley, P. (2012). \"Phytochemical diversity of cranberry (Vaccinium macrocarpon Aiton) cultivars by anthocyanin determination and metabolomic profiling with chemometric analysis\". Journal of Agricultural and Food Chemistry 60 (1): 261\u201371. doi:10.1021\/jf2033335. PMID 22148867.   \n\n\u2191 Scherling, C.; Roscher, C.; Giavalisco, P. et al. (2010). \"Metabolomics unravel contrasting effects of biodiversity on the performance of individual plant species\". PLoS One 5 (9): e12569. doi:10.1371\/journal.pone.0012569. PMC PMC2935349. PMID 20830202. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2935349 .   \n\n\u2191 20.0 20.1 Mudge, E.M.; Murch, S.J.; Brown, P.N. (2017). \"Leaner and greener analysis of cannabinoids\". Analytical and Bioanalytical Chemistry 409 (12): 3153\u201363. doi:10.1007\/s00216-017-0256-3. PMC PMC5395585. PMID 28233028. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395585 .   \n\n\u2191 21.0 21.1 Grof, C.P.L. (2018). \"Cannabis, from plant to pill\". British Journal of Clinical Pharmacology 84 (11): 2463\u201367. doi:10.1111\/bcp.13618.   \n\n\u2191 Andre, C.M.; Hausman, J.F.; Guerriero, G. (2016). \"Cannabis sativa: The Plant of the Thousand and One Molecules\". Frontiers in Plant Science 7: 19. doi:10.3389\/fpls.2016.00019. PMC PMC4740396. PMID 26870049. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396 .   \n\n\u2191 23.0 23.1 Piomelli, D.; Russo, E.B. (2016). \"The Cannabis sativa Versus Cannabis indica Debate: An Interview with Ethan Russo, MD\". Cannabis and Cannabinoid Research 1 (1): 44\u20136. doi:10.1089\/can.2015.29003.ebr. PMC PMC5576603. PMID 28861479. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603 .   \n\n\u2191 Sexton, M.; Shelton, K.; Haley, P. et al. (2018). \"Evaluation of Cannabinoid and Terpenoid Content: Cannabis Flower Compared to Supercritical CO2 Concentrate\". Planta Medica 84 (4): 234-241. doi:10.1055\/s-0043-119361. PMID 28926863.   \n\n\u2191 de Meijer, E.P.; Bagatta, M.; Carboni, A. et al. (2003). \"The inheritance of chemical phenotype in Cannabis sativa L.\". Genetics 163 (1): 335\u201346. PMC PMC1462421. PMID 12586720. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421 .   \n\n\u2191 Hazekamp, A.; Fischedick, J.T. (2012). \"Cannabis - From cultivar to chemovar\". Drug testing and analysis 4 (7\u20138): 660\u20137. doi:10.1002\/dta.407. PMID 22362625.   \n\n\u2191 27.0 27.1 27.2 Meyer, R.S.; DuVal, A.E.; Jensen, H.R. (2012). \"Patterns and processes in crop domestication: An historical review and quantitative analysis of 203 global food crops\". New Phytologist 196 (1): 29\u201348. doi:10.1111\/j.1469-8137.2012.04253.x.   \n\n\u2191 28.0 28.1 McKey, D.; Elias, M.; Pujol, B. et al. (2010). \"The evolutionary ecology of clonally propagated domesticated plants\". New Phytologist 186 (2): 318\u201332. doi:10.1111\/j.1469-8137.2010.03210.x. PMID 20202131.   \n\n\u2191 ElSohly, M.A.; Ross, S.A.; Mehmedic, Z. et al. (2000). \"Potency trends of delta9-THC and other cannabinoids in confiscated marijuana from 1980-1997\". Journal of Forensic Sciences 45 (1): 24\u201330. PMID 10641915.   \n\n\u2191 Mehmedic, Z.; Chandra, S.; Slade, D. et al. (2010). \"Potency trends of \u03949-THC and other cannabinoids in confiscated cannabis preparations from 1993 to 2008\". Journal of Forensic Sciences 55 (5): 1209\u201317. doi:10.1111\/j.1556-4029.2010.01441.x. PMID 20487147.   \n\n\u2191 Fiehn, O. (2002). \"Metabolomics--The link between genotypes and phenotypes\". Plant Molecular Biology 48 (1\u20132): 155\u201371. PMID 11860207.   \n\n\u2191 Smith, R.M. (1997). \"Identification of Butyl Cannabinoids in Marijuanas\". Journal of Forensic Sciences 42 (4): 610\u201318. doi:10.1520\/JFS14173J.   \n\n\u2191 Small, E.; Beckstead, H.D. (1973). \"Letter: Cannabinoid phenotypes in Cannabis sativa\". Nature 245 (5421): 147\u20138. PMID 4582664.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\">https:\/\/www.limswiki.org\/index.php\/Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles on cannabis testingLIMSwiki journal articles on cannabis research\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 29 May 2019, at 00:15.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 70 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","ec56e6ba701339e54fcca672efb020ae_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Chemometric_analysis_of_cannabinoids_Chemotaxonomy_and_domestication_syndrome skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Chemometric analysis of cannabinoids: Chemotaxonomy and domestication syndrome<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis\" data-key=\"ae3a33525e4682427d4498e16c586f9e\">Cannabis<\/a><\/i> is an interesting domesticated crop with a long history of cultivation and use. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_strains\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis strains\" data-key=\"39bbebad56bc4dce75022ec5f2049cad\">Strains<\/a> have been selected through informal breeding programs with undisclosed parentage and criteria. The term \u201cstrain\u201d refers to minor morphological differences and grower branding rather than distinct cultivated varieties. We hypothesized that strains sold by different licensed producers are chemotaxonomically indistinguishable and that the commercial practice of identifying strains by the ratio of total \u03949-<a href=\"https:\/\/en.wikipedia.org\/wiki\/Tetrahydrocannabinol\" class=\"extiw wiki-link\" title=\"wikipedia:Tetrahydrocannabinol\" data-key=\"c63b7f849adf168f4b4ff293132f1e53\">tetrahydrocannabinol<\/a> (THC) and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidiol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidiol\" data-key=\"0c46162c3d7b370d7448646c25334265\">cannabidiol<\/a> (CBD) is insufficient to account for the reported human health outcomes. We used targeted <a href=\"https:\/\/en.wikipedia.org\/wiki\/Metabolomics\" class=\"extiw wiki-link\" title=\"wikipedia:Metabolomics\" data-key=\"f87c063772bebe02c6c6b0f8095ee256\">metabolomics<\/a> to analyze 11 known <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinoid\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinoid\" data-key=\"fe5c4e73e0c21e16db393a214691296b\">cannabinoids<\/a> and an untargeted metabolomics approach to identify 21 unknown cannabinoids. Five clusters of chemotaxonomically indistinguishable strains were identified from the 33 commercial products. Only three of the clusters produce cannabidiolic acid (CBDA) in significant quantities, while the other two clusters redirect metabolic resources toward the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tetrahydrocannabinolic_acid\" class=\"extiw wiki-link\" title=\"wikipedia:Tetrahydrocannabinolic acid\" data-key=\"1a480d0caf9f4be51bd48f4ab8347c2b\">tetrahydrocannabinolic acid<\/a> (THCA) production pathways. Six unknown metabolites were unique to CBD-rich strains and\/or correlated to CBDA, and three unknowns were found only in THC-rich strains. Together, these data indicate the domestication of the <i>Cannabis<\/i> <a href=\"https:\/\/en.wikipedia.org\/wiki\/Germplasm\" class=\"extiw wiki-link\" title=\"wikipedia:Germplasm\" data-key=\"8d6b6a3d54b8449e7f9f457bee4d0701\">germplasm<\/a> has resulted in a loss of the CBDA pathway in some strains and reallocation of resources between CBDA and THCA pathways in others. The impact of domestication is a lack of chemical diversity and loss of biodiversity in modern <i>Cannabis<\/i> strains.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p><i>Cannabis sativa<\/i> L. (marijuana) is a <a href=\"https:\/\/en.wikipedia.org\/wiki\/Dioecy\" class=\"extiw wiki-link\" title=\"wikipedia:Dioecy\" data-key=\"25d05a7dc3b43fe9e66a29234c03e374\">dioecious<\/a>, annual plant from Central Asia that has been used medicinally and recreationally for thousands of years.<sup id=\"rdp-ebb-cite_ref-ClarkeCanna17_1-0\" class=\"reference\"><a href=\"#cite_note-ClarkeCanna17-1\">[1]<\/a><\/sup> The domestication of <i>Cannabis<\/i> has included human selection, inbreeding, and cross breeding, as well as natural outcrossing and genome mixing.<sup id=\"rdp-ebb-cite_ref-ClarkeCanna17_1-1\" class=\"reference\"><a href=\"#cite_note-ClarkeCanna17-1\">[1]<\/a><\/sup> Strains are not easily delineated by genotype, and only moderate correlations have been observed between <i>C. indica<\/i> and <i>C. sativa<\/i> ancestry. In addition, large genetic variance has been observed within identically named strains.<sup id=\"rdp-ebb-cite_ref-SawlerTheGenetic15_2-0\" class=\"reference\"><a href=\"#cite_note-SawlerTheGenetic15-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SolerGenetic17_3-0\" class=\"reference\"><a href=\"#cite_note-SolerGenetic17-3\">[3]<\/a><\/sup> Standardized, highly controlled programs to breed elite varieties or cultivars by selection of phytochemical profile have been limited.<sup id=\"rdp-ebb-cite_ref-SmallEvo15_4-0\" class=\"reference\"><a href=\"#cite_note-SmallEvo15-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-deMeijerTheChemical14_5-0\" class=\"reference\"><a href=\"#cite_note-deMeijerTheChemical14-5\">[5]<\/a><\/sup> It is estimated that there are several hundred or perhaps thousands of strains of cannabis currently being cultivated in legal and illegal markets.<sup id=\"rdp-ebb-cite_ref-SmallEvo15_4-1\" class=\"reference\"><a href=\"#cite_note-SmallEvo15-4\">[4]<\/a><\/sup> It is possible that chemically identical or very closely related plant material is being sold under several different names by different producers, with no clear definition of the concept of a \u201cstrain.\u201d\n<\/p><p><i>Cannabis<\/i> producers market their products based on the amounts of total THC and CBD with the assumption that the overall phytochemical composition of the material can be extrapolated from these values, but there is considerable anecdotal evidence suggesting that strains with similar THC\/CBD content have different effects on human physiology.<sup id=\"rdp-ebb-cite_ref-McParlandCannabis08_6-0\" class=\"reference\"><a href=\"#cite_note-McParlandCannabis08-6\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RussoTaming11_7-0\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-7\">[7]<\/a><\/sup> More than 120 different cannabinoids have been described in <i>Cannabis<\/i><sup id=\"rdp-ebb-cite_ref-ElSohlyConst14_8-0\" class=\"reference\"><a href=\"#cite_note-ElSohlyConst14-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TurnerConst80_9-0\" class=\"reference\"><a href=\"#cite_note-TurnerConst80-9\">[9]<\/a><\/sup>, with the most interesting phytochemistry found in the glandular <a href=\"https:\/\/en.wikipedia.org\/wiki\/Trichrome\" class=\"extiw wiki-link\" title=\"wikipedia:Trichrome\" data-key=\"4a5c3452359d3ad15f56e5c6455d4098\">trichomes<\/a> on the flowers of the female inflorescences.<sup id=\"rdp-ebb-cite_ref-TurnerQuant78_10-0\" class=\"reference\"><a href=\"#cite_note-TurnerQuant78-10\">[10]<\/a><\/sup> THC is the most researched cannabinoid, and there are 10 additional classes of cannabinoids with varying chemical structures.<sup id=\"rdp-ebb-cite_ref-ElSohlyConst14_8-1\" class=\"reference\"><a href=\"#cite_note-ElSohlyConst14-8\">[8]<\/a><\/sup> Cannabinoids are synthesized in acidic forms through the condensation of geranyl diphosphate (GPP) and most commonly olivetolic acid, products of the methylerythritol phosphate (MEP) and polyketide pathways.<sup id=\"rdp-ebb-cite_ref-Flores-SanchezSecond08_11-0\" class=\"reference\"><a href=\"#cite_note-Flores-SanchezSecond08-11\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FellermeierBiosynth01_12-0\" class=\"reference\"><a href=\"#cite_note-FellermeierBiosynth01-12\">[12]<\/a><\/sup> There are several other polyketides that can be used in place of olivetolic acid, which contribute to the wide variation within this chemical class.<sup id=\"rdp-ebb-cite_ref-DegenhardtChapter17_13-0\" class=\"reference\"><a href=\"#cite_note-DegenhardtChapter17-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShoyamaBiosynth84_14-0\" class=\"reference\"><a href=\"#cite_note-ShoyamaBiosynth84-14\">[14]<\/a><\/sup> Neutral cannabinoids are products of <a href=\"https:\/\/en.wikipedia.org\/wiki\/Decarboxylation\" class=\"extiw wiki-link\" title=\"wikipedia:Decarboxylation\" data-key=\"1b2f03c2feb1f0d2c530732788090504\">decarboxylation<\/a> from processing and handling harvested flowers.\n<\/p><p>Chemometric models are used to evaluate metabolite datasets to delineate relationships and identify potential influences on phytochemical diversity.<sup id=\"rdp-ebb-cite_ref-TuriMeta15_15-0\" class=\"reference\"><a href=\"#cite_note-TuriMeta15-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WorleyMulti13_16-0\" class=\"reference\"><a href=\"#cite_note-WorleyMulti13-16\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HagelPlant08_17-0\" class=\"reference\"><a href=\"#cite_note-HagelPlant08-17\">[17]<\/a><\/sup> These approaches can be classified as targeted analysis, untargeted phytochemical discovery, metabolomic profiling. or fingerprinting.<sup id=\"rdp-ebb-cite_ref-TuriMeta15_15-1\" class=\"reference\"><a href=\"#cite_note-TuriMeta15-15\">[15]<\/a><\/sup> Targeted metabolomics determines differences in known phytochemicals, while the untargeted approaches evaluate unidentified compounds in the phytochemical profiles.<sup id=\"rdp-ebb-cite_ref-TuriMeta15_15-2\" class=\"reference\"><a href=\"#cite_note-TuriMeta15-15\">[15]<\/a><\/sup> Targeted-untargeted approaches combine known metabolites with the untargeted datasets as a hypothesis-generating tool to discover metabolite relationships, clusters, families and biochemical pathways.<sup id=\"rdp-ebb-cite_ref-TuriMeta15_15-3\" class=\"reference\"><a href=\"#cite_note-TuriMeta15-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BrownPhyto12_18-0\" class=\"reference\"><a href=\"#cite_note-BrownPhyto12-18\">[18]<\/a><\/sup> The use of these models and algorithms enables a better understanding of metabolite commonality and diversity within plant species.<sup id=\"rdp-ebb-cite_ref-ScherlingMeta10_19-0\" class=\"reference\"><a href=\"#cite_note-ScherlingMeta10-19\">[19]<\/a><\/sup>\n<\/p><p>We hypothesized that the total THC and CBD content is not sufficient to distinguish strains and that a combination of targeted and untargeted chemometric approaches can be used to predict cannabinoid composition and to better understand the impact of informal breeding program and selection on the phytochemical diversity of <i>Cannabis<\/i>. To investigate these hypotheses, we assembled a collection of <i>Cannabis<\/i> strains sold by licensed producers in Canada primarily based on total THC\/CBD content, and analyzed the strains for known cannabinoids using a previously validated analytical method<sup id=\"rdp-ebb-cite_ref-MudgeLeaner17_20-0\" class=\"reference\"><a href=\"#cite_note-MudgeLeaner17-20\">[20]<\/a><\/sup> to establish clusters of similar plant materials.\n<\/p><p>We then used an untargeted metabolomics approach to identify previously uncharacterized compounds and chemical relationships. We identified five clusters of chemotaxonomically indistinguishable strains within the collection. Our results show that the variation in less abundant cannabinoids between <i>Cannabis<\/i> strains was not dependent on the total THC and CBD content. These data suggest that the domestication of the <i>Cannabis<\/i> germplasm has resulted in the loss of the CBDA pathway in some strains and the reallocation of resources between CBDA and THCA pathways in others.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Targeted_metabolomics_of_cannabinoids\">Targeted metabolomics of cannabinoids<\/span><\/h3>\n<p>Two cannabinoids for which standards were obtained, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidivarin\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidivarin\" data-key=\"db5283e51af01194d2a0c178cc3b9d3b\">cannabidivarin<\/a> (CBDV) and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabicyclol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabicyclol\" data-key=\"a6001a6f4dd603285dc8da0318595f7a\">cannabicyclol<\/a> (CBL), were not detected in any strain. The 11 remaining cannabinoids with available chemical reference standards were identified and quantified. THCA content ranged from 0.76 to 20.71% w\/w, with almost a linear increase in content from the lowest to highest strain with an r2 of 0.97, while CBDA content ranged from <MDL to 18.11% w\/w, with the highest CBDA strains having the lowest THCA contents (Fig. 1). In THC-abundant strains, the CBDA levels were less than 0.15%, while in CBD abundant strains the content was greater than 5%. THC, the decarboxylated form of THCA, was present in strains from <LOQ up to 2% by weight in some strains, while CBD contents ranged from <MDL to 0.8%. CBD was most prevalent in high-CBDA strains. In addition, seven cannabinoids present at lower levels were quantified using individual calibration standards: <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tetrahydrocannabivarin\" class=\"extiw wiki-link\" title=\"wikipedia:Tetrahydrocannabivarin\" data-key=\"23a3568f832f9af716d7af86b7988cbd\">tetrahydrocannabivarin<\/a> (THCV), <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabigerol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabigerol\" data-key=\"ff3a6dcc74cd7fecd03bf2b259b03105\">cannabigerol<\/a> (CBG), <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinol\" data-key=\"8fa80479b70a31dacda0d3de9d8880ad\">cannabinol<\/a> (CBN), <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabichromene\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabichromene\" data-key=\"2f3d934372efdfa61bdc2db408011a6c\">cannabichromene<\/a> (CBC), cannabidivarinic acid (CBDVA), cannabigerolic acid (CBGA), and \u03948-THC.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Mudge_ScientificReports2018_8.png\" class=\"image wiki-link\" data-key=\"89f6ad0b75f6b4d8dd0a2c8a42fab833\"><img alt=\"Fig1 Mudge ScientificReports2018 8.png\" src=\"https:\/\/www.limswiki.org\/images\/c\/c4\/Fig1_Mudge_ScientificReports2018_8.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> Biosynthetic pathway of cannabinoids originating from olivetolic acid and geranyl pyrophosphate. Graphs describe the cannabinoid contents within the 33 strains obtained arranged from lowest to highest total THC.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Classification_of_strains\">Classification of strains<\/span><\/h3>\n<p>We hypothesized that individual plant breeders selected for <i>Cannabis<\/i> strains by up-regulating and down-regulating specific enzymes within the biosynthetic pathways, resulting in a redirection of metabolites between THCA and CBDA. Our data analysis identified five clusters of strains that fall within a narrow range of total CBD\/THC values consistent with this hypothesis (Table 1). The branch of the biosynthetic pathway with olivetolic acid and geranyl pyrophosphate as precursors produces CBGA, CBG, CBCA, CBC, THCA, THC, CBDA and CBD (Fig. 1). Strains from all clusters contained measurable amounts of CBGA, CBG, THCA and THC (Fig. 1). Nine strains from the clusters with higher concentrations of THCA (blue and purple) did not contain detectable levels of CBC (Fig. 1). Two of the clusters were not found to contain significant quantities of CBDA and CBD (Fig. 1; blue and purple). One strain was different from all others and had a greater CBDA content and detectable levels of CBGA, CBG, CBC, and CBD with minimal THCA and THC (Fig. 1; red).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> Strains of <i>Cannabis<\/i> were clustered into five distinct groups that could be separated by the flow of metabolites through the CBD and THC pathways.\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Group\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Color code\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CBD range (% w\/w)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">THC range (% w\/w)\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\"># of strains\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Blue\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><MDL\u20130.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.3\u201319.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">B\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Purple\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><MDL\u20130.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.0\u20139.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">C\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Orange\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.1\u20139.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.0\u20136.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">D\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Green\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.3\u20138.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.7\u20133.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">E\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Red\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Compounds produced from the precursors divarinolic acid and geranyl pyrophosphate via cannabigerovarin acid (CBGVA) were also found to differ by strain cluster (Fig. 2). CBGVA appears to be a branch point for allocation of resources in <i>Cannabis<\/i> between THCV and CBDVA, indicating that the enzyme activity or the resource allocation mechanism for production of THCV was lost in the breeding process of strains clustered in the red, orange, and green groups (Fig. 2).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Mudge_ScientificReports2018_8.png\" class=\"image wiki-link\" data-key=\"9a8159c8c8509ad9d1da1751035eb476\"><img alt=\"Fig2 Mudge ScientificReports2018 8.png\" src=\"https:\/\/www.limswiki.org\/images\/6\/69\/Fig2_Mudge_ScientificReports2018_8.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> Biosynthetic pathway of cannabinoids originating from divarinolic acid and geranyl pyrophosphate. Graphs describe the cannabinoid contents within the 33 strains obtained arranged from lowest to highest total THC.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Untargeted_metabolomics_analysis\">Untargeted metabolomics analysis<\/span><\/h3>\n<p>In addition to the 11 cannabinoids that corresponded with authentic standards, 21 peaks were identified in the <a href=\"https:\/\/www.limswiki.org\/index.php\/Chromatography\" title=\"Chromatography\" class=\"wiki-link\" data-key=\"2615535d1f14c6cffdfad7285999ad9d\">chromatograms<\/a> with UV spectra characteristic of cannabinoids. By comparison to THC, the contents were estimated from <MDL up to 0.34% by weight. Two unknown cannabinoids (CMPD-7 and CMPD-11) were detected in all strains, while CMPD-3 and CMPD-20 were each only detected in a single strain.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Relationships_between_known_and_unknown_cannabinoids\">Relationships between known and unknown cannabinoids<\/span><\/h3>\n<p>A principal component analysis (PCA) of the autoscaled cannabinoid data was plotted to show the clustering of the samples in an unsupervised fashion (Fig. 3). In the PCA plot, the first two principal components (PC) captured 36.6% of the variance in the data. Based on the loadings plot, the first PC was most highly influenced by the THCA and CBDA content of the strains, which are negatively correlated. There are two high THC strains (CAN17 and CAN21) and one CBD strain (CAN34) that were separated from the data clustered within the 95% confidence limit of the total data variance. Based on the loadings plot (Fig. 3B), CAN17 and CAN21 may be influenced by a significant number of low abundance cannabinoids including CBGA, CMPD-12, and CMPD-11. CAN34 is likely due to its significantly higher CBDA content relative to the other strains and because it contained less than 1% total THC.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Mudge_ScientificReports2018_8.png\" class=\"image wiki-link\" data-key=\"6956db046b486538214350e0153d6a6e\"><img alt=\"Fig3 Mudge ScientificReports2018 8.png\" src=\"https:\/\/www.limswiki.org\/images\/1\/1f\/Fig3_Mudge_ScientificReports2018_8.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3<\/b> Principal Component Analysis (PCA) of cannabinoid profiles classified according to THC\/CBD contents (<b>a<\/b>) scores plot (<b>b<\/b>) loadings plot.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>While the first two principal components of PCA describe 36% of the variance, there is a remaining 64% of the variance in the cannabinoids not being described with this model. Therefore, additional models were employed to understand the relationships between cannabinoids and to identify additional strain classes based on the content of these 32 different cannabinoids. Multiple linear regression (MLR) analysis showed that 14 cannabinoids were better suited compared to all cannabinoids for predicting THCA content with validation r<sup>2<\/sup> values improving from 0.02 and 0.88, respectively and for predicting CBDA content 14 cannabinoids improved the validation r<sup>2<\/sup> values from 0.49 to 0.95 when compared with using the entire data set.\n<\/p><p>Pearson correlations were used to determine whether any of the unidentified cannabinoids could be associated with the major cannabinoids THCA, THC, CBDA, and CBD (Table 2). There was no significant correlation of THCA or THC and any of the unknown compounds (Table 2). The CBDA content was positively correlated with CMPD1, CBDVA, CMPD5, CMPD6, CMPD16, and CMPD18 (Table 2). CBD was potentially weakly correlated with CMPD1, CMPD6, and CBDA (Table 2).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><b>Table 2.<\/b> Pearson correlation coefficients of all cannabinoids relative to the four major cannabinoids (THCA, CBDA, THC and CBD) in addition to UV spectral analysis describing cannabinoids as acidic or neutral.\n<\/td><\/tr>\n\n<tr>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Cannabinoid\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">UV Spectrum: Acidic\/Neutral\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">THCA\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CBDA\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">THC\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">CBD\n<\/th>\n<th style=\"background-color:#e2e2e2; padding-left:10px; padding-right:10px;\">Putative ID\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.71\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.93\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBDA-C1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.22\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBDVA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.70\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.93\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.36\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.49\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.36\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.61\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.65\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.84\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.58\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBDA-C4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBDA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.81\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.68\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.53\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.41\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBGA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.52\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.22\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.35\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">THCV\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.36\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u22120.68\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.68\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBG\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.66\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.35\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.43\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBDMA\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.55\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.68\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.44\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">THCA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.81\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.68\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.26\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.41\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.73\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.91\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.72\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBDM\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">THC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8-THC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CBC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.63\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.59\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Acidic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CMPD21\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.05\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Putative_identifications_and_pathways\">Putative identifications and pathways<\/span><\/h3>\n<p>Ten of the unknowns were found across multiple strains from all of the clusters (Fig. 4). CMPD1 was strongly correlated with CBDA according to Pearson\u2019s correlation (Table 2), and although it was found in many of the strains classified as blue or purple, it was at much higher concentrations in the red, green, and orange clusters (Fig. 5a). Compounds 3, 5, 6, 15, and 18 were found only in the CBD-rich clusters red, green, and orange (Fig. 5b\u2013f). Compounds 2, 12, and 20 were found only in THC-dominant strains (Fig. 6a\u2013c).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Mudge_ScientificReports2018_8.png\" class=\"image wiki-link\" data-key=\"08f9f9ff4020f91dae2156caf3a40c0f\"><img alt=\"Fig4 Mudge ScientificReports2018 8.png\" src=\"https:\/\/www.limswiki.org\/images\/2\/23\/Fig4_Mudge_ScientificReports2018_8.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4<\/b> Unknown cannabinoids determined by untargeted metabolomics analysis to be common to all clusters of strains. (<b>a<\/b>) CMPD4, (<b>b<\/b>) CMPD7, (<b>c<\/b>) CMPD8, (<b>d<\/b>) CMPD9, (<b>e<\/b>) CMPD10, (<b>f<\/b>) CMPD11, (<b>g<\/b>) CMPD14, (<b>h<\/b>) CMPD16, (<b>i<\/b>) CMPD19, (<b>j<\/b>) CMPD21.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Mudge_ScientificReports2018_8.png\" class=\"image wiki-link\" data-key=\"c73301038595e5b4fed03ac95754bb35\"><img alt=\"Fig5 Mudge ScientificReports2018 8.png\" src=\"https:\/\/www.limswiki.org\/images\/b\/b4\/Fig5_Mudge_ScientificReports2018_8.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 5<\/b> Unidentified cannabinoids determined by untargeted metabolomics analysis to be unique to CBD-rich strains. (<b>a<\/b>) CMPD1, (<b>b<\/b>) CMPD3, (<b>c<\/b>) CMPD5, (<b>d<\/b>) CMPD6, (<b>e<\/b>) CMPD15, (<b>f<\/b>) CMPD18.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Mudge_ScientificReports2018_8.png\" class=\"image wiki-link\" data-key=\"5fafc7909118df6ade0ebae6ba86f072\"><img alt=\"Fig6 Mudge ScientificReports2018 8.png\" src=\"https:\/\/www.limswiki.org\/images\/0\/09\/Fig6_Mudge_ScientificReports2018_8.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 6<\/b> Unidentified cannabinoids determined by untargeted metabolomics analysis to be unique to THC dominant strains. (<b>a<\/b>) CMPD2, (<b>b<\/b>) CMPD12, (<b>c<\/b>) CMPD20.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>The long history of human use has made the exact region of origin for <i>Cannabis<\/i> difficult to establish, though literature supports Northeast Asia.<sup id=\"rdp-ebb-cite_ref-ClarkeCanna17_1-2\" class=\"reference\"><a href=\"#cite_note-ClarkeCanna17-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmallEvo15_4-2\" class=\"reference\"><a href=\"#cite_note-SmallEvo15-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrofCanna18_21-0\" class=\"reference\"><a href=\"#cite_note-GrofCanna18-21\">[21]<\/a><\/sup> Breeding of <i>Cannabis<\/i> cultivars in the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hemp\" class=\"extiw wiki-link\" title=\"wikipedia:Hemp\" data-key=\"2e896be8a228178ae105f8b468061a0f\">hemp<\/a> industry has focused on morphological improvements through established breeding programs, while marijuana, or <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_(drug)\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis (drug)\" data-key=\"43fbd34351979f1e17186f202a2b1e49\">drug-type cannabis<\/a>, has primarily taken place in underground\/clandestine programs through crossing <a href=\"https:\/\/en.wikipedia.org\/wiki\/Landrace\" class=\"extiw wiki-link\" title=\"wikipedia:Landrace\" data-key=\"8ce613e5280ad2c03ce6bf605536c96f\">landraces<\/a> and\/or <i>indica<\/i> and <i>sativa<\/i> lineages.<sup id=\"rdp-ebb-cite_ref-AndreCanna16_22-0\" class=\"reference\"><a href=\"#cite_note-AndreCanna16-22\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PiomelliTheCanna16_23-0\" class=\"reference\"><a href=\"#cite_note-PiomelliTheCanna16-23\">[23]<\/a><\/sup> The major focus of breeding was increasing the yield of THC, although other features were considered, including organoleptics (aroma), morphology, color, and trichome density.<sup id=\"rdp-ebb-cite_ref-ClarkeCanna17_1-3\" class=\"reference\"><a href=\"#cite_note-ClarkeCanna17-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmallEvo15_4-3\" class=\"reference\"><a href=\"#cite_note-SmallEvo15-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SextonEval18_24-0\" class=\"reference\"><a href=\"#cite_note-SextonEval18-24\">[24]<\/a><\/sup> The genetic diversity between marijuana strains is lower in comparison with hemp varieties due to crossing closely related varieties.<sup id=\"rdp-ebb-cite_ref-SawlerTheGenetic15_2-1\" class=\"reference\"><a href=\"#cite_note-SawlerTheGenetic15-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SolerGenetic17_3-1\" class=\"reference\"><a href=\"#cite_note-SolerGenetic17-3\">[3]<\/a><\/sup> CBDA and THCA synthases are thought to be controlled by two alleles on a single locus (B), crossing of CBDA and THCA dominant strains will produce offspring with intermediate total THC:CBD ratios.<sup id=\"rdp-ebb-cite_ref-deMeijerTheChemical14_5-1\" class=\"reference\"><a href=\"#cite_note-deMeijerTheChemical14-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-deMeijerTheInher03_25-0\" class=\"reference\"><a href=\"#cite_note-deMeijerTheInher03-25\">[25]<\/a><\/sup> With the prevalence of propagation through cuttings of mother plants, feminization of seeds, and production of sensimilla, the need for male plants has decreased, resulting in potential loss of genetic and phytochemical diversity.<sup id=\"rdp-ebb-cite_ref-ClarkeCanna17_1-4\" class=\"reference\"><a href=\"#cite_note-ClarkeCanna17-1\">[1]<\/a><\/sup> The <i>sativa<\/i> and <i>indica<\/i> lineages used to describe <i>Cannabis<\/i> throughout the industry are based on postulation that <i>sativa<\/i> strains originated from European hemp cultivars, while <i>indica<\/i> are from potent, resinous Indian <i>Cannabis<\/i><sup id=\"rdp-ebb-cite_ref-SmallEvo15_4-4\" class=\"reference\"><a href=\"#cite_note-SmallEvo15-4\">[4]<\/a><\/sup>, but given the use and trade of the plant in ancient times, the exact origin is unknown and these may not be distinct species.<sup id=\"rdp-ebb-cite_ref-GrofCanna18_21-1\" class=\"reference\"><a href=\"#cite_note-GrofCanna18-21\">[21]<\/a><\/sup> Modern strains are considered dominant in either of these two \u201clineages\u201d or hybrids between close relatives. These classifications focus on the pharmacological effects associated with the strains where <i>sativa<\/i> plants are considered stimulating and <i>indica<\/i> plants are associated with relaxation and sedation, but this is not a botanical or chemotaxonomical classification.<sup id=\"rdp-ebb-cite_ref-PiomelliTheCanna16_23-1\" class=\"reference\"><a href=\"#cite_note-PiomelliTheCanna16-23\">[23]<\/a><\/sup> Comparisons of cannabinoid contents of these classifications have shown that the THC content can be identical between these two classification groups.<sup id=\"rdp-ebb-cite_ref-SolerGenetic17_3-2\" class=\"reference\"><a href=\"#cite_note-SolerGenetic17-3\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HazekampCanna12_26-0\" class=\"reference\"><a href=\"#cite_note-HazekampCanna12-26\">[26]<\/a><\/sup> Many questions remain: What is a \u201cstrain\u201d? Does a \u201cstrain\u201d represent a phytochemically unique variety? Are \u201cstrains\u201d from different growers actually different? Is there a more appropriate way to classify \u201cstrains\u201d? Are these cultivars, varietals, landraces or even species? What is the impact of domestication on the ecological fitness of the species?\n<\/p><p>Breeding closely related plants potentially leads to loss of genetic diversity within the genome.<sup id=\"rdp-ebb-cite_ref-MeyerPatterns12_27-0\" class=\"reference\"><a href=\"#cite_note-MeyerPatterns12-27\">[27]<\/a><\/sup> Traits signifying domestication syndrome include phenotypic changes such as increase seed size, loss of shattering, changes in reproduction, changes in secondary metabolites and loss of pest resistance compared with wild ancestors.<sup id=\"rdp-ebb-cite_ref-MeyerPatterns12_27-1\" class=\"reference\"><a href=\"#cite_note-MeyerPatterns12-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McKeyTheEvo10_28-0\" class=\"reference\"><a href=\"#cite_note-McKeyTheEvo10-28\">[28]<\/a><\/sup> Reviews of <i>Cannabis<\/i> breeding have summarized domestication in terms of morphology, while focus on secondary metabolism has focused primarily on THC content.<sup id=\"rdp-ebb-cite_ref-ClarkeCanna17_1-5\" class=\"reference\"><a href=\"#cite_note-ClarkeCanna17-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmallEvo15_4-5\" class=\"reference\"><a href=\"#cite_note-SmallEvo15-4\">[4]<\/a><\/sup> Recent forensic evaluations of confiscated sensimilla <i>Cannabis<\/i> in the U.S. has shown dramatic increases in total THC content over the last 30 years, from 6.3% to 11.5%<sup id=\"rdp-ebb-cite_ref-ElSohlyPot00_29-0\" class=\"reference\"><a href=\"#cite_note-ElSohlyPot00-29\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MehmedicPot10_30-0\" class=\"reference\"><a href=\"#cite_note-MehmedicPot10-30\">[30]<\/a><\/sup>, but strains with greater than 20% total THC are available in the marketplace. This artificial increase in THCA production has resulted in the loss of CBDA synthase activity in THC dominant strains. Although crossbreeding will result in THC:CBD hybrid offspring, the loss of other biosynthetic pathways is unknown due to the non-rigorous breeding programs focusing primarily on the production of a single metabolite. Our data indicate that these breeding programs have also impacted unknown related metabolites with undetermined function.\n<\/p><p>Metabolomics analysis can generate chemotaxonomic classifications of plants in addition to hypothesis-generating insight of data correlations, metabolite identification, and relationships that would not be possible through single metabolite evaluation.<sup id=\"rdp-ebb-cite_ref-TuriMeta15_15-4\" class=\"reference\"><a href=\"#cite_note-TuriMeta15-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FiehnMeta02_31-0\" class=\"reference\"><a href=\"#cite_note-FiehnMeta02-31\">[31]<\/a><\/sup> Using the correlation data and PCA loadings plots, we can hypothesize the putative identity of some of these unknown cannabinoids. For example, CMPD6 had a Pearson correlation coefficient of 0.89 with CBDA and occupies the same space within the PCA loadings plot. The UV spectra with a maximum of 224\u2009nm identifies this compound as an acidic cannabinoid which was only detected in the presence of CBDA. Further evaluation showed that it eluted between CBDVA and CBDA; therefore, is hypothesized to be CBDA-C4 with a butyl side chain on the polyketide (Table 2).<sup id=\"rdp-ebb-cite_ref-SmithIdent97_32-0\" class=\"reference\"><a href=\"#cite_note-SmithIdent97-32\">[32]<\/a><\/sup> Likewise, we putatively identified CBDA-C1, CBDM, and CBDMA among the unknowns separated by our chromatography protocol (Table 2). Due to the presence of THCA synthase in all strains, the correlation cannabinoids produced with this enzyme is less obvious. It was previously reported that low abundance cannabinoids may be regulated by upstream biosynthesis of precursor polyketides.<sup id=\"rdp-ebb-cite_ref-ShoyamaBiosynth84_14-1\" class=\"reference\"><a href=\"#cite_note-ShoyamaBiosynth84-14\">[14]<\/a><\/sup> We found fewer unknown cannabinoids in the strains selected for higher THC content. With such strong emphasis on the synthesis of a single metabolite, there is a strong possibility that other biosynthetic pathways have been lost in the process.<sup id=\"rdp-ebb-cite_ref-MeyerPatterns12_27-2\" class=\"reference\"><a href=\"#cite_note-MeyerPatterns12-27\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McKeyTheEvo10_28-1\" class=\"reference\"><a href=\"#cite_note-McKeyTheEvo10-28\">[28]<\/a><\/sup>\n<\/p><p>Several classification systems have been proposed for <i>Cannabis<\/i> based on a limited number of phenotypic attributes.<sup id=\"rdp-ebb-cite_ref-ClarkeCanna17_1-6\" class=\"reference\"><a href=\"#cite_note-ClarkeCanna17-1\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmallEvo15_4-6\" class=\"reference\"><a href=\"#cite_note-SmallEvo15-4\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmallLetter73_33-0\" class=\"reference\"><a href=\"#cite_note-SmallLetter73-33\">[33]<\/a><\/sup> The concept of a \u201cstrain\u201d does not reflect the crop domestication, breeding programs, or plant chemistry. The strains available in the Canadian marketplace are closely related, and evaluating single metabolite classes does not provide sufficient information to understand the phytochemical diversity available. The abundance of secondary metabolites within plants does not necessarily correlate with pharmacological significance, and with <i>Cannabis<\/i> there is the postulated \u201centourage effect\u201d describing the synergistic effects of many metabolites for anecdotal medical efficacy.<sup id=\"rdp-ebb-cite_ref-RussoTaming11_7-1\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-7\">[7]<\/a><\/sup> Domestication of the crop has limited the genetic variability in the crop, and the impact on crop diversity, physiology, and metabolism is not fully understood. Further research is needed to evaluate the low abundance cannabinoids for potential medicinal efficacy and to determine their roles in plant metabolism.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Reagents\">Reagents<\/span><\/h3>\n<p>Methanol, acetonitrile, ammonium formate and formic acid (98%) were HPLC grade. Water was deionized and purified to 18.2\u2009M\u03a9 using a Barnstead Smart2Pure nanopure system (Thermo Scientific). Cannabinoid standards for quantification were purchased from Cerilliant Corp. (Round Rock, TX) for tetrahydrocannabinolic acid (THCA), THC, cannabidiolic acid (CBDA), CBD, cannabigerol (CBG), cannabichromene (CBC), tetrahydrocannabivarin (THCV) and cannabinol (CBN), \u03948-THC, cannabidivarinic acid (CBDVA), cannabidivarin (CBDV), cannabigerolic acid (CBGA) and cannabicyclol (CBL). All standards were provided as 1.0\u2009mg\/mL solutions in either methanol or acetonitrile.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Plant_materials\">Plant materials<\/span><\/h3>\n<p>Thirty-three strains of <i>Cannabis<\/i> were purchased from five licensed producers in Canada under the Access to Cannabis for Medical Purposes Regulations, and laboratory analysis was performed under a Health Canada Controlled Drugs and Substances License. The test samples were provided as whole or milled flowers in five, 10, and 15 gram packages and stored at room temperature until use. Due to the legal restrictions pertaining to the storage of <i>Cannabis<\/i> strains, submission of voucher specimens to a herbarium were not possible, but given the regulatory framework associated with these plants, their identify has been confirmed as <i>Cannabis sativa<\/i> L.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Targeted_metabolomics_of_cannabinoids_2\">Targeted metabolomics of cannabinoids<\/span><\/h3>\n<p>The content of 13 cannabinoids was determined according to a previously validated analytical method.<sup id=\"rdp-ebb-cite_ref-MudgeLeaner17_20-1\" class=\"reference\"><a href=\"#cite_note-MudgeLeaner17-20\">[20]<\/a><\/sup> In brief, ground <i>Cannabis<\/i> flowers (0.200\u2009g) were extracted with 25\u2009mL of 80% methanol in a 50\u2009mL amber centrifuge tube for 15\u2009minutes by sonication at room temperature with vortexing every five\u2009minutes, followed by centrifugation at 4500\u2009g for five\u2009minutes and filtration with a 0.22\u2009\u00b5m PTFE filter. Extracts were diluted to within the calibration range using the extraction solvent and placed in the 4\u00b0C sample holder for same-day analysis. Chromatographic separation was performed on an Agilent 1200 UHPLC with a Kinetex C18 100\u2009mm\u2009\u00d7\u20093.0\u2009mm, 1.8\u2009\u00b5m column (Phenomenex; Torrance, CA) using gradient elution with 10\u2009mM ammonium formate (pH 3.6) and acetonitrile. The autosampler was maintained at 4\u00b0C and detection was at 220\u2009nm. The peak areas for peaks with typical acidic or neutral cannabinoid UV spectra eluting between 2.5 and 14.5\u2009minutes were collected using Chemstation software (Agilent Technologies), and known cannabinoids were identified. Known cannabinoids were quantified in % w\/w against their individual calibration curves using external calibration in Microsoft Excel. The total THC content was determined as the sum of THC and THCA in addition to the total degradation products of THC: CBN and \u03948-THC, adjusted by the molar mass ratios. Total CBD content was determined as the sum of CBDA and CBD adjusted by molar mass ratios.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Untargeted_metabolomics\">Untargeted metabolomics<\/span><\/h3>\n<p>Unknown cannabinoids were identified and numbered in sequential order as they appeared in the chromatogram. Unknown cannabinoids were quantified as THC equivalents using the THC calibration curves and ordered in sequential order in the chromatogram as CMPD#.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_analysis\">Data analysis<\/span><\/h3>\n<p>For multivariate analysis, missing values were replaced with the method detection limit (MDL) divided by two for each assigned cannabinoid. In the cannabinoid profiles, where the MDL has not been determined for unassigned peaks, the missing data was replaced with half of the MDL of THC. Pearson correlation coefficients to determine relationships between metabolites were calculated using the <i>cor<\/i> script in R. As the concentration of a given metabolite does not necessarily correlate with pharmacological activity, the data were autoscaled by mean centering and scaling to unit variance in order to give each metabolite equal weight prior to multivariate analyses. Principal component analysis (PCA) and multiple linear regression (MLR) analysis were subsequently modeled using Solo\u2009+\u2009MIA (Eigenvector Research).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Data_availability\">Data availability<\/span><\/h2>\n<p>The datasets generated during and\/or analysed during the current study are available from the corresponding author on reasonable request.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This research was undertaken, in part, thanks to funding from the Canada Research Chairs program.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>E.M., S.M. and P.B. designed the study. E.M. conducted the laboratory analysis. All authors analyzed the data and wrote the manuscript. All authors gave final approval for publication.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competiting_interests\">Competiting interests<\/span><\/h3>\n<p>The authors declare no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-ClarkeCanna17-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ClarkeCanna17_1-0\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-ClarkeCanna17_1-1\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-ClarkeCanna17_1-2\">1.2<\/a><\/sup> <sup><a href=\"#cite_ref-ClarkeCanna17_1-3\">1.3<\/a><\/sup> <sup><a href=\"#cite_ref-ClarkeCanna17_1-4\">1.4<\/a><\/sup> <sup><a href=\"#cite_ref-ClarkeCanna17_1-5\">1.5<\/a><\/sup> <sup><a href=\"#cite_ref-ClarkeCanna17_1-6\">1.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Clarke, R.C.; Merlin, M.D. (2016). \"<i>Cannabis<\/i> Domestication, Breeding History, Present-day Genetic Diversity, and Future Prospects\". <i>Critical Reviews in Plant Sciences<\/i> <b>35<\/b> (5\u20136): 293\u2013327. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F07352689.2016.1267498\" target=\"_blank\">10.1080\/07352689.2016.1267498<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%27%27Cannabis%27%27+Domestication%2C+Breeding+History%2C+Present-day+Genetic+Diversity%2C+and+Future+Prospects&rft.jtitle=Critical+Reviews+in+Plant+Sciences&rft.aulast=Clarke%2C+R.C.%3B+Merlin%2C+M.D.&rft.au=Clarke%2C+R.C.%3B+Merlin%2C+M.D.&rft.date=2016&rft.volume=35&rft.issue=5%E2%80%936&rft.pages=293%E2%80%93327&rft_id=info:doi\/10.1080%2F07352689.2016.1267498&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SawlerTheGenetic15-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SawlerTheGenetic15_2-0\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-SawlerTheGenetic15_2-1\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sawler, J.; Stout, J.M.; Fardner, K.M. et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350\" target=\"_blank\">\"The Genetic Structure of Marijuana and Hemp\"<\/a>. <i>PLoS One<\/i> <b>10<\/b> (8): e0133292. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0133292\" target=\"_blank\">10.1371\/journal.pone.0133292<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4550350\/\" target=\"_blank\">PMC4550350<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26308334\" target=\"_blank\">26308334<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4550350<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Genetic+Structure+of+Marijuana+and+Hemp&rft.jtitle=PLoS+One&rft.aulast=Sawler%2C+J.%3B+Stout%2C+J.M.%3B+Fardner%2C+K.M.+et+al.&rft.au=Sawler%2C+J.%3B+Stout%2C+J.M.%3B+Fardner%2C+K.M.+et+al.&rft.date=2015&rft.volume=10&rft.issue=8&rft.pages=e0133292&rft_id=info:doi\/10.1371%2Fjournal.pone.0133292&rft_id=info:pmc\/PMC4550350&rft_id=info:pmid\/26308334&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4550350&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SolerGenetic17-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SolerGenetic17_3-0\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-SolerGenetic17_3-1\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-SolerGenetic17_3-2\">3.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Soler, S.; Gramazio, P.; Fig\u00e0s, M.R. et al. (2017). \"Genetic structure of <i>Cannabis sativa<\/i> var. <i>indica<\/i> cultivars based on genomic SSR (gSSR) markers: Implications for breeding and germplasm management\". <i>Industrial Crops and Products<\/i> <b>104<\/b>: 171\u201378. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.indcrop.2017.04.043\" target=\"_blank\">10.1016\/j.indcrop.2017.04.043<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genetic+structure+of+%27%27Cannabis+sativa%27%27+var.+%27%27indica%27%27+cultivars+based+on+genomic+SSR+%28gSSR%29+markers%3A+Implications+for+breeding+and+germplasm+management&rft.jtitle=Industrial+Crops+and+Products&rft.aulast=Soler%2C+S.%3B+Gramazio%2C+P.%3B+Fig%C3%A0s%2C+M.R.+et+al.&rft.au=Soler%2C+S.%3B+Gramazio%2C+P.%3B+Fig%C3%A0s%2C+M.R.+et+al.&rft.date=2017&rft.volume=104&rft.pages=171%E2%80%9378&rft_id=info:doi\/10.1016%2Fj.indcrop.2017.04.043&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmallEvo15-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SmallEvo15_4-0\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-SmallEvo15_4-1\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-SmallEvo15_4-2\">4.2<\/a><\/sup> <sup><a href=\"#cite_ref-SmallEvo15_4-3\">4.3<\/a><\/sup> <sup><a href=\"#cite_ref-SmallEvo15_4-4\">4.4<\/a><\/sup> <sup><a href=\"#cite_ref-SmallEvo15_4-5\">4.5<\/a><\/sup> <sup><a href=\"#cite_ref-SmallEvo15_4-6\">4.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Small, E. (2015). \"Evolution and Classification of <i>Cannabis sativa<\/i> (Marijuana, Hemp) in Relation to Human Utilization\". <i>The Botanical Review<\/i> <b>81<\/b> (3): 189\u2013294. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs12229-015-9157-3\" target=\"_blank\">10.1007\/s12229-015-9157-3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evolution+and+Classification+of+%27%27Cannabis+sativa%27%27+%28Marijuana%2C+Hemp%29+in+Relation+to+Human+Utilization&rft.jtitle=The+Botanical+Review&rft.aulast=Small%2C+E.&rft.au=Small%2C+E.&rft.date=2015&rft.volume=81&rft.issue=3&rft.pages=189%E2%80%93294&rft_id=info:doi\/10.1007%2Fs12229-015-9157-3&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-deMeijerTheChemical14-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-deMeijerTheChemical14_5-0\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-deMeijerTheChemical14_5-1\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">de Meijer, E. (2014). \"Chapter 5: The Chemical Phenotypes (Chemotypes) of Cannabis\". In Pertwee, R.. <i>Handbook of Cannabis<\/i>. Oxford Scholarship Online. pp. 89\u2013110. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Facprof%3Aoso%2F9780199662685.003.0005\" target=\"_blank\">10.1093\/acprof:oso\/9780199662685.003.0005<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780199662685.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+5%3A+The+Chemical+Phenotypes+%28Chemotypes%29+of+Cannabis&rft.atitle=Handbook+of+Cannabis&rft.aulast=de+Meijer%2C+E.&rft.au=de+Meijer%2C+E.&rft.date=2014&rft.pages=pp.%26nbsp%3B89%E2%80%93110&rft.pub=Oxford+Scholarship+Online&rft_id=info:doi\/10.1093%2Facprof%3Aoso%2F9780199662685.003.0005&rft.isbn=9780199662685&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McParlandCannabis08-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McParlandCannabis08_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McPartland, J.M.; Russo, E.B. (2001). \"Cannabis and Cannabis Extracts\". <i>Journal of Cannabis Therapeutics<\/i> <b>1<\/b> (3\u20134): 103\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1300%2FJ175v01n03_08\" target=\"_blank\">10.1300\/J175v01n03_08<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+and+Cannabis+Extracts&rft.jtitle=Journal+of+Cannabis+Therapeutics&rft.aulast=McPartland%2C+J.M.%3B+Russo%2C+E.B.&rft.au=McPartland%2C+J.M.%3B+Russo%2C+E.B.&rft.date=2001&rft.volume=1&rft.issue=3%E2%80%934&rft.pages=103%E2%80%9332&rft_id=info:doi\/10.1300%2FJ175v01n03_08&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RussoTaming11-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RussoTaming11_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-RussoTaming11_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Russo, E.B. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\" target=\"_blank\">\"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\"<\/a>. <i>British Journal of Pharmacology<\/i> <b>163<\/b> (7): 1344\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1476-5381.2011.01238.x\" target=\"_blank\">10.1111\/j.1476-5381.2011.01238.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3165946\/\" target=\"_blank\">PMC3165946<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21749363\" target=\"_blank\">21749363<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Taming+THC%3A+potential+cannabis+synergy+and+phytocannabinoid-terpenoid+entourage+effects&rft.jtitle=British+Journal+of+Pharmacology&rft.aulast=Russo%2C+E.B.&rft.au=Russo%2C+E.B.&rft.date=2011&rft.volume=163&rft.issue=7&rft.pages=1344%E2%80%9364&rft_id=info:doi\/10.1111%2Fj.1476-5381.2011.01238.x&rft_id=info:pmc\/PMC3165946&rft_id=info:pmid\/21749363&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3165946&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElSohlyConst14-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ElSohlyConst14_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-ElSohlyConst14_8-1\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">ElSohly, M.a.; Gul, W. (2014). \"Chapter 1: Constituents of <i>Cannabis sativa<\/i>\". In Pertwee, R.. <i>Handbook of Cannabis<\/i>. Oxford Scholarship Online. pp. 3\u201322. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Facprof%3Aoso%2F9780199662685.003.0001\" target=\"_blank\">10.1093\/acprof:oso\/9780199662685.003.0001<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780199662685.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+1%3A+Constituents+of+%27%27Cannabis+sativa%27%27&rft.atitle=Handbook+of+Cannabis&rft.aulast=ElSohly%2C+M.a.%3B+Gul%2C+W.&rft.au=ElSohly%2C+M.a.%3B+Gul%2C+W.&rft.date=2014&rft.pages=pp.%26nbsp%3B3%E2%80%9322&rft.pub=Oxford+Scholarship+Online&rft_id=info:doi\/10.1093%2Facprof%3Aoso%2F9780199662685.003.0001&rft.isbn=9780199662685&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TurnerConst80-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TurnerConst80_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Turner, C.E.; ElSohly, M.A.; Boeren, E.G. (1980). \"Constituents of <i>Cannabis sativa<\/i> L. XVII. A review of the natural constituents\". <i>Journal of Natural Products<\/i> <b>43<\/b> (2): 169-234. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/6991645\" target=\"_blank\">6991645<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Constituents+of+%27%27Cannabis+sativa%27%27+L.+XVII.+A+review+of+the+natural+constituents&rft.jtitle=Journal+of+Natural+Products&rft.aulast=Turner%2C+C.E.%3B+ElSohly%2C+M.A.%3B+Boeren%2C+E.G.&rft.au=Turner%2C+C.E.%3B+ElSohly%2C+M.A.%3B+Boeren%2C+E.G.&rft.date=1980&rft.volume=43&rft.issue=2&rft.pages=169-234&rft_id=info:pmid\/6991645&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TurnerQuant78-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TurnerQuant78_10-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Turner, J.C.; Hemphill, J.K.; Mahlberg, P.G. (1978). \"Quantitative determination of cannabinoids in individual glandular trichomes of <i>Cannabis sativa<\/i> L. (Cannabacaea)\". <i>American Journal of Botany<\/i> <b>65<\/b> (10): 1103\u201306. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fj.1537-2197.1978.tb06177.x\" target=\"_blank\">10.1002\/j.1537-2197.1978.tb06177.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quantitative+determination+of+cannabinoids+in+individual+glandular+trichomes+of+%27%27Cannabis+sativa%27%27+L.+%28Cannabacaea%29&rft.jtitle=American+Journal+of+Botany&rft.aulast=Turner%2C+J.C.%3B+Hemphill%2C+J.K.%3B+Mahlberg%2C+P.G.&rft.au=Turner%2C+J.C.%3B+Hemphill%2C+J.K.%3B+Mahlberg%2C+P.G.&rft.date=1978&rft.volume=65&rft.issue=10&rft.pages=1103%E2%80%9306&rft_id=info:doi\/10.1002%2Fj.1537-2197.1978.tb06177.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Flores-SanchezSecond08-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Flores-SanchezSecond08_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Flores-Sanchez, I.J.; Verpoorte, R. (2008). \"Secondary metabolism in cannabis\". <i>Phytochemistry Reviews<\/i> <b>7<\/b> (3): 615\u2013639. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11101-008-9094-4\" target=\"_blank\">10.1007\/s11101-008-9094-4<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Secondary+metabolism+in+cannabis&rft.jtitle=Phytochemistry+Reviews&rft.aulast=Flores-Sanchez%2C+I.J.%3B+Verpoorte%2C+R.&rft.au=Flores-Sanchez%2C+I.J.%3B+Verpoorte%2C+R.&rft.date=2008&rft.volume=7&rft.issue=3&rft.pages=615%E2%80%93639&rft_id=info:doi\/10.1007%2Fs11101-008-9094-4&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FellermeierBiosynth01-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FellermeierBiosynth01_12-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fellermeier, M.; Eisenreich, W.; Bacher, A. et al. (2001). \"Biosynthesis of cannabinoids. Incorporation experiments with (13)C-labeled glucoses\". <i>European Journal of Biochemistry<\/i> <b>268<\/b> (6): 1596-604. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11248677\" target=\"_blank\">11248677<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Biosynthesis+of+cannabinoids.+Incorporation+experiments+with+%2813%29C-labeled+glucoses&rft.jtitle=European+Journal+of+Biochemistry&rft.aulast=Fellermeier%2C+M.%3B+Eisenreich%2C+W.%3B+Bacher%2C+A.+et+al.&rft.au=Fellermeier%2C+M.%3B+Eisenreich%2C+W.%3B+Bacher%2C+A.+et+al.&rft.date=2001&rft.volume=268&rft.issue=6&rft.pages=1596-604&rft_id=info:pmid\/11248677&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DegenhardtChapter17-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DegenhardtChapter17_13-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Degenhardt, F.; Stehle, F.; Kayser, O. (2017). \"Chapter 2: The Biosynthesis of Cannabinoids\". In Preedy, V.R.. <i>Handbook of Cannabis and Related Pathologies<\/i>. Elsevier. pp. 13\u201323. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FB978-0-12-800756-3.00002-8\" target=\"_blank\">10.1016\/B978-0-12-800756-3.00002-8<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780128007563.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+2%3A+The+Biosynthesis+of+Cannabinoids&rft.atitle=Handbook+of+Cannabis+and+Related+Pathologies&rft.aulast=Degenhardt%2C+F.%3B+Stehle%2C+F.%3B+Kayser%2C+O.&rft.au=Degenhardt%2C+F.%3B+Stehle%2C+F.%3B+Kayser%2C+O.&rft.date=2017&rft.pages=pp.%26nbsp%3B13%E2%80%9323&rft.pub=Elsevier&rft_id=info:doi\/10.1016%2FB978-0-12-800756-3.00002-8&rft.isbn=9780128007563&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShoyamaBiosynth84-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShoyamaBiosynth84_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShoyamaBiosynth84_14-1\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shoyama, Y.; Hirano, H.; Nishioka, I. (1984). \"Biosynthesis of propyl cannabinoid acid and its biosynthetic relationship with pentyl and methyl cannabinoid acids\". <i>Phytochemistry<\/i> <b>23<\/b> (9): 1909\u201312. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0031-9422%2800%2984939-0\" target=\"_blank\">10.1016\/S0031-9422(00)84939-0<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Biosynthesis+of+propyl+cannabinoid+acid+and+its+biosynthetic+relationship+with+pentyl+and+methyl+cannabinoid+acids&rft.jtitle=Phytochemistry&rft.aulast=Shoyama%2C+Y.%3B+Hirano%2C+H.%3B+Nishioka%2C+I.&rft.au=Shoyama%2C+Y.%3B+Hirano%2C+H.%3B+Nishioka%2C+I.&rft.date=1984&rft.volume=23&rft.issue=9&rft.pages=1909%E2%80%9312&rft_id=info:doi\/10.1016%2FS0031-9422%2800%2984939-0&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TuriMeta15-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TuriMeta15_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-TuriMeta15_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-TuriMeta15_15-2\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-TuriMeta15_15-3\">15.3<\/a><\/sup> <sup><a href=\"#cite_ref-TuriMeta15_15-4\">15.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Turi, C.E.; Finley, J.; Shipley, P.R. et al. (2015). \"Metabolomics for phytochemical discovery: development of statistical approaches using a cranberry model system\". <i>Journal of Natural Products<\/i> <b>78<\/b> (4): 953-66. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fnp500667z\" target=\"_blank\">10.1021\/np500667z<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25751407\" target=\"_blank\">25751407<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Metabolomics+for+phytochemical+discovery%3A+development+of+statistical+approaches+using+a+cranberry+model+system&rft.jtitle=Journal+of+Natural+Products&rft.aulast=Turi%2C+C.E.%3B+Finley%2C+J.%3B+Shipley%2C+P.R.+et+al.&rft.au=Turi%2C+C.E.%3B+Finley%2C+J.%3B+Shipley%2C+P.R.+et+al.&rft.date=2015&rft.volume=78&rft.issue=4&rft.pages=953-66&rft_id=info:doi\/10.1021%2Fnp500667z&rft_id=info:pmid\/25751407&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WorleyMulti13-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WorleyMulti13_16-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Worley, B.; Powers, R. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4465187\" target=\"_blank\">\"Multivariate Analysis in Metabolomics\"<\/a>. <i>Current Metabolomics<\/i> <b>1<\/b> (1): 92\u2013107. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2174%2F2213235X11301010092\" target=\"_blank\">10.2174\/2213235X11301010092<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4465187\/\" target=\"_blank\">PMC4465187<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26078916\" target=\"_blank\">26078916<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4465187\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4465187<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multivariate+Analysis+in+Metabolomics&rft.jtitle=Current+Metabolomics&rft.aulast=Worley%2C+B.%3B+Powers%2C+R.&rft.au=Worley%2C+B.%3B+Powers%2C+R.&rft.date=2013&rft.volume=1&rft.issue=1&rft.pages=92%E2%80%93107&rft_id=info:doi\/10.2174%2F2213235X11301010092&rft_id=info:pmc\/PMC4465187&rft_id=info:pmid\/26078916&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4465187&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HagelPlant08-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HagelPlant08_17-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hagel, J.M.; Facchini, P.J. (2008). \"Plant metabolomics: analytical platforms and integration with functional genomics\". <i>Phytochemistry Reviews<\/i> <b>7<\/b> (3): 479\u2013497. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11101-007-9086-9\" target=\"_blank\">10.1007\/s11101-007-9086-9<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Plant+metabolomics%3A+analytical+platforms+and+integration+with+functional+genomics&rft.jtitle=Phytochemistry+Reviews&rft.aulast=Hagel%2C+J.M.%3B+Facchini%2C+P.J.&rft.au=Hagel%2C+J.M.%3B+Facchini%2C+P.J.&rft.date=2008&rft.volume=7&rft.issue=3&rft.pages=479%E2%80%93497&rft_id=info:doi\/10.1007%2Fs11101-007-9086-9&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrownPhyto12-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrownPhyto12_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brown, P.N;. Murch, S.J.; Shipley, P. (2012). \"Phytochemical diversity of cranberry (Vaccinium macrocarpon Aiton) cultivars by anthocyanin determination and metabolomic profiling with chemometric analysis\". <i>Journal of Agricultural and Food Chemistry<\/i> <b>60<\/b> (1): 261\u201371. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fjf2033335\" target=\"_blank\">10.1021\/jf2033335<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22148867\" target=\"_blank\">22148867<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Phytochemical+diversity+of+cranberry+%28Vaccinium+macrocarpon+Aiton%29+cultivars+by+anthocyanin+determination+and+metabolomic+profiling+with+chemometric+analysis&rft.jtitle=Journal+of+Agricultural+and+Food+Chemistry&rft.aulast=Brown%2C+P.N%3B.+Murch%2C+S.J.%3B+Shipley%2C+P.&rft.au=Brown%2C+P.N%3B.+Murch%2C+S.J.%3B+Shipley%2C+P.&rft.date=2012&rft.volume=60&rft.issue=1&rft.pages=261%E2%80%9371&rft_id=info:doi\/10.1021%2Fjf2033335&rft_id=info:pmid\/22148867&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScherlingMeta10-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScherlingMeta10_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scherling, C.; Roscher, C.; Giavalisco, P. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2935349\" target=\"_blank\">\"Metabolomics unravel contrasting effects of biodiversity on the performance of individual plant species\"<\/a>. <i>PLoS One<\/i> <b>5<\/b> (9): e12569. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0012569\" target=\"_blank\">10.1371\/journal.pone.0012569<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2935349\/\" target=\"_blank\">PMC2935349<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20830202\" target=\"_blank\">20830202<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2935349\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2935349<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Metabolomics+unravel+contrasting+effects+of+biodiversity+on+the+performance+of+individual+plant+species&rft.jtitle=PLoS+One&rft.aulast=Scherling%2C+C.%3B+Roscher%2C+C.%3B+Giavalisco%2C+P.+et+al.&rft.au=Scherling%2C+C.%3B+Roscher%2C+C.%3B+Giavalisco%2C+P.+et+al.&rft.date=2010&rft.volume=5&rft.issue=9&rft.pages=e12569&rft_id=info:doi\/10.1371%2Fjournal.pone.0012569&rft_id=info:pmc\/PMC2935349&rft_id=info:pmid\/20830202&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2935349&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MudgeLeaner17-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MudgeLeaner17_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-MudgeLeaner17_20-1\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mudge, E.M.; Murch, S.J.; Brown, P.N. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395585\" target=\"_blank\">\"Leaner and greener analysis of cannabinoids\"<\/a>. <i>Analytical and Bioanalytical Chemistry<\/i> <b>409<\/b> (12): 3153\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00216-017-0256-3\" target=\"_blank\">10.1007\/s00216-017-0256-3<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5395585\/\" target=\"_blank\">PMC5395585<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28233028\" target=\"_blank\">28233028<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395585\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395585<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Leaner+and+greener+analysis+of+cannabinoids&rft.jtitle=Analytical+and+Bioanalytical+Chemistry&rft.aulast=Mudge%2C+E.M.%3B+Murch%2C+S.J.%3B+Brown%2C+P.N.&rft.au=Mudge%2C+E.M.%3B+Murch%2C+S.J.%3B+Brown%2C+P.N.&rft.date=2017&rft.volume=409&rft.issue=12&rft.pages=3153%E2%80%9363&rft_id=info:doi\/10.1007%2Fs00216-017-0256-3&rft_id=info:pmc\/PMC5395585&rft_id=info:pmid\/28233028&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5395585&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrofCanna18-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GrofCanna18_21-0\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-GrofCanna18_21-1\">21.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grof, C.P.L. (2018). \"Cannabis, from plant to pill\". <i>British Journal of Clinical Pharmacology<\/i> <b>84<\/b> (11): 2463\u201367. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fbcp.13618\" target=\"_blank\">10.1111\/bcp.13618<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis%2C+from+plant+to+pill&rft.jtitle=British+Journal+of+Clinical+Pharmacology&rft.aulast=Grof%2C+C.P.L.&rft.au=Grof%2C+C.P.L.&rft.date=2018&rft.volume=84&rft.issue=11&rft.pages=2463%E2%80%9367&rft_id=info:doi\/10.1111%2Fbcp.13618&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AndreCanna16-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AndreCanna16_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Andre, C.M.; Hausman, J.F.; Guerriero, G. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396\" target=\"_blank\">\"<i>Cannabis sativa<\/i>: The Plant of the Thousand and One Molecules\"<\/a>. <i>Frontiers in Plant Science<\/i> <b>7<\/b>: 19. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffpls.2016.00019\" target=\"_blank\">10.3389\/fpls.2016.00019<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4740396\/\" target=\"_blank\">PMC4740396<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26870049\" target=\"_blank\">26870049<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%27%27Cannabis+sativa%27%27%3A+The+Plant+of+the+Thousand+and+One+Molecules&rft.jtitle=Frontiers+in+Plant+Science&rft.aulast=Andre%2C+C.M.%3B+Hausman%2C+J.F.%3B+Guerriero%2C+G.&rft.au=Andre%2C+C.M.%3B+Hausman%2C+J.F.%3B+Guerriero%2C+G.&rft.date=2016&rft.volume=7&rft.pages=19&rft_id=info:doi\/10.3389%2Ffpls.2016.00019&rft_id=info:pmc\/PMC4740396&rft_id=info:pmid\/26870049&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4740396&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PiomelliTheCanna16-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PiomelliTheCanna16_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-PiomelliTheCanna16_23-1\">23.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Piomelli, D.; Russo, E.B. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603\" target=\"_blank\">\"The <i>Cannabis sativa<\/i> Versus <i>Cannabis indica<\/i> Debate: An Interview with Ethan Russo, MD\"<\/a>. <i>Cannabis and Cannabinoid Research<\/i> <b>1<\/b> (1): 44\u20136. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fcan.2015.29003.ebr\" target=\"_blank\">10.1089\/can.2015.29003.ebr<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5576603\/\" target=\"_blank\">PMC5576603<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28861479\" target=\"_blank\">28861479<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+%27%27Cannabis+sativa%27%27+Versus+%27%27Cannabis+indica%27%27+Debate%3A+An+Interview+with+Ethan+Russo%2C+MD&rft.jtitle=Cannabis+and+Cannabinoid+Research&rft.aulast=Piomelli%2C+D.%3B+Russo%2C+E.B.&rft.au=Piomelli%2C+D.%3B+Russo%2C+E.B.&rft.date=2016&rft.volume=1&rft.issue=1&rft.pages=44%E2%80%936&rft_id=info:doi\/10.1089%2Fcan.2015.29003.ebr&rft_id=info:pmc\/PMC5576603&rft_id=info:pmid\/28861479&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5576603&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SextonEval18-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SextonEval18_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sexton, M.; Shelton, K.; Haley, P. et al. (2018). \"Evaluation of Cannabinoid and Terpenoid Content: Cannabis Flower Compared to Supercritical CO2 Concentrate\". <i>Planta Medica<\/i> <b>84<\/b> (4): 234-241. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1055%2Fs-0043-119361\" target=\"_blank\">10.1055\/s-0043-119361<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28926863\" target=\"_blank\">28926863<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+of+Cannabinoid+and+Terpenoid+Content%3A+Cannabis+Flower+Compared+to+Supercritical+CO2+Concentrate&rft.jtitle=Planta+Medica&rft.aulast=Sexton%2C+M.%3B+Shelton%2C+K.%3B+Haley%2C+P.+et+al.&rft.au=Sexton%2C+M.%3B+Shelton%2C+K.%3B+Haley%2C+P.+et+al.&rft.date=2018&rft.volume=84&rft.issue=4&rft.pages=234-241&rft_id=info:doi\/10.1055%2Fs-0043-119361&rft_id=info:pmid\/28926863&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-deMeijerTheInher03-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-deMeijerTheInher03_25-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">de Meijer, E.P.; Bagatta, M.; Carboni, A. et al. (2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421\" target=\"_blank\">\"The inheritance of chemical phenotype in <i>Cannabis sativa<\/i> L.\"<\/a>. <i>Genetics<\/i> <b>163<\/b> (1): 335\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1462421\/\" target=\"_blank\">PMC1462421<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12586720\" target=\"_blank\">12586720<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+inheritance+of+chemical+phenotype+in+%27%27Cannabis+sativa%27%27+L.&rft.jtitle=Genetics&rft.aulast=de+Meijer%2C+E.P.%3B+Bagatta%2C+M.%3B+Carboni%2C+A.+et+al.&rft.au=de+Meijer%2C+E.P.%3B+Bagatta%2C+M.%3B+Carboni%2C+A.+et+al.&rft.date=2003&rft.volume=163&rft.issue=1&rft.pages=335%E2%80%9346&rft_id=info:pmc\/PMC1462421&rft_id=info:pmid\/12586720&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1462421&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HazekampCanna12-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HazekampCanna12_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hazekamp, A.; Fischedick, J.T. (2012). \"Cannabis - From cultivar to chemovar\". <i>Drug testing and analysis<\/i> <b>4<\/b> (7\u20138): 660\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fdta.407\" target=\"_blank\">10.1002\/dta.407<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22362625\" target=\"_blank\">22362625<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+-+From+cultivar+to+chemovar&rft.jtitle=Drug+testing+and+analysis&rft.aulast=Hazekamp%2C+A.%3B+Fischedick%2C+J.T.&rft.au=Hazekamp%2C+A.%3B+Fischedick%2C+J.T.&rft.date=2012&rft.volume=4&rft.issue=7%E2%80%938&rft.pages=660%E2%80%937&rft_id=info:doi\/10.1002%2Fdta.407&rft_id=info:pmid\/22362625&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MeyerPatterns12-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MeyerPatterns12_27-0\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-MeyerPatterns12_27-1\">27.1<\/a><\/sup> <sup><a href=\"#cite_ref-MeyerPatterns12_27-2\">27.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Meyer, R.S.; DuVal, A.E.; Jensen, H.R. (2012). \"Patterns and processes in crop domestication: An historical review and quantitative analysis of 203 global food crops\". <i>New Phytologist<\/i> <b>196<\/b> (1): 29\u201348. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1469-8137.2012.04253.x\" target=\"_blank\">10.1111\/j.1469-8137.2012.04253.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patterns+and+processes+in+crop+domestication%3A+An+historical+review+and+quantitative+analysis+of+203+global+food+crops&rft.jtitle=New+Phytologist&rft.aulast=Meyer%2C+R.S.%3B+DuVal%2C+A.E.%3B+Jensen%2C+H.R.&rft.au=Meyer%2C+R.S.%3B+DuVal%2C+A.E.%3B+Jensen%2C+H.R.&rft.date=2012&rft.volume=196&rft.issue=1&rft.pages=29%E2%80%9348&rft_id=info:doi\/10.1111%2Fj.1469-8137.2012.04253.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McKeyTheEvo10-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McKeyTheEvo10_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-McKeyTheEvo10_28-1\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McKey, D.; Elias, M.; Pujol, B. et al. (2010). \"The evolutionary ecology of clonally propagated domesticated plants\". <i>New Phytologist<\/i> <b>186<\/b> (2): 318\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1469-8137.2010.03210.x\" target=\"_blank\">10.1111\/j.1469-8137.2010.03210.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20202131\" target=\"_blank\">20202131<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+evolutionary+ecology+of+clonally+propagated+domesticated+plants&rft.jtitle=New+Phytologist&rft.aulast=McKey%2C+D.%3B+Elias%2C+M.%3B+Pujol%2C+B.+et+al.&rft.au=McKey%2C+D.%3B+Elias%2C+M.%3B+Pujol%2C+B.+et+al.&rft.date=2010&rft.volume=186&rft.issue=2&rft.pages=318%E2%80%9332&rft_id=info:doi\/10.1111%2Fj.1469-8137.2010.03210.x&rft_id=info:pmid\/20202131&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElSohlyPot00-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ElSohlyPot00_29-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">ElSohly, M.A.; Ross, S.A.; Mehmedic, Z. et al. (2000). \"Potency trends of delta9-THC and other cannabinoids in confiscated marijuana from 1980-1997\". <i>Journal of Forensic Sciences<\/i> <b>45<\/b> (1): 24\u201330. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10641915\" target=\"_blank\">10641915<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Potency+trends+of+delta9-THC+and+other+cannabinoids+in+confiscated+marijuana+from+1980-1997&rft.jtitle=Journal+of+Forensic+Sciences&rft.aulast=ElSohly%2C+M.A.%3B+Ross%2C+S.A.%3B+Mehmedic%2C+Z.+et+al.&rft.au=ElSohly%2C+M.A.%3B+Ross%2C+S.A.%3B+Mehmedic%2C+Z.+et+al.&rft.date=2000&rft.volume=45&rft.issue=1&rft.pages=24%E2%80%9330&rft_id=info:pmid\/10641915&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MehmedicPot10-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MehmedicPot10_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mehmedic, Z.; Chandra, S.; Slade, D. et al. (2010). \"Potency trends of \u03949-THC and other cannabinoids in confiscated cannabis preparations from 1993 to 2008\". <i>Journal of Forensic Sciences<\/i> <b>55<\/b> (5): 1209\u201317. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1556-4029.2010.01441.x\" target=\"_blank\">10.1111\/j.1556-4029.2010.01441.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20487147\" target=\"_blank\">20487147<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Potency+trends+of+%CE%949-THC+and+other+cannabinoids+in+confiscated+cannabis+preparations+from+1993+to+2008&rft.jtitle=Journal+of+Forensic+Sciences&rft.aulast=Mehmedic%2C+Z.%3B+Chandra%2C+S.%3B+Slade%2C+D.+et+al.&rft.au=Mehmedic%2C+Z.%3B+Chandra%2C+S.%3B+Slade%2C+D.+et+al.&rft.date=2010&rft.volume=55&rft.issue=5&rft.pages=1209%E2%80%9317&rft_id=info:doi\/10.1111%2Fj.1556-4029.2010.01441.x&rft_id=info:pmid\/20487147&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FiehnMeta02-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FiehnMeta02_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fiehn, O. (2002). \"Metabolomics--The link between genotypes and phenotypes\". <i>Plant Molecular Biology<\/i> <b>48<\/b> (1\u20132): 155\u201371. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11860207\" target=\"_blank\">11860207<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Metabolomics--The+link+between+genotypes+and+phenotypes&rft.jtitle=Plant+Molecular+Biology&rft.aulast=Fiehn%2C+O.&rft.au=Fiehn%2C+O.&rft.date=2002&rft.volume=48&rft.issue=1%E2%80%932&rft.pages=155%E2%80%9371&rft_id=info:pmid\/11860207&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithIdent97-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmithIdent97_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smith, R.M. (1997). \"Identification of Butyl Cannabinoids in Marijuanas\". <i>Journal of Forensic Sciences<\/i> <b>42<\/b> (4): 610\u201318. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1520%2FJFS14173J\" target=\"_blank\">10.1520\/JFS14173J<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identification+of+Butyl+Cannabinoids+in+Marijuanas&rft.jtitle=Journal+of+Forensic+Sciences&rft.aulast=Smith%2C+R.M.&rft.au=Smith%2C+R.M.&rft.date=1997&rft.volume=42&rft.issue=4&rft.pages=610%E2%80%9318&rft_id=info:doi\/10.1520%2FJFS14173J&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmallLetter73-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmallLetter73_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Small, E.; Beckstead, H.D. (1973). \"Letter: Cannabinoid phenotypes in <i>Cannabis sativa<\/i>\". <i>Nature<\/i> <b>245<\/b> (5421): 147\u20138. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/4582664\" target=\"_blank\">4582664<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Letter%3A+Cannabinoid+phenotypes+in+%27%27Cannabis+sativa%27%27&rft.jtitle=Nature&rft.aulast=Small%2C+E.%3B+Beckstead%2C+H.D.&rft.au=Small%2C+E.%3B+Beckstead%2C+H.D.&rft.date=1973&rft.volume=245&rft.issue=5421&rft.pages=147%E2%80%938&rft_id=info:pmid\/4582664&rfr_id=info:sid\/en.wikipedia.org:Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165448\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.893 seconds\nReal time usage: 0.938 seconds\nPreprocessor visited node count: 27100\/1000000\nPreprocessor generated node count: 36797\/1000000\nPost\u2010expand include size: 204239\/2097152 bytes\nTemplate argument size: 62780\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 872.755 1 - -total\n 82.30% 718.312 1 - Template:Reflist\n 72.73% 634.738 33 - Template:Citation\/core\n 69.03% 602.493 30 - Template:Cite_journal\n 11.15% 97.352 1 - Template:Infobox_journal_article\n 10.82% 94.429 1 - Template:Infobox\n 9.67% 84.438 57 - Template:Citation\/identifier\n 7.50% 65.419 3 - Template:Cite_book\n 6.94% 60.556 80 - Template:Infobox\/row\n 3.92% 34.186 122 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11055-0!*!0!!en!5!* and timestamp 20190701165447 and revision id 35665\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome\">https:\/\/www.limswiki.org\/index.php\/Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","ec56e6ba701339e54fcca672efb020ae_images":["https:\/\/www.limswiki.org\/images\/c\/c4\/Fig1_Mudge_ScientificReports2018_8.png","https:\/\/www.limswiki.org\/images\/6\/69\/Fig2_Mudge_ScientificReports2018_8.png","https:\/\/www.limswiki.org\/images\/1\/1f\/Fig3_Mudge_ScientificReports2018_8.png","https:\/\/www.limswiki.org\/images\/2\/23\/Fig4_Mudge_ScientificReports2018_8.png","https:\/\/www.limswiki.org\/images\/b\/b4\/Fig5_Mudge_ScientificReports2018_8.png","https:\/\/www.limswiki.org\/images\/0\/09\/Fig6_Mudge_ScientificReports2018_8.png"],"ec56e6ba701339e54fcca672efb020ae_timestamp":1562000087,"500951db854008d04f1cf96423731db5_type":"article","500951db854008d04f1cf96423731db5_title":"The cannabinoid content of legal cannabis in Washington State varies systematically across testing facilities and popular consumer products (Jikomes and Zoorob 2018)","500951db854008d04f1cf96423731db5_url":"https:\/\/www.limswiki.org\/index.php\/Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products","500951db854008d04f1cf96423731db5_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:The cannabinoid content of legal cannabis in Washington State varies systematically across testing facilities and popular consumer products\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nThe cannabinoid content of legal cannabis in Washington State varies\r\nsystematically across testing facilities and popular consumer productsJournal\n \nScientific ReportsAuthor(s)\n \nJikomes, Nick; Zoorob, MichaelAuthor affiliation(s)\n \nLeafly Holdings, Harvard UniversityPrimary contact\n \nEmail: Contact author via journalYear published\n \n2018Volume and issue\n \n8Page(s)\n \n4519DOI\n \n10.1038\/s41598-018-22755-2ISSN\n \n2045-2322Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/www.nature.com\/articles\/s41598-018-22755-2Download\n \nhttps:\/\/www.nature.com\/articles\/s41598-018-22755-2.pdf (PDF)\n\n\n\n\n \n This article contains rendered mathematical formulae. You may require the TeX All the Things plugin for Chrome or the Native MathML add-on and fonts for Firefox if they don't render properly for you. \n\n\nContents\n\n1 Abstract \n2 Introduction \n3 Results \n\n3.1 The basic chemotype landscape of commercial cannabis \n3.2 THC and CBD measurements vary widely across testing laboratories \n3.3 Interlab differences persist after controlling for plausible confounds \n\n3.3.1 Low-level cannabinoid measurements vary widely across laboratories \n\n\n3.4 Changes in THC content of commercial cannabis products over time \n3.5 THC content across popular commercial categories: indica, sativa, and hybrid \n3.6 Cannabinoid variation within and across popular commercial strain names \n\n\n4 Discussion \n5 Methods \n\n5.1 Data \n\n5.1.1 Washington I-502 cannabis test data \n5.1.2 Calculation of THC and CBD levels \n5.1.3 Leafly data \n\n\n5.2 Analytic methods \n\n5.2.1 Matching strain names \n5.2.2 Chemotype cutoffs \n5.2.3 Statistical significance and effect size for very large samples \n5.2.4 Cannabinoid inflation regression models \n5.2.5 THC and CBD variation across popular commercial strains \n5.2.6 Strain category cannabinoid estimation \n\n\n\n\n6 Electronic supplementary material \n7 Acknowledgements \n\n7.1 Author contributions \n7.2 Competing interests \n7.3 Data availability \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nThe majority of adults in the U.S. now have state-legal access to medical or recreational cannabis products, despite their federal prohibition. Given the wide array of pharmacologically active compounds in these products, it is essential that their biochemical profile is measured and reported to consumers, which requires accurate laboratory testing. However, no universal standards for laboratory testing protocols currently exist, and there is controversy as to whether all reported results are legitimate. To investigate these concerns, we analyzed a publicly available seed-to-sale traceability dataset from Washington State containing measurements of the cannabinoid content of legal cannabis products from state-certified laboratories. Consistent with previous work, we found that commercial Cannabis strains fall into three broad chemotypes defined by the tetrahydrocannabinol:cannabidiol (THC:CBD) ratio. Moreover, we documented systematic differences in the cannabinoid content reported by different laboratories, relative stability in cannabinoid levels of commercial flower and concentrates over time, and differences between popular commercial strains. Importantly, interlab differences in cannabinoid reporting persisted even after controlling for plausible confounds. Our results underscore the need for standardized laboratory methodologies in the legal cannabis industry and provide a framework for quantitatively assessing laboratory quality.\n\nIntroduction \nFor millennia, Cannabis has been cultivated for medicinal, recreational, and industrial purposes.[1] Despite mounting evidence for the legitimate medical utility of cannabis products and their principal psychoactive constituents[2][3], they remain classified as Schedule I controlled substances by the U.S. federal government. Nonetheless, public opinion on legal cannabis has changed dramatically in recent years[4], and a majority of U.S. states now allow legal access to medical cannabis for approved patients, with several states also allowing recreational adult-use.[5][6] This dynamic legal landscape has given rise to a rapidly growing legal cannabis industry that offers a wide variety of products to consumers.\nBecause the core product of this burgeoning industry contains multiple compounds with psychoactive and medicinal properties[7], it is imperative that the major biochemical constituents of cannabis are accurately quantified, and the results made accessible to consumers. Because recreational cannabis products may differ substantially from cannabis grown for federally-sanctioned research[8] or found on the black market[9], there is a particular need to study the commercial cannabis being consumed today by millions of adults in states allowing legal adult-use consumption.\nThe adoption of universal industry testing standards will be crucial for comparing data across the many existing testing laboratories. However, standardized procedures have yet to be adopted, and controversy exists about whether all laboratories are accurately measuring and reporting cannabinoid content.[10] Most of these labs were not established quality control labs with a track record of testing food or pharmaceutical products, but rather started specifically to focus on cannabis products. At present, there is limited published data[8] on the content of commercial cannabis products in the U.S., including quantification of potential differences in the measurements reported across these testing laboratories. Reliable testing data will also shed light on questions important to consumers and regulators, such as whether cannabinoid levels are changing over time or differ systematically between commercial products.\nTo investigate these concerns, we analyzed a large dataset from Washington State\u2019s seed-to-sale traceability system. This dataset comprises hundreds of thousands of measurements of the principal cannabinoids in commercial cannabis, including tetrahydrocannabinol (THC) and cannabidiol (CBD). These measurements are available for commercial products tested across all state-licensed laboratories since 2014, which allowed us to assess the cannabinoid composition of commercial products between laboratories, over time, and across strains.\n\nResults \nThe basic chemotype landscape of commercial cannabis \nCannabis likely evolved in Central Asia, and landraces native to regions including Afghanistan, Pakistan, India and China[11] have been found to fall into three general chemotypes based on genetically-constrained THC:CBD ratios.[12][13] Consistent with previous work in landraces and commercial Dutch Cannabis[13][14], we found that commercial Cannabis grown in Washington also conforms to this pattern (Fig. 1a\u2013c). Unlike landraces, which are more likely to fall into the chemotype III (CBD-dominant) category and generally display lower overall levels of total THC[13], most commercial Cannabis falls into the chemotype I category, characterized by relatively high total THC and low total CBD levels (Fig. 1d\u2013f; see \"Methods\" section at the end for definition of total THC and CBD levels). While studying the chemotype landscape of these commercial samples, we observed striking differences in THC:CBD distributions across laboratories for both flower (Fig. 1d\u2013f, Figure S1) and concentrates (Figure S2). This prompted us to examine interlab differences in more detail. In particular, we wished to assess whether this variation stemmed from intrinsic (e.g., methodological) differences between laboratories or from heterogeneity in the products submitted to those labs.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 1 The THC:CBD ratio defines three broad chemotypes of commercial cannabis flower measured by testing labs in Washington. Left column: Scatterplots of total THC vs. total CBD levels for cannabis flower. Right column: Histograms showing the THC:CBD ratio on a log scale and indicating the proportion of flower samples for each chemotype. Data are displayed for measurements batched across all Labs A-F (panels a-b; n\u2009=\u2009175,136), for the lab reporting the lowest mean total THC levels (Lab A; panels c-d; n\u2009=\u200962,719), and the lab reporting the highest mean total THC levels (Lab F; panels e-f; n=26,664). Histograms for each of the six labs contributing to batched data in panels a-b are shown in Figure S1. Panels a and c were subsampled to n=50,000 for visualization purposes.\n\n\n\nTHC and CBD measurements vary widely across testing laboratories \nTo compare cannabinoid measurements across labs, we looked at distributions of total THC and CBD levels for the six largest laboratories by data volume for different chemotypes and product categories. These labs, referred to henceforth as labs A-F, are Confidence Analytics (Lab A), Analytical 360 (Lab B), Green Grower Labs (Lab C), Integrity Labs (Lab D), Testing Technologies (Lab E), and Peak Analytics (Lab F). We observed differences in reported values of both THC and CBD (Fig. 2). For example, the median total THC content for chemotype I flower products ranged from 17.7% to 23.2% between the labs reporting the lowest and highest THC levels, respectively (Fig. 2a; labs A-F ordered from lowest to highest median reported THC levels). Pairwise differences in mean THC content between labs were statistically significant (p\u2009<\u20090.001 for each pairwise comparison in Fig. 2a, two-sided t-test). To quantify the magnitude of differences between labs, we calculated the effect sizes of pairwise differences using two metrics: Cohen\u2019s d, the standardized difference between two means[15], and a \u201cCommon Language\u201d (CL) effect size, the probability that a random value from one sample will be greater than a random value from the other[16] (Fig. 2b; see Analytical Methods).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 2 Total THC and CBD Measurements Differ Between Labs Across Chemotypes and Product Categories. Left column: Violin plots showing the distribution of total THC or CBD levels across labs A-F. Black lines denote median values, which are printed below the x-axis for each lab. Right column: Effect size matrices displaying the effect size of pairwise differences in distributions between labs. Matrices are color-coded according to one measure of effect size (Cohen\u2019s d), and a second measure (Common Language) is printed for each comparison.\n\n\n\nCalculating effect sizes allows a more intuitive assessment of the magnitude of interlab differences, especially when very large sample sizes allow even trivial differences between means to reach statistical significance. For example, mean THC levels of the chemotype I flower for Lab B and Lab A were 18.4% and 17.7%, respectively (Fig. 2a and b). While this difference was highly significant due to the large sample sizes, the effect size was small (d\u2009=\u20090.13; see the \"Methods\" section). The common language effect size (CL) for this comparison was 0.54, indicating a 54% chance that a random THC measurement from Lab B will be larger than a random measurement from Lab A. In contrast, when comparing Lab F to Lab A, which reported the highest mean THC levels, the effect size was considerably larger (d\u2009=\u20091.28, CL\u2009=\u20090.82).\nWe observed a similar pattern when comparing CBD measurements across labs for chemotype II and III flower samples (Fig. 2c and d) and THC levels for concentrates (Fig. 2e and f). The labs reporting the highest levels of THC for chemotype I flower products also reported the highest levels of CBD for other flower chemotypes and THC levels for concentrates (Fig. 2), indicating a systematic tendency for certain labs to report higher levels of cannabinoids across chemotypes and product categories. This may be explained by differences in laboratory protocols. While most labs report using high-performance liquid chromatography (HPLC) to detect cannabinoids, the details of each protocol likely differ. Alternatively, interlab differences may be driven by labs receiving distinct sets of cannabis products for testing.\n\nInterlab differences persist after controlling for plausible confounds \nTo investigate potential determinants of interlab differences, we quantified the average cannabinoid levels reported by each lab after accounting for strain name, the producer-processor submitting samples for testing, and time of measurement (see \"Methods\" section). Four separate regression models were estimated: (1) THC levels in chemotype I flower products (n\u2009=\u2009161,933); (2) THC levels in chemotype I concentrate products (n\u2009=\u200933,888); (3) CBD levels in chemotype II and III flower products (n\u2009=\u20094,661); and (4) CBD levels in chemotype II and III concentrate products (n\u2009=\u20092,156). Large interlab variability in reported THC and CBD levels persisted across product categories after controlling for these factors (Fig. 3). Differences were observed for both flower (Fig. 3a,c) and concentrates (Fig. 3b,d). For chemotype I flower, the average adjusted total THC level for Peak Analytics (~23%) was significantly higher (p\u2009<\u20090.001; Wald test) than all other labs (Fig. 3a).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 3 THC and CBD Levels Vary Between Labs After Controlling for Plausible Confounds. Average predicted values (+\/\u2212 99% confidence intervals) are shown, by lab, for (a) THC levels in chemotype I flower products (n\u2009=\u2009161,933); (b) THC levels in chemotype I concentrate products (n\u2009=\u200933,888); (c) CBD levels in chemotype II and III flower products (n\u2009=\u20094,661); and (d) CBD levels in chemotype II and III concentrate products (n\u2009=\u20092,156) after adjusting for grower, strain-name, and time of measurement. Predicted values were generated from fixed-effects regressions with cluster-robust standard errors (see Methods).\n\n\n\nFor chemotype I concentrates, Lab F\u2019s average reported total THC (~75%) exceeded all other labs, including the lab reporting the second-highest average total THC (Lab E, ~70%). For total CBD levels in chemotype II and chemotype III flower, Lab F again reported the largest mean quantity, at about 13%, significantly higher (p\u2009<\u20090.01) than all other labs (Fig. 3c). For chemotype II and chemotype III concentrates, Lab F\u2019s average products reported the highest CBD, but these estimates were uncertain due to the relatively small sample size (Fig. 3d). Overall, these results suggest that the observed differences between laboratories cannot be explained by differences in the producers, product types, or strain names of the samples being processed by each lab.\n\nLow-level cannabinoid measurements vary widely across laboratories \nExamination of THC:CBD distributions across laboratories indicated substantial variation in their propensity to report chemotype I strains with low total CBD levels (Figure S1, far right bins). To investigate this further, we plotted the density of chemotype I flower products with less than 1% CBD by dry weight (Fig. 4). The shape of these distributions varied somewhat across labs, likely due to methodological differences determining their limit of quantification (LOQ). Similar to what we observed in the THC:CBD histograms (Fig. 1b), these density plots indicate that differences exist between labs\u2019 propensity to detect low levels of CBD in chemotype I flower; they tend to display local maxima near 0.1%, which is the LOQ most labs report for cannabinoids.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 4 Labs differ in the propensity to detect low levels of CBD in chemotype I flower. (a) Kernel density plots of each lab\u2019s distribution of total CBD levels below 1.0% dry weight for chemotype I flower (y-axis scaled to one). Most labs show a local maximum near 0.1% total CBD, which is a commonly reported LOQ. (b) Fraction of chemotype I flower with total CBD levels below 0.1% dry weight. Bars indicate proportions +\/\u2212 95% CI for a binomial proportion. (c) Effect size matrix indicating the magnitude of interlab differences shown in panel B. Effect size is quantified as Cohen\u2019s h (see \"Methods\" section).\n\n\n\nTo further quantify these differences, we compared the proportion of chemotype I flower having <0.1% total CBD across laboratories. There were dramatic differences between labs (Fig. 4b), with some reporting substantially more chemotype I flower with total CBD <0.1% than others. The volume of data caused even tiny interlab differences to reach high levels of statistical significance (p\u2009<\u20090.001 for all pairwise comparisons, except Lab E vs. Lab F, Mann-Whitney U test). Thus, we quantified the effect size of these differences by computing Cohen\u2019s h for all pairwise comparisons (Fig. 4c; see Analytic Methods). Many of these interlab differences were of very large effect size (|h| >0.80, and often much greater), confirming that there are substantial differences in labs\u2019 propensity to detect low levels of CBD in chemotype I flower. The analyses so far indicate that cannabinoid inflation and differences in the ability of labs to detect low-level cannabinoids both contribute to systematic differences in their reported measurements.\n\nChanges in THC content of commercial cannabis products over time \nWhile modern commercial strains contain higher THC levels than recreational cannabis from past decades[17][18], it is unclear whether THC levels have continued climbing since Washington permitted adult-use cannabis. Thus, we looked for potential changes in the total THC content of commercial products in recent years. Because our previous analyses revealed systematic interlab variability in cannabinoid measurements, we sought to minimize the potential confound of lab-specific \u201ccannabinoid inflation.\u201d Thus, we quantified cannabinoid levels over time, separately for different subsets of laboratories: the three labs reporting the lowest mean THC levels (low THC reporting, LTR), the three reporting the highest mean THC levels (high THC reporting, HTR), as well as data pooled across laboratories. Figure 5a shows mean THC levels, averaged across labs, for chemotype I flower products from June 2014 through May 2017. While there was an upward trend from 2014 to early 2015, mean THC levels appear to have largely plateaued, with modest fluctuations since 2015 (Fig. 5a). This trend was evident in the pooled data as well as in LTR and HTR labs, although HTR labs showed a steeper increase in total THC levels from 2014 to 2015.\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 5 Mean THC Levels for Chemotype I Flower Products Over Time. (a) Total THC levels over time averaged across all labs or those reporting the highest or lowest mean THC levels. (b) Distribution of THC levels for each year on record for low THC reporting (LTR) labs. (c) Effect size matrix quantifying the mean difference in THC levels across years for LTR labs. (d) Distribution of THC levels for each year for high THC reporting (HTR) labs, and (e) the effect size matrix quantifying the magnitude of yearly differences.\n\n\n\nTo further quantify changes in THC levels over time, we compared total THC levels for each year of data (Fig. 5b and c). Median THC levels for chemotype I flower rose from 2014 to 2015 but changed only slightly between 2015 and 2017. This was true whether we looked at the three LTR labs (Fig. 5b and c) or the three HTR labs (Fig. 5d and e). Again, large sample sizes allowed small differences in mean THC levels to reach statistical significance for all pairwise comparisons (p\u2009<\u20090.001, Mann-Whitney U test), except 2015 to 2016 for the HTR cohort (p\u2009=\u20090.334, Fig. 5d). After 2014, the effect sizes for year-to-year comparisons were small (Fig. 5b; |Cohen\u2019s d| <0.23 for each comparison). Thus, we conclude that there has not been a substantial increase in the THC content of Washington state\u2019s commercial cannabis flower from since 2015, although there were notable differences between LTR and HTR labs. For example, THC distributions from HTR labs were much more skewed (Fig. 5d; skew\u2009=\u2009\u22120.2) than for low-LTR labs (Fig. 5b; skew\u2009=\u20090.06). In addition, the increase in mean THC values from 2014 to 2015 was much larger for HTR than LTR labs (4.5% vs. 2.3%, respectively).\nWe were also interested in whether concentrates have increased in THC levels since 2014, as these products contain a much higher THC concentration. Mean THC levels across labs appeared to be relatively flat from 2014 to 2017 (Figure S3A). There was a small increase in THC levels from 2014 to 2015 for both cohorts of labs (Figure S3), although this was smaller than the increase observed for flower. From 2015 onward, there was a small decrease in mean THC levels for LTR and HTR labs (Figure S3). Thus, we conclude that, since 2015, there has not been a substantial increase in mean THC levels for commercial flower and concentrate products in Washington.\n\nTHC content across popular commercial categories: indica, sativa, and hybrid \nThe vernacular among cannabis users involves a triad of \u201cindica,\u201d \u201csativa,\u201d and \u201chybrid\u201d strains.[19][20] Recreational consumers and popular educational resources often attribute distinctive psychoactive effects to indica and sativa strains[21], while scholars tend to be more skeptical of these claims.[20][22] In landraces, accessions from indica strains have been associated with more THC than sativas[13] but indica and sativa recreational products sold in the Netherlands had similar THC content.[14] The term \u201cstrain,\u201d although widely used, is not a botanically-accepted term for distinguishing plant varieties, and many scholars prefer the term \u201cchemovar\u201d in order to emphasize biochemical differences between specific Cannabis varieties[23] (see \"Discussion\" section). To investigate potential differences in cannabinoid content among commercial strain categories used by consumers, we looked at the distribution of THC content of indica, sativa, and hybrid flower samples in Washington\u2019s commercial market. We matched test results using their producer-given strain name from the I-502 dataset to the Leafly.com strain database to retrieve their popular indica, sativa, or hybrid categorization (see \"Methods\" section). This matching process yielded 166,594 flower results for analysis: 42,711 indica (25.6%), 31,822 sativa (19.1%), and 92,061 hybrid (55.3%) products. While hybrids had higher mean levels of THC compared to indicas and sativas, the distributions of THC content among indicas, sativas, and hybrids overlapped considerably (Fig. 6a\u2013d).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 6 Total THC and CBD Levels Across Popular Consumer Strain Categories for Flower Products. (a) Distribution of THC levels across popular strain categories for chemotype I flower and (b) effect size matrix quantifying the magnitude of differences between them. (c) Distribution of CBD levels across the same categories for chemotype II and III flower and (d) effect size matrix quantifying the magnitude of differences between them.\n\n\n\nTo further quantify differences in THC content, we estimated a bivariate regression model of THC on strain category across all labs. The model indicates that hybrid strains have modestly greater THC content, on average, than either indica or sativa strains (Fig. 6e; hybrid vs indica: 1.22%, p\u2009<\u20090.001; hybrid vs sativa: 0.89%, p\u2009<\u20090.01). The difference in THC between sativa and indica could not be distinguished from sampling variability (sativa vs indica: 0.33%, p\u2009=\u20090.230). Moreover, the indica, sativa, hybrid distinction explained only a tiny fraction of THC variability between flower samples (r2\u2009=\u20090.016), and the differences in mean THC content had modest effect sizes (Hybrid vs Sativa: Cohen\u2019s d\u2009=\u20090.283; Hybrid vs Indica: Cohen\u2019s d\u2009=\u20090.206; Indica vs Sativa: Cohen\u2019s d\u2009=\u2009\u22120.079). An analogous test for variability in CBD content across strain categories among Chemotype II and Chemotype III flower yielded similar results (Fig. 6f; hybrid vs indica: 2.17%, p\u2009<\u20090.01; hybrid vs sativa: 0.90%, p\u2009=\u20090.261; sativa vs indica: 1.26%, p\u2009<\u20090.05) with modest effect sizes (Hybrid vs Indica: Cohen\u2019s d\u2009=\u20090.148, Hybrid vs Sativa: Cohen\u2019s d\u2009=\u20090.068, Sativa vs Indica: Cohen\u2019s d\u2009=\u20090.268).\nImportantly, the above results are sensitive to laboratory measurements. Data from most labs reflect the general pattern of hybrids having somewhat higher THC than indica and sativa, which are very similar. However, performing the same regression solely using flower products from Lab F (n\u2009=\u200922,049), which reports the highest mean THC levels, would not detect the higher average THC levels of hybrid flowers (hybrid vs indica: 0.47%, p\u2009=\u20090.21; hybrid vs sativa: 0.28%, p\u2009=\u20090.425). Repeating the analysis with solely flower products from Lab A (n\u2009=\u200950,610), the lab reporting the lowest mean THC levels, replicates the overall result, with hybrids having slightly higher THC than indica (1.13%, p\u2009<\u20090.001) and sativa (0.80%, p\u2009<\u20090.01).\n\nCannabinoid variation within and across popular commercial strain names \nWhile commercial flower products fall into one of three chemotypes based on their THC:CBD ratio (Fig. 1), we wondered how much THC:CBD ratios varied within and between the popular commercial strain names that flower samples are given. Because the strain names of flower products submitted for laboratory testing are simply given by the producer-processor, they do not guarantee the true identity of the strain. In fact, personal correspondence with industry professionals indicated that we should expect flower samples submitted for testing to be mislabeled to some extent, perhaps due to business motives driving products to be given certain strain names based purely on their popularity and hence potential market value.\nTo visualize differences between samples based on their popular commercial strain names, we plotted the THC:CBD ratio for 23 labeled strains (fifteen Chemotype I strains, three Chemotype II strains, and five Chemotype III strains) based on their consumer popularity (determined by cumulative pageviews on Leafly.com; see \"Methods\" section) using data from the labs reporting the lowest (Fig. 7a; Lab A) and highest (Fig. 7b; Lab F) mean levels of THC across all flower products. This revealed clear differences in the THC:CBD profiles reported by each lab, as well as differences between the THC:CBD ratios of samples labeled with different strain names. Moreover, multi-modal distributions were apparent for many strains, with peaks sometimes at drastically different THC:CBD ratios. For example, \u201cCharlotte\u2019s Web\u201d is a popular Chemotype III strain that was specifically bred to have high total CBD and low total THC levels. Figure 7a clearly shows that many Charlotte\u2019s Web flower samples tested in Washington commonly fit this profile, but many also fit the profile of Chemotype II and even Chemotype I products with high total THC levels. Thus, for subsequent analyses, we quantified data before and after filtering by modal chemotype. For example, for a strain name like Charlotte\u2019s Web, this would mean considering only measurements within the Chemotype III cutoffs (see \"Methods\" section).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 7 Distribution of THC-to-CBD Ratios Vary Across Popular Commercial Strain Names and Between Labs. THC-to-CBD ratios plotted on a logarithmic scale for cannabis flower samples across twenty-three popular commercial strain names for the single lab (Lab A) reporting the lowest (a) and the single lab (Lab F) reporting the highest (b) overall THC levels for cannabis flower.\n\n\n\nTo formalize how well colloquial strain names capture variation in THC:CBD profiles, we estimated a series of multilevel models with random intercepts for each strain[24] to estimate the share of the total variation in the logged THC:CBD ratio explained by the strain name, before and after filtering by modal chemotype. This information is contained in the intraclass correlation coefficient (ICC), a measure of similarity within-groups calculated as the ratio of the within-strain variance to the total variance (see \"Methods\" section). The ICC is bounded by 0 and 1, where a value of 0 indicates that a sample\u2019s strain name is completely uninformative of its THC:CBD ratio and a value of 1 indicates that a product\u2019s strain name is perfectly predictive of its THC:CBD ratio. In these data, calculating the ICC is complicated by the relatively large number of test results with reported zero CBD (and thus an unbounded THC:CBD ratio). Thus, ICCs pre-and-post filtering are shown both after omitting results with 0 CBD and coercing these results to tail values (Fig. 8a; see \"Methods\" section). When using tail values, results with zero reported CBD were coerced to a ratio of 3.5 and results with zero THC coerced to \u22122.0. These values correspond approximately to the most extreme values observed in the data (see Fig. 1b).\n\r\n\n\n\n\n\n\n\n\n\n\n Fig. 8 Popular Strain Names as Signal for THC and CBD Content. (a) Proportion of variation in log10 THC:CBD ratio explained by popular strain names (Intraclass Correlation Coefficient). 99% CIs are shown, by lab, before (black circles) and after (gray squares) filtering test results by the modal chemotype of each strain name. The ICC is shown both for dropping values for which 0% CBD or 0% THC is reported (left) and coercing cannabinoid ratios for these tests (see Methods). (b) Mean THC level of popular chemotype I strains. 99% CIs are shown after filtering by modal chemotype, for the lab reporting the lowest THC levels. (c) Mean CBD levels for popular chemotype II (above dotted line) and chemotype III (below dotted line) strain names. Results shown for the lab reporting the lowest mean THC levels.\n\n\n\nAcross all labs, the pre-filtered strain ICC was 0.57 (omitting results with zero reported CBD or THC) or 0.41 (coercing results with zero reported CBD to 3.5 and zero THC to \u22122.0). After filtering, the ICCs increase, respectively, to about 0.71 and 0.51. This overall test conceals significant variation between labs. Figure 8a shows the ICCs with 99% confidence intervals, by lab, before and after strain name filtering, separately for the two methods of handling results with reported zero CBD. In general, a large portion of the variation in THC:CBD ratio was attributable to strain name. This quantity varied substantially between labs, and filtering results outside a strain\u2019s modal chemotype typically boosted the proportion of variation explained by strain by between about 0.10 and 0.15, depending on the model. In all but one case\u2014Lab F, with missing CBD results coerced to a ratio of 3.5\u2014filtering resulted in a statistically significant (p\u2009<\u20090.01, z test) increase in the proportion of variation explained by strain. Labs varied substantially in the proportion of zero CBD test results, and, consequently, the ICC was more sensitive to the handling of missing data for those labs. For Lab F, the lab with the highest proportion of zero CBD results, the ICC was very sensitive to the handling of missing data; the ICC was 0.76 and 0.48, respectively, before filtering, and 0.92 and 0.53, after filtering. In contrast, Lab A, which had few zero CBD values, had a stable ICC across both methods, with pre-filtered ICCs of 0.65 and 0.64, respectively, and post-filtering ICCs of 0.79 and 0.78.\nGiven these observations, we compared total THC and CBD levels across strains only after filtering data by laboratory and modal chemotype. Using data from only one laboratory ensured that the same laboratory testing protocol was used to measure cannabinoids across flower samples. We chose to use data from Lab A, which had the largest number of flower samples. Comparing mean THC or CBD levels revealed clear and statistically significant differences (p\u2009<\u20090.01, Wald test) between many strain names, even within chemotypes (Fig. 8). These results suggest that strain names can provide meaningful, though variable, signals of the composition of flower samples. Filtering results to the strain\u2019s modal chemotype boosted the proportion of variation explained by strain clustering by about 0.15. Furthermore, the strength of this signal varies between labs, particularly with respect to the CBD content of high-THC flower.\n\nDiscussion \nOur results confirm that commercial cannabis strains in Washington fall into three principal chemotypes defined by their THC:CBD ratio, similar to landrace[13] and commercial Dutch strains.[14] While this result is unsurprising given the biological constraints on cannabinoid production[25][26], we were able to use this dataset to investigate outstanding questions about commercial cannabis products widely used by consumers today. These included formal analysis of discrepancies in the cannabinoid levels reported by different laboratories, trends in THC content over time, and systematic differences in the THC:CBD profiles between flower samples with different labeled strain names.\nA key area of concern for legal cannabis consumers, industry professionals, and state regulators is the accuracy of the state-mandated testing data that is required to be displayed on product packaging. Recent media reports of \u201cTHC inflation\u201d and high-profile suspensions of state-licensed testing facilities have prompted concern over the accuracy of the cannabinoid content of legal cannabis products.[27] Our analyses revealed clear, systematic differences in the results obtained by different testing facilities in Washington, with some labs consistently reporting higher or lower levels of cannabinoids than others. Moreover, these differences could not be explained by differences in the producer or strain name associated with the samples being submitted, suggesting that discrepancies between labs are likely caused by systematic differences in their testing methodologies. It is crucial that precise standards are adopted by the industry to ensure that laboratories produce results that are reproducible across labs, independent of the exact testing method used, with the ultimate goal of reporting results that consumers can trust. Our analyses provide a potential framework for quantitatively evaluating laboratory quality, including labs\u2019 sensitivity to detecting low-level cannabinoids (Fig. 4), agreement with independent measurements of cannabis products in scholarly journals (discussed below), and differences between labs after accounting for other characteristics.\nIn our analyses of Washington labs, median THC levels for Chemotype I flower varied considerably (Fig. 2), ranging from 17.7% to 23.2%. The results reported by labs with lower median THC levels are in better agreement with independent measurements of cannabis flower from legal markets.[8][28][29] For example, Vergara et al.[8] reported total THC levels for commercial flower samples in several U.S. cities, finding that flower samples averaged approximately 19% total THC in Seattle, WA. Even lower levels were (~15%) reported for Denver, CO; Sacramento, CA; and Oakland, CA.[8] Other measurements of flower samples originating from medical cannabis patients in California found median total THC levels for Chemotype I samples to be approximately 17% dry weight.[28] Another study measured THCA across a variety of popular cultivars and found mean THCA levels to be just over 16% dry weight for specimens with a broad-leaflet phenotype vs. 14% for those with a narrow leaflet phenotype (total THC levels were not reported).[29]\nReliable cannabis laboratory testing is an attainable goal. In the absence of federal regulations in the United States for the foreseeable future, it will be incumbent on state regulators to implement universal testing standards for cannabis laboratories. But states have extensive experience in this arena, regulating laboratories that analyze drinking water and evidence from crime scenes. They need only hold cannabis laboratories to similar standards. A first step may be to require that cannabis labs, like other testing facilities, receive third-party accreditation of compliance with ISO\/IEC 17025 guidelines for testing and calibration laboratories by auditors who are themselves ISO-accredited.[30] Second, states could require that cannabis labs adhere to a standardized testing protocol for analyzing cannabis to ensure consistency between labs. Such protocols have already been developed. For example, the American Herbal Pharmacopoeia published a 64-page monograph called Cannabis Inflorescence detailing analytical procedures for cannabinoid detection and quantification to establish benchmark methods.[31] The American Herbal Products Association\u2019s points readers to Cannabis Inflorescence for guidance regarding \u201cspecific analytical methods\u201d in its recommendations for regulators.[32] Finally, states should implement regular \u201cround-robin\u201d audits, sending a blinded, common sample for testing across various laboratories.\nOverall, our results are consistent with reports of \u201ccannabinoid inflation\u201d by certain laboratories[10] and point to intrinsic characteristics of labs as driving the observed variability, rather than confounds introduced from different producers, product types, or strain names being processed by each lab. Previous analysis of data from Washington\u2019s traceability dataset found a statistically significant relationship between the price per gram of flower and both total THC and total CBD levels[33], which may provide an economic incentive for cannabis producers to seek test results with higher total THC or CBD levels.\nRestricted to the lab reporting the lowest THC levels for Chemotype I flower (median\u2009=\u200917.7%), the 99th percentile THC level is 27.0%, compared to 31.8% for the lab reporting the highest THC levels. This suggests that flower samples with total THC levels approaching or exceeding 30% are very rare, and that many flower products labeled as containing >30% total THC by dry weight may be inaccurate. Our analysis of changes in THC content over time suggest that THC levels have not risen substantially, at least since 2015 (Fig. 5). While it is well-known that modern recreational cannabis contains higher levels of THC than it did in previous decades[17], this apparent plateauing in recent years may be expected due to biological limits imposed on cannabinoid production. In Cannabis, total potential THC levels are determined by levels of THCA synthesized within the plant, which is under genetic control.[12][34]\nIn general, our results do not suggest that flower samples labeled as indica, sativa, and hybrid differ substantially in terms of total THC content or THC:CBD profiles, as the indica\/sativa\/hybrid typology accounted for only about 1% of the relative variability in THC content (r\u00b2\u2009=\u20090.016). Samples labeled sativa vs. indica in Washington\u2019s commercial market do not differ in THC content, similar to what was documented among commercial strains in the Netherlands.[14] However, we did find that hybrid strains have slightly higher levels of total THC. Since hybrid strains are produced by crossing indica and sativa varieties, this result is somewhat counter-intuitive. We interpret it as potentially owing to the selective breeding of hybrid strains to maximize THC content for commercial purposes. It is also possible that, on average, these group are grown under different growing conditions, such as indoors vs. outdoors.\nWhile there were no substantial differences in the THC or CBD content between flower samples labeled as indica vs. sativa, our analyses were limited by the contents of the Washington traceability dataset, which did not include measurements of other phytocannabinoid or terpenes, a major class of aromatic compounds produced by Cannabis that likely modulate the effects of phytocannabinoids[7][23][25][35] It remains possible that indica and sativa samples differ systematically in their full phytocannabinoid or terpene profile. Indeed, a recent analysis of cannabinoid and terpene profiles from Dutch flower samples found several terpenes that may serve as markers for indica- vs. sativa-type samples.[14] Other recent work[23] has shown how more complete sets of biochemical measurements, including terpenoid profiles, can be used to define multiple distinct Cannabis chemovars.\nSince flower samples with different strain names within each of the three principal chemotypes can be differentiated somewhat by their THC:CBD ratio, more detailed analysis of the biochemical composition of commercial strains may uncover further differences. Indeed, previous work has attempted to differentiate strains based on analysis of more complete cannabinoid and terpene profiles[14][28][36][37] in an effort to define Cannabis \u201cchemovars.\u201d[23][38] An important area of future research will be to define the chemovars for commercially available cannabis products in legal U.S. markets. Applying computational approaches to large-scale cannabinoid and terpene datasets will allow cannabis strains and products to be organized according to their biochemical constituents rather than colloquial strain names. Accurate data quantifying the principal constituents of cannabis could foster the development of \u201cproduct maps\u201d that organize commercial products based on levels of their psychoactive and medically-relevant compounds. However, doing so requires having accurate lab testing data in the first place, which requires the establishment of laboratory testing standards that ensure the correct identification and quantification of the contents of cannabis products. Our results highlight the need for such standards given the large, systematic differences observed between laboratories in Washington.\n\nMethods \nData \nWashington I-502 cannabis test data \nWe submitted public records requests for all test results registered with the Washington State Liquor and Cannabis Board (LCB), the agency which regulates cannabis sales. The request was made on August 7, 2017 and the LCB provided us the raw data files on August 14, 2017. These data contained cannabinoid profiles, grower and laboratory information, and test date for 304,123 test results from June 2014 through May 2017 (excluding a very small number of test results that were dropped from the dataset for containing mathematically or biologically impossible values, such as THC content greater than 100%). The data files from which we extracted data for our analyses are attached online.\nWe focused our analyses on two product categories: flower and concentrates. \u201cFlower\u201d refers to products refer to those comprised of the mature female flower of the cannabis plant, which most cannabis products are derived from. Our analyses of flower products came from data in the \u201cFlower Lot\u201d group, which can be found in the \"InventoryLabel\" column of the I-502 dataset accompanying this paper. Our analyses of concentrate products refer to data with the \u201cHydrocarbon Wax\u201d InventoryLabel, which was the largest single group for product types that would normally be considered concentrates. We did not explore data from other product categories in this study, including \u201cHash,\u201d \u201cMarijuana Infused Edibles,\u201d and a variety of others. We encourage other scholars to mine this dataset for further insights.\n\nCalculation of THC and CBD levels \nCannabis plants do not synthesize THC or CBD; instead, they synthesize the cannabinoid acids THCA and CBDA, which are made from a common precursor[12] and must be decarboxylated (e.g., by heat energy) to yield the phytocannabinoids THC and CBD. Thus, cannabis products, especially flower samples, contain mainly THCA and\/or CBDA, as well as small levels of THC and CBD that result from spontaneous decarboxylation during the cultivation process. Total THC or CBD levels, in units of percent of dry weight, are typically calculated as:\n\n \n \n \n \n \n T\n o\n t\n a\n l\n \n \n \n C\n B\n D\n \n =\n (\n 0.877\n ×\n \n C\n B\n D\n A\n \n )\n +\n \n C\n B\n D\n \n \n \n {\\displaystyle {Total}\\,{CBD}=(0.877\\times {CBDA})+{CBD}}\n \n \n \n \n \n \n T\n o\n t\n a\n l\n \n T\n H\n C\n =\n (\n 0.877\n ×\n T\n H\n C\n A\n )\n +\n T\n H\n C\n \n \n {\\displaystyle Total\\,THC=(0.877\\times THCA)+THC}\n \n \n...where THCA and THC refer to the percent dry weight concentration of each cannabinoid present in a cannabis product, and 0.877 is the scaling factor accounting for the difference in molecular weight between THCA and THC. \u201cTotal THC\u201d or \u201cTotal CBD\u201d refers to this value, which is the maximum potential THC or CBD content of a cannabis product, assuming 100% decarboxylation of cannabinoid acids. This is the standard way of reporting \u201ctotal\u201d cannabinoid levels for legal cannabis products in Washington. Unless otherwise noted, \u201cTHC levels\u201d or \u201ctotal THC levels\u201d refers to this value (similar for CBD levels).\n\nLeafly data \nLeafly\u2019s database of strains was used to group test results, as described in the matching process below. The \"Indica,\" \"Sativa,\" or \"Hybrid\" categorization from Leafly.com allowed us to categorize strains according to their popular consumer designations. Our data matched to 1,318 unique commercial strains. The popularity of consumer strain names were determined by cumulative pageviews of the strain pages on Leafly.com.\n\nAnalytic methods \nMatching strain names \nStrain names in the Washington I-502 seed-to-sale traceability data were matched to a database of Leafly strains to retrieve information about strain popularity and indica\/sativa categorization. The matching process had two stages. First, I-502 data strain names were standardized using regular expressions to delete reference to grower\u2019s names and cities, correct spelling irregularities, and expand acronyms (e.g., \u201cGDP\u201d referring to the strain \u201cGranddaddy Purple\u201d). Second, the processed I-502 strain names were matched to the Leafly database using open-source implementations of two different approximate string matching algorithms (Ratcliff\/Obershelp and Levenshtein Distance). If the two algorithms agreed, then the test result was assigned the Leafly strain. Through this process, used to minimize the number of false positive matches, 214,747 (70.6%) results were matched to Leafly strains, including 166,594 for flower samples and the remainder for other product types. A complete list of raw strain names and their corresponding Leafly match can be found in the \"Electronic supplementary material\" section.\n\nChemotype cutoffs \nFollowing previous scholars[13], test results were classified into chemotypes based on the distinct groups visible after plotting the log10 THC-to-CBD ratio. For our analyses, we defined Chemotype I strains as any with a 5:1 THC:CBD ratio or greater, Chemotype III strains as any with a 1:5 THC:CBD ratio or lower, and the remainder of products were classified as Chemotype II. We called the \u201cstrain chemotype\u201d the modal chemotype among test results of that strain (in the matched Leafly data).\n\nStatistical significance and effect size for very large samples \nDue to the size of the I-502 dataset we analyzed, statistical comparisons often involved samples with tens of thousands of data points each, and statistically significant results were often achieved even for trivial differences between sample means with highly overlapping distributions. Thus, we calculated effect size in a variety of ways to determine whether differences between samples were relatively large or small. When comparing continuous distributions, we computed effect size in two ways: Cohen\u2019s d[15][39] and a \u201cCommon Language\u201d (CL) effect size metric.[16]\nThe CL effect size measures the probability that a randomly selected value from one population will be greater than a randomly selected value from another, while Cohen\u2019s d measures effect size as the difference between two samples means divided by their pooled standard deviation[15]:\n\n \n \n \n d\n =\n \n \n \n \n \n y\n \n i\n \n \n −\n \n y\n \n j\n \n \n \n \n \n \n (\n \n n\n \n i\n \n \n −\n 1\n )\n \n s\n \n i\n \n \n 2\n \n \n +\n (\n \n n\n \n j\n \n \n −\n 1\n )\n \n s\n \n j\n \n \n 2\n \n \n \n \n \n n\n \n i\n \n \n +\n \n n\n \n j\n \n \n −\n 2\n \n \n \n \n \n \n \n {\\displaystyle d=\\,{\\frac {y_{i}-y_{j}}{\\sqrt {\\frac {(n_{i}-1)s_{i}^{2}+(n_{j}-1)s_{j}^{2}}{n_{i}+n_{j}-2}}}}}\n \n \nwhere y denotes each sample mean, and n and s2 are the size and variance of each sample, respectively.\nFor example, Fig. 2a shows that Lab A (Confidence Analytics) and Lab B (Analytical 360) report slightly different median THC values for flower (17.6% and 18.3%, respectively) and display highly overlapping distributions. Performing a t-test to measure a difference in the means (17.5% and 18.0%, respectively) returns p-value\u2009<\u20091.1\u2009\u00d7\u200910-52. While the t-test indicates a statistically significant result, |Cohen\u2019s d|\u2009=\u20090.13, indicating a \u201csmall effect.\u201d[15] Similarly, the CL effect size is 0.46, indicating that a random value from Confidence Analytics will be greater than a random value from Analytical 360 46% of the time, which is close to what identical distributions would display (50%). The effect size matrices displayed in this paper allow one to assess whether differences between samples are relatively large or small, which is especially useful when large sample sizes allow small differences to reach statistical significance.\nFor comparing different proportions, we computed Cohen\u2019s h[15], which is the difference between two proportions\u2019 arcsine transformation:\n\n \n \n \n h\n =\n 2\n a\n r\n c\n s\n i\n n\n \n \n \n p\n \n i\n \n \n \n \n −\n 2\n a\n r\n c\n s\n i\n n\n \n \n \n p\n \n j\n \n \n \n \n \n \n {\\displaystyle h=2arcsin{\\sqrt {p_{i}}}-2arcsin{\\sqrt {p_{j}}}}\n \n \n\nCannabinoid inflation regression models \nEach of the four regression models shown employed a fixed-effect transformation for each grower-strain combination to absorb heterogeneity in cannabinoid content attributable to these factors.[40] Written in the Least Squares Dummy Variable formulation, each equation takes the form:\n\n \n \n \n \n y\n \n t\n l\n s\n \n \n =\n \n β\n \n 1\n \n \n D\n a\n t\n \n e\n \n t\n \n \n +\n \n ∑\n \n s\n =\n 1\n \n \n S\n \n \n \n δ\n \n s\n \n \n \n D\n \n t\n s\n \n \n +\n \n ∑\n \n l\n =\n 1\n \n \n L\n \n \n \n γ\n \n l\n \n \n \n D\n \n t\n l\n \n \n +\n \n η\n \n t\n l\n s\n \n \n \n \n {\\displaystyle y_{tls}=\\beta _{1}Date_{t}+\\sum \\limits _{s=1}^{S}\\delta _{s}D_{ts}+\\sum \\limits _{l=1}^{L}\\gamma _{l}D_{tl}+\\eta _{tls}}\n \n \nwhere y denotes the cannabinoid content of a particular test t from lab l of producer-strain s, S indexes each producer-strain combination (omitting one), and L indexes each lab (omitting one); D\u2019s are indicator variables denoting group (lab and strain) membership, and \u03b7 is the idiosyncratic model error. The primary quantity of interest is \u03b3, the expected deviation in cannabinoid content for each lab relative to the omitted lab, adjusting for grower\/strain and test date. As a robustness check, for regressions subsetted to concentrate products, an additional dummy variable for type of concentrate (e.g., \u201cBubble Hash,\u201d \u201cButane Hash Oil,\u201d and \u201cCO2 Hash Oil\u201d) was included to control for potential heterogeneity between labs in their stock of concentrate products, since concentrates differ in potency by their production method[41]; inclusion of this variable does not materially alter the results. Standard errors were clustered by grower-strain, since the number of clusters is large (there were 22,716 grower-strain combinations) and residuals are likely to be correlated within clusters.[42] Average covariate-adjusted cannabinoid values across labs are plotted with 99% confidence intervals estimated through the delta-method using Stata\u2019s margins command, holding covariates at their actual values.[43]\n\nTHC and CBD variation across popular commercial strains \nTo examine clustering in THC:CBD ratio by strain name, we estimated unconditional two-level hierarchical models with random intercepts for each strain, across all labs and separately by lab. Each specification took the following form:\n\n \n \n \n \n r\n \n t\n s\n \n \n =\n γ\n +\n \n μ\n \n s\n \n \n (\n 2\n )\n \n \n +\n \n ε\n \n t\n s\n \n \n \n \n {\\displaystyle r_{ts}=\\gamma +\\mu _{s}^{(2)}+\\varepsilon _{ts}}\n \n \nwhere r is the THC:CBD ratio for each test t of strain s, \u03b3 is the \u201cglobal-mean\u201d THC:CBD ratio across all strain names, \u03bc is the level-two random intercept for each strain (capturing the variance of the mean for each strain around the overall mean), and \u03b5 is the deviation of each test from its strain mean. The intraclass correlation coefficient (ICC) is calculated as the ratio of the within-group variance to the total variance, which can be decomposed into the within-group variance and the between group variance. Here, the ICC \u03c1 is the ratio of the within-strain variance to the sum of the between-strain variance and within-strain variance:\n\n \n \n \n ρ\n =\n \n \n \n σ\n \n w\n \n \n 2\n \n \n \n σ\n \n 2\n \n \n \n \n =\n \n \n \n σ\n \n w\n \n \n 2\n \n \n \n \n σ\n \n w\n \n \n 2\n \n \n +\n \n σ\n \n b\n \n \n 2\n \n \n \n \n \n \n \n {\\displaystyle \\rho ={\\frac {\\sigma _{w}^{2}}{\\sigma ^{2}}}={\\frac {\\sigma _{w}^{2}}{\\sigma _{w}^{2}+\\sigma _{b}^{2}}}}\n \n \nwhich is strictly bounded between 0 (when there is zero within-strain variance) and 1 (when there is zero between-strain variance). The random intercept models were estimated using Stata\u2019s mixed command; the post-estimation command estat icc, l(99) was used to calculate 99% confidence intervals with a logit transformation to accommodate the restricted domain of the ICC and the delta method to estimate standard errors.\n\nStrain category cannabinoid estimation \nRegression was used to estimate whether indica, sativa, and hybrid flower products differed systematically in THC and CBD. Standard errors were clustered by strain, since indica, sativa, hybrid is a strain-level property. There were 1,318 unique strains.\n\nElectronic supplementary material \n Supplemental figures (.pdf)\n Supplementary dataset I (.csv)\nAcknowledgements \nThe authors would like to thank Dr. Brad Douglass and Dr. David Roberson for helpful comments and feedback on the manuscript.\n\nAuthor contributions \nNick Jikomes and Michael Zoorob contributed equally to this work. N.J. and M.Z. jointly conceived of the analyses and wrote the manuscript.\n\nCompeting interests \nNick Jikomes is employed by Leafly Holdings, Inc. which is wholly-owned by Privateer Holdings, a for-profit firm in the legal cannabis industry. Neither Leafly nor Privateer have a direct financial interest in any third-party laboratory testing facilities in the legal cannabis industry.\n\nData availability \nAll data generated or analyzed during this study are included in this published article (and its supplementary material). They are also available in the Harvard Data Repository (https:\/\/doi.org\/10.7910\/DVN\/E8TQSD).\n\nReferences \n\n\n\u2191 Grinspoon, L. (16 August 2005). \"History of Cannabis as a Medicine\" (PDF). MAPS. http:\/\/www.maps.org\/research-archive\/mmj\/grinspoon_history_cannabis_medicine.pdf .   \n\n\u2191 Whiting, P.F.; Wolff, R.F.; Deshpande, S. et al. (2015). \"Cannabinoids for Medical Use: A Systematic Review and meta-analysis\". JAMA 313 (24): 2456\u201373. doi:10.1001\/jama.2015.6358. PMID 26103030.   \n\n\u2191 National Academies of Sciences, Engineering, and Medicine (2017). The Health Effects of Cannabis and Cannabinoids: The Current State of Evidence and Recommendations for Research. National Academies Press. doi:10.17226\/24625. ISBN 9780309453073.   \n\n\u2191 Geiger, A. (12 October 2016). \"Support for marijuana legalization continues to rise\". Fact Tank. Pew Research Center. http:\/\/www.pewresearch.org\/fact-tank\/2016\/10\/12\/support-for-marijuana-legalization-continues-to-rise . Retrieved 29 September 2017 .   \n\n\u2191 Compton, W.M.; Han, B.; Highes, A. et al. (2017). \"Use of Marijuana for Medical Purposes Among Adults in the United States\". JAMA 317 (2): 209\u201311. doi:10.1001\/jama.2016.18900. PMID 27992636.   \n\n\u2191 Barry, R.A.; Glantz, S. (2016). \"A Public Health Framework for Legalized Retail Marijuana Based on the US Experience: Avoiding a New Tobacco Industry\". PLoS Medicine 13 (9): e1002131. doi:10.1371\/journal.pmed.1002131. PMC PMC5038957. PMID 27676176. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5038957 .   \n\n\u2191 7.0 7.1 Andre, C.M.; Hausman, J.-F.; Guerriero, G. (2016). \"Cannabis sativa: The plant of the thousand and one molecules\". Frontiers in Plant Medicine 7: 19. doi:10.3389\/fpls.2016.00019. PMC PMC4740396. PMID 26870049. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396 .   \n\n\u2191 8.0 8.1 8.2 8.3 8.4 Vergara, D.; Bidwell, L.C.; Gaudino, R. et al. (2017). \"Compromised External Validity: Federally Produced Cannabis Does Not Reflect Legal Markets\". Scientific Reports 7: 46528. doi:10.1038\/srep46528. PMC PMC5395929. PMID 28422145. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395929 .   \n\n\u2191 Morgan, C.J.; Schafer, G.; Freeman, T.P. et al. (2010). \"Impact of cannabidiol on the acute memory and psychotomimetic effects of smoked cannabis: Naturalistic study\". The British Journal of Psychiatry 197: 4. doi:10.1192\/bjp.bp.110.077503. PMID 20884951.   \n\n\u2191 10.0 10.1 Coughlin-Bogue, T. (28 April 2017). \"Leafly Investigation: Is Washington's Top Cannabis Lab Inflating THC Numbers?\". Leafly. https:\/\/www.leafly.com\/news\/industry\/leafly-investigation-washingtons-top-cannabis-lab-inflating-thc-numbers . Retrieved 13 September 2017 .   \n\n\u2191 Hillig, K.W. (2005). \"Genetic evidence for speciation in Cannabis (Cannabaceae)\". Genetic Resources and Crop Evolution 52 (2): 161\u201380. doi:10.1007\/s10722-003-4452-y.   \n\n\u2191 12.0 12.1 12.2 de Meijer, E.P.; Bagatta, M.; Carboni, A. et al. (2003). \"The inheritance of chemical phenotype in Cannabis sativa L\". Genetics 163 (1): 335\u201346. PMC PMC1462421. PMID 12586720. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421 .   \n\n\u2191 13.0 13.1 13.2 13.3 13.4 13.5 Hillig, K.W.; Mahlberg, P.G. (2004). \"A chemotaxonomic analysis of cannabinoid variation in Cannabis (Cannabaceae)\". American Journal of Botany 91 (6): 966-75. doi:10.3732\/ajb.91.6.966. PMID 21653452.   \n\n\u2191 14.0 14.1 14.2 14.3 14.4 14.5 Hazekamp, A.; Tejkalov\u00e1, K.; Papadimitriou, S. (2016). \"Cannabis: From Cultivar to Chemovar II\u2014A Metabolomics Approach to Cannabis Classification\". Cannabis and Cannabinoid Research 1 (1). doi:10.1089\/can.2016.0017.   \n\n\u2191 15.0 15.1 15.2 15.3 15.4 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). L. Eribaum Associates. ISBN 9780805802832.   \n\n\u2191 16.0 16.1 McGraw, K.O.; Wong, S.P. (1992). \"A common language effect size statistic\". Psychological Bulletin 111 (2): 361\u201365. doi:10.1037\/0033-2909.111.2.361.   \n\n\u2191 17.0 17.1 ElSohly, M.A.; Mehmedic, Z.; Foster, S. et al. (2016). \"Changes in Cannabis Potency Over the Last 2 Decades (1995-2014): Analysis of Current Data in the United States\". Biological Psychiatry 79 (7): 613\u20139. doi:10.1016\/j.biopsych.2016.01.004. PMC PMC4987131. PMID 26903403. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987131 .   \n\n\u2191 Mehmedic, Z.; Chandra, S.; Slade, D. et al. (2010). \"Potency trends of \u03949-THC and other cannabinoids in confiscated cannabis preparations from 1993 to 2008\". Journal of Forensic Sciences 55 (5): 1209\u201317. doi:10.1111\/j.1556-4029.2010.01441.x. PMID 20487147.   \n\n\u2191 Cervantes, J. (2015). The Cannabis Encyclopedia: The Definitive Guide to Cultivation & Consumption of Medical Marijuana. Van Patten Publishing. ISBN 9781878823342.   \n\n\u2191 20.0 20.1 McPartland, J.M. (2017). \"Cannabis sativa and Cannabis indica versus \u201cSativa\u201d and \u201cIndica\u201d\". In Chandra, S.; Lata, H.; ElSholy, M.A.. Cannabis sativa L. - Botany and Biotechnology. Springer. pp. 101\u201321. ISBN 9783319545646.   \n\n\u2191 Pearce, D.D.; Mitsouras, K.; Irizarry, K.J. (2014). \"Discriminating the effects of Cannabis sativa and Cannabis indica: A web survey of medical cannabis users\". Journal of Alternative 20 (10): 787\u201391. doi:10.1089\/acm.2013.0190. PMID 25191852.   \n\n\u2191 Piomeli, D.; Russo, E.B. (2016). \"The Cannabis sativa Versus Cannabis indica Debate: An Interview with Ethan Russo, MD\". Cannabis and Cannabinoid Research 1 (1): 44\u201346. doi:10.1089\/can.2015.29003.ebr. PMC PMC5576603. PMID 28861479. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603 .   \n\n\u2191 23.0 23.1 23.2 23.3 Lewis, M.A.; Russo, E.B.; Smith, K.M. (2018). \"Pharmacological Foundations of Cannabis Chemovars\". Planta Medica 84 (4): 225\u201333. doi:10.1055\/s-0043-122240. PMID 29161743.   \n\n\u2191 Gelman, A.; Hill, J. (2007). Data Analysis Using Regression and Multilevel\/Hierarchical Models. Cambridge University Press. ISBN 9780521686891.   \n\n\u2191 25.0 25.1 Russo, E.B. (2011). \"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\". British Journal of Pharmacology 163 (7): 1344\u201364. doi:10.1111\/j.1476-5381.2011.01238.x. PMC PMC3165946. PMID 21749363. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946 .   \n\n\u2191 de Meijer, E. (2015). \"Chapter 5: The Chemical Phenotypes (Chemotypes) of Cannabis\". In Pertwee, R.. Handbook of Cannabis. Oxford University Press. pp. 89\u2013110. ISBN 9780199662685.   \n\n\u2191 Coughlin-Bogue, T. (15 September 2017). \"Washington Labs Launch Effort to Address Credibility Crisis\". Leafly. https:\/\/www.leafly.com\/news\/industry\/washington-labs-launch-effort-to-address-credibility-crisis . Retrieved 16 September 2017 .   \n\n\u2191 28.0 28.1 28.2 Elzinga, S.; Fischedick, J.; Podkolinski, R. et al. (2015). \"Cannabinoids and Terpenes as Chemotaxonomic Markers in Cannabis\". Natural Products Chemistry & Research 3: 181. doi:10.4172\/2329-6836.1000181.   \n\n\u2191 29.0 29.1 Lynch, R.C.; Vergara, D.; Tittes, S. et al. (2017). \"Genomic and Chemical Diversity in Cannabis\". Critical Reviews in Plant Sciences 35 (5\u20136): 349\u201363. doi:10.1080\/07352689.2016.1265363.   \n\n\u2191 Unger, P.; Brauninger, R.; Hudalla, C. et al. (December 2014). \"Standards for Cannabis Testing Laboratories\" (PDF). Cannabis Safety Institute. http:\/\/cannabissafetyinstitute.org\/wp-content\/uploads\/2015\/01\/Standards-for-Cannabis-Testing-Laboratories.pdf .   \n\n\u2191 Upton, R.; Craker, L.; ElSohly, M. et al., ed. (2014). Cannabis Inflorescence: Cannabis spp.. American Herbal Pharmacopoeia. ISBN 1929425333. http:\/\/www.herbal-ahp.org\/order_online.htm .   \n\n\u2191 Cannabis Committee (02 February 2016). \"Recommendations for Regulators - Cannabis Operations\" (PDF). American Herbal Products Association. http:\/\/www.ahpa.org\/Portals\/0\/PDFs\/Committee\/CC\/Cannabis_Dispensing_Recommendations_Regulators.pdf .   \n\n\u2191 Smart, R.; Caulkins, J.P.; Kilmer, B. et al. (2017). \"Variation in cannabis potency and prices in a newly legal market: evidence from 30 million cannabis sales in Washington state\". Addiction 112 (12): 2167\u201377. doi:10.1111\/add.13886. PMC PMC5673542. PMID 28556310. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5673542 .   \n\n\u2191 Weiblen, G.D.; Wenger, J.P.; Craft, K.J. et al. (2015). \"Gene duplication and divergence affecting drug content in Cannabis sativa\". The New Phytologist 208 (4): 1241-50. doi:10.1111\/nph.13562. PMID 26189495.   \n\n\u2191 McPartland, J.M.; Russo, E.B. (2008). \"Cannabis and Cannabis Extracts: Greater Than the Sum of Their Parts?\". Journal of Cannabis Therapeutics 1 (3\u20134): 103\u201332. doi:10.1300\/J175v01n03_08.   \n\n\u2191 Aizpurua-Olaizola, O.; Soydaner, U;. \u00d6zt\u00fcrk, E. et al. (2016). \"Evolution of the Cannabinoid and Terpene Content during the Growth of Cannabis sativa Plants from Different Chemotypes\". Journal of Natural Products 79 (2): 324\u201331. doi:10.1021\/acs.jnatprod.5b00949. PMID 26836472.   \n\n\u2191 Casano, S.; Grassi, G.; Martini, V. et al. (2010). \"Variations in terpene profiles of different strains of Cannabis sativa\". ISHS Acta Horticulturae 925: 115\u201321. doi:10.17660\/ActaHortic.2011.925.15.   \n\n\u2191 Hazekamp, A.; Fischedick, J.T. (2012). \"Cannabis - from cultivar to chemovar\". Drug Testing and Analysis 4 (7\u20138): 660\u20137. doi:10.1002\/dta.407. PMID 22362625.   \n\n\u2191 Sawilowsky, S.S. (2009). \"New Effect Size Rules of Thumb\". Journal of Modern Applied Statistical Methods 8 (2). doi:10.22237\/jmasm\/1257035100.   \n\n\u2191 Wooldridge, J.M. (2015). Introductory Econometrics: A Modern Approach (6th ed.). Cengage Learning. pp. 484\u20135. ISBN 9781305270107.   \n\n\u2191 Raber, J.C.; Elzinga, S.; Kaplan, C. (2015). \"Understanding dabs: Contamination concerns of cannabis concentrates and cannabinoid transfer during the act of dabbing\". Journal of Toxicological Sciences 40 (6): 797\u2013803. doi:10.2131\/jts.40.797. PMID 26558460.   \n\n\u2191 Cameron, A.C.; Miller, D.L. (2015). \"A Practitioner\u2019s Guide to Cluster-Robust Inference\". Journal of Human Resources 50 (2): 317-372. doi:10.3368\/jhr.50.2.317.   \n\n\u2191 Williams, R. (2012). \"Using the margins command to estimate and interpret adjusted predictions and marginal effects\". The Stata Journal 12 (2): 308-31. https:\/\/www.stata-journal.com\/article.html?article=st0260 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\">https:\/\/www.limswiki.org\/index.php\/Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2019)LIMSwiki journal articles (all)LIMSwiki journal articles (with rendered math)LIMSwiki journal articles on cannabis testingLIMSwiki journal articles on laws and regulations\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 15 April 2019, at 21:29.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 269 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","500951db854008d04f1cf96423731db5_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:The cannabinoid content of legal cannabis in Washington State varies systematically across testing facilities and popular consumer products<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The majority of adults in the U.S. now have state-legal access to medical or recreational cannabis products, despite their federal prohibition. Given the wide array of pharmacologically active compounds in these products, it is essential that their <a href=\"https:\/\/en.wikipedia.org\/wiki\/Biomolecule\" class=\"extiw wiki-link\" title=\"wikipedia:Biomolecule\" data-key=\"53fe12e43c42f8137f63158d1ae5572e\">biochemical<\/a> profile is measured and reported to consumers, which requires accurate <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> testing. However, no universal <a href=\"https:\/\/www.limswiki.org\/index.php\/Specification_(technical_standard)\" title=\"Specification (technical standard)\" class=\"wiki-link\" data-key=\"dc2050725de4ea7ca520800e4f969b20\">standards<\/a> for laboratory testing <a href=\"https:\/\/en.wikipedia.org\/wiki\/Protocol_(science)\" class=\"extiw wiki-link\" title=\"wikipedia:Protocol (science)\" data-key=\"dd4b3d67b8ba976fcaaa9ea7b8f7f1e7\">protocols<\/a> currently exist, and there is controversy as to whether all reported results are legitimate. To investigate these concerns, we analyzed a publicly available <a href=\"https:\/\/www.cannaqa.wiki\/index.php?title=Seed-to-sale\" class=\"extiw\" title=\"cannaqawiki:Seed-to-sale\" rel=\"external_link\" target=\"_blank\">seed-to-sale<\/a> traceability dataset from Washington State containing measurements of the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinoid\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinoid\" data-key=\"fe5c4e73e0c21e16db393a214691296b\">cannabinoid<\/a> content of legal cannabis products from state-certified laboratories. Consistent with previous work, we found that commercial <i><a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis\" data-key=\"ae3a33525e4682427d4498e16c586f9e\">Cannabis<\/a><\/i> strains fall into three broad <a href=\"https:\/\/en.wikipedia.org\/wiki\/Chemotype\" class=\"extiw wiki-link\" title=\"wikipedia:Chemotype\" data-key=\"b162b44fa2e0a0a877f825fbb66f3c7f\">chemotypes<\/a> defined by the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Tetrahydrocannabinol\" class=\"extiw wiki-link\" title=\"wikipedia:Tetrahydrocannabinol\" data-key=\"c63b7f849adf168f4b4ff293132f1e53\">tetrahydrocannabinol<\/a>:<a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabidiol\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabidiol\" data-key=\"0c46162c3d7b370d7448646c25334265\">cannabidiol<\/a> (THC:CBD) ratio. Moreover, we documented systematic differences in the cannabinoid content reported by different laboratories, relative stability in cannabinoid levels of commercial flower and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_concentrate\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis concentrate\" data-key=\"e1754480a85316820c041e0f7f409adc\">concentrates<\/a> over time, and differences between popular commercial <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_strains\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis strains\" data-key=\"39bbebad56bc4dce75022ec5f2049cad\">strains<\/a>. Importantly, interlab differences in cannabinoid reporting persisted even after controlling for plausible confounds. Our results underscore the need for standardized laboratory methodologies in the legal <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_industry\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis industry\" data-key=\"0db7feaccdfb0220df32143798713a34\">cannabis industry<\/a> and provide a framework for quantitatively assessing <a href=\"https:\/\/en.wikipedia.org\/wiki\/Laboratory_quality_control\" class=\"extiw wiki-link\" title=\"wikipedia:Laboratory quality control\" data-key=\"9bc16e3a44106160b234b01e679e3db5\">laboratory quality<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>For millennia, <i>Cannabis<\/i> has been cultivated for medicinal, recreational, and industrial purposes.<sup id=\"rdp-ebb-cite_ref-GrinspoonHist05_1-0\" class=\"reference\"><a href=\"#cite_note-GrinspoonHist05-1\">[1]<\/a><\/sup> Despite mounting evidence for the legitimate medical utility of cannabis products and their principal <a href=\"https:\/\/en.wikipedia.org\/wiki\/Psychoactive_drug\" class=\"extiw wiki-link\" title=\"wikipedia:Psychoactive drug\" data-key=\"67dd6c48c70dc595ceef1258759e4e28\">psychoactive<\/a> constituents<sup id=\"rdp-ebb-cite_ref-WhitingCanna15_2-0\" class=\"reference\"><a href=\"#cite_note-WhitingCanna15-2\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-NASEM_TheHealth17_3-0\" class=\"reference\"><a href=\"#cite_note-NASEM_TheHealth17-3\">[3]<\/a><\/sup>, they remain classified as <a href=\"https:\/\/en.wikipedia.org\/wiki\/Controlled_Substances_Act\" class=\"extiw wiki-link\" title=\"wikipedia:Controlled Substances Act\" data-key=\"dc9b3ee3a1db6f2642725d442821d995\">Schedule I<\/a> controlled substances by the U.S. federal government. Nonetheless, public opinion on legal cannabis has changed dramatically in recent years<sup id=\"rdp-ebb-cite_ref-GeigerSupport16_4-0\" class=\"reference\"><a href=\"#cite_note-GeigerSupport16-4\">[4]<\/a><\/sup>, and a majority of U.S. states now allow legal access to <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_(drug)\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis (drug)\" data-key=\"43fbd34351979f1e17186f202a2b1e49\">medical cannabis<\/a> for approved patients, with several states also allowing recreational adult-use.<sup id=\"rdp-ebb-cite_ref-ComptonUseOf17_5-0\" class=\"reference\"><a href=\"#cite_note-ComptonUseOf17-5\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BarryAPublic16_6-0\" class=\"reference\"><a href=\"#cite_note-BarryAPublic16-6\">[6]<\/a><\/sup> This dynamic legal landscape has given rise to a rapidly growing legal cannabis industry that offers a wide variety of products to consumers.\n<\/p><p>Because the core product of this burgeoning industry contains multiple compounds with psychoactive and medicinal properties<sup id=\"rdp-ebb-cite_ref-AndreCannabis16_7-0\" class=\"reference\"><a href=\"#cite_note-AndreCannabis16-7\">[7]<\/a><\/sup>, it is imperative that the major biochemical constituents of cannabis are accurately quantified, and the results made accessible to consumers. Because recreational cannabis products may differ substantially from cannabis grown for federally-sanctioned research<sup id=\"rdp-ebb-cite_ref-VergaraComp17_8-0\" class=\"reference\"><a href=\"#cite_note-VergaraComp17-8\">[8]<\/a><\/sup> or found on the black market<sup id=\"rdp-ebb-cite_ref-MorganImpact10_9-0\" class=\"reference\"><a href=\"#cite_note-MorganImpact10-9\">[9]<\/a><\/sup>, there is a particular need to study the commercial cannabis being consumed today by millions of adults in states allowing legal adult-use consumption.\n<\/p><p>The adoption of universal industry testing standards will be crucial for comparing data across the many existing testing laboratories. However, standardized procedures have yet to be adopted, and controversy exists about whether all laboratories are accurately measuring and reporting cannabinoid content.<sup id=\"rdp-ebb-cite_ref-Coughlin-BogueLeafly17_10-0\" class=\"reference\"><a href=\"#cite_note-Coughlin-BogueLeafly17-10\">[10]<\/a><\/sup> Most of these labs were not established quality control labs with a track record of testing food or pharmaceutical products, but rather started specifically to focus on cannabis products. At present, there is limited published data<sup id=\"rdp-ebb-cite_ref-VergaraComp17_8-1\" class=\"reference\"><a href=\"#cite_note-VergaraComp17-8\">[8]<\/a><\/sup> on the content of commercial cannabis products in the U.S., including quantification of potential differences in the measurements reported across these testing laboratories. Reliable testing data will also shed light on questions important to consumers and regulators, such as whether cannabinoid levels are changing over time or differ systematically between commercial products.\n<\/p><p>To investigate these concerns, we analyzed a large dataset from Washington State\u2019s seed-to-sale traceability system. This dataset comprises hundreds of thousands of measurements of the principal cannabinoids in commercial cannabis, including tetrahydrocannabinol (THC) and cannabidiol (CBD). These measurements are available for commercial products tested across all state-licensed laboratories since 2014, which allowed us to assess the cannabinoid composition of commercial products between laboratories, over time, and across strains.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"The_basic_chemotype_landscape_of_commercial_cannabis\">The basic chemotype landscape of commercial cannabis<\/span><\/h3>\n<p><i>Cannabis<\/i> likely evolved in Central Asia, and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Landrace\" class=\"extiw wiki-link\" title=\"wikipedia:Landrace\" data-key=\"8ce613e5280ad2c03ce6bf605536c96f\">landraces<\/a> native to regions including Afghanistan, Pakistan, India and China<sup id=\"rdp-ebb-cite_ref-HilligGenetic05_11-0\" class=\"reference\"><a href=\"#cite_note-HilligGenetic05-11\">[11]<\/a><\/sup> have been found to fall into three general chemotypes based on genetically-constrained THC:CBD ratios.<sup id=\"rdp-ebb-cite_ref-deMeijerTheInher03_12-0\" class=\"reference\"><a href=\"#cite_note-deMeijerTheInher03-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HilligAChemo04_13-0\" class=\"reference\"><a href=\"#cite_note-HilligAChemo04-13\">[13]<\/a><\/sup> Consistent with previous work in landraces and commercial Dutch <i>Cannabis<\/i><sup id=\"rdp-ebb-cite_ref-HilligAChemo04_13-1\" class=\"reference\"><a href=\"#cite_note-HilligAChemo04-13\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HazekampCannabis16_14-0\" class=\"reference\"><a href=\"#cite_note-HazekampCannabis16-14\">[14]<\/a><\/sup>, we found that commercial <i>Cannabis<\/i> grown in Washington also conforms to this pattern (Fig. 1a\u2013c). Unlike landraces, which are more likely to fall into the chemotype III (CBD-dominant) category and generally display lower overall levels of total THC<sup id=\"rdp-ebb-cite_ref-HilligAChemo04_13-2\" class=\"reference\"><a href=\"#cite_note-HilligAChemo04-13\">[13]<\/a><\/sup>, most commercial <i>Cannabis<\/i> falls into the chemotype I category, characterized by relatively high total THC and low total CBD levels (Fig. 1d\u2013f; see \"Methods\" section at the end for definition of total THC and CBD levels). While studying the chemotype landscape of these commercial samples, we observed striking differences in THC:CBD distributions across laboratories for both flower (Fig. 1d\u2013f, Figure S1) and concentrates (Figure S2). This prompted us to examine interlab differences in more detail. In particular, we wished to assess whether this variation stemmed from intrinsic (e.g., methodological) differences between laboratories or from heterogeneity in the products submitted to those labs.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"2ab2efada60f5fcffccd13d86d7c3fdc\"><img alt=\"Fig1 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/93\/Fig1_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 1<\/b> The THC:CBD ratio defines three broad chemotypes of commercial cannabis flower measured by testing labs in Washington. Left column: Scatterplots of total THC vs. total CBD levels for cannabis flower. Right column: Histograms showing the THC:CBD ratio on a log scale and indicating the proportion of flower samples for each chemotype. Data are displayed for measurements batched across all Labs A-F (panels a-b; n\u2009=\u2009175,136), for the lab reporting the lowest mean total THC levels (Lab A; panels c-d; n\u2009=\u200962,719), and the lab reporting the highest mean total THC levels (Lab F; panels e-f; n=26,664). Histograms for each of the six labs contributing to batched data in panels a-b are shown in Figure S1. Panels a and c were subsampled to n=50,000 for visualization purposes.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"THC_and_CBD_measurements_vary_widely_across_testing_laboratories\">THC and CBD measurements vary widely across testing laboratories<\/span><\/h3>\n<p>To compare cannabinoid measurements across labs, we looked at distributions of total THC and CBD levels for the six largest laboratories by data volume for different chemotypes and product categories. These labs, referred to henceforth as labs A-F, are Confidence Analytics (Lab A), Analytical 360 (Lab B), Green Grower Labs (Lab C), Integrity Labs (Lab D), Testing Technologies (Lab E), and Peak Analytics (Lab F). We observed differences in reported values of both THC and CBD (Fig. 2). For example, the median total THC content for chemotype I flower products ranged from 17.7% to 23.2% between the labs reporting the lowest and highest THC levels, respectively (Fig. 2a; labs A-F ordered from lowest to highest median reported THC levels). Pairwise differences in mean THC content between labs were statistically significant (p\u2009<\u20090.001 for each pairwise comparison in Fig. 2a, two-sided t-test). To quantify the magnitude of differences between labs, we calculated the effect sizes of pairwise differences using two metrics: Cohen\u2019s d, the standardized difference between two means<sup id=\"rdp-ebb-cite_ref-CohenStatistical88_15-0\" class=\"reference\"><a href=\"#cite_note-CohenStatistical88-15\">[15]<\/a><\/sup>, and a \u201cCommon Language\u201d (CL) effect size, the probability that a random value from one sample will be greater than a random value from the other<sup id=\"rdp-ebb-cite_ref-McGrawAComm92_16-0\" class=\"reference\"><a href=\"#cite_note-McGrawAComm92-16\">[16]<\/a><\/sup> (Fig. 2b; see Analytical Methods).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"ec68d9fcfb1ca33cbe1889562f99f165\"><img alt=\"Fig2 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/2\/25\/Fig2_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 2<\/b> Total THC and CBD Measurements Differ Between Labs Across Chemotypes and Product Categories. Left column: Violin plots showing the distribution of total THC or CBD levels across labs A-F. Black lines denote median values, which are printed below the x-axis for each lab. Right column: Effect size matrices displaying the effect size of pairwise differences in distributions between labs. Matrices are color-coded according to one measure of effect size (Cohen\u2019s d), and a second measure (Common Language) is printed for each comparison.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Calculating effect sizes allows a more intuitive assessment of the magnitude of interlab differences, especially when very large sample sizes allow even trivial differences between means to reach statistical significance. For example, mean THC levels of the chemotype I flower for Lab B and Lab A were 18.4% and 17.7%, respectively (Fig. 2a and b). While this difference was highly significant due to the large sample sizes, the effect size was small (d\u2009=\u20090.13; see the \"Methods\" section). The common language effect size (CL) for this comparison was 0.54, indicating a 54% chance that a random THC measurement from Lab B will be larger than a random measurement from Lab A. In contrast, when comparing Lab F to Lab A, which reported the highest mean THC levels, the effect size was considerably larger (d\u2009=\u20091.28, CL\u2009=\u20090.82).\n<\/p><p>We observed a similar pattern when comparing CBD measurements across labs for chemotype II and III flower samples (Fig. 2c and d) and THC levels for concentrates (Fig. 2e and f). The labs reporting the highest levels of THC for chemotype I flower products also reported the highest levels of CBD for other flower chemotypes and THC levels for concentrates (Fig. 2), indicating a systematic tendency for certain labs to report higher levels of cannabinoids across chemotypes and product categories. This may be explained by differences in laboratory protocols. While most labs report using <a href=\"https:\/\/www.limswiki.org\/index.php\/High-performance_liquid_chromatography\" title=\"High-performance liquid chromatography\" class=\"wiki-link\" data-key=\"dc274e995eb18113903eebaef69c0cf9\">high-performance liquid chromatography<\/a> (HPLC) to detect cannabinoids, the details of each protocol likely differ. Alternatively, interlab differences may be driven by labs receiving distinct sets of cannabis products for testing.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Interlab_differences_persist_after_controlling_for_plausible_confounds\">Interlab differences persist after controlling for plausible confounds<\/span><\/h3>\n<p>To investigate potential determinants of interlab differences, we quantified the average cannabinoid levels reported by each lab after accounting for strain name, the producer-processor submitting samples for testing, and time of measurement (see \"Methods\" section). Four separate regression models were estimated: (1) THC levels in chemotype I flower products (n\u2009=\u2009161,933); (2) THC levels in chemotype I concentrate products (n\u2009=\u200933,888); (3) CBD levels in chemotype II and III flower products (n\u2009=\u20094,661); and (4) CBD levels in chemotype II and III concentrate products (n\u2009=\u20092,156). Large interlab variability in reported THC and CBD levels persisted across product categories after controlling for these factors (Fig. 3). Differences were observed for both flower (Fig. 3a,c) and concentrates (Fig. 3b,d). For chemotype I flower, the average adjusted total THC level for Peak Analytics (~23%) was significantly higher (p\u2009<\u20090.001; Wald test) than all other labs (Fig. 3a).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"33f90eaf419fbfd8ffbb13a4dd3609b1\"><img alt=\"Fig3 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/37\/Fig3_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 3<\/b> THC and CBD Levels Vary Between Labs After Controlling for Plausible Confounds. Average predicted values (+\/\u2212 99% confidence intervals) are shown, by lab, for <b>(a)<\/b> THC levels in chemotype I flower products (n\u2009=\u2009161,933); <b>(b)<\/b> THC levels in chemotype I concentrate products (n\u2009=\u200933,888); <b>(c)<\/b> CBD levels in chemotype II and III flower products (n\u2009=\u20094,661); and <b>(d)<\/b> CBD levels in chemotype II and III concentrate products (n\u2009=\u20092,156) after adjusting for grower, strain-name, and time of measurement. Predicted values were generated from fixed-effects regressions with cluster-robust standard errors (see Methods).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For chemotype I concentrates, Lab F\u2019s average reported total THC (~75%) exceeded all other labs, including the lab reporting the second-highest average total THC (Lab E, ~70%). For total CBD levels in chemotype II and chemotype III flower, Lab F again reported the largest mean quantity, at about 13%, significantly higher (p\u2009<\u20090.01) than all other labs (Fig. 3c). For chemotype II and chemotype III concentrates, Lab F\u2019s average products reported the highest CBD, but these estimates were uncertain due to the relatively small sample size (Fig. 3d). Overall, these results suggest that the observed differences between laboratories cannot be explained by differences in the producers, product types, or strain names of the samples being processed by each lab.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Low-level_cannabinoid_measurements_vary_widely_across_laboratories\">Low-level cannabinoid measurements vary widely across laboratories<\/span><\/h4>\n<p>Examination of THC:CBD distributions across laboratories indicated substantial variation in their propensity to report chemotype I strains with low total CBD levels (Figure S1, far right bins). To investigate this further, we plotted the density of chemotype I flower products with less than 1% CBD by dry weight (Fig. 4). The shape of these distributions varied somewhat across labs, likely due to methodological differences determining their limit of quantification (LOQ). Similar to what we observed in the THC:CBD histograms (Fig. 1b), these density plots indicate that differences exist between labs\u2019 propensity to detect low levels of CBD in chemotype I flower; they tend to display local maxima near 0.1%, which is the LOQ most labs report for cannabinoids.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"df209b2b505de487b6b62afe3ecad30d\"><img alt=\"Fig4 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/a\/a8\/Fig4_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 4<\/b> Labs differ in the propensity to detect low levels of CBD in chemotype I flower. <b>(a)<\/b> Kernel density plots of each lab\u2019s distribution of total CBD levels below 1.0% dry weight for chemotype I flower (y-axis scaled to one). Most labs show a local maximum near 0.1% total CBD, which is a commonly reported LOQ. <b>(b)<\/b> Fraction of chemotype I flower with total CBD levels below 0.1% dry weight. Bars indicate proportions +\/\u2212 95% CI for a binomial proportion. <b>(c)<\/b> Effect size matrix indicating the magnitude of interlab differences shown in panel B. Effect size is quantified as Cohen\u2019s h (see \"Methods\" section).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>To further quantify these differences, we compared the proportion of chemotype I flower having <0.1% total CBD across laboratories. There were dramatic differences between labs (Fig. 4b), with some reporting substantially more chemotype I flower with total CBD <0.1% than others. The volume of data caused even tiny interlab differences to reach high levels of statistical significance (p\u2009<\u20090.001 for all pairwise comparisons, except Lab E vs. Lab F, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Mann-Whitney_U_test\" class=\"extiw wiki-link\" title=\"wikipedia:Mann-Whitney U test\" data-key=\"b81cf7d8bfe28db6432bb8e3871bd383\">Mann-Whitney U test<\/a>). Thus, we quantified the effect size of these differences by computing Cohen\u2019s h for all pairwise comparisons (Fig. 4c; see Analytic Methods). Many of these interlab differences were of very large effect size (|h| >0.80, and often much greater), confirming that there are substantial differences in labs\u2019 propensity to detect low levels of CBD in chemotype I flower. The analyses so far indicate that cannabinoid inflation and differences in the ability of labs to detect low-level cannabinoids both contribute to systematic differences in their reported measurements.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Changes_in_THC_content_of_commercial_cannabis_products_over_time\">Changes in THC content of commercial cannabis products over time<\/span><\/h3>\n<p>While modern commercial strains contain higher THC levels than recreational cannabis from past decades<sup id=\"rdp-ebb-cite_ref-ElSohlyChanges16_17-0\" class=\"reference\"><a href=\"#cite_note-ElSohlyChanges16-17\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MehmedicPotency10_18-0\" class=\"reference\"><a href=\"#cite_note-MehmedicPotency10-18\">[18]<\/a><\/sup>, it is unclear whether THC levels have continued climbing since Washington permitted adult-use cannabis. Thus, we looked for potential changes in the total THC content of commercial products in recent years. Because our previous analyses revealed systematic interlab variability in cannabinoid measurements, we sought to minimize the potential confound of lab-specific \u201ccannabinoid inflation.\u201d Thus, we quantified cannabinoid levels over time, separately for different subsets of laboratories: the three labs reporting the lowest mean THC levels (low THC reporting, LTR), the three reporting the highest mean THC levels (high THC reporting, HTR), as well as data pooled across laboratories. Figure 5a shows mean THC levels, averaged across labs, for chemotype I flower products from June 2014 through May 2017. While there was an upward trend from 2014 to early 2015, mean THC levels appear to have largely plateaued, with modest fluctuations since 2015 (Fig. 5a). This trend was evident in the pooled data as well as in LTR and HTR labs, although HTR labs showed a steeper increase in total THC levels from 2014 to 2015.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"ef6bf14c59065c1927d3c18980401d09\"><img alt=\"Fig5 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/b3\/Fig5_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 5<\/b> Mean THC Levels for Chemotype I Flower Products Over Time. <b>(a)<\/b> Total THC levels over time averaged across all labs or those reporting the highest or lowest mean THC levels. <b>(b)<\/b> Distribution of THC levels for each year on record for low THC reporting (LTR) labs. <b>(c)<\/b> Effect size matrix quantifying the mean difference in THC levels across years for LTR labs. <b>(d)<\/b> Distribution of THC levels for each year for high THC reporting (HTR) labs, and <b>(e)<\/b> the effect size matrix quantifying the magnitude of yearly differences.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>To further quantify changes in THC levels over time, we compared total THC levels for each year of data (Fig. 5b and c). Median THC levels for chemotype I flower rose from 2014 to 2015 but changed only slightly between 2015 and 2017. This was true whether we looked at the three LTR labs (Fig. 5b and c) or the three HTR labs (Fig. 5d and e). Again, large sample sizes allowed small differences in mean THC levels to reach statistical significance for all pairwise comparisons (p\u2009<\u20090.001, Mann-Whitney U test), except 2015 to 2016 for the HTR cohort (p\u2009=\u20090.334, Fig. 5d). After 2014, the effect sizes for year-to-year comparisons were small (Fig. 5b; |Cohen\u2019s d| <0.23 for each comparison). Thus, we conclude that there has not been a substantial increase in the THC content of Washington state\u2019s commercial cannabis flower from since 2015, although there were notable differences between LTR and HTR labs. For example, THC distributions from HTR labs were much more skewed (Fig. 5d; skew\u2009=\u2009\u22120.2) than for low-LTR labs (Fig. 5b; skew\u2009=\u20090.06). In addition, the increase in mean THC values from 2014 to 2015 was much larger for HTR than LTR labs (4.5% vs. 2.3%, respectively).\n<\/p><p>We were also interested in whether concentrates have increased in THC levels since 2014, as these products contain a much higher THC concentration. Mean THC levels across labs appeared to be relatively flat from 2014 to 2017 (Figure S3A). There was a small increase in THC levels from 2014 to 2015 for both cohorts of labs (Figure S3), although this was smaller than the increase observed for flower. From 2015 onward, there was a small decrease in mean THC levels for LTR and HTR labs (Figure S3). Thus, we conclude that, since 2015, there has not been a substantial increase in mean THC levels for commercial flower and concentrate products in Washington.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"THC_content_across_popular_commercial_categories:_indica.2C_sativa.2C_and_hybrid\">THC content across popular commercial categories: indica, sativa, and hybrid<\/span><\/h3>\n<p>The vernacular among cannabis users involves a triad of \u201cindica,\u201d \u201csativa,\u201d and \u201chybrid\u201d strains.<sup id=\"rdp-ebb-cite_ref-CervantesTheCann15_19-0\" class=\"reference\"><a href=\"#cite_note-CervantesTheCann15-19\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McPartlandCanna17_20-0\" class=\"reference\"><a href=\"#cite_note-McPartlandCanna17-20\">[20]<\/a><\/sup> Recreational consumers and popular educational resources often attribute distinctive psychoactive effects to indica and sativa strains<sup id=\"rdp-ebb-cite_ref-PearceDiscrim14_21-0\" class=\"reference\"><a href=\"#cite_note-PearceDiscrim14-21\">[21]<\/a><\/sup>, while scholars tend to be more skeptical of these claims.<sup id=\"rdp-ebb-cite_ref-McPartlandCanna17_20-1\" class=\"reference\"><a href=\"#cite_note-McPartlandCanna17-20\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PiomeliTheCanna16_22-0\" class=\"reference\"><a href=\"#cite_note-PiomeliTheCanna16-22\">[22]<\/a><\/sup> In landraces, accessions from indica strains have been associated with more THC than sativas<sup id=\"rdp-ebb-cite_ref-HilligAChemo04_13-3\" class=\"reference\"><a href=\"#cite_note-HilligAChemo04-13\">[13]<\/a><\/sup> but indica and sativa recreational products sold in the Netherlands had similar THC content.<sup id=\"rdp-ebb-cite_ref-HazekampCannabis16_14-1\" class=\"reference\"><a href=\"#cite_note-HazekampCannabis16-14\">[14]<\/a><\/sup> The term \u201cstrain,\u201d although widely used, is not a botanically-accepted term for distinguishing plant varieties, and many scholars prefer the term \u201cchemovar\u201d in order to emphasize biochemical differences between specific Cannabis varieties<sup id=\"rdp-ebb-cite_ref-LewisPharma18_23-0\" class=\"reference\"><a href=\"#cite_note-LewisPharma18-23\">[23]<\/a><\/sup> (see \"Discussion\" section). To investigate potential differences in cannabinoid content among commercial strain categories used by consumers, we looked at the distribution of THC content of indica, sativa, and hybrid flower samples in Washington\u2019s commercial market. We matched test results using their producer-given strain name from the I-502 dataset to the Leafly.com strain database to retrieve their popular indica, sativa, or hybrid categorization (see \"Methods\" section). This matching process yielded 166,594 flower results for analysis: 42,711 indica (25.6%), 31,822 sativa (19.1%), and 92,061 hybrid (55.3%) products. While hybrids had higher mean levels of THC compared to indicas and sativas, the distributions of THC content among indicas, sativas, and hybrids overlapped considerably (Fig. 6a\u2013d).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"7c217f647e893cdd1ad5f3a6de57d98c\"><img alt=\"Fig6 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/4\/45\/Fig6_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 6<\/b> Total THC and CBD Levels Across Popular Consumer Strain Categories for Flower Products. <b>(a)<\/b> Distribution of THC levels across popular strain categories for chemotype I flower and <b>(b)<\/b> effect size matrix quantifying the magnitude of differences between them. <b>(c)<\/b> Distribution of CBD levels across the same categories for chemotype II and III flower and <b>(d)<\/b> effect size matrix quantifying the magnitude of differences between them.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>To further quantify differences in THC content, we estimated a bivariate regression model of THC on strain category across all labs. The model indicates that hybrid strains have modestly greater THC content, on average, than either indica or sativa strains (Fig. 6e; hybrid vs indica: 1.22%, p\u2009<\u20090.001; hybrid vs sativa: 0.89%, p\u2009<\u20090.01). The difference in THC between sativa and indica could not be distinguished from sampling variability (sativa vs indica: 0.33%, p\u2009=\u20090.230). Moreover, the indica, sativa, hybrid distinction explained only a tiny fraction of THC variability between flower samples (r<sup>2<\/sup>\u2009=\u20090.016), and the differences in mean THC content had modest effect sizes (Hybrid vs Sativa: Cohen\u2019s d\u2009=\u20090.283; Hybrid vs Indica: Cohen\u2019s d\u2009=\u20090.206; Indica vs Sativa: Cohen\u2019s d\u2009=\u2009\u22120.079). An analogous test for variability in CBD content across strain categories among Chemotype II and Chemotype III flower yielded similar results (Fig. 6f; hybrid vs indica: 2.17%, p\u2009<\u20090.01; hybrid vs sativa: 0.90%, p\u2009=\u20090.261; sativa vs indica: 1.26%, p\u2009<\u20090.05) with modest effect sizes (Hybrid vs Indica: Cohen\u2019s d\u2009=\u20090.148, Hybrid vs Sativa: Cohen\u2019s d\u2009=\u20090.068, Sativa vs Indica: Cohen\u2019s d\u2009=\u20090.268).\n<\/p><p>Importantly, the above results are sensitive to laboratory measurements. Data from most labs reflect the general pattern of hybrids having somewhat higher THC than indica and sativa, which are very similar. However, performing the same regression solely using flower products from Lab F (n\u2009=\u200922,049), which reports the highest mean THC levels, would not detect the higher average THC levels of hybrid flowers (hybrid vs indica: 0.47%, p\u2009=\u20090.21; hybrid vs sativa: 0.28%, p\u2009=\u20090.425). Repeating the analysis with solely flower products from Lab A (n\u2009=\u200950,610), the lab reporting the lowest mean THC levels, replicates the overall result, with hybrids having slightly higher THC than indica (1.13%, p\u2009<\u20090.001) and sativa (0.80%, p\u2009<\u20090.01).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Cannabinoid_variation_within_and_across_popular_commercial_strain_names\">Cannabinoid variation within and across popular commercial strain names<\/span><\/h3>\n<p>While commercial flower products fall into one of three chemotypes based on their THC:CBD ratio (Fig. 1), we wondered how much THC:CBD ratios varied within and between the popular commercial strain names that flower samples are given. Because the strain names of flower products submitted for laboratory testing are simply given by the producer-processor, they do not guarantee the true identity of the strain. In fact, personal correspondence with industry professionals indicated that we should expect flower samples submitted for testing to be mislabeled to some extent, perhaps due to business motives driving products to be given certain strain names based purely on their popularity and hence potential market value.\n<\/p><p>To visualize differences between samples based on their popular commercial strain names, we plotted the THC:CBD ratio for 23 labeled strains (fifteen Chemotype I strains, three Chemotype II strains, and five Chemotype III strains) based on their consumer popularity (determined by cumulative pageviews on Leafly.com; see \"Methods\" section) using data from the labs reporting the lowest (Fig. 7a; Lab A) and highest (Fig. 7b; Lab F) mean levels of THC across all flower products. This revealed clear differences in the THC:CBD profiles reported by each lab, as well as differences between the THC:CBD ratios of samples labeled with different strain names. Moreover, multi-modal distributions were apparent for many strains, with peaks sometimes at drastically different THC:CBD ratios. For example, \u201c<a href=\"https:\/\/en.wikipedia.org\/wiki\/Charlotte%27s_web_(cannabis)\" class=\"extiw wiki-link\" title=\"wikipedia:Charlotte's web (cannabis)\" data-key=\"04b6d1d347d0f4d8df351e2b09e08cab\">Charlotte\u2019s Web<\/a>\u201d is a popular Chemotype III strain that was specifically bred to have high total CBD and low total THC levels. Figure 7a clearly shows that many Charlotte\u2019s Web flower samples tested in Washington commonly fit this profile, but many also fit the profile of Chemotype II and even Chemotype I products with high total THC levels. Thus, for subsequent analyses, we quantified data before and after filtering by modal chemotype. For example, for a strain name like Charlotte\u2019s Web, this would mean considering only measurements within the Chemotype III cutoffs (see \"Methods\" section).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"6e1a01274437838b1e18ca6583a51721\"><img alt=\"Fig7 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/b9\/Fig7_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 7<\/b> Distribution of THC-to-CBD Ratios Vary Across Popular Commercial Strain Names and Between Labs. THC-to-CBD ratios plotted on a logarithmic scale for cannabis flower samples across twenty-three popular commercial strain names for the single lab (Lab A) reporting the lowest <b>(a)<\/b> and the single lab (Lab F) reporting the highest <b>(b)<\/b> overall THC levels for cannabis flower.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>To formalize how well colloquial strain names capture variation in THC:CBD profiles, we estimated a series of multilevel models with random intercepts for each strain<sup id=\"rdp-ebb-cite_ref-GelmanData07_24-0\" class=\"reference\"><a href=\"#cite_note-GelmanData07-24\">[24]<\/a><\/sup> to estimate the share of the total variation in the logged THC:CBD ratio explained by the strain name, before and after filtering by modal chemotype. This information is contained in the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Intraclass_correlation\" class=\"extiw wiki-link\" title=\"wikipedia:Intraclass correlation\" data-key=\"92622bd513fe86e523489350c9c85496\">intraclass correlation coefficient<\/a> (ICC), a measure of similarity within-groups calculated as the ratio of the within-strain variance to the total variance (see \"Methods\" section). The ICC is bounded by 0 and 1, where a value of 0 indicates that a sample\u2019s strain name is completely uninformative of its THC:CBD ratio and a value of 1 indicates that a product\u2019s strain name is perfectly predictive of its THC:CBD ratio. In these data, calculating the ICC is complicated by the relatively large number of test results with reported zero CBD (and thus an unbounded THC:CBD ratio). Thus, ICCs pre-and-post filtering are shown both after omitting results with 0 CBD and coercing these results to tail values (Fig. 8a; see \"Methods\" section). When using tail values, results with zero reported CBD were coerced to a ratio of 3.5 and results with zero THC coerced to \u22122.0. These values correspond approximately to the most extreme values observed in the data (see Fig. 1b).\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Jikomes_SciReports2018_8.jpg\" class=\"image wiki-link\" data-key=\"49e9421214fea1e690d6203669d64afe\"><img alt=\"Fig8 Jikomes SciReports2018 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/9e\/Fig8_Jikomes_SciReports2018_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Fig. 8<\/b> Popular Strain Names as Signal for THC and CBD Content. <b>(a)<\/b> Proportion of variation in log10 THC:CBD ratio explained by popular strain names (Intraclass Correlation Coefficient). 99% CIs are shown, by lab, before (black circles) and after (gray squares) filtering test results by the modal chemotype of each strain name. The ICC is shown both for dropping values for which 0% CBD or 0% THC is reported (left) and coercing cannabinoid ratios for these tests (see Methods). <b>(b)<\/b> Mean THC level of popular chemotype I strains. 99% CIs are shown after filtering by modal chemotype, for the lab reporting the lowest THC levels. <b>(c)<\/b> Mean CBD levels for popular chemotype II (above dotted line) and chemotype III (below dotted line) strain names. Results shown for the lab reporting the lowest mean THC levels.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Across all labs, the pre-filtered strain ICC was 0.57 (omitting results with zero reported CBD or THC) or 0.41 (coercing results with zero reported CBD to 3.5 and zero THC to \u22122.0). After filtering, the ICCs increase, respectively, to about 0.71 and 0.51. This overall test conceals significant variation between labs. Figure 8a shows the ICCs with 99% confidence intervals, by lab, before and after strain name filtering, separately for the two methods of handling results with reported zero CBD. In general, a large portion of the variation in THC:CBD ratio was attributable to strain name. This quantity varied substantially between labs, and filtering results outside a strain\u2019s modal chemotype typically boosted the proportion of variation explained by strain by between about 0.10 and 0.15, depending on the model. In all but one case\u2014Lab F, with missing CBD results coerced to a ratio of 3.5\u2014filtering resulted in a statistically significant (p\u2009<\u20090.01, z test) increase in the proportion of variation explained by strain. Labs varied substantially in the proportion of zero CBD test results, and, consequently, the ICC was more sensitive to the handling of missing data for those labs. For Lab F, the lab with the highest proportion of zero CBD results, the ICC was very sensitive to the handling of missing data; the ICC was 0.76 and 0.48, respectively, before filtering, and 0.92 and 0.53, after filtering. In contrast, Lab A, which had few zero CBD values, had a stable ICC across both methods, with pre-filtered ICCs of 0.65 and 0.64, respectively, and post-filtering ICCs of 0.79 and 0.78.\n<\/p><p>Given these observations, we compared total THC and CBD levels across strains only after filtering data by laboratory and modal chemotype. Using data from only one laboratory ensured that the same laboratory testing protocol was used to measure cannabinoids across flower samples. We chose to use data from Lab A, which had the largest number of flower samples. Comparing mean THC or CBD levels revealed clear and statistically significant differences (p\u2009<\u20090.01, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Wald_test\" class=\"extiw wiki-link\" title=\"wikipedia:Wald test\" data-key=\"175bb23355926cfba9871692b2341ead\">Wald test<\/a>) between many strain names, even within chemotypes (Fig. 8). These results suggest that strain names can provide meaningful, though variable, signals of the composition of flower samples. Filtering results to the strain\u2019s modal chemotype boosted the proportion of variation explained by strain clustering by about 0.15. Furthermore, the strength of this signal varies between labs, particularly with respect to the CBD content of high-THC flower.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Our results confirm that commercial cannabis strains in Washington fall into three principal chemotypes defined by their THC:CBD ratio, similar to landrace<sup id=\"rdp-ebb-cite_ref-HilligAChemo04_13-4\" class=\"reference\"><a href=\"#cite_note-HilligAChemo04-13\">[13]<\/a><\/sup> and commercial Dutch strains.<sup id=\"rdp-ebb-cite_ref-HazekampCannabis16_14-2\" class=\"reference\"><a href=\"#cite_note-HazekampCannabis16-14\">[14]<\/a><\/sup> While this result is unsurprising given the biological constraints on cannabinoid production<sup id=\"rdp-ebb-cite_ref-RussoTaming11_25-0\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PertweeHandbook15_26-0\" class=\"reference\"><a href=\"#cite_note-PertweeHandbook15-26\">[26]<\/a><\/sup>, we were able to use this dataset to investigate outstanding questions about commercial cannabis products widely used by consumers today. These included formal analysis of discrepancies in the cannabinoid levels reported by different laboratories, trends in THC content over time, and systematic differences in the THC:CBD profiles between flower samples with different labeled strain names.\n<\/p><p>A key area of concern for legal cannabis consumers, industry professionals, and state regulators is the accuracy of the state-mandated testing data that is required to be displayed on product packaging. Recent media reports of \u201cTHC inflation\u201d and high-profile suspensions of state-licensed testing facilities have prompted concern over the accuracy of the cannabinoid content of legal cannabis products.<sup id=\"rdp-ebb-cite_ref-Coughlin-BogueWash17_27-0\" class=\"reference\"><a href=\"#cite_note-Coughlin-BogueWash17-27\">[27]<\/a><\/sup> Our analyses revealed clear, systematic differences in the results obtained by different testing facilities in Washington, with some labs consistently reporting higher or lower levels of cannabinoids than others. Moreover, these differences could not be explained by differences in the producer or strain name associated with the samples being submitted, suggesting that discrepancies between labs are likely caused by systematic differences in their testing methodologies. It is crucial that precise standards are adopted by the industry to ensure that laboratories produce results that are reproducible across labs, independent of the exact testing method used, with the ultimate goal of reporting results that consumers can trust. Our analyses provide a potential framework for quantitatively evaluating laboratory quality, including labs\u2019 sensitivity to detecting low-level cannabinoids (Fig. 4), agreement with independent measurements of cannabis products in scholarly journals (discussed below), and differences between labs after accounting for other characteristics.\n<\/p><p>In our analyses of Washington labs, median THC levels for Chemotype I flower varied considerably (Fig. 2), ranging from 17.7% to 23.2%. The results reported by labs with lower median THC levels are in better agreement with independent measurements of cannabis flower from legal markets.<sup id=\"rdp-ebb-cite_ref-VergaraComp17_8-2\" class=\"reference\"><a href=\"#cite_note-VergaraComp17-8\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ElzingaCanna15_28-0\" class=\"reference\"><a href=\"#cite_note-ElzingaCanna15-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LynchGenomic17_29-0\" class=\"reference\"><a href=\"#cite_note-LynchGenomic17-29\">[29]<\/a><\/sup> For example, Vergara <i>et al.<\/i><sup id=\"rdp-ebb-cite_ref-VergaraComp17_8-3\" class=\"reference\"><a href=\"#cite_note-VergaraComp17-8\">[8]<\/a><\/sup> reported total THC levels for commercial flower samples in several U.S. cities, finding that flower samples averaged approximately 19% total THC in Seattle, WA. Even lower levels were (~15%) reported for Denver, CO; Sacramento, CA; and Oakland, CA.<sup id=\"rdp-ebb-cite_ref-VergaraComp17_8-4\" class=\"reference\"><a href=\"#cite_note-VergaraComp17-8\">[8]<\/a><\/sup> Other measurements of flower samples originating from medical cannabis patients in California found median total THC levels for Chemotype I samples to be approximately 17% dry weight.<sup id=\"rdp-ebb-cite_ref-ElzingaCanna15_28-1\" class=\"reference\"><a href=\"#cite_note-ElzingaCanna15-28\">[28]<\/a><\/sup> Another study measured THCA across a variety of popular cultivars and found mean THCA levels to be just over 16% dry weight for specimens with a broad-leaflet phenotype vs. 14% for those with a narrow leaflet phenotype (total THC levels were not reported).<sup id=\"rdp-ebb-cite_ref-LynchGenomic17_29-1\" class=\"reference\"><a href=\"#cite_note-LynchGenomic17-29\">[29]<\/a><\/sup>\n<\/p><p>Reliable cannabis laboratory testing is an attainable goal. In the absence of federal regulations in the United States for the foreseeable future, it will be incumbent on state regulators to implement universal testing standards for cannabis laboratories. But states have extensive experience in this arena, regulating laboratories that analyze drinking water and evidence from crime scenes. They need only hold cannabis laboratories to similar standards. A first step may be to require that cannabis labs, like other testing facilities, receive third-party accreditation of compliance with <a href=\"https:\/\/www.limswiki.org\/index.php\/ISO\/IEC_17025\" title=\"ISO\/IEC 17025\" class=\"wiki-link\" data-key=\"0a89cebb34370dd860cce86881cbf29c\">ISO\/IEC 17025<\/a> guidelines for testing and <a href=\"https:\/\/www.limswiki.org\/index.php\/Reference_laboratory\" title=\"Reference laboratory\" class=\"wiki-link\" data-key=\"f719f408e1660f86b53857eef2f13f32\">calibration laboratories<\/a> by auditors who are themselves ISO-accredited.<sup id=\"rdp-ebb-cite_ref-UngerStand14_30-0\" class=\"reference\"><a href=\"#cite_note-UngerStand14-30\">[30]<\/a><\/sup> Second, states could require that cannabis labs adhere to a standardized testing protocol for analyzing cannabis to ensure consistency between labs. Such protocols have already been developed. For example, the American Herbal Pharmacopoeia published a 64-page monograph called Cannabis Inflorescence detailing analytical procedures for cannabinoid detection and quantification to establish benchmark methods.<sup id=\"rdp-ebb-cite_ref-AHPCanna14_31-0\" class=\"reference\"><a href=\"#cite_note-AHPCanna14-31\">[31]<\/a><\/sup> The American Herbal Products Association\u2019s points readers to Cannabis Inflorescence for guidance regarding \u201cspecific analytical methods\u201d in its recommendations for regulators.<sup id=\"rdp-ebb-cite_ref-AHPARecomm16_32-0\" class=\"reference\"><a href=\"#cite_note-AHPARecomm16-32\">[32]<\/a><\/sup> Finally, states should implement regular \u201cround-robin\u201d audits, sending a blinded, common sample for testing across various laboratories.\n<\/p><p>Overall, our results are consistent with reports of \u201ccannabinoid inflation\u201d by certain laboratories<sup id=\"rdp-ebb-cite_ref-Coughlin-BogueLeafly17_10-1\" class=\"reference\"><a href=\"#cite_note-Coughlin-BogueLeafly17-10\">[10]<\/a><\/sup> and point to intrinsic characteristics of labs as driving the observed variability, rather than confounds introduced from different producers, product types, or strain names being processed by each lab. Previous analysis of data from Washington\u2019s traceability dataset found a statistically significant relationship between the price per gram of flower and both total THC and total CBD levels<sup id=\"rdp-ebb-cite_ref-SmartVariation17_33-0\" class=\"reference\"><a href=\"#cite_note-SmartVariation17-33\">[33]<\/a><\/sup>, which may provide an economic incentive for cannabis producers to seek test results with higher total THC or CBD levels.\n<\/p><p>Restricted to the lab reporting the lowest THC levels for Chemotype I flower (median\u2009=\u200917.7%), the 99th percentile THC level is 27.0%, compared to 31.8% for the lab reporting the highest THC levels. This suggests that flower samples with total THC levels approaching or exceeding 30% are very rare, and that many flower products labeled as containing >30% total THC by dry weight may be inaccurate. Our analysis of changes in THC content over time suggest that THC levels have not risen substantially, at least since 2015 (Fig. 5). While it is well-known that modern recreational cannabis contains higher levels of THC than it did in previous decades<sup id=\"rdp-ebb-cite_ref-ElSohlyChanges16_17-1\" class=\"reference\"><a href=\"#cite_note-ElSohlyChanges16-17\">[17]<\/a><\/sup>, this apparent plateauing in recent years may be expected due to biological limits imposed on cannabinoid production. In <i>Cannabis<\/i>, total potential THC levels are determined by levels of THCA synthesized within the plant, which is under genetic control.<sup id=\"rdp-ebb-cite_ref-deMeijerTheInher03_12-1\" class=\"reference\"><a href=\"#cite_note-deMeijerTheInher03-12\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WeiblenGene15_34-0\" class=\"reference\"><a href=\"#cite_note-WeiblenGene15-34\">[34]<\/a><\/sup>\n<\/p><p>In general, our results do not suggest that flower samples labeled as indica, sativa, and hybrid differ substantially in terms of total THC content or THC:CBD profiles, as the indica\/sativa\/hybrid typology accounted for only about 1% of the relative variability in THC content (r\u00b2\u2009=\u20090.016). Samples labeled sativa vs. indica in Washington\u2019s commercial market do not differ in THC content, similar to what was documented among commercial strains in the Netherlands.<sup id=\"rdp-ebb-cite_ref-HazekampCannabis16_14-3\" class=\"reference\"><a href=\"#cite_note-HazekampCannabis16-14\">[14]<\/a><\/sup> However, we did find that hybrid strains have slightly higher levels of total THC. Since hybrid strains are produced by crossing indica and sativa varieties, this result is somewhat counter-intuitive. We interpret it as potentially owing to the selective breeding of hybrid strains to maximize THC content for commercial purposes. It is also possible that, on average, these group are grown under different growing conditions, such as indoors vs. outdoors.\n<\/p><p>While there were no substantial differences in the THC or CBD content between flower samples labeled as indica vs. sativa, our analyses were limited by the contents of the Washington traceability dataset, which did not include measurements of other <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabinoid#Phytocannabinoids\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabinoid\" data-key=\"4ecae6863238a39cbf28d8a1e2d33b7f\">phytocannabinoid<\/a> or <a href=\"https:\/\/en.wikipedia.org\/wiki\/Terpene\" class=\"extiw wiki-link\" title=\"wikipedia:Terpene\" data-key=\"5378b8663069e9faf390dc59f6cdd279\">terpenes<\/a>, a major class of aromatic compounds produced by <i>Cannabis<\/i> that likely modulate the effects of phytocannabinoids<sup id=\"rdp-ebb-cite_ref-AndreCannabis16_7-1\" class=\"reference\"><a href=\"#cite_note-AndreCannabis16-7\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LewisPharma18_23-1\" class=\"reference\"><a href=\"#cite_note-LewisPharma18-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RussoTaming11_25-1\" class=\"reference\"><a href=\"#cite_note-RussoTaming11-25\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McPartlandCann08_35-0\" class=\"reference\"><a href=\"#cite_note-McPartlandCann08-35\">[35]<\/a><\/sup> It remains possible that indica and sativa samples differ systematically in their full phytocannabinoid or terpene profile. Indeed, a recent analysis of cannabinoid and terpene profiles from Dutch flower samples found several terpenes that may serve as markers for indica- vs. sativa-type samples.<sup id=\"rdp-ebb-cite_ref-HazekampCannabis16_14-4\" class=\"reference\"><a href=\"#cite_note-HazekampCannabis16-14\">[14]<\/a><\/sup> Other recent work<sup id=\"rdp-ebb-cite_ref-LewisPharma18_23-2\" class=\"reference\"><a href=\"#cite_note-LewisPharma18-23\">[23]<\/a><\/sup> has shown how more complete sets of biochemical measurements, including <a href=\"https:\/\/en.wikipedia.org\/wiki\/Terpenoid\" class=\"extiw wiki-link\" title=\"wikipedia:Terpenoid\" data-key=\"2318a85945e9c683392a375f0ee2a609\">terpenoid<\/a> profiles, can be used to define multiple distinct <i>Cannabis<\/i> chemovars.\n<\/p><p>Since flower samples with different strain names within each of the three principal chemotypes can be differentiated somewhat by their THC:CBD ratio, more detailed analysis of the biochemical composition of commercial strains may uncover further differences. Indeed, previous work has attempted to differentiate strains based on analysis of more complete cannabinoid and terpene profiles<sup id=\"rdp-ebb-cite_ref-HazekampCannabis16_14-5\" class=\"reference\"><a href=\"#cite_note-HazekampCannabis16-14\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ElzingaCanna15_28-2\" class=\"reference\"><a href=\"#cite_note-ElzingaCanna15-28\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Aizpurua-OlaizolaEvo16_36-0\" class=\"reference\"><a href=\"#cite_note-Aizpurua-OlaizolaEvo16-36\">[36]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CasanoVari10_37-0\" class=\"reference\"><a href=\"#cite_note-CasanoVari10-37\">[37]<\/a><\/sup> in an effort to define Cannabis \u201cchemovars.\u201d<sup id=\"rdp-ebb-cite_ref-LewisPharma18_23-3\" class=\"reference\"><a href=\"#cite_note-LewisPharma18-23\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HazekampCanna12_38-0\" class=\"reference\"><a href=\"#cite_note-HazekampCanna12-38\">[38]<\/a><\/sup> An important area of future research will be to define the chemovars for commercially available cannabis products in legal U.S. markets. Applying computational approaches to large-scale cannabinoid and terpene datasets will allow cannabis strains and products to be organized according to their biochemical constituents rather than colloquial strain names. Accurate data quantifying the principal constituents of cannabis could foster the development of \u201cproduct maps\u201d that organize commercial products based on levels of their psychoactive and medically-relevant compounds. However, doing so requires having accurate lab testing data in the first place, which requires the establishment of laboratory testing standards that ensure the correct identification and quantification of the contents of cannabis products. Our results highlight the need for such standards given the large, systematic differences observed between laboratories in Washington.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Data\">Data<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Washington_I-502_cannabis_test_data\">Washington I-502 cannabis test data<\/span><\/h4>\n<p>We submitted public records requests for all test results registered with the Washington State Liquor and Cannabis Board (LCB), the agency which regulates cannabis sales. The request was made on August 7, 2017 and the LCB provided us the raw data files on August 14, 2017. These data contained cannabinoid profiles, grower and laboratory information, and test date for 304,123 test results from June 2014 through May 2017 (excluding a very small number of test results that were dropped from the dataset for containing mathematically or biologically impossible values, such as THC content greater than 100%). The data files from which we extracted data for our analyses are attached online.\n<\/p><p>We focused our analyses on two product categories: flower and concentrates. \u201cFlower\u201d refers to products refer to those comprised of the mature female flower of the cannabis plant, which most cannabis products are derived from. Our analyses of flower products came from data in the \u201cFlower Lot\u201d group, which can be found in the \"InventoryLabel\" column of the I-502 dataset accompanying this paper. Our analyses of concentrate products refer to data with the \u201cHydrocarbon Wax\u201d InventoryLabel, which was the largest single group for product types that would normally be considered concentrates. We did not explore data from other product categories in this study, including \u201c<a href=\"https:\/\/en.wikipedia.org\/wiki\/Hashish\" class=\"extiw wiki-link\" title=\"wikipedia:Hashish\" data-key=\"9e510ba46144ed7245a372fc8451d6b3\">Hash<\/a>,\u201d \u201c<a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_edible\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis edible\" data-key=\"36b6417223e7b6de5b8d46843f4d2c4e\">Marijuana Infused Edibles<\/a>,\u201d and a variety of others. We encourage other scholars to mine this dataset for further insights.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Calculation_of_THC_and_CBD_levels\">Calculation of THC and CBD levels<\/span><\/h4>\n<p><i>Cannabis<\/i> plants do not synthesize THC or CBD; instead, they synthesize the cannabinoid acids THCA and CBDA, which are made from a common precursor<sup id=\"rdp-ebb-cite_ref-deMeijerTheInher03_12-2\" class=\"reference\"><a href=\"#cite_note-deMeijerTheInher03-12\">[12]<\/a><\/sup> and must be <a href=\"https:\/\/en.wikipedia.org\/wiki\/Decarboxylation\" class=\"extiw wiki-link\" title=\"wikipedia:Decarboxylation\" data-key=\"1b2f03c2feb1f0d2c530732788090504\">decarboxylated<\/a> (e.g., by heat energy) to yield the phytocannabinoids THC and CBD. Thus, cannabis products, especially flower samples, contain mainly THCA and\/or CBDA, as well as small levels of THC and CBD that result from spontaneous decarboxylation during the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cannabis_cultivation\" class=\"extiw wiki-link\" title=\"wikipedia:Cannabis cultivation\" data-key=\"4dd8a55b106e9dead428ae3b92556976\">cultivation<\/a> process. Total THC or CBD levels, in units of percent of dry weight, are typically calculated as:\n<\/p>\n<ul><li> <span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/a479fd10a694ea0083e6c82c8f5376cd9db95dd5'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -0.838ex; width:39.906ex; height:2.843ex;\" \/><\/span><\/li>\n<li> <span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/ffa1776a19e8527276c3f576c4e0b9614766d7cf'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -0.838ex; width:39.941ex; height:2.843ex;\" \/><\/span><\/li><\/ul>\n<p>...where THCA and THC refer to the percent dry weight concentration of each cannabinoid present in a cannabis product, and 0.877 is the scaling factor accounting for the difference in molecular weight between THCA and THC. \u201cTotal THC\u201d or \u201cTotal CBD\u201d refers to this value, which is the maximum potential THC or CBD content of a cannabis product, assuming 100% decarboxylation of cannabinoid acids. This is the standard way of reporting \u201ctotal\u201d cannabinoid levels for legal cannabis products in Washington. Unless otherwise noted, \u201cTHC levels\u201d or \u201ctotal THC levels\u201d refers to this value (similar for CBD levels).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Leafly_data\">Leafly data<\/span><\/h4>\n<p>Leafly\u2019s database of strains was used to group test results, as described in the matching process below. The \"Indica,\" \"Sativa,\" or \"Hybrid\" categorization from Leafly.com allowed us to categorize strains according to their popular consumer designations. Our data matched to 1,318 unique commercial strains. The popularity of consumer strain names were determined by cumulative pageviews of the strain pages on Leafly.com.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Analytic_methods\">Analytic methods<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Matching_strain_names\">Matching strain names<\/span><\/h4>\n<p>Strain names in the Washington I-502 seed-to-sale traceability data were matched to a database of Leafly strains to retrieve information about strain popularity and indica\/sativa categorization. The matching process had two stages. First, I-502 data strain names were standardized using regular expressions to delete reference to grower\u2019s names and cities, correct spelling irregularities, and expand acronyms (e.g., \u201cGDP\u201d referring to the strain \u201cGranddaddy Purple\u201d). Second, the processed I-502 strain names were matched to the Leafly database using open-source implementations of two different approximate string matching algorithms (Ratcliff\/Obershelp and Levenshtein Distance). If the two algorithms agreed, then the test result was assigned the Leafly strain. Through this process, used to minimize the number of false positive matches, 214,747 (70.6%) results were matched to Leafly strains, including 166,594 for flower samples and the remainder for other product types. A complete list of raw strain names and their corresponding Leafly match can be found in the \"Electronic supplementary material\" section.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Chemotype_cutoffs\">Chemotype cutoffs<\/span><\/h4>\n<p>Following previous scholars<sup id=\"rdp-ebb-cite_ref-HilligAChemo04_13-5\" class=\"reference\"><a href=\"#cite_note-HilligAChemo04-13\">[13]<\/a><\/sup>, test results were classified into chemotypes based on the distinct groups visible after plotting the log<sub>10<\/sub> THC-to-CBD ratio. For our analyses, we defined Chemotype I strains as any with a 5:1 THC:CBD ratio or greater, Chemotype III strains as any with a 1:5 THC:CBD ratio or lower, and the remainder of products were classified as Chemotype II. We called the \u201cstrain chemotype\u201d the modal chemotype among test results of that strain (in the matched Leafly data).\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Statistical_significance_and_effect_size_for_very_large_samples\">Statistical significance and effect size for very large samples<\/span><\/h4>\n<p>Due to the size of the I-502 dataset we analyzed, statistical comparisons often involved samples with tens of thousands of data points each, and statistically significant results were often achieved even for trivial differences between sample means with highly overlapping distributions. Thus, we calculated effect size in a variety of ways to determine whether differences between samples were relatively large or small. When comparing continuous distributions, we computed effect size in two ways: Cohen\u2019s d<sup id=\"rdp-ebb-cite_ref-CohenStatistical88_15-1\" class=\"reference\"><a href=\"#cite_note-CohenStatistical88-15\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SawilowksyNew09_39-0\" class=\"reference\"><a href=\"#cite_note-SawilowksyNew09-39\">[39]<\/a><\/sup> and a \u201cCommon Language\u201d (CL) effect size metric.<sup id=\"rdp-ebb-cite_ref-McGrawAComm92_16-1\" class=\"reference\"><a href=\"#cite_note-McGrawAComm92-16\">[16]<\/a><\/sup>\n<\/p><p>The CL effect size measures the probability that a randomly selected value from one population will be greater than a randomly selected value from another, while Cohen\u2019s d measures effect size as the difference between two samples means divided by their pooled standard deviation<sup id=\"rdp-ebb-cite_ref-CohenStatistical88_15-2\" class=\"reference\"><a href=\"#cite_note-CohenStatistical88-15\">[15]<\/a><\/sup>:\n<\/p><p><span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/c5afa357a38d6a46070facbd9fadfe553ae91639'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -6.005ex; width:23.253ex; height:9.509ex;\" \/><\/span>\n<\/p><p>where <i>y<\/i> denotes each sample mean, and <i>n<\/i> and <i>s<sup>2<\/sup><\/i> are the size and variance of each sample, respectively.\n<\/p><p>For example, Fig. 2a shows that Lab A (Confidence Analytics) and Lab B (Analytical 360) report slightly different median THC values for flower (17.6% and 18.3%, respectively) and display highly overlapping distributions. Performing a t-test to measure a difference in the means (17.5% and 18.0%, respectively) returns p-value\u2009<\u20091.1\u2009\u00d7\u200910<sup>-52<\/sup>. While the t-test indicates a statistically significant result, |Cohen\u2019s d|\u2009=\u20090.13, indicating a \u201csmall effect.\u201d<sup id=\"rdp-ebb-cite_ref-CohenStatistical88_15-3\" class=\"reference\"><a href=\"#cite_note-CohenStatistical88-15\">[15]<\/a><\/sup> Similarly, the CL effect size is 0.46, indicating that a random value from Confidence Analytics will be greater than a random value from Analytical 360 46% of the time, which is close to what identical distributions would display (50%). The effect size matrices displayed in this paper allow one to assess whether differences between samples are relatively large or small, which is especially useful when large sample sizes allow small differences to reach statistical significance.\n<\/p><p>For comparing different proportions, we computed Cohen\u2019s h<sup id=\"rdp-ebb-cite_ref-CohenStatistical88_15-4\" class=\"reference\"><a href=\"#cite_note-CohenStatistical88-15\">[15]<\/a><\/sup>, which is the difference between two proportions\u2019 arcsine transformation:\n<\/p><p><span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/6704a3c9e3d32b2da38f939ea7a5a2373ee668a0'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -1.338ex; width:30.669ex; height:3.176ex;\" \/><\/span>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Cannabinoid_inflation_regression_models\">Cannabinoid inflation regression models<\/span><\/h4>\n<p>Each of the four regression models shown employed a fixed-effect transformation for each grower-strain combination to absorb heterogeneity in cannabinoid content attributable to these factors.<sup id=\"rdp-ebb-cite_ref-WooldridgeIntro15_40-0\" class=\"reference\"><a href=\"#cite_note-WooldridgeIntro15-40\">[40]<\/a><\/sup> Written in the Least Squares Dummy Variable formulation, each equation takes the form:\n<\/p><p><span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/6c01086cb28bc2678d8c10079a00271f37841d0a'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -3.005ex; width:44.57ex; height:7.343ex;\" \/><\/span>\n<\/p><p>where <i>y<\/i> denotes the cannabinoid content of a particular test <i>t<\/i> from lab <i>l<\/i> of producer-strain <i>s<\/i>, <i>S<\/i> indexes each producer-strain combination (omitting one), and <i>L<\/i> indexes each lab (omitting one); <i>D<\/i>\u2019s are indicator variables denoting group (lab and strain) membership, and <i>\u03b7<\/i> is the idiosyncratic model error. The primary quantity of interest is <i>\u03b3<\/i>, the expected deviation in cannabinoid content for each lab relative to the omitted lab, adjusting for grower\/strain and test date. As a robustness check, for regressions subsetted to concentrate products, an additional dummy variable for type of concentrate (e.g., \u201cBubble Hash,\u201d \u201cButane <a href=\"https:\/\/en.wikipedia.org\/wiki\/Hash_oil\" class=\"extiw wiki-link\" title=\"wikipedia:Hash oil\" data-key=\"9364bc33fafecf120facc38438eb2048\">Hash Oil<\/a>,\u201d and \u201cCO<sub>2<\/sub> Hash Oil\u201d) was included to control for potential heterogeneity between labs in their stock of concentrate products, since concentrates differ in potency by their production method<sup id=\"rdp-ebb-cite_ref-RaberUnder15_41-0\" class=\"reference\"><a href=\"#cite_note-RaberUnder15-41\">[41]<\/a><\/sup>; inclusion of this variable does not materially alter the results. Standard errors were clustered by grower-strain, since the number of clusters is large (there were 22,716 grower-strain combinations) and residuals are likely to be correlated within clusters.<sup id=\"rdp-ebb-cite_ref-CameronAPract15_42-0\" class=\"reference\"><a href=\"#cite_note-CameronAPract15-42\">[42]<\/a><\/sup> Average covariate-adjusted cannabinoid values across labs are plotted with 99% confidence intervals estimated through the delta-method using Stata\u2019s margins command, holding covariates at their actual values.<sup id=\"rdp-ebb-cite_ref-WilliamsUsing12_43-0\" class=\"reference\"><a href=\"#cite_note-WilliamsUsing12-43\">[43]<\/a><\/sup>\n<\/p>\n<h4><span class=\"mw-headline\" id=\"THC_and_CBD_variation_across_popular_commercial_strains\">THC and CBD variation across popular commercial strains<\/span><\/h4>\n<p>To examine clustering in THC:CBD ratio by strain name, we estimated unconditional two-level hierarchical models with random intercepts for each strain, across all labs and separately by lab. Each specification took the following form:\n<\/p><p><span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/ea9d493317a97570d8c6800ab3b73fb0d4121c3c'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -0.838ex; width:19.103ex; height:3.509ex;\" \/><\/span>\n<\/p><p>where <i>r<\/i> is the THC:CBD ratio for each test <i>t<\/i> of strain <i>s<\/i>, <i>\u03b3<\/i> is the \u201cglobal-mean\u201d THC:CBD ratio across all strain names, <i>\u03bc<\/i> is the level-two random intercept for each strain (capturing the variance of the mean for each strain around the overall mean), and <i>\u03b5<\/i> is the deviation of each test from its strain mean. The intraclass correlation coefficient (ICC) is calculated as the ratio of the within-group variance to the total variance, which can be decomposed into the within-group variance and the between group variance. Here, the ICC <i>\u03c1<\/i> is the ratio of the within-strain variance to the sum of the between-strain variance and within-strain variance:\n<\/p><p><span><span class=\"mwe-math-mathml-inline mwe-math-mathml-a11y\" style=\"display: none;\"><\/span><meta class=\"mwe-math-fallback-image-inline\" aria-hidden=\"true\" style=\"background-image: url('https:\/\/en.wikipedia.org\/api\/rest_v1\/media\/math\/render\/svg\/9705bf583945aae8ec1514196905e6a7af80eab3'); background-repeat: no-repeat; background-size: 100% 100%; vertical-align: -3.005ex; width:19.769ex; height:6.676ex;\" \/><\/span>\n<\/p><p>which is strictly bounded between 0 (when there is zero within-strain variance) and 1 (when there is zero between-strain variance). The random intercept models were estimated using Stata\u2019s <code>mixed<\/code> command; the post-estimation command <code>estat icc, l(99)<\/code> was used to calculate 99% confidence intervals with a logit transformation to accommodate the restricted domain of the ICC and the delta method to estimate standard errors.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Strain_category_cannabinoid_estimation\">Strain category cannabinoid estimation<\/span><\/h4>\n<p>Regression was used to estimate whether indica, sativa, and hybrid flower products differed systematically in THC and CBD. Standard errors were clustered by strain, since indica, sativa, hybrid is a strain-level property. There were 1,318 unique strains.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Electronic_supplementary_material\">Electronic supplementary material<\/span><\/h2>\n<ul><li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1038%2Fs41598-018-22755-2\/MediaObjects\/41598_2018_22755_MOESM1_ESM.pdf\" target=\"_blank\">Supplemental figures<\/a> (.pdf)<\/li>\n<li> <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/static-content.springer.com\/esm\/art%3A10.1038%2Fs41598-018-22755-2\/MediaObjects\/41598_2018_22755_MOESM2_ESM.csv\" target=\"_blank\">Supplementary dataset I<\/a> (.csv)<\/li><\/ul>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors would like to thank Dr. Brad Douglass and Dr. David Roberson for helpful comments and feedback on the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Nick Jikomes and Michael Zoorob contributed equally to this work. N.J. and M.Z. jointly conceived of the analyses and wrote the manuscript.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>Nick Jikomes is employed by Leafly Holdings, Inc. which is wholly-owned by Privateer Holdings, a for-profit firm in the legal cannabis industry. Neither Leafly nor Privateer have a direct financial interest in any third-party laboratory testing facilities in the legal cannabis industry.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_availability\">Data availability<\/span><\/h3>\n<p>All data generated or analyzed during this study are included in this published article (and its supplementary material). They are also available in the Harvard Data Repository (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/doi.org\/10.7910\/DVN\/E8TQSD\" target=\"_blank\">https:\/\/doi.org\/10.7910\/DVN\/E8TQSD<\/a>).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-GrinspoonHist05-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GrinspoonHist05_1-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Grinspoon, L. (16 August 2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.maps.org\/research-archive\/mmj\/grinspoon_history_cannabis_medicine.pdf\" target=\"_blank\">\"History of Cannabis as a Medicine\"<\/a> (PDF). MAPS<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.maps.org\/research-archive\/mmj\/grinspoon_history_cannabis_medicine.pdf\" target=\"_blank\">http:\/\/www.maps.org\/research-archive\/mmj\/grinspoon_history_cannabis_medicine.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=History+of+Cannabis+as+a+Medicine&rft.atitle=&rft.aulast=Grinspoon%2C+L.&rft.au=Grinspoon%2C+L.&rft.date=16+August+2005&rft.pub=MAPS&rft_id=http%3A%2F%2Fwww.maps.org%2Fresearch-archive%2Fmmj%2Fgrinspoon_history_cannabis_medicine.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WhitingCanna15-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WhitingCanna15_2-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Whiting, P.F.; Wolff, R.F.; Deshpande, S. et al. (2015). \"Cannabinoids for Medical Use: A Systematic Review and meta-analysis\". <i>JAMA<\/i> <b>313<\/b> (24): 2456\u201373. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2015.6358\" target=\"_blank\">10.1001\/jama.2015.6358<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26103030\" target=\"_blank\">26103030<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabinoids+for+Medical+Use%3A+A+Systematic+Review+and+meta-analysis&rft.jtitle=JAMA&rft.aulast=Whiting%2C+P.F.%3B+Wolff%2C+R.F.%3B+Deshpande%2C+S.+et+al.&rft.au=Whiting%2C+P.F.%3B+Wolff%2C+R.F.%3B+Deshpande%2C+S.+et+al.&rft.date=2015&rft.volume=313&rft.issue=24&rft.pages=2456%E2%80%9373&rft_id=info:doi\/10.1001%2Fjama.2015.6358&rft_id=info:pmid\/26103030&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NASEM_TheHealth17-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NASEM_TheHealth17_3-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">National Academies of Sciences, Engineering, and Medicine (2017). <i>The Health Effects of Cannabis and Cannabinoids: The Current State of Evidence and Recommendations for Research<\/i>. National Academies Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.17226%2F24625\" target=\"_blank\">10.17226\/24625<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780309453073.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Health+Effects+of+Cannabis+and+Cannabinoids%3A+The+Current+State+of+Evidence+and+Recommendations+for+Research&rft.aulast=National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rft.au=National+Academies+of+Sciences%2C+Engineering%2C+and+Medicine&rft.date=2017&rft.pub=National+Academies+Press&rft_id=info:doi\/10.17226%2F24625&rft.isbn=9780309453073&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeigerSupport16-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeigerSupport16_4-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Geiger, A. (12 October 2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pewresearch.org\/fact-tank\/2016\/10\/12\/support-for-marijuana-legalization-continues-to-rise\" target=\"_blank\">\"Support for marijuana legalization continues to rise\"<\/a>. <i>Fact Tank<\/i>. Pew Research Center<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pewresearch.org\/fact-tank\/2016\/10\/12\/support-for-marijuana-legalization-continues-to-rise\" target=\"_blank\">http:\/\/www.pewresearch.org\/fact-tank\/2016\/10\/12\/support-for-marijuana-legalization-continues-to-rise<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 29 September 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Support+for+marijuana+legalization+continues+to+rise&rft.atitle=Fact+Tank&rft.aulast=Geiger%2C+A.&rft.au=Geiger%2C+A.&rft.date=12+October+2016&rft.pub=Pew+Research+Center&rft_id=http%3A%2F%2Fwww.pewresearch.org%2Ffact-tank%2F2016%2F10%2F12%2Fsupport-for-marijuana-legalization-continues-to-rise&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ComptonUseOf17-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ComptonUseOf17_5-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Compton, W.M.; Han, B.; Highes, A. et al. (2017). \"Use of Marijuana for Medical Purposes Among Adults in the United States\". <i>JAMA<\/i> <b>317<\/b> (2): 209\u201311. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2016.18900\" target=\"_blank\">10.1001\/jama.2016.18900<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27992636\" target=\"_blank\">27992636<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+Marijuana+for+Medical+Purposes+Among+Adults+in+the+United+States&rft.jtitle=JAMA&rft.aulast=Compton%2C+W.M.%3B+Han%2C+B.%3B+Highes%2C+A.+et+al.&rft.au=Compton%2C+W.M.%3B+Han%2C+B.%3B+Highes%2C+A.+et+al.&rft.date=2017&rft.volume=317&rft.issue=2&rft.pages=209%E2%80%9311&rft_id=info:doi\/10.1001%2Fjama.2016.18900&rft_id=info:pmid\/27992636&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarryAPublic16-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BarryAPublic16_6-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barry, R.A.; Glantz, S. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5038957\" target=\"_blank\">\"A Public Health Framework for Legalized Retail Marijuana Based on the US Experience: Avoiding a New Tobacco Industry\"<\/a>. <i>PLoS Medicine<\/i> <b>13<\/b> (9): e1002131. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1002131\" target=\"_blank\">10.1371\/journal.pmed.1002131<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5038957\/\" target=\"_blank\">PMC5038957<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/27676176\" target=\"_blank\">27676176<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5038957\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5038957<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Public+Health+Framework+for+Legalized+Retail+Marijuana+Based+on+the+US+Experience%3A+Avoiding+a+New+Tobacco+Industry&rft.jtitle=PLoS+Medicine&rft.aulast=Barry%2C+R.A.%3B+Glantz%2C+S.&rft.au=Barry%2C+R.A.%3B+Glantz%2C+S.&rft.date=2016&rft.volume=13&rft.issue=9&rft.pages=e1002131&rft_id=info:doi\/10.1371%2Fjournal.pmed.1002131&rft_id=info:pmc\/PMC5038957&rft_id=info:pmid\/27676176&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5038957&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AndreCannabis16-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AndreCannabis16_7-0\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-AndreCannabis16_7-1\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Andre, C.M.; Hausman, J.-F.; Guerriero, G. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396\" target=\"_blank\">\"<i>Cannabis sativa<\/i>: The plant of the thousand and one molecules\"<\/a>. <i>Frontiers in Plant Medicine<\/i> <b>7<\/b>: 19. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3389%2Ffpls.2016.00019\" target=\"_blank\">10.3389\/fpls.2016.00019<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4740396\/\" target=\"_blank\">PMC4740396<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26870049\" target=\"_blank\">26870049<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4740396<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%27%27Cannabis+sativa%27%27%3A+The+plant+of+the+thousand+and+one+molecules&rft.jtitle=Frontiers+in+Plant+Medicine&rft.aulast=Andre%2C+C.M.%3B+Hausman%2C+J.-F.%3B+Guerriero%2C+G.&rft.au=Andre%2C+C.M.%3B+Hausman%2C+J.-F.%3B+Guerriero%2C+G.&rft.date=2016&rft.volume=7&rft.pages=19&rft_id=info:doi\/10.3389%2Ffpls.2016.00019&rft_id=info:pmc\/PMC4740396&rft_id=info:pmid\/26870049&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4740396&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VergaraComp17-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VergaraComp17_8-0\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-VergaraComp17_8-1\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-VergaraComp17_8-2\">8.2<\/a><\/sup> <sup><a href=\"#cite_ref-VergaraComp17_8-3\">8.3<\/a><\/sup> <sup><a href=\"#cite_ref-VergaraComp17_8-4\">8.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vergara, D.; Bidwell, L.C.; Gaudino, R. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395929\" target=\"_blank\">\"Compromised External Validity: Federally Produced Cannabis Does Not Reflect Legal Markets\"<\/a>. <i>Scientific Reports<\/i> <b>7<\/b>: 46528. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsrep46528\" target=\"_blank\">10.1038\/srep46528<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5395929\/\" target=\"_blank\">PMC5395929<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28422145\" target=\"_blank\">28422145<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395929\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5395929<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Compromised+External+Validity%3A+Federally+Produced+Cannabis+Does+Not+Reflect+Legal+Markets&rft.jtitle=Scientific+Reports&rft.aulast=Vergara%2C+D.%3B+Bidwell%2C+L.C.%3B+Gaudino%2C+R.+et+al.&rft.au=Vergara%2C+D.%3B+Bidwell%2C+L.C.%3B+Gaudino%2C+R.+et+al.&rft.date=2017&rft.volume=7&rft.pages=46528&rft_id=info:doi\/10.1038%2Fsrep46528&rft_id=info:pmc\/PMC5395929&rft_id=info:pmid\/28422145&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5395929&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MorganImpact10-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MorganImpact10_9-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Morgan, C.J.; Schafer, G.; Freeman, T.P. et al. (2010). \"Impact of cannabidiol on the acute memory and psychotomimetic effects of smoked cannabis: Naturalistic study\". <i>The British Journal of Psychiatry<\/i> <b>197<\/b>: 4. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1192%2Fbjp.bp.110.077503\" target=\"_blank\">10.1192\/bjp.bp.110.077503<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20884951\" target=\"_blank\">20884951<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Impact+of+cannabidiol+on+the+acute+memory+and+psychotomimetic+effects+of+smoked+cannabis%3A+Naturalistic+study&rft.jtitle=The+British+Journal+of+Psychiatry&rft.aulast=Morgan%2C+C.J.%3B+Schafer%2C+G.%3B+Freeman%2C+T.P.+et+al.&rft.au=Morgan%2C+C.J.%3B+Schafer%2C+G.%3B+Freeman%2C+T.P.+et+al.&rft.date=2010&rft.volume=197&rft.pages=4&rft_id=info:doi\/10.1192%2Fbjp.bp.110.077503&rft_id=info:pmid\/20884951&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Coughlin-BogueLeafly17-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Coughlin-BogueLeafly17_10-0\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-Coughlin-BogueLeafly17_10-1\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Coughlin-Bogue, T. (28 April 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.leafly.com\/news\/industry\/leafly-investigation-washingtons-top-cannabis-lab-inflating-thc-numbers\" target=\"_blank\">\"Leafly Investigation: Is Washington's Top Cannabis Lab Inflating THC Numbers?\"<\/a>. <i>Leafly<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.leafly.com\/news\/industry\/leafly-investigation-washingtons-top-cannabis-lab-inflating-thc-numbers\" target=\"_blank\">https:\/\/www.leafly.com\/news\/industry\/leafly-investigation-washingtons-top-cannabis-lab-inflating-thc-numbers<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 September 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Leafly+Investigation%3A+Is+Washington%27s+Top+Cannabis+Lab+Inflating+THC+Numbers%3F&rft.atitle=Leafly&rft.aulast=Coughlin-Bogue%2C+T.&rft.au=Coughlin-Bogue%2C+T.&rft.date=28+April+2017&rft_id=https%3A%2F%2Fwww.leafly.com%2Fnews%2Findustry%2Fleafly-investigation-washingtons-top-cannabis-lab-inflating-thc-numbers&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HilligGenetic05-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HilligGenetic05_11-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hillig, K.W. (2005). \"Genetic evidence for speciation in <i>Cannabis<\/i> (Cannabaceae)\". <i>Genetic Resources and Crop Evolution<\/i> <b>52<\/b> (2): 161\u201380. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10722-003-4452-y\" target=\"_blank\">10.1007\/s10722-003-4452-y<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genetic+evidence+for+speciation+in+%27%27Cannabis%27%27+%28Cannabaceae%29&rft.jtitle=Genetic+Resources+and+Crop+Evolution&rft.aulast=Hillig%2C+K.W.&rft.au=Hillig%2C+K.W.&rft.date=2005&rft.volume=52&rft.issue=2&rft.pages=161%E2%80%9380&rft_id=info:doi\/10.1007%2Fs10722-003-4452-y&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-deMeijerTheInher03-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-deMeijerTheInher03_12-0\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-deMeijerTheInher03_12-1\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-deMeijerTheInher03_12-2\">12.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">de Meijer, E.P.; Bagatta, M.; Carboni, A. et al. (2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421\" target=\"_blank\">\"The inheritance of chemical phenotype in <i>Cannabis sativa<\/i> L\"<\/a>. <i>Genetics<\/i> <b>163<\/b> (1): 335\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1462421\/\" target=\"_blank\">PMC1462421<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12586720\" target=\"_blank\">12586720<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1462421<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+inheritance+of+chemical+phenotype+in+%27%27Cannabis+sativa%27%27+L&rft.jtitle=Genetics&rft.aulast=de+Meijer%2C+E.P.%3B+Bagatta%2C+M.%3B+Carboni%2C+A.+et+al.&rft.au=de+Meijer%2C+E.P.%3B+Bagatta%2C+M.%3B+Carboni%2C+A.+et+al.&rft.date=2003&rft.volume=163&rft.issue=1&rft.pages=335%E2%80%9346&rft_id=info:pmc\/PMC1462421&rft_id=info:pmid\/12586720&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1462421&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HilligAChemo04-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HilligAChemo04_13-0\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-HilligAChemo04_13-1\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-HilligAChemo04_13-2\">13.2<\/a><\/sup> <sup><a href=\"#cite_ref-HilligAChemo04_13-3\">13.3<\/a><\/sup> <sup><a href=\"#cite_ref-HilligAChemo04_13-4\">13.4<\/a><\/sup> <sup><a href=\"#cite_ref-HilligAChemo04_13-5\">13.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hillig, K.W.; Mahlberg, P.G. (2004). \"A chemotaxonomic analysis of cannabinoid variation in <i>Cannabis<\/i> (Cannabaceae)\". <i>American Journal of Botany<\/i> <b>91<\/b> (6): 966-75. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3732%2Fajb.91.6.966\" target=\"_blank\">10.3732\/ajb.91.6.966<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21653452\" target=\"_blank\">21653452<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+chemotaxonomic+analysis+of+cannabinoid+variation+in+%27%27Cannabis%27%27+%28Cannabaceae%29&rft.jtitle=American+Journal+of+Botany&rft.aulast=Hillig%2C+K.W.%3B+Mahlberg%2C+P.G.&rft.au=Hillig%2C+K.W.%3B+Mahlberg%2C+P.G.&rft.date=2004&rft.volume=91&rft.issue=6&rft.pages=966-75&rft_id=info:doi\/10.3732%2Fajb.91.6.966&rft_id=info:pmid\/21653452&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HazekampCannabis16-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HazekampCannabis16_14-0\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-HazekampCannabis16_14-1\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-HazekampCannabis16_14-2\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-HazekampCannabis16_14-3\">14.3<\/a><\/sup> <sup><a href=\"#cite_ref-HazekampCannabis16_14-4\">14.4<\/a><\/sup> <sup><a href=\"#cite_ref-HazekampCannabis16_14-5\">14.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hazekamp, A.; Tejkalov\u00e1, K.; Papadimitriou, S. (2016). \"Cannabis: From Cultivar to Chemovar II\u2014A Metabolomics Approach to <i>Cannabis<\/i> Classification\". <i>Cannabis and Cannabinoid Research<\/i> <b>1<\/b> (1). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fcan.2016.0017\" target=\"_blank\">10.1089\/can.2016.0017<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis%3A+From+Cultivar+to+Chemovar+II%E2%80%94A+Metabolomics+Approach+to+%27%27Cannabis%27%27+Classification&rft.jtitle=Cannabis+and+Cannabinoid+Research&rft.aulast=Hazekamp%2C+A.%3B+Tejkalov%C3%A1%2C+K.%3B+Papadimitriou%2C+S.&rft.au=Hazekamp%2C+A.%3B+Tejkalov%C3%A1%2C+K.%3B+Papadimitriou%2C+S.&rft.date=2016&rft.volume=1&rft.issue=1&rft_id=info:doi\/10.1089%2Fcan.2016.0017&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CohenStatistical88-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CohenStatistical88_15-0\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-CohenStatistical88_15-1\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-CohenStatistical88_15-2\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-CohenStatistical88_15-3\">15.3<\/a><\/sup> <sup><a href=\"#cite_ref-CohenStatistical88_15-4\">15.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Cohen, J. (1988). <i>Statistical power analysis for the behavioral sciences<\/i> (2nd ed.). L. Eribaum Associates. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780805802832.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Statistical+power+analysis+for+the+behavioral+sciences&rft.aulast=Cohen%2C+J.&rft.au=Cohen%2C+J.&rft.date=1988&rft.edition=2nd&rft.pub=L.+Eribaum+Associates&rft.isbn=9780805802832&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McGrawAComm92-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McGrawAComm92_16-0\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-McGrawAComm92_16-1\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McGraw, K.O.; Wong, S.P. (1992). \"A common language effect size statistic\". <i>Psychological Bulletin<\/i> <b>111<\/b> (2): 361\u201365. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1037%2F0033-2909.111.2.361\" target=\"_blank\">10.1037\/0033-2909.111.2.361<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+common+language+effect+size+statistic&rft.jtitle=Psychological+Bulletin&rft.aulast=McGraw%2C+K.O.%3B+Wong%2C+S.P.&rft.au=McGraw%2C+K.O.%3B+Wong%2C+S.P.&rft.date=1992&rft.volume=111&rft.issue=2&rft.pages=361%E2%80%9365&rft_id=info:doi\/10.1037%2F0033-2909.111.2.361&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElSohlyChanges16-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ElSohlyChanges16_17-0\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-ElSohlyChanges16_17-1\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">ElSohly, M.A.; Mehmedic, Z.; Foster, S. et al. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987131\" target=\"_blank\">\"Changes in Cannabis Potency Over the Last 2 Decades (1995-2014): Analysis of Current Data in the United States\"<\/a>. <i>Biological Psychiatry<\/i> <b>79<\/b> (7): 613\u20139. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.biopsych.2016.01.004\" target=\"_blank\">10.1016\/j.biopsych.2016.01.004<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4987131\/\" target=\"_blank\">PMC4987131<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26903403\" target=\"_blank\">26903403<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987131\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4987131<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Changes+in+Cannabis+Potency+Over+the+Last+2+Decades+%281995-2014%29%3A+Analysis+of+Current+Data+in+the+United+States&rft.jtitle=Biological+Psychiatry&rft.aulast=ElSohly%2C+M.A.%3B+Mehmedic%2C+Z.%3B+Foster%2C+S.+et+al.&rft.au=ElSohly%2C+M.A.%3B+Mehmedic%2C+Z.%3B+Foster%2C+S.+et+al.&rft.date=2016&rft.volume=79&rft.issue=7&rft.pages=613%E2%80%939&rft_id=info:doi\/10.1016%2Fj.biopsych.2016.01.004&rft_id=info:pmc\/PMC4987131&rft_id=info:pmid\/26903403&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4987131&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MehmedicPotency10-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MehmedicPotency10_18-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mehmedic, Z.; Chandra, S.; Slade, D. et al. (2010). \"Potency trends of \u03949-THC and other cannabinoids in confiscated cannabis preparations from 1993 to 2008\". <i>Journal of Forensic Sciences<\/i> <b>55<\/b> (5): 1209\u201317. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1556-4029.2010.01441.x\" target=\"_blank\">10.1111\/j.1556-4029.2010.01441.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20487147\" target=\"_blank\">20487147<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Potency+trends+of+%CE%949-THC+and+other+cannabinoids+in+confiscated+cannabis+preparations+from+1993+to+2008&rft.jtitle=Journal+of+Forensic+Sciences&rft.aulast=Mehmedic%2C+Z.%3B+Chandra%2C+S.%3B+Slade%2C+D.+et+al.&rft.au=Mehmedic%2C+Z.%3B+Chandra%2C+S.%3B+Slade%2C+D.+et+al.&rft.date=2010&rft.volume=55&rft.issue=5&rft.pages=1209%E2%80%9317&rft_id=info:doi\/10.1111%2Fj.1556-4029.2010.01441.x&rft_id=info:pmid\/20487147&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CervantesTheCann15-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CervantesTheCann15_19-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Cervantes, J. (2015). <i>The Cannabis Encyclopedia: The Definitive Guide to Cultivation & Consumption of Medical Marijuana<\/i>. Van Patten Publishing. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781878823342.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Cannabis+Encyclopedia%3A+The+Definitive+Guide+to+Cultivation+%26+Consumption+of+Medical+Marijuana&rft.aulast=Cervantes%2C+J.&rft.au=Cervantes%2C+J.&rft.date=2015&rft.pub=Van+Patten+Publishing&rft.isbn=9781878823342&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McPartlandCanna17-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McPartlandCanna17_20-0\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-McPartlandCanna17_20-1\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">McPartland, J.M. (2017). \"Cannabis sativa and Cannabis indica versus \u201cSativa\u201d and \u201cIndica\u201d\". In Chandra, S.; Lata, H.; ElSholy, M.A.. Cannabis sativa<i> L. - Botany and Biotechnology<\/i>. Springer. pp. 101\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9783319545646.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cannabis+sativa+and+Cannabis+indica+versus+%E2%80%9CSativa%E2%80%9D+and+%E2%80%9CIndica%E2%80%9D&rft.atitle=%27%27Cannabis+sativa%27%27+L.+-+Botany+and+Biotechnology&rft.aulast=McPartland%2C+J.M.&rft.au=McPartland%2C+J.M.&rft.date=2017&rft.pages=pp.%26nbsp%3B101%E2%80%9321&rft.pub=Springer&rft.isbn=9783319545646&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PearceDiscrim14-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PearceDiscrim14_21-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pearce, D.D.; Mitsouras, K.; Irizarry, K.J. (2014). \"Discriminating the effects of Cannabis sativa and Cannabis indica: A web survey of medical cannabis users\". <i>Journal of Alternative<\/i> <b>20<\/b> (10): 787\u201391. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Facm.2013.0190\" target=\"_blank\">10.1089\/acm.2013.0190<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25191852\" target=\"_blank\">25191852<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Discriminating+the+effects+of+Cannabis+sativa+and+Cannabis+indica%3A+A+web+survey+of+medical+cannabis+users&rft.jtitle=Journal+of+Alternative&rft.aulast=Pearce%2C+D.D.%3B+Mitsouras%2C+K.%3B+Irizarry%2C+K.J.&rft.au=Pearce%2C+D.D.%3B+Mitsouras%2C+K.%3B+Irizarry%2C+K.J.&rft.date=2014&rft.volume=20&rft.issue=10&rft.pages=787%E2%80%9391&rft_id=info:doi\/10.1089%2Facm.2013.0190&rft_id=info:pmid\/25191852&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PiomeliTheCanna16-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PiomeliTheCanna16_22-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Piomeli, D.; Russo, E.B. (2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603\" target=\"_blank\">\"The <i>Cannabis sativa<\/i> Versus <i>Cannabis indica<\/i> Debate: An Interview with Ethan Russo, MD\"<\/a>. <i>Cannabis and Cannabinoid Research<\/i> <b>1<\/b> (1): 44\u201346. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fcan.2015.29003.ebr\" target=\"_blank\">10.1089\/can.2015.29003.ebr<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5576603\/\" target=\"_blank\">PMC5576603<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28861479\" target=\"_blank\">28861479<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5576603<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+%27%27Cannabis+sativa%27%27+Versus+%27%27Cannabis+indica%27%27+Debate%3A+An+Interview+with+Ethan+Russo%2C+MD&rft.jtitle=Cannabis+and+Cannabinoid+Research&rft.aulast=Piomeli%2C+D.%3B+Russo%2C+E.B.&rft.au=Piomeli%2C+D.%3B+Russo%2C+E.B.&rft.date=2016&rft.volume=1&rft.issue=1&rft.pages=44%E2%80%9346&rft_id=info:doi\/10.1089%2Fcan.2015.29003.ebr&rft_id=info:pmc\/PMC5576603&rft_id=info:pmid\/28861479&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5576603&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LewisPharma18-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LewisPharma18_23-0\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-LewisPharma18_23-1\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-LewisPharma18_23-2\">23.2<\/a><\/sup> <sup><a href=\"#cite_ref-LewisPharma18_23-3\">23.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lewis, M.A.; Russo, E.B.; Smith, K.M. (2018). \"Pharmacological Foundations of Cannabis Chemovars\". <i>Planta Medica<\/i> <b>84<\/b> (4): 225\u201333. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1055%2Fs-0043-122240\" target=\"_blank\">10.1055\/s-0043-122240<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/29161743\" target=\"_blank\">29161743<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pharmacological+Foundations+of+Cannabis+Chemovars&rft.jtitle=Planta+Medica&rft.aulast=Lewis%2C+M.A.%3B+Russo%2C+E.B.%3B+Smith%2C+K.M.&rft.au=Lewis%2C+M.A.%3B+Russo%2C+E.B.%3B+Smith%2C+K.M.&rft.date=2018&rft.volume=84&rft.issue=4&rft.pages=225%E2%80%9333&rft_id=info:doi\/10.1055%2Fs-0043-122240&rft_id=info:pmid\/29161743&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GelmanData07-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GelmanData07_24-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Gelman, A.; Hill, J. (2007). <i>Data Analysis Using Regression and Multilevel\/Hierarchical Models<\/i>. Cambridge University Press. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780521686891.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Data+Analysis+Using+Regression+and+Multilevel%2FHierarchical+Models&rft.aulast=Gelman%2C+A.%3B+Hill%2C+J.&rft.au=Gelman%2C+A.%3B+Hill%2C+J.&rft.date=2007&rft.pub=Cambridge+University+Press&rft.isbn=9780521686891&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RussoTaming11-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RussoTaming11_25-0\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-RussoTaming11_25-1\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Russo, E.B. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\" target=\"_blank\">\"Taming THC: potential cannabis synergy and phytocannabinoid-terpenoid entourage effects\"<\/a>. <i>British Journal of Pharmacology<\/i> <b>163<\/b> (7): 1344\u201364. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1476-5381.2011.01238.x\" target=\"_blank\">10.1111\/j.1476-5381.2011.01238.x<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3165946\/\" target=\"_blank\">PMC3165946<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21749363\" target=\"_blank\">21749363<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3165946<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Taming+THC%3A+potential+cannabis+synergy+and+phytocannabinoid-terpenoid+entourage+effects&rft.jtitle=British+Journal+of+Pharmacology&rft.aulast=Russo%2C+E.B.&rft.au=Russo%2C+E.B.&rft.date=2011&rft.volume=163&rft.issue=7&rft.pages=1344%E2%80%9364&rft_id=info:doi\/10.1111%2Fj.1476-5381.2011.01238.x&rft_id=info:pmc\/PMC3165946&rft_id=info:pmid\/21749363&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3165946&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PertweeHandbook15-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PertweeHandbook15_26-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">de Meijer, E. (2015). \"Chapter 5: The Chemical Phenotypes (Chemotypes) of Cannabis\". In Pertwee, R.. <i>Handbook of Cannabis<\/i>. Oxford University Press. pp. 89\u2013110. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9780199662685.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chapter+5%3A+The+Chemical+Phenotypes+%28Chemotypes%29+of+Cannabis&rft.atitle=Handbook+of+Cannabis&rft.aulast=de+Meijer%2C+E.&rft.au=de+Meijer%2C+E.&rft.date=2015&rft.pages=pp.%26nbsp%3B89%E2%80%93110&rft.pub=Oxford+University+Press&rft.isbn=9780199662685&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Coughlin-BogueWash17-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Coughlin-BogueWash17_27-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Coughlin-Bogue, T. (15 September 2017). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.leafly.com\/news\/industry\/washington-labs-launch-effort-to-address-credibility-crisis\" target=\"_blank\">\"Washington Labs Launch Effort to Address Credibility Crisis\"<\/a>. <i>Leafly<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.leafly.com\/news\/industry\/washington-labs-launch-effort-to-address-credibility-crisis\" target=\"_blank\">https:\/\/www.leafly.com\/news\/industry\/washington-labs-launch-effort-to-address-credibility-crisis<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 September 2017<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Washington+Labs+Launch+Effort+to+Address+Credibility+Crisis&rft.atitle=Leafly&rft.aulast=Coughlin-Bogue%2C+T.&rft.au=Coughlin-Bogue%2C+T.&rft.date=15+September+2017&rft_id=https%3A%2F%2Fwww.leafly.com%2Fnews%2Findustry%2Fwashington-labs-launch-effort-to-address-credibility-crisis&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElzingaCanna15-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ElzingaCanna15_28-0\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-ElzingaCanna15_28-1\">28.1<\/a><\/sup> <sup><a href=\"#cite_ref-ElzingaCanna15_28-2\">28.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Elzinga, S.; Fischedick, J.; Podkolinski, R. et al. (2015). \"Cannabinoids and Terpenes as Chemotaxonomic Markers in Cannabis\". <i>Natural Products Chemistry & Research<\/i> <b>3<\/b>: 181. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4172%2F2329-6836.1000181\" target=\"_blank\">10.4172\/2329-6836.1000181<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabinoids+and+Terpenes+as+Chemotaxonomic+Markers+in+Cannabis&rft.jtitle=Natural+Products+Chemistry+%26+Research&rft.aulast=Elzinga%2C+S.%3B+Fischedick%2C+J.%3B+Podkolinski%2C+R.+et+al.&rft.au=Elzinga%2C+S.%3B+Fischedick%2C+J.%3B+Podkolinski%2C+R.+et+al.&rft.date=2015&rft.volume=3&rft.pages=181&rft_id=info:doi\/10.4172%2F2329-6836.1000181&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LynchGenomic17-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LynchGenomic17_29-0\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-LynchGenomic17_29-1\">29.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lynch, R.C.; Vergara, D.; Tittes, S. et al. (2017). \"Genomic and Chemical Diversity in Cannabis\". <i>Critical Reviews in Plant Sciences<\/i> <b>35<\/b> (5\u20136): 349\u201363. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F07352689.2016.1265363\" target=\"_blank\">10.1080\/07352689.2016.1265363<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genomic+and+Chemical+Diversity+in+Cannabis&rft.jtitle=Critical+Reviews+in+Plant+Sciences&rft.aulast=Lynch%2C+R.C.%3B+Vergara%2C+D.%3B+Tittes%2C+S.+et+al.&rft.au=Lynch%2C+R.C.%3B+Vergara%2C+D.%3B+Tittes%2C+S.+et+al.&rft.date=2017&rft.volume=35&rft.issue=5%E2%80%936&rft.pages=349%E2%80%9363&rft_id=info:doi\/10.1080%2F07352689.2016.1265363&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UngerStand14-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UngerStand14_30-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Unger, P.; Brauninger, R.; Hudalla, C. et al. (December 2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/cannabissafetyinstitute.org\/wp-content\/uploads\/2015\/01\/Standards-for-Cannabis-Testing-Laboratories.pdf\" target=\"_blank\">\"Standards for Cannabis Testing Laboratories\"<\/a> (PDF). Cannabis Safety Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/cannabissafetyinstitute.org\/wp-content\/uploads\/2015\/01\/Standards-for-Cannabis-Testing-Laboratories.pdf\" target=\"_blank\">http:\/\/cannabissafetyinstitute.org\/wp-content\/uploads\/2015\/01\/Standards-for-Cannabis-Testing-Laboratories.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Standards+for+Cannabis+Testing+Laboratories&rft.atitle=&rft.aulast=Unger%2C+P.%3B+Brauninger%2C+R.%3B+Hudalla%2C+C.+et+al.&rft.au=Unger%2C+P.%3B+Brauninger%2C+R.%3B+Hudalla%2C+C.+et+al.&rft.date=December+2014&rft.pub=Cannabis+Safety+Institute&rft_id=http%3A%2F%2Fcannabissafetyinstitute.org%2Fwp-content%2Fuploads%2F2015%2F01%2FStandards-for-Cannabis-Testing-Laboratories.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AHPCanna14-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AHPCanna14_31-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Upton, R.; Craker, L.; ElSohly, M. et al., ed. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.herbal-ahp.org\/order_online.htm\" target=\"_blank\">Cannabis Inflorescence<i>: <\/i>Cannabis<i> spp.<\/i><\/a>. American Herbal Pharmacopoeia. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 1929425333<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.herbal-ahp.org\/order_online.htm\" target=\"_blank\">http:\/\/www.herbal-ahp.org\/order_online.htm<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=%27%27Cannabis+Inflorescence%27%27%3A+%27%27Cannabis%27%27+spp.&rft.date=2014&rft.pub=American+Herbal+Pharmacopoeia&rft.isbn=1929425333&rft_id=http%3A%2F%2Fwww.herbal-ahp.org%2Forder_online.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AHPARecomm16-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AHPARecomm16_32-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Cannabis Committee (02 February 2016). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ahpa.org\/Portals\/0\/PDFs\/Committee\/CC\/Cannabis_Dispensing_Recommendations_Regulators.pdf\" target=\"_blank\">\"Recommendations for Regulators - Cannabis Operations\"<\/a> (PDF). American Herbal Products Association<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ahpa.org\/Portals\/0\/PDFs\/Committee\/CC\/Cannabis_Dispensing_Recommendations_Regulators.pdf\" target=\"_blank\">http:\/\/www.ahpa.org\/Portals\/0\/PDFs\/Committee\/CC\/Cannabis_Dispensing_Recommendations_Regulators.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Recommendations+for+Regulators+-+Cannabis+Operations&rft.atitle=&rft.aulast=Cannabis+Committee&rft.au=Cannabis+Committee&rft.date=02+February+2016&rft.pub=American+Herbal+Products+Association&rft_id=http%3A%2F%2Fwww.ahpa.org%2FPortals%2F0%2FPDFs%2FCommittee%2FCC%2FCannabis_Dispensing_Recommendations_Regulators.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmartVariation17-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmartVariation17_33-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smart, R.; Caulkins, J.P.; Kilmer, B. et al. (2017). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5673542\" target=\"_blank\">\"Variation in cannabis potency and prices in a newly legal market: evidence from 30 million cannabis sales in Washington state\"<\/a>. <i>Addiction<\/i> <b>112<\/b> (12): 2167\u201377. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fadd.13886\" target=\"_blank\">10.1111\/add.13886<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" data-key=\"c85bdffd69dd30e02024b9cc3d7679e2\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC5673542\/\" target=\"_blank\">PMC5673542<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/28556310\" target=\"_blank\">28556310<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5673542\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC5673542<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Variation+in+cannabis+potency+and+prices+in+a+newly+legal+market%3A+evidence+from+30+million+cannabis+sales+in+Washington+state&rft.jtitle=Addiction&rft.aulast=Smart%2C+R.%3B+Caulkins%2C+J.P.%3B+Kilmer%2C+B.+et+al.&rft.au=Smart%2C+R.%3B+Caulkins%2C+J.P.%3B+Kilmer%2C+B.+et+al.&rft.date=2017&rft.volume=112&rft.issue=12&rft.pages=2167%E2%80%9377&rft_id=info:doi\/10.1111%2Fadd.13886&rft_id=info:pmc\/PMC5673542&rft_id=info:pmid\/28556310&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC5673542&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeiblenGene15-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WeiblenGene15_34-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Weiblen, G.D.; Wenger, J.P.; Craft, K.J. et al. (2015). \"Gene duplication and divergence affecting drug content in Cannabis sativa\". <i>The New Phytologist<\/i> <b>208<\/b> (4): 1241-50. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fnph.13562\" target=\"_blank\">10.1111\/nph.13562<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26189495\" target=\"_blank\">26189495<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+duplication+and+divergence+affecting+drug+content+in+Cannabis+sativa&rft.jtitle=The+New+Phytologist&rft.aulast=Weiblen%2C+G.D.%3B+Wenger%2C+J.P.%3B+Craft%2C+K.J.+et+al.&rft.au=Weiblen%2C+G.D.%3B+Wenger%2C+J.P.%3B+Craft%2C+K.J.+et+al.&rft.date=2015&rft.volume=208&rft.issue=4&rft.pages=1241-50&rft_id=info:doi\/10.1111%2Fnph.13562&rft_id=info:pmid\/26189495&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McPartlandCann08-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McPartlandCann08_35-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McPartland, J.M.; Russo, E.B. (2008). \"Cannabis and Cannabis Extracts: Greater Than the Sum of Their Parts?\". <i>Journal of Cannabis Therapeutics<\/i> <b>1<\/b> (3\u20134): 103\u201332. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1300%2FJ175v01n03_08\" target=\"_blank\">10.1300\/J175v01n03_08<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+and+Cannabis+Extracts%3A+Greater+Than+the+Sum+of+Their+Parts%3F&rft.jtitle=Journal+of+Cannabis+Therapeutics&rft.aulast=McPartland%2C+J.M.%3B+Russo%2C+E.B.&rft.au=McPartland%2C+J.M.%3B+Russo%2C+E.B.&rft.date=2008&rft.volume=1&rft.issue=3%E2%80%934&rft.pages=103%E2%80%9332&rft_id=info:doi\/10.1300%2FJ175v01n03_08&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Aizpurua-OlaizolaEvo16-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Aizpurua-OlaizolaEvo16_36-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aizpurua-Olaizola, O.; Soydaner, U;. \u00d6zt\u00fcrk, E. et al. (2016). \"Evolution of the Cannabinoid and Terpene Content during the Growth of Cannabis sativa Plants from Different Chemotypes\". <i>Journal of Natural Products<\/i> <b>79<\/b> (2): 324\u201331. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Facs.jnatprod.5b00949\" target=\"_blank\">10.1021\/acs.jnatprod.5b00949<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26836472\" target=\"_blank\">26836472<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evolution+of+the+Cannabinoid+and+Terpene+Content+during+the+Growth+of+Cannabis+sativa+Plants+from+Different+Chemotypes&rft.jtitle=Journal+of+Natural+Products&rft.aulast=Aizpurua-Olaizola%2C+O.%3B+Soydaner%2C+U%3B.+%C3%96zt%C3%BCrk%2C+E.+et+al.&rft.au=Aizpurua-Olaizola%2C+O.%3B+Soydaner%2C+U%3B.+%C3%96zt%C3%BCrk%2C+E.+et+al.&rft.date=2016&rft.volume=79&rft.issue=2&rft.pages=324%E2%80%9331&rft_id=info:doi\/10.1021%2Facs.jnatprod.5b00949&rft_id=info:pmid\/26836472&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CasanoVari10-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CasanoVari10_37-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Casano, S.; Grassi, G.; Martini, V. et al. (2010). \"Variations in terpene profiles of different strains of <i>Cannabis sativa<\/i>\". <i>ISHS Acta Horticulturae<\/i> <b>925<\/b>: 115\u201321. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.17660%2FActaHortic.2011.925.15\" target=\"_blank\">10.17660\/ActaHortic.2011.925.15<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Variations+in+terpene+profiles+of+different+strains+of+%27%27Cannabis+sativa%27%27&rft.jtitle=ISHS+Acta+Horticulturae&rft.aulast=Casano%2C+S.%3B+Grassi%2C+G.%3B+Martini%2C+V.+et+al.&rft.au=Casano%2C+S.%3B+Grassi%2C+G.%3B+Martini%2C+V.+et+al.&rft.date=2010&rft.volume=925&rft.pages=115%E2%80%9321&rft_id=info:doi\/10.17660%2FActaHortic.2011.925.15&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HazekampCanna12-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HazekampCanna12_38-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hazekamp, A.; Fischedick, J.T. (2012). \"Cannabis - from cultivar to chemovar\". <i>Drug Testing and Analysis<\/i> <b>4<\/b> (7\u20138): 660\u20137. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fdta.407\" target=\"_blank\">10.1002\/dta.407<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22362625\" target=\"_blank\">22362625<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cannabis+-+from+cultivar+to+chemovar&rft.jtitle=Drug+Testing+and+Analysis&rft.aulast=Hazekamp%2C+A.%3B+Fischedick%2C+J.T.&rft.au=Hazekamp%2C+A.%3B+Fischedick%2C+J.T.&rft.date=2012&rft.volume=4&rft.issue=7%E2%80%938&rft.pages=660%E2%80%937&rft_id=info:doi\/10.1002%2Fdta.407&rft_id=info:pmid\/22362625&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SawilowksyNew09-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SawilowksyNew09_39-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sawilowsky, S.S. (2009). \"New Effect Size Rules of Thumb\". <i>Journal of Modern Applied Statistical Methods<\/i> <b>8<\/b> (2). <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.22237%2Fjmasm%2F1257035100\" target=\"_blank\">10.22237\/jmasm\/1257035100<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=New+Effect+Size+Rules+of+Thumb&rft.jtitle=Journal+of+Modern+Applied+Statistical+Methods&rft.aulast=Sawilowsky%2C+S.S.&rft.au=Sawilowsky%2C+S.S.&rft.date=2009&rft.volume=8&rft.issue=2&rft_id=info:doi\/10.22237%2Fjmasm%2F1257035100&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WooldridgeIntro15-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WooldridgeIntro15_40-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Wooldridge, J.M. (2015). <i>Introductory Econometrics: A Modern Approach<\/i> (6th ed.). Cengage Learning. pp. 484\u20135. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" data-key=\"f64947ba21e884434bd70e8d9e60bae6\">ISBN<\/a> 9781305270107.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Introductory+Econometrics%3A+A+Modern+Approach&rft.aulast=Wooldridge%2C+J.M.&rft.au=Wooldridge%2C+J.M.&rft.date=2015&rft.pages=pp.%26nbsp%3B484%E2%80%935&rft.edition=6th&rft.pub=Cengage+Learning&rft.isbn=9781305270107&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaberUnder15-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RaberUnder15_41-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Raber, J.C.; Elzinga, S.; Kaplan, C. (2015). \"Understanding dabs: Contamination concerns of cannabis concentrates and cannabinoid transfer during the act of dabbing\". <i>Journal of Toxicological Sciences<\/i> <b>40<\/b> (6): 797\u2013803. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2131%2Fjts.40.797\" target=\"_blank\">10.2131\/jts.40.797<\/a>. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" data-key=\"1d34e999f13d8801964a6b3e9d7b4e30\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/26558460\" target=\"_blank\">26558460<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+dabs%3A+Contamination+concerns+of+cannabis+concentrates+and+cannabinoid+transfer+during+the+act+of+dabbing&rft.jtitle=Journal+of+Toxicological+Sciences&rft.aulast=Raber%2C+J.C.%3B+Elzinga%2C+S.%3B+Kaplan%2C+C.&rft.au=Raber%2C+J.C.%3B+Elzinga%2C+S.%3B+Kaplan%2C+C.&rft.date=2015&rft.volume=40&rft.issue=6&rft.pages=797%E2%80%93803&rft_id=info:doi\/10.2131%2Fjts.40.797&rft_id=info:pmid\/26558460&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CameronAPract15-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CameronAPract15_42-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cameron, A.C.; Miller, D.L. (2015). \"A Practitioner\u2019s Guide to Cluster-Robust Inference\". <i>Journal of Human Resources<\/i> <b>50<\/b> (2): 317-372. <a rel=\"nofollow\" class=\"external text wiki-link\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" data-key=\"ae6d69c760ab710abc2dd89f3937d2f4\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3368%2Fjhr.50.2.317\" target=\"_blank\">10.3368\/jhr.50.2.317<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Practitioner%E2%80%99s+Guide+to+Cluster-Robust+Inference&rft.jtitle=Journal+of+Human+Resources&rft.aulast=Cameron%2C+A.C.%3B+Miller%2C+D.L.&rft.au=Cameron%2C+A.C.%3B+Miller%2C+D.L.&rft.date=2015&rft.volume=50&rft.issue=2&rft.pages=317-372&rft_id=info:doi\/10.3368%2Fjhr.50.2.317&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilliamsUsing12-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilliamsUsing12_43-0\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Williams, R. (2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.stata-journal.com\/article.html?article=st0260\" target=\"_blank\">\"Using the margins command to estimate and interpret adjusted predictions and marginal effects\"<\/a>. <i>The Stata Journal<\/i> <b>12<\/b> (2): 308-31<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.stata-journal.com\/article.html?article=st0260\" target=\"_blank\">https:\/\/www.stata-journal.com\/article.html?article=st0260<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+the+margins+command+to+estimate+and+interpret+adjusted+predictions+and+marginal+effects&rft.jtitle=The+Stata+Journal&rft.aulast=Williams%2C+R.&rft.au=Williams%2C+R.&rft.date=2012&rft.volume=12&rft.issue=2&rft.pages=308-31&rft_id=https%3A%2F%2Fwww.stata-journal.com%2Farticle.html%3Farticle%3Dst0260&rfr_id=info:sid\/en.wikipedia.org:Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Some grammar and punctuation was cleaned up to improve readability. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20190701165447\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.218 seconds\nReal time usage: 2.106 seconds\nPreprocessor visited node count: 33627\/1000000\nPreprocessor generated node count: 41767\/1000000\nPost\u2010expand include size: 258829\/2097152 bytes\nTemplate argument size: 80994\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 1088.652 1 - -total\n 84.63% 921.371 1 - Template:Reflist\n 74.23% 808.124 43 - Template:Citation\/core\n 53.88% 586.522 29 - Template:Cite_journal\n 13.79% 150.130 8 - Template:Cite_book\n 10.84% 118.028 6 - Template:Cite_web\n 8.15% 88.755 63 - Template:Citation\/identifier\n 7.62% 82.938 1 - Template:Infobox_journal_article\n 7.27% 79.155 1 - Template:Infobox\n 4.39% 47.822 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:11013-0!*!0!!en!5!*!math=5 and timestamp 20190701165445 and revision id 35431\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products\">https:\/\/www.limswiki.org\/index.php\/Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","500951db854008d04f1cf96423731db5_images":["https:\/\/www.limswiki.org\/images\/9\/93\/Fig1_Jikomes_SciReports2018_8.jpg","https:\/\/www.limswiki.org\/images\/2\/25\/Fig2_Jikomes_SciReports2018_8.jpg","https:\/\/www.limswiki.org\/images\/3\/37\/Fig3_Jikomes_SciReports2018_8.jpg","https:\/\/www.limswiki.org\/images\/a\/a8\/Fig4_Jikomes_SciReports2018_8.jpg","https:\/\/www.limswiki.org\/images\/b\/b3\/Fig5_Jikomes_SciReports2018_8.jpg","https:\/\/www.limswiki.org\/images\/4\/45\/Fig6_Jikomes_SciReports2018_8.jpg","https:\/\/www.limswiki.org\/images\/b\/b9\/Fig7_Jikomes_SciReports2018_8.jpg","https:\/\/www.limswiki.org\/images\/9\/9e\/Fig8_Jikomes_SciReports2018_8.jpg"],"500951db854008d04f1cf96423731db5_timestamp":1562000085,"190d254a3f08deee60acd9e3bc1a09ee":{"type":"chapter","title":"1. Analytical testing","key":"190d254a3f08deee60acd9e3bc1a09ee"}},"link":"https:\/\/www.limswiki.org\/index.php\/Book:LIMSjournal_-_Summer_2019","price_currency":"","price_amount":"","book_size":"","download_url":"https:\/\/www.limsforum.com?ebb_action=book_download&book_id=85264","language":"","cta_button_content":"","toc":[{"type":"chapter","name":"1. Analytical testing","id":"190d254a3f08deee60acd9e3bc1a09ee","children":[{"type":"article","name":"The cannabinoid content of legal cannabis in Washington State varies systematically across testing facilities and popular consumer products (Jikomes and Zoorob 2018)","id":"500951db854008d04f1cf96423731db5","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:The_cannabinoid_content_of_legal_cannabis_in_Washington_State_varies_systematically_across_testing_facilities_and_popular_consumer_products"},{"type":"article","name":"Chemometric analysis of cannabinoids: Chemotaxonomy and domestication syndrome (Mudge et al. 2018)","id":"ec56e6ba701339e54fcca672efb020ae","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Chemometric_analysis_of_cannabinoids:_Chemotaxonomy_and_domestication_syndrome"}]},{"type":"chapter","name":"2. Cybersecurity","id":"f893de7a74a72d6e2be4c072ada96aa1","children":[{"type":"article","name":"National and transnational security implications of asymmetric access to and use of biological data (Berger and Schneck 2019)","id":"74131eedf92ecea39da4d1e4c07690b3","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:National_and_transnational_security_implications_of_asymmetric_access_to_and_use_of_biological_data"},{"type":"article","name":"Defending our public biological databases as a global critical infrastructure (Caswell et al. 2019)","id":"484090b9c88bd26fd5f834f6ab7cbd8a","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Defending_our_public_biological_databases_as_a_global_critical_infrastructure"},{"type":"article","name":"Cyberbiosecurity: A new perspective on protecting U.S. food and agricultural system (Duncan et al. 2019)","id":"e69b956327faef5b93c17a708e883f73","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_A_new_perspective_on_protecting_U.S._food_and_agricultural_system"},{"type":"article","name":"Smart information systems in cybersecurity: An ethical analysis (Macnish et al. 2019)","id":"c35e087aee1f0ed0d5c790bbd387ef29","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Smart_information_systems_in_cybersecurity:_An_ethical_analysis"},{"type":"article","name":"Cyberbiosecurity: An emerging new discipline to help safeguard the bioeconomy (Murch et al. 2018)","id":"d53838ebf5172963d6c6496807f9617e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Cyberbiosecurity:_An_emerging_new_discipline_to_help_safeguard_the_bioeconomy"},{"type":"article","name":"Assessing cyberbiosecurity vulnerabilities and infrastructure resilience (Schabacker et al. 2019)","id":"7685e865ec42d1c3178d9cb2493521d4","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Assessing_cyberbiosecurity_vulnerabilities_and_infrastructure_resilience"}]},{"type":"chapter","name":"3. Health and laboratory informatics","id":"c4ea21fa8b0c570001e11a4dc3676456","children":[{"type":"article","name":"Laboratory information management software for engineered mini-protein therapeutic workflow (Brusniak et al. 2019)","id":"c2dd7f033df4bcb56d74373a750e4a88","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Laboratory_information_management_software_for_engineered_mini-protein_therapeutic_workflow"},{"type":"article","name":"Determining the hospital information system (HIS) success rate: Development of a new instrument and case study (Ebnehoseini et al. 2019)","id":"36d4da5bfaa11917913bb8842c095eca","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Determining_the_hospital_information_system_(HIS)_success_rate:_Development_of_a_new_instrument_and_case_study"},{"type":"article","name":"Developing workforce capacity in public health informatics: Core competencies and curriculum design (Wholey et al. 2019)","id":"97b51c665da19cc225b05dd3ff621e96","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Developing_workforce_capacity_in_public_health_informatics:_Core_competencies_and_curriculum_design"}]},{"type":"chapter","name":"4. Information management","id":"f31e6b190999c118ef2b710307b7466e","children":[{"type":"article","name":"What is the meaning of sharing: Informing, being informed or information overload? (Haugsbakken 2018)","id":"4be2f1cd180a132f1a4ce3ca4fde9d6e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:What_is_the_meaning_of_sharing:_Informing,_being_informed_or_information_overload%3F"},{"type":"article","name":"DAQUA-MASS: An ISO 8000-61-based data quality management methodology for sensor data (Perez-Castillo et al. 2018)","id":"f59eece05cdc9b79c1305c6bea6b72d6","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:DAQUA-MASS:_An_ISO_8000-61-based_data_quality_management_methodology_for_sensor_data"}]}],"settings":{"show_cover":0,"show_title":0,"show_subtitle":0,"show_full_title":0,"show_editor":0,"show_editor_pic":0,"show_publisher":0,"show_language":0,"show_size":0,"show_toc":0,"show_content_beneath_cover":0,"toc_links":"","cta_button":"","content_location":"","log_in_msg":"","cover_size":""},"title_image":"https:\/\/s3.limsforum.com\/www.limsforum.com\/wp-content\/uploads\/Fig1_Schabacker_FrontBioengBiotech2019_7.jpg"}}
LIMSjournal - Summer 2019
Volume 5, Issue 2
Editor: Shawn Douglas
Publisher: LabLynx Press
Copyright LabLynx Inc. All rights reserved.