{"ID":78004,"post_author":"9208550","post_date":"2018-12-13 16:02:11","post_date_gmt":"0000-00-00 00:00:00","post_content":"","post_title":"LIMSjournal - 2015 Edition","post_excerpt":"","post_status":"draft","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"","to_ping":"","pinged":"","post_modified":"2018-12-13 16:02:11","post_modified_gmt":"2018-12-13 21:02:11","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=78004","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","_ebook_metadata":{"enabled":"on","private":"0","guid":"A7260F82-C570-4D27-997E-6EE74C766B54","title":"LIMSjournal - 2015 Edition","subtitle":"","cover_theme":"nico_3","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_3&subtitle=&editor=Shawn+Douglas&title=LIMSjournal+-+2015+Edition&title_image=https%3A%2F%2Fs3.limsforum.com%2Fwww.limsforum.com%2Fwp-content%2Fuploads%2FFig1_Dander_BMCBioinformatics2014_15.jpg&publisher=LabLynx+Press","editor":"Shawn Douglas","publisher":"LabLynx Press","author_id":"26","image_url":"","items":{"be30bf8a40599ea6d6b707ba00d0baa1_type":"article","be30bf8a40599ea6d6b707ba00d0baa1_title":"Human\u2013information interaction with complex information for decision-making (Albers 2015)","be30bf8a40599ea6d6b707ba00d0baa1_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making","be30bf8a40599ea6d6b707ba00d0baa1_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Human\u2013information interaction with complex information for decision-making\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nHuman\u2013information interaction with complex information for decision-makingJournal\n \nInformaticsAuthor(s)\n \nAlbers, Michael J.Author affiliation(s)\n \nDepartment of English, East Carolina UniversityPrimary contact\n \nE-Mail: albersm@ecu.edu; Tel.: +1-252-328-6374Editors\n \nSedig, Kamran; Parsons, PaulYear published\n \n2015Volume and issue\n \n2 (2)Page(s)\n \n4\u201319DOI\n \n10.3390\/informatics2020004ISSN\n \n2227-9709Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.mdpi.com\/2227-9709\/2\/2\/4Download\n \nhttp:\/\/www.mdpi.com\/2227-9709\/2\/2\/4\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Explanation of terms \n\n3.1 Simple information \n3.2 Complex information \n3.3 Information relationships \n3.4 Contextual awareness \n\n\n4 HII is information interaction, not data interaction \n5 Communication of simple and complex information \n6 HII for decision-making \n\n6.1 Decision-making strategies \n\n6.1.1 Experience-based \n6.1.2 Knowledge-based \n\n\n\n\n7 Conclusions \n8 Conflict of interests \n9 References \n10 Notes \n\n\n\nAbstract \nHuman\u2013information interaction (HII) for simple information and for complex information is different because people's goals and information needs differ between the two cases. With complex information, comprehension comes from understanding the relationships and interactions within the information and factors outside of a design team's control. Yet, a design team must consider all these within an HII design in order to maximize the communication potential. This paper considers how simple and complex information requires different design strategies and how those strategies differ.\nKeywords: human\u2013information interaction, decision-making, complex communication, information design\n\nIntroduction \nA primary goal of human\u2013information interaction (HII) for complex information is to communicate concepts and ideas to be used for decisions. That is, information that a reader uses to develop an understanding of a situation and to make decisions about a complex situation.[1][2][3]\nIn any complex communication situation, the proper amount of information should result in maximum communication; of course, too little, too much, poor organization, or inappropriate content reduces the communication. Stating that the communication needs to provide the proper amount of information could considered tautological. It is easy to claim \u2014 and adds little to a discussion \u2014 a text should have the proper information (I use text within the article in a generic sense: paper, web pages, etc). However, defining and measuring the communication value of a document (and the information it contains) proves to be a much more difficult task.\nThe interaction and presentation needs of complex information are very different from those of simple information, where a person looks for single information elements. For example, consider the difference in looking up the score of yesterday\u2019s game (simple information) and considering house remodeling to help care for a chronically sick parent (complex information). When we move from a communication viewpoint to an HII viewpoint, we also bring into play issues of how a person can manipulate the information.[2][4][5] The base text may contain the \u201cproper information\u201d but for maximum communication, it may need to be reordered or otherwise manipulated to meet individual needs.\nA large percentage of the HII literature either looks at simple information search and interaction or conflates simple and complex information. It also tends to take a web-based focus with the person having an essentially infinite number of sources to explore.[6] While this is true in many cases, within this article, I take a much narrower focus on HII and look at the construction of content within a single set of information that is under a content development team\u2019s control, such as the production of corporate reports or public information for a specific goal (an example presented later will be on communicating hurricane information). A view focused on the design team\u2019s task needs to consider what information to create and how people will interact with it. The more commonly discussed view of people searching the Internet (or other sources) for existing information[7] will not be directly considered since a design team has no control over how that search proceeds or what information the person may view.\n\nExplanation of terms \nIn the introduction, I used the terms simple and complex information. Here I provide a working definition for each. \n\nSimple information \nSimple information can be characterized by existing as a single information element. A high percentage of the \u201clook it up on Google\u201d amounts to simple information. For example, a person can look up when a movie was released or a recipe for a carrot cake. Both fit the definition of simple information. Factors that distinguish simple information include:\n\n Single path. One path can be fully defined that will result in the answer.\n Right\/wrong answer. The correctness can be tested and the information declared correct or not correct.\n Complete information. The completeness can be tested and the information declared complete or not complete.\n Closed system. All of the factors that might influence the answer can be defined and accounted for.\nComplex information \nComplex information exists at the opposite end from simple answers. There is no single \u201canswer\u201d. Addressing complex questions requires using and integrating multiple information elements, which often conflict. People have a complex web of information needs and interactions to fill those needs.[8] For example, analyzing monthly business reports and making decisions about this month\u2019s production, or making health care choices. Factors that distinguish complex information include:\n\n Multiple paths. There is no single path to an answer. A person can take many different paths and all will work. The effectiveness of the paths may, of course, vary.\n Open-ended. The idea of \u201ccomplete\u201d is undefined. A person can continue to collect information and refine their understanding with an essentially infinite amount of information. Instead, a person has to pick a stopping point and make a decision.\n Needs cannot be predefined. The information that a person needs cannot be predefined. Of course the major or essential information can be predefined, but the many smaller information elements that can exert a strong influence vary too much between individuals.\n History. The information exists within a continuum and that history influences how it gets interpreted and used.\n Non-linear. The overall situation shows a non-linear response with small differences in some information elements resulting in very large differences in appropriate decisions.\n Open system. All of the factors that might influence the decision cannot be defined. There are too many and the information is dynamic, changing on time scales relevant to the decision situation.\nMost of the simple information that gets communicated tends to be single facts or procedural type of information. Complex information, on the other hand, deals with a broader scope.\n\n Complex information communicates concepts and ideas.\n Complex information communicates an understanding of a situation.\n Complex information communicates relationships and interactions.\nClearly, the HII required to maximize the communication of complex information is very different between the two. Most writing and UX guidelines deal with communicating simple information. But to really address people\u2019s information needs, we need to consider why and how they interact with complex information. Consider the case of electronic health records \u2014 they clearly represent complex information \u2014 and represent an example where the design team has control over a significant part of how the HII occurs. The design has to consider many potentially conflicting sets of priorities (development time, maintenance cost, hospital administration goals, government regulations, medical needs, etc.), which, in turn, affect how well the users can interact with the information. The input methods must be efficient for data entry, but presenting the information the same way to a nurse or physician may not be the best for integrating it into a clear view of the patient\u2019s condition. The relationships of information (connecting the results of various medical tests, etc.) should reflect the initial diagnosis and should also reflect the changing patient status as treatment progresses.\nIn previous work, I have defined a situation as \u201cthe current world state which the user needs to understand. An underlying assumption is that the user needs to interact with a system to gain the necessary information in order to understand the situation\u201d[1] (p. 11). The information relevant to the HII exists both inside and outside any circle drawn to enclose \u201cthe situation\u201d (Figure 1). Honestly, the difficulty of drawing that enclosing circle is a characteristic of a complex situation.\n\n\n\nFigure 1. Complex situation with information both inside and outside of a design team\u2019s control.\r\nHuman\u2013information interaction (HII) only directly applies to the information a design team controls.\nOnce an upfront analysis has defined the people\u2019s goals and information needs, then the HII factors which support manipulating the information and revealing the relationships can be considered and entered into the design.[2][9]\n\nTypically, the failure of these technical documents [used here in a generic sense for any information source] comes not from a lack of information; the text probably contains an excess of information. Post hoc studies of communication failures find many sources to blame: poor information architecture, poor organization, wrong grade level or writing style, or poor presentation. But instead of seeing these problems as a root cause, let\u2019s consider them as symptoms of a more fundamental problem: a problem stemming from the underlying complexity of the situational context and a failure of the information presentation to match that complexity.[10](p. 110)\nThe task assigned to an HII design team is to ensure the underlying complexity is not overly simplified and that the information presentation and manipulation meets the needs of the people and the demands of the situation. As others have discussed, the HII problem for maximizing the communication for complex information is not a tools question, but one of understanding what people need and how they come to understand the information.[11]\n\nInformation relationships \nUnderstanding complex information depends on understanding relationships, but the potential relationships are essentially infinite. Comprehension of a complex situation occurs when people can mentally integrate those relationships into their view of the situation.\n\nInformation integration lies, not in a text element itself, but in the relationships between those elements. A reader needs to figure out what information is relevant and how to connect it to the current problem. Without proper information relationships, the reader does not gain an integrated understanding of information, but instead gains a collection of facts. Without relationships, information exists as a bunch of interesting factoids which do not help a person form an adequate mental picture of the situation. Collections of facts are less than useful for understanding and working with the open-ended problems that people encounter in complex situations.[12][13] Without the relationships, a person learns about X and Y, but not how X and Y relate to each other or to Z in terms of their current problem or situation.[14] The text fails to communicate because the reader can\u2019t form the necessary information relationships.[10](p. 111)\nA patient trying to understand a medical problem often has difficulty understanding the situation because they don\u2019t understand the relationships. They can look at a set of lab results and know the numbers, but lack the medical practitioner\u2019s knowledge of what the values should be for their condition. The patient may be worried because some values are high, but the physician is satisfied with them because the specific condition often has even higher values. Likewise, a junior manager needs to develop the knowledge of how to interpret multiple values that may appear on a collection of monthly reports. Good HII can help by providing a more integrated view and providing help with the integration, without interfering with more experienced people\u2019s information interaction.\nBuilding relationships means developing a deep-level understanding of the text rather than just a surface-level knowledge. Deep-level knowledge involves seeing the macrostructure of the text and being able to apply prior knowledge to it and fit the text within the reader\u2019s prior knowledge framework. Surface-level knowledge involves knowing the basic text. People with surface-level knowledge can quote the text and if asked recall-type questions would respond with answers very close to the text language, but they would not be able to elaborate on the answer or connect it to other information. People with deep-level understanding would be able to place the text\u2019s information into their own words. A major element in the difference between people being able to develop deep-level understanding versus a surface-level understanding is their prior knowledge.\nCommunicating complex information for decision-making can be viewed as working to help build relationships within the information.[8] Unfortunately, building relationships is a great theoretical concept, but one that does not lend itself to a direct operational definition. One way to define judging or qualifying relationships can be how they reduce the uncertainty a person has about the situation.[15] This reshapes an HII design team\u2019s goal to one of focusing content creation in terms of what information does the reader need to reduce their uncertainty and, consequently, to build a web of relationships between the information elements. \n\nContextual awareness \nExcept for training material, the readers of technical information typically understand the basics but need to know specific information about the current situation (as opposed to the general situation) in order to make decisions. The understanding of a complex situation needed to make informed decisions comes when people can distinguish the information structure, grasp the relationships within it[16][17][18], and make inferences on the future evolution of the situation.\nBuilding on Endsley\u2019s (1995)[19] situation awareness work, we can call this contextual awareness. Contextual awareness is the understanding of the information within an informational situation which forms the basis for how to interpret new information and how to make decisions for interacting with that situation. With poor contextual awareness, people can know something is occurring or that a particular piece of information exists, but they cannot easily find relevant related information or they have the information but do not understand how it relates to the overall situation. On the other hand, good contextual awareness does not guarantee a person will form the proper intention or make the proper decisions; the error analysis literature is filled with cases where people understood a situation, but still made incorrect choices. Unfortunately for design teams, the concept of relevance itself is highly nuanced and multifaceted[17], creating a complex interplay that must be understood to engineer high quality HII.\nElements of good contextual awareness are (Figure 2): (1) Understands how the information fits within the current situation. (2) Understands the information relationships.[10] Information comprehension requires knowing how information relates to other information. (3) Understands the future development of the situation and can make predictions about the ripple effects of any decision across the entire situation. HII that supports complex information needs to provide the interactions that support people developing high quality contextual awareness. \nKain, de Jong, and Smith (2010)[20] study into how to communicate hurricane risks and warnings highlighted the issues of how people interpret information and make decisions about how they will react. The hurricane experts had their view of what information was needed and how it should be presented, but the research showed the people wanted\/needed a different presentation. Their mental methods of forming relationships and of interpreting complex information differed from what the experts thought. The process of building contextual awareness differs and a design team\u2019s analysis must capture those differences.\nA design team working on developing the HII for a country\u2019s hurricane awareness plan needs to balance both the expert\u2019s \u201chere is what the people must hear\u201d against the more pragmatical \u201chere\u2019s what I want to know\u201d as well as the local people\u2019s opinion of how they react to hurricane warnings. Many of them have been through multiple hurricanes and have strongly held views that often conflict with the authorities. Notice how in this case, the design team has control of the information. A general search on hurricane warnings will have sentences such as \u201cconsult your local authorities for evacuation routes\u201d but, here, the design team will be tasked with providing that evacuation route information. At the same time, they need to ensure the relationships between evacuation route, getting ready for evacuation, and planning on returning are all clearly laid out and connected. \n\n\n\nFigure 2. Stages of developing contextual awareness\nHII is information interaction, not data interaction \nPeople who are engaged with decision-making and complex information need to be presented with information, not data (Figure 3). Quick definition of my terms here, which I discuss in more detail elsewhere.[1]\n\n Data: Raw numbers, facts, and figures.\n Information: Information is data in context. It relates to the situation and contains the relationships that connect the information to the situation.\n Knowledge: Interconnected web of the relevant information and the relationships linking the information within the situation. \n\n\n\nFigure 3. Data, information, and knowledge hierarchy. The higher the HII works in the hierarchy,\r\nthe better it fits building an understanding of the situation.\nAs an example, I heard a presentation that looked at the effect of sea level rise on the Norfolk, VA area.[21] A software program had nice manipulation that let a person dynamically see the effects of different amounts of sea level rise from global warming and how it would affect the city. The problem was not with the implementation, but with the underlying assumptions of a design team. They were assuming that by providing a tool and letting people see how different sea level changes would affect the area that it would bring about understanding and increase long-term preparedness. But the tool was presenting content at the data level. Yes, I can change the sliders and see how the sea level changes, but it was devoid of supporting content (in the metaphor of this paper, the tool was a single puzzle piece). It not only didn\u2019t connect with other pieces, those other pieces were not provided. As a result, a global warming denier could play with the model, agree that a four foot rise would be a catastrophic problem, but then reject it as something that would never happen. One of the software\u2019s goals was to help people prepare, but without giving them the content and relationships to build their contextual awareness; it was a single data point and not part of a coherent presentation of information. \nThe transition from information to knowledge is important since it involves comprehending the relationships within the data and placing them within the context of the situation. Moving to knowledge is essential to building contextual awareness and must be the goal of an HII design team.\n\nCommunication of simple and complex information \nMany discussions on communication start with Shannon information theory (1948)[22] (Figure 4), which works for analyzing simple information. With clearly defined information to be conveyed, a design team can concentrate on efficient communication methods. Unfortunately, Shannon was really discussing the minimum information to convey a message, not how to communicate in a natural language. It works well for computer-computer communication, but is not as applicable to human-human communication, especially for complex information, with or without a computer mediator. \n\n\n\nFigure 4. Basic block diagram of Shannon information theory. Information moves from the source\r\nto the final destination with the overall goal to be minimizing both the noise and total amount of required content.\nDriven in part by a Shannon-based desire to efficiently communication information, design teams try to break the communication into individual components. This fits with our reductionist nature to break any problem into individual components. Standard approaches for communicating complex information (and analyzing any complex system) are to assume it is assembled from smaller systems[3][13][23][24]. Thus, it is hardly a surprising statement that we see a complex system as the sum of its components: 1 + 1 + 1 + 1 + \u2026.. = THE SYSTEM.\nUnfortunately, assuming a complex system is the sum of its components at best redefines it as a simple system. The complexity that people need to understand and the HII must support is not just a sum of individual components, but the interactions and relationships between components. The redefinition to a sum of individual components ignores the complex interactions that give rise to more than the sum of the parts: 1 + 1 + 1 + 1 + \u2026..+ in >> \u01a9n. \nThe communication problem, and consequently the HII problem, is that redefining the problem to be a simple problem fundamentally changes the problem itself. As a result, the communication itself changes and the relationships and interactions that are essential to understanding get lost. The communication-related literature seems to generally miss this important concept, although the computer science literature consistently contains quotes such as \u201cthe fundamental reason today\u2019s software engineering cannot effectively manage inherent complexity is that its basis is in developing individual programs rather than in interacting systems.\u201d[24] (p. 73). In designs that strive to communicate complex information, an over-privileging of developing and\/or organizing content while discounting the interactions and relationships within and between content leads to results similar to those critiqued by Sommerville. \nIn the introduction, I described most simple information as being single information elements that can be looked up (Figure 5). Complex information on the other hand, has a multitude of information elements that a person must understand, some of which are outside of the bounds of a HII system. Factors such as asking a person down the hall, having heard of a bad experience with product X, etc.[10] (p. 115) \n\n\n\nFigure 5. Relationships and understanding a situation\nFigure 5 shows content as nice rectangles. But they could be better viewed as jigsaw pieces with the interlocks on the pieces representing the relationships (Figure 6), how the pieces interconnect with each other into a coherent understanding. \n\n\n\nFigure 6. Relationships and the complexities of assembling information into a coherent understanding.\r\nThe understanding comes from both the content and knowing how it fits together.\nUnfortunately, the pieces in Figure 6 are not static jigsaw shapes. They are better viewed as a collection of puzzle pieces that change shape as they move away\/toward other pieces, and as they merge together (Figure 7). As a result, changes to one piece can ripple through the entire information web. A change to a piece in the upper left corner results in many of the pieces changing in some relevant manner. The information and HII appropriate and relevant for each point of a situation can be different. Situations are not static; they have a beginning, middle, and end. The puzzle pieces of Figure 7 are not just dynamic from a person\u2019s interaction, but also change over time and over the situation\u2019s evolution. \n\n\n\nFigure 7. Dynamic evolution of information. The relationships between information elements\r\ncan change over both time and by the act of connecting them.\nLet\u2019s reexamine the six characteristics of complex information with respect to the puzzle metaphor.\n\nMultiple paths: People have multiple paths through the information. The order in which they move between the puzzle pieces cannot be predefined. Each HII with a piece can change how it interconnects; thus, different paths through the information change how people build the relationships and, consequently, how the pieces fit together. \nOpen-ended: No clear point of enough information. The HII cannot move toward a predefined point of \u201cnow you have all the information.\u201d With a goal of communicating concepts and ideas, how many pieces a person needs, cannot be defined. Of course, the issue of information needs versus information wants also comes into play. Coupled with this is the fact that information search and problem-solving are sufficing processing.[25][26] People stop once they are comfortable with their understanding of the found information. Unfortunately, people are poor judges of knowing they have found adequate information.\nCannot be predefined: How many pieces a person will interact with remains unknowable to a design team. Interestingly, the size (content) of individual pieces is dynamic; some people need smaller\/larger pieces to effectively comprehend the overall situation and tend to make choices in terms of immediate, rather than long-term, efficiency and effort of the HII.[27][28]\nHistory: Past history of a situation affects how the pieces will evolve and the past history of the people interacting with the information affects how they interpret it. Two situations may appear to be the same but the past strongly influences how the piece will change. For example, six months ago sales in the southeast region were down by 30%. A set of decisions was made and sales have steadily increased since them. They are still 10% below desired levels, but are increasing at an acceptable rate. This situation is much different than if sales are 10% below the desired level and not conforming to predictions.\nNon-linear: The relationships between pieces can show non-linear response to changes. Minor differences in past history or the HII path can result in widely different \u2014 but appropriate \u2014 end points. In individual piece can morph into very different final shapes, even though they started from similar initial conditions.\nOpen system: As the system evolves, the overall content within the system changes. Some new information gets introduced and some information drops out. The number of pieces, their content, and their shape can change.\nWhen the information is viewed as dynamic puzzle pieces, how those pieces change shape helps to emphasize how many solutions to problems introduce other problems, sometimes worse than the one they were intended to solve. For example, many environmental solutions inevitably end up doing more harm than good. The future changes to relationships are not understood when decisions are made, so the evolution of the situation is poorly predicted (poor contextual awareness). Worse, too many people make assumptions that they can change one piece with nothing else changing. In complex ecosystems \u2014 biological, educational or industrial \u2014 actions always have consequences that are hard to predict; a change to one piece ripples through all the others. The \u201chard to predict\u201d is a hallmark of a complex system. HII that supports interacting with the information must consider those ramifications.\nBeyond the need for the HII to handle dynamic puzzle pieces, it must also allow for easy trimming of the information space. All of the available information is not relevant to any specific situation. Instead, a person needs a select collection of information (Figure 8).\n\n\n\nFigure 8. Information relevant to a situation. Each situation (or class of situations) only needs\r\na subset of the information. High quality HII helps to keep the focus on only the relevant subset.\nHII for decision-making \nThe overall scope of the information flow for the HII of complex information moves to prominence when the interaction goal is decision-making. People need to collect information, analyze it, and make a decision; a process that depends on the flow of information both within and outside the control of an HII design team.\nDecision-making occurs as a result of comparing what is perceived in the environment and what is known by the decision-maker.[29] Essentially all of our current literature discusses considering the needs of people when designing information. However, current practice does not typically consider how people\u2019s questions depend on how they make decisions and interact with the system.[30][31] HII attempts to take the entire cycle of information interpretation and decision-making and place it within people\u2019s current situation.\nMirel (1998)[12] follows the same line of thinking as Conklin\u2019s (2003)[32] wicked problems when she points out that analyzing complex tasks requires seeing more than a single path:\n\nThis broader view is necessary to capture the following traits of complex tasks: paths of action that are unpredictable, paths that are never completely visible from any one vantage point, and nuance judgments and interpretations that involve multiple factors and that yield many solutions.[12] (p. 14)\nMirel may have focused on the analysis aspects, but that is an early and vital step in any information interaction, especially for HII of complex information. Unless a design team clearly understands the people\u2019s goals and information needs, there is minimal chance for the HII to support it.\nThe issue of wicked problems and unpredictable paths that both Conklin and Mirel discuss brings to light an interesting point. As design teams work toward HII of complex information, they encounter an interesting and confounding recursion: the HII of complex information is itself a complex problem and, as such, does not lend itself to easy answers. Yet, many design teams try to address those issues with the same process that works for simple information, which leads to an easy-answer mentality. As a result, they reduce the scope and avoid confronting the complex information HII issues head-on. Design teams who try to reduce all problems to a collection of individual parts or who assume they should consider only the information they control fall into this trap. They have redefined the complex HII problem into a simple problem. \n\nDecision-making strategies \nPeople use the available information to make a decision about how to proceed in their current situation; information use is always based upon the context within which it is used.[16] This hard connection between use and context greatly complicates a design team\u2019s task since they must understand both factors. And, depending on audience, the base information may exist within multiple contexts.\nBased on the results of the decision, people take action and then need to re-analyze the situation. Closely related to decision-making is monitoring the situation to ensure it continues along the predicted path. If the information fits the expected progress, then a decision of \u201cdo nothing\u201d is correct. However, if the information indicates that it is deviating, then people cycle back into decision-making.\nThere are two broad decision-making strategies that people use. \n\nExperience-based \nExperience-based decision-making is used the most and is a person\u2019s default strategy. It is a rule-based strategy: \u201cif X happens, then do Y.\u201d Most decision-making, at least in the corporate world, is\nexperienced-based.[12] The reader is skilled in the overall area (such as a CFO making financial decisions), but does need specific information for the current situation. They need to know how this month\u2019s production or sales numbers differ from last month\u2019s or between different regions, but they do have a set of mental rules on how to interpret those numbers.\nBased on past experience, people develop a rule-based strategy so they can reliably expect that \u201cwhen X happens, then do Y.\u201d Rule-based decisions reduce cognitive effort and work well within normal situations because the rules themselves evolved from past experience. However, that same rule-based strategy can cause problems if a person jumps to a decision based on incomplete information. X can happen in multiple situations but other factors differ\u2014the relationships of what\/how X is interconnected within the situation\u2014which cause solution Y to only be an appropriate response in some situations.\nKlein\u2019s (1999)[25] recognition-primed model of decision-making posits that experienced people do little problem solving, especially in routine situations. Instead, once they recognize a situation, they form a possible intention very rapidly, mentally evaluate it, and, if no major problems are evident, take action on it. Alternatives are not considered. In other words, they tend to use their prior knowledge and initial assessment to immediately pick a solution, and if it seems workable, then that is considered the answer. (Concerns about whether or not the solution is optimal are not part of the mental evaluation.)\nPost-failure analysis tends to highlight that the decision-makers had the proper information, but improperly interpreted it. In many cases, it is a result of using experience-based decision-making and not realizing the \u201cif x, then y\u201d rule did not apply to the situation. \n\nKnowledge-based \nExperience-based decision-making can break down, typically when the situation contains unexpected factors or relationships. When people recognize their rules no longer apply, they have to shift to knowledge-based decision-making. Unfortunately, the shift to knowledge-based decision-making often happens after the monitoring phase of an experienced-based strategy reveals the situation is not developing as predicted.\nIn knowledge-based decision, a person knows their experience-based rule set has broken down and they are no longer working within a normal situation. Something is not right in the current situation and they are trying to figure out what to change to fix the problem. In other words, the expected relationships are not being found and a person needs to understand why. It tends to happen when people do not understand what caused the base problem. Unlike decision-making based on rules of \u201cwhen X happens, then do Y,\u201d a person has to fully develop their contextual awareness so they can make a decision.\nThe HII of complex information applies more for knowledge-based strategies than it does for experience-based ones. Knowledge-based strategies require people to engage in a systematic interaction with the information and work to build up their contextual awareness. \nSomeone working on a global warming problem is working at the knowledge-based and not experienced-based level. Allowing people to manipulate data as if it was a simple information \u2014 a collection of individual components \u2014 can misguide them into thinking it\u2019s an experience-based problem and they can draw on their prior knowledge. Worse, defining the problem as \u201cif they can work this sea level rise model, then they will understand the problem\u201d redefines the problem from a complex problem to a simple problem. Unfortunately, but inevitable with the history component of complex information, prior knowledge and biases strongly influence how information gets interpreted. Of course, with global warming\u2014and many other situations\u2014the political aspects, factors outside of an HII design team\u2019s control, come to the forefront. \n\nConclusions \nProviding people with any collection of information, especially complex information, can easily result in information overload. With the large amounts of information and its dynamic nature, high quality HII is essential for people to comprehend that information. \nRather than a lack of information, the failure to anticipate people\u2019s needs forms the basis of most information problems and poor decision-making; an issue design teams must explicitly address. Managers typically suffer from a shortage of decision-relevant information and a simultaneous overabundance of irrelevant information.[33] People are forced to make decisions based on an unfiltered avalanche of information that may or may not be relevant to their needs.\nUnderstanding a complex situation comes from understanding the relationships within the information. Building that understanding results not in knowing the individual pieces of information but in knowing how those pieces of information interconnect. That, in turn, requires looking at the system as not just a large collection of information, but as a highly dynamic integration of information and relationships between information. As a design team works on developing the system HII, they must ensure they capture those relationships and interconnections so they can be explicitly made to the people using the system. As such, a design team must:\n\n Define the situations that must be understood\n Determine the information people need to understand the situation\n Determine how that information is connected and how people see those connections as they build their contextual awareness\n Understand how the information and relationships change as the situation evolves\n Understand the biases, interaction expectations, and decision-making styles of the audience\n Determine the best HII for presenting the information to achieve the best information communication while allowing for those biases, interaction expectations, and decision-making styles\nHII plays a fundamental part in ensuring the overall information content is usable and that people can form the relationships needed to build their contextual awareness. As that understanding develops, people develop their contextual awareness of the situation and can make informed decisions. \n\nConflict of interests \nThe author declares there is no conflict of interest.\n\nReferences \n\n\u2191 1.0 1.1 1.2 Albers, M. (2004). Communication of Complex Information: User Goals and Information Needs for Dynamic Web Information. Erlbaum.   \n\n\u2191 2.0 2.1 2.2 Albers, M. (2012). Human-Information Interaction and Technical Communication: Concepts and Frameworks. IGI Global.   \n\n\u2191 3.0 3.1 Mirel, B. (2003). Interaction Design for Complex Problem Solving: Developing Useful and Usable Software. Morgan Kaufmann.   \n\n\u2191 Albers, M. (July 2011). \"Human-information interactions with complex software\". Proceedings of the HCI International.   \n\n\u2191 Sedig, K.; Parsons, P. (2013). \"Interaction design for complex cognitive activities with visual representations: A pattern-based approach\". AIS Transactions on Human-Computer Interaction 5: 84\u2013133.   \n\n\u2191 Cole, C. (2014). \"Google, tear down this wall to exploratory search!\". Bulletin of the Association for Information Science and Technology 40: 50\u201354.   \n\n\u2191 Ingwersen, P.; Jarvelin, K. (2005). The Turn: Integration of Information Seeking and Retrieval in Context. Springer.   \n\n\u2191 8.0 8.1 Cole, C. (2012). Information Need: A Theory Connecting Information Search to Knowledge Formation. Information Today.   \n\n\u2191 Parsons, P.; Sedig, K. (2014). \"Adjustable properties of visual representations: Improving the quality of human-information interaction\". Journal of the Association for Information Science and Technology 65: 455\u2013482.   \n\n\u2191 10.0 10.1 10.2 10.3 Albers, M.; Still, B. (Eds.) (2010). \"Usability and information relationships: Considering content relationships when testing complex information\". Usability of Complex Information Systems: Evaluation of User Interaction. CRC Press. pp. 109\u2013132.   \n\n\u2191 Taylor, R.S. (1968). \"Question-negotiation and information seeking in libraries\". College & Research Libraries 29: 178\u2013194.   \n\n\u2191 12.0 12.1 12.2 12.3 \"Applied constructivism for user documentation\". Journal of Business and Technical Communication 12: 7\u201349. 1998.   \n\n\u2191 13.0 13.1 Mirel, B.; Albers, M. (Ed.); Mazur, B. (Ed.) (2003). \"Dynamic usability: Designing usefulness into systems for complex tasks\". Content and Complexity: Information Design in Software Development and Documentation. Erlbaum. pp. 233\u2013261.   \n\n\u2191 \"Can we ever escape from data overload? A cognitive systems diagnosis\". Cognition, Technology & Work 4: 22\u201336. 2002.   \n\n\u2191 Albers, M. (October 2009). \"Information relationships: The source of useful and usable content\". Proceedings of the 29th Annual International Conference on Computer Documentation.   \n\n\u2191 16.0 16.1 Fidel, R. (2012). Human Information Interaction: An Ecological Approach to Information Behavior. MIT.   \n\n\u2191 17.0 17.1 Saracevic, T. (2007). \"Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II. Nature and manifestations of relevance\". Journal of the Association for Information Science and Technology 58: 1915\u20131933.   \n\n\u2191 Thuring, M.; Hannemann, J.; Haake, J. (1995). \"Hypermedia and cognition: Designing for comprehension\". Communications of the ACM 38: 57\u201366.   \n\n\u2191 Endsley, M. (1995). \"Toward a theory of situation awareness in dynamic systems\". Human Factors 37: 32\u201364.   \n\n\u2191 Kain, K. de Jong, M.; Smith, C.; Albers, M. (Ed.); Still, B. (Ed.) (2010). \"Information Usability testing as audience and context analysis for risk communication\". Usability of Complex Information Systems: Evaluation of User Interaction. CRC Press. pp. 305\u2013332.   \n\n\u2191 Richards, D. (2015). \"Testing the waters: Local Users, sea level rise, and the productive usability of interactive geovisualizations\". Communication Design Quarterly 3: 24\u201332.   \n\n\u2191 Shannon, C. (1948). \"A mathematical theory of communication\". Bell System Technical Journal 27: 379\u2013423.   \n\n\u2191 Redish, J. (2007). \"Expanding usability testing to evaluate complex systems\". The Journal of Usability Studies 2: 102\u2013111.   \n\n\u2191 24.0 24.1 Sommerville, O.; Cliff, D.; Calinescu, R.; Keen, J.; Kelly, T.; Kwiatkowska, M.; Mcdermid, J.; Paige, R. (2012). \"Large scale complex IT systems\". Communications of the ACM 55: 71\u201377.   \n\n\u2191 25.0 25.1 Klein, G. (1999). Sources of Power: How People make Decisions. MIT.   \n\n\u2191 Simon, H. (1979). Models of Thought. Yale University Press.   \n\n\u2191 Fennema, M.; Kleinmuntz, D. (1995). \"Anticipation of effort and accuracy in multi-attribute choice\". Organizational Behavior and Human Decision Processes 63: 21\u201332.   \n\n\u2191 Payne, S.; Howes. A.; Reader, W. (2001). \"Adaptively distributing cognition: A decision making perspective on human-computer interaction\". Behaviour & Information Technology 20: 339\u2013346.   \n\n\u2191 Mason, R.; Mitroff, I. (1973). \"A program for research on management information systems\". Management Science 19: 475\u2013487.   \n\n\u2191 Gerlach, K.; Kuo, F. (1991). \"Understanding human-computer interaction for information system design\". MIS Quarterly 15: 527\u2013550.   \n\n\u2191 Woods, D.; Roth, E. (1988). \"Cognitive engineering: Human problem solving with tools\". Human Factors 30: 415\u2013430.   \n\n\u2191 Conklin, J.. \"Wicked problems and fragmentation\". http:\/\/www.cognexus.org\/id26.htm . Retrieved 24 April 2015 .   \n\n\u2191 Ackoff, R. (1967). \"Management misinformation systems\". Management Science 14: 147\u2013156.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation and content. The two most notable changes involve citations. Albers lists the following citation in his references as citation 20:\n\"20. Albers, M. Contextual awareness as measure of human-information interaction in usability and design. In Proceedings of the HCI International, Orlando, FL, USA, 12\u201314 July 2011.\" \nHowever, he never referenced that citation number in the paper. We have thus omitted it. Additionally, under the subsection \"Experience-based\", he references \"Klein\u2019s (1999)\" but seemingly uses the incorrect citation number of 12, for Mirel 1998. We've corrected the citation number.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\">https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:55.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,714 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","be30bf8a40599ea6d6b707ba00d0baa1_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Human\u2013information_interaction_with_complex_information_for_decision-making skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Human\u2013information interaction with complex information for decision-making<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Human\u2013information interaction (HII) for simple <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> and for complex information is different because people's goals and information needs differ between the two cases. With complex information, comprehension comes from understanding the relationships and interactions within the information and factors outside of a design team's control. Yet, a design team must consider all these within an HII design in order to maximize the communication potential. This paper considers how simple and complex information requires different design strategies and how those strategies differ.\n<\/p><p><b>Keywords<\/b>: human\u2013information interaction, decision-making, complex communication, information design\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>A primary goal of human\u2013information interaction (HII) for complex information is to communicate concepts and ideas to be used for decisions. That is, information that a reader uses to develop an understanding of a situation and to make decisions about a complex situation.<sup id=\"rdp-ebb-cite_ref-AlbersCCI04_1-0\" class=\"reference\"><a href=\"#cite_note-AlbersCCI04-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlbersHII12_2-0\" class=\"reference\"><a href=\"#cite_note-AlbersHII12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MirelCPS03_3-0\" class=\"reference\"><a href=\"#cite_note-MirelCPS03-3\" rel=\"external_link\">[3]<\/a><\/sup>\n<\/p><p>In any complex communication situation, the proper amount of information should result in maximum communication; of course, too little, too much, poor organization, or inappropriate content reduces the communication. Stating that the communication needs to provide the proper amount of information could considered tautological. It is easy to claim \u2014 and adds little to a discussion \u2014 a text should have the proper information (I use text within the article in a generic sense: paper, web pages, <i>etc<\/i>). However, defining and measuring the communication value of a document (and the information it contains) proves to be a much more difficult task.\n<\/p><p>The interaction and presentation needs of complex information are very different from those of simple information, where a person looks for single information elements. For example, consider the difference in looking up the score of yesterday\u2019s game (simple information) and considering house remodeling to help care for a chronically sick parent (complex information). When we move from a communication viewpoint to an HII viewpoint, we also bring into play issues of how a person can manipulate the information.<sup id=\"rdp-ebb-cite_ref-AlbersHII12_2-1\" class=\"reference\"><a href=\"#cite_note-AlbersHII12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlbersHII11_4-0\" class=\"reference\"><a href=\"#cite_note-AlbersHII11-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SedigID13_5-0\" class=\"reference\"><a href=\"#cite_note-SedigID13-5\" rel=\"external_link\">[5]<\/a><\/sup> The base text may contain the \u201cproper information\u201d but for maximum communication, it may need to be reordered or otherwise manipulated to meet individual needs.\n<\/p><p>A large percentage of the HII literature either looks at simple information search and interaction or conflates simple and complex information. It also tends to take a web-based focus with the person having an essentially infinite number of sources to explore.<sup id=\"rdp-ebb-cite_ref-6\" class=\"reference\"><a href=\"#cite_note-6\" rel=\"external_link\">[6]<\/a><\/sup> While this is true in many cases, within this article, I take a much narrower focus on HII and look at the <i>construction of content<\/i> within a single set of information that is under a content development team\u2019s control, such as the production of corporate reports or public information for a specific goal (an example presented later will be on communicating hurricane information). A view focused on the design team\u2019s task needs to consider what information to create and how people will interact with it. The more commonly discussed view of people searching the Internet (or other sources) for existing information<sup id=\"rdp-ebb-cite_ref-IngwersenTurn05_7-0\" class=\"reference\"><a href=\"#cite_note-IngwersenTurn05-7\" rel=\"external_link\">[7]<\/a><\/sup> will not be directly considered since a design team has no control over how that search proceeds or what information the person may view.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Explanation_of_terms\">Explanation of terms<\/span><\/h2>\n<p>In the introduction, I used the terms simple and complex information. Here I provide a working definition for each. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Simple_information\">Simple information<\/span><\/h3>\n<p>Simple information can be characterized by existing as a single information element. A high percentage of the \u201clook it up on Google\u201d amounts to simple information. For example, a person can look up when a movie was released or a recipe for a carrot cake. Both fit the definition of simple information. Factors that distinguish simple information include:\n<\/p>\n<ul><li> Single path. One path can be fully defined that will result in the answer.<\/li>\n<li> Right\/wrong answer. The correctness can be tested and the information declared correct or not correct.<\/li>\n<li> Complete information. The completeness can be tested and the information declared complete or not complete.<\/li>\n<li> Closed system. All of the factors that might influence the answer can be defined and accounted for.<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Complex_information\">Complex information<\/span><\/h3>\n<p>Complex information exists at the opposite end from simple answers. There is no single \u201canswer\u201d. Addressing complex questions requires using and integrating multiple information elements, which often conflict. People have a complex web of information needs and interactions to fill those needs.<sup id=\"rdp-ebb-cite_ref-ColeIN12_8-0\" class=\"reference\"><a href=\"#cite_note-ColeIN12-8\" rel=\"external_link\">[8]<\/a><\/sup> For example, analyzing monthly business reports and making decisions about this month\u2019s production, or making health care choices. Factors that distinguish complex information include:\n<\/p>\n<ul><li> Multiple paths. There is no single path to an answer. A person can take many different paths and all will work. The effectiveness of the paths may, of course, vary.<\/li>\n<li> Open-ended. The idea of \u201ccomplete\u201d is undefined. A person can continue to collect information and refine their understanding with an essentially infinite amount of information. Instead, a person has to pick a stopping point and make a decision.<\/li>\n<li> Needs cannot be predefined. The information that a person needs cannot be predefined. Of course the major or essential information can be predefined, but the many smaller information elements that can exert a strong influence vary too much between individuals.<\/li>\n<li> History. The information exists within a continuum and that history influences how it gets interpreted and used.<\/li>\n<li> Non-linear. The overall situation shows a non-linear response with small differences in some information elements resulting in very large differences in appropriate decisions.<\/li>\n<li> Open system. All of the factors that might influence the decision cannot be defined. There are too many and the information is dynamic, changing on time scales relevant to the decision situation.<\/li><\/ul>\n<p>Most of the simple information that gets communicated tends to be single facts or procedural type of information. Complex information, on the other hand, deals with a broader scope.\n<\/p>\n<ul><li> Complex information communicates concepts and ideas.<\/li>\n<li> Complex information communicates an understanding of a situation.<\/li>\n<li> Complex information communicates relationships and interactions.<\/li><\/ul>\n<p>Clearly, the HII required to maximize the communication of complex information is very different between the two. Most writing and UX guidelines deal with communicating simple information. But to really address people\u2019s information needs, we need to consider why and how they interact with complex information. Consider the case of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> \u2014 they clearly represent complex information \u2014 and represent an example where the design team has control over a significant part of how the HII occurs. The design has to consider many potentially conflicting sets of priorities (development time, maintenance cost, hospital administration goals, government regulations, medical needs, <i>etc.<\/i>), which, in turn, affect how well the users can interact with the information. The input methods must be efficient for data entry, but presenting the information the same way to a nurse or physician may not be the best for integrating it into a clear view of the patient\u2019s condition. The relationships of information (connecting the results of various medical tests, <i>etc.<\/i>) should reflect the initial diagnosis and should also reflect the changing patient status as treatment progresses.\n<\/p><p>In previous work, I have defined a situation as \u201cthe current world state which the user needs to understand. An underlying assumption is that the user needs to interact with a <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_system\" title=\"Data system\" target=\"_blank\" class=\"wiki-link\" data-key=\"686de8e8c43733586ac2ff962f3838a3\">system<\/a> to gain the necessary information in order to understand the situation\u201d<sup id=\"rdp-ebb-cite_ref-AlbersCCI04_1-1\" class=\"reference\"><a href=\"#cite_note-AlbersCCI04-1\" rel=\"external_link\">[1]<\/a><\/sup> (p. 11). The information relevant to the HII exists both inside and outside any circle drawn to enclose \u201cthe situation\u201d (Figure 1). Honestly, the difficulty of drawing that enclosing circle is a characteristic of a complex situation.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"32620c019e2eeded96f213d31c31afa0\"><img alt=\"Fig1 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f8\/Fig1_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Complex situation with information both inside and outside of a design team\u2019s control.<br \/>Human\u2013information interaction (HII) only directly applies to the information a design team controls.<\/blockquote>\n<p>Once an upfront analysis has defined the people\u2019s goals and information needs, then the HII factors which support manipulating the information and revealing the relationships can be considered and entered into the design.<sup id=\"rdp-ebb-cite_ref-AlbersHII12_2-2\" class=\"reference\"><a href=\"#cite_note-AlbersHII12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ParsonsAdj14_9-0\" class=\"reference\"><a href=\"#cite_note-ParsonsAdj14-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/p>\n<blockquote>Typically, the failure of these technical documents [used here in a generic sense for any information source] comes not from a lack of information; the text probably contains an excess of information. Post hoc studies of communication failures find many sources to blame: poor information architecture, poor organization, wrong grade level or writing style, or poor presentation. But instead of seeing these problems as a root cause, let\u2019s consider them as symptoms of a more fundamental problem: a problem stemming from the underlying complexity of the situational context and a failure of the information presentation to match that complexity.<sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-0\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup>(p. 110)<\/blockquote>\n<p>The task assigned to an HII design team is to ensure the underlying complexity is not overly simplified and that the information presentation and manipulation meets the needs of the people and the demands of the situation. As others have discussed, the HII problem for maximizing the communication for complex information is not a tools question, but one of understanding what people need and how they come to understand the information.<sup id=\"rdp-ebb-cite_ref-TaylorQN68_11-0\" class=\"reference\"><a href=\"#cite_note-TaylorQN68-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Information_relationships\">Information relationships<\/span><\/h3>\n<p>Understanding complex information depends on understanding relationships, but the potential relationships are essentially infinite. Comprehension of a complex situation occurs when people can mentally integrate those relationships into their view of the situation.\n<\/p>\n<blockquote>Information integration lies, not in a text element itself, but in the relationships between those elements. A reader needs to figure out what information is relevant and how to connect it to the current problem. Without proper information relationships, the reader does not gain an integrated understanding of information, but instead gains a collection of facts. Without relationships, information exists as a bunch of interesting factoids which do not help a person form an adequate mental picture of the situation. Collections of facts are less than useful for understanding and working with the open-ended problems that people encounter in complex situations.<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-0\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MirelDes03_13-0\" class=\"reference\"><a href=\"#cite_note-MirelDes03-13\" rel=\"external_link\">[13]<\/a><\/sup> Without the relationships, a person learns about <i>X<\/i> and <i>Y<\/i>, but not how <i>X<\/i> and <i>Y<\/i> relate to each other or to <i>Z<\/i> in terms of their current problem or situation.<sup id=\"rdp-ebb-cite_ref-WoodsCan02_14-0\" class=\"reference\"><a href=\"#cite_note-WoodsCan02-14\" rel=\"external_link\">[14]<\/a><\/sup> The text fails to communicate because the reader can\u2019t form the necessary information relationships.<sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-1\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup>(p. 111)<\/blockquote>\n<p>A patient trying to understand a medical problem often has difficulty understanding the situation because they don\u2019t understand the relationships. They can look at a set of lab results and know the numbers, but lack the medical practitioner\u2019s knowledge of what the values should be for their condition. The patient may be worried because some values are high, but the physician is satisfied with them because the specific condition often has even higher values. Likewise, a junior manager needs to develop the knowledge of how to interpret multiple values that may appear on a collection of monthly reports. Good HII can help by providing a more integrated view and providing help with the integration, without interfering with more experienced people\u2019s information interaction.\n<\/p><p>Building relationships means developing a deep-level understanding of the text rather than just a surface-level knowledge. Deep-level knowledge involves seeing the macrostructure of the text and being able to apply prior knowledge to it and fit the text within the reader\u2019s prior knowledge framework. Surface-level knowledge involves knowing the basic text. People with surface-level knowledge can quote the text and if asked recall-type questions would respond with answers very close to the text language, but they would not be able to elaborate on the answer or connect it to other information. People with deep-level understanding would be able to place the text\u2019s information into their own words. A major element in the difference between people being able to develop deep-level understanding versus a surface-level understanding is their prior knowledge.\n<\/p><p>Communicating complex information for decision-making can be viewed as working to help build relationships within the information.<sup id=\"rdp-ebb-cite_ref-ColeIN12_8-1\" class=\"reference\"><a href=\"#cite_note-ColeIN12-8\" rel=\"external_link\">[8]<\/a><\/sup> Unfortunately, building relationships is a great theoretical concept, but one that does not lend itself to a direct operational definition. One way to define judging or qualifying relationships can be how they reduce the uncertainty a person has about the situation.<sup id=\"rdp-ebb-cite_ref-AlbersIR09_15-0\" class=\"reference\"><a href=\"#cite_note-AlbersIR09-15\" rel=\"external_link\">[15]<\/a><\/sup> This reshapes an HII design team\u2019s goal to one of focusing content creation in terms of what information does the reader need to reduce their uncertainty and, consequently, to build a web of relationships between the information elements. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Contextual_awareness\">Contextual awareness<\/span><\/h3>\n<p>Except for training material, the readers of technical information typically understand the basics but need to know specific information about the current situation (as opposed to the general situation) in order to make decisions. The understanding of a complex situation needed to make informed decisions comes when people can distinguish the information structure, grasp the relationships within it<sup id=\"rdp-ebb-cite_ref-FodelHII12_16-0\" class=\"reference\"><a href=\"#cite_note-FodelHII12-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SaracevicRel07_17-0\" class=\"reference\"><a href=\"#cite_note-SaracevicRel07-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ThuringHyp95_18-0\" class=\"reference\"><a href=\"#cite_note-ThuringHyp95-18\" rel=\"external_link\">[18]<\/a><\/sup>, and make inferences on the future evolution of the situation.\n<\/p><p>Building on Endsley\u2019s (1995)<sup id=\"rdp-ebb-cite_ref-EndsleyTow95_19-0\" class=\"reference\"><a href=\"#cite_note-EndsleyTow95-19\" rel=\"external_link\">[19]<\/a><\/sup> situation awareness work, we can call this contextual awareness. Contextual awareness is the understanding of the information within an informational situation which forms the basis for how to interpret new information and how to make decisions for interacting with that situation. With poor contextual awareness, people can know something is occurring or that a particular piece of information exists, but they cannot easily find relevant related information or they have the information but do not understand how it relates to the overall situation. On the other hand, good contextual awareness does not guarantee a person will form the proper intention or make the proper decisions; the error analysis literature is filled with cases where people understood a situation, but still made incorrect choices. Unfortunately for design teams, the concept of relevance itself is highly nuanced and multifaceted<sup id=\"rdp-ebb-cite_ref-SaracevicRel07_17-1\" class=\"reference\"><a href=\"#cite_note-SaracevicRel07-17\" rel=\"external_link\">[17]<\/a><\/sup>, creating a complex interplay that must be understood to engineer high quality HII.\n<\/p><p>Elements of good contextual awareness are (Figure 2): (1) Understands how the information fits within the current situation. (2) Understands the information relationships.<sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-2\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup> Information comprehension requires knowing how information relates to other information. (3) Understands the future development of the situation and can make predictions about the ripple effects of any decision across the entire situation. HII that supports complex information needs to provide the interactions that support people developing high quality contextual awareness. \n<\/p><p>Kain, de Jong, and Smith (2010)<sup id=\"rdp-ebb-cite_ref-KainIU10_20-0\" class=\"reference\"><a href=\"#cite_note-KainIU10-20\" rel=\"external_link\">[20]<\/a><\/sup> study into how to communicate hurricane risks and warnings highlighted the issues of how people interpret information and make decisions about how they will react. The hurricane experts had their view of what information was needed and how it should be presented, but the research showed the people wanted\/needed a different presentation. Their mental methods of forming relationships and of interpreting complex information differed from what the experts thought. The process of building contextual awareness differs and a design team\u2019s analysis must capture those differences.\n<\/p><p>A design team working on developing the HII for a country\u2019s hurricane awareness plan needs to balance both the expert\u2019s \u201chere is what the people must hear\u201d against the more pragmatical \u201chere\u2019s what I want to know\u201d as well as the local people\u2019s opinion of how they react to hurricane warnings. Many of them have been through multiple hurricanes and have strongly held views that often conflict with the authorities. Notice how in this case, the design team has control of the information. A general search on hurricane warnings will have sentences such as \u201cconsult your local authorities for evacuation routes\u201d but, here, the design team will be tasked with providing that evacuation route information. At the same time, they need to ensure the relationships between evacuation route, getting ready for evacuation, and planning on returning are all clearly laid out and connected. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"4cb8c42b6e866a5cb861e21402624426\"><img alt=\"Fig2 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/7\/71\/Fig2_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> Stages of developing contextual awareness<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"HII_is_information_interaction.2C_not_data_interaction\">HII is information interaction, not data interaction<\/span><\/h2>\n<p>People who are engaged with decision-making and complex information need to be presented with information, not data (Figure 3). Quick definition of my terms here, which I discuss in more detail elsewhere.<sup id=\"rdp-ebb-cite_ref-AlbersCCI04_1-2\" class=\"reference\"><a href=\"#cite_note-AlbersCCI04-1\" rel=\"external_link\">[1]<\/a><\/sup>\n<\/p>\n<ul><li> Data: Raw numbers, facts, and figures.<\/li>\n<li> Information: Information is data in context. It relates to the situation and contains the relationships that connect the information to the situation.<\/li>\n<li> Knowledge: Interconnected web of the relevant information and the relationships linking the information within the situation. <\/li><\/ul>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a0702e0c4b311eec5fe513fe33a4bcde\"><img alt=\"Fig3 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/8\/83\/Fig3_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3.<\/b> Data, information, and knowledge hierarchy. The higher the HII works in the hierarchy,<br \/>the better it fits building an understanding of the situation.<\/blockquote>\n<p>As an example, I heard a presentation that looked at the effect of sea level rise on the Norfolk, VA area.<sup id=\"rdp-ebb-cite_ref-RichardsUsa10_21-0\" class=\"reference\"><a href=\"#cite_note-RichardsUsa10-21\" rel=\"external_link\">[21]<\/a><\/sup> A software program had nice manipulation that let a person dynamically see the effects of different amounts of sea level rise from global warming and how it would affect the city. The problem was not with the implementation, but with the underlying assumptions of a design team. They were assuming that by providing a tool and letting people see how different sea level changes would affect the area that it would bring about understanding and increase long-term preparedness. But the tool was presenting content at the data level. Yes, I can change the sliders and see how the sea level changes, but it was devoid of supporting content (in the metaphor of this paper, the tool was a single puzzle piece). It not only didn\u2019t connect with other pieces, those other pieces were not provided. As a result, a global warming denier could play with the model, agree that a four foot rise would be a catastrophic problem, but then reject it as something that would never happen. One of the software\u2019s goals was to help people prepare, but without giving them the content and relationships to build their contextual awareness; it was a single data point and not part of a coherent presentation of information. \n<\/p><p>The transition from information to knowledge is important since it involves comprehending the relationships within the data and placing them within the context of the situation. Moving to knowledge is essential to building contextual awareness and must be the goal of an HII design team.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Communication_of_simple_and_complex_information\">Communication of simple and complex information<\/span><\/h2>\n<p>Many discussions on communication start with Shannon information theory (1948)<sup id=\"rdp-ebb-cite_ref-ShannonMath48_22-0\" class=\"reference\"><a href=\"#cite_note-ShannonMath48-22\" rel=\"external_link\">[22]<\/a><\/sup> (Figure 4), which works for analyzing simple information. With clearly defined information to be conveyed, a design team can concentrate on efficient communication methods. Unfortunately, Shannon was really discussing the minimum information to convey a message, not how to communicate in a natural language. It works well for computer-computer communication, but is not as applicable to human-human communication, especially for complex information, with or without a computer mediator. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5d31159b79e6cb60a6709b135d431527\"><img alt=\"Fig4 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/be\/Fig4_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4.<\/b> Basic block diagram of Shannon information theory. Information moves from the source<br \/>to the final destination with the overall goal to be minimizing both the noise and total amount of required content.<\/blockquote>\n<p>Driven in part by a Shannon-based desire to efficiently communication information, design teams try to break the communication into individual components. This fits with our reductionist nature to break any problem into individual components. Standard approaches for communicating complex information (and analyzing any complex system) are to assume it is assembled from smaller systems<sup id=\"rdp-ebb-cite_ref-MirelCPS03_3-1\" class=\"reference\"><a href=\"#cite_note-MirelCPS03-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MirelDes03_13-1\" class=\"reference\"><a href=\"#cite_note-MirelDes03-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RedishExp07_23-0\" class=\"reference\"><a href=\"#cite_note-RedishExp07-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SommervilleLar12_24-0\" class=\"reference\"><a href=\"#cite_note-SommervilleLar12-24\" rel=\"external_link\">[24]<\/a><\/sup>. Thus, it is hardly a surprising statement that we see a complex system as the sum of its components: 1 + 1 + 1 + 1 + \u2026.. = THE SYSTEM.\n<\/p><p>Unfortunately, assuming a complex system is the sum of its components at best redefines it as a simple system. The complexity that people need to understand and the HII must support is not just a sum of individual components, but the interactions and relationships between components. The redefinition to a sum of individual components ignores the complex interactions that give rise to more than the sum of the parts: 1 + 1 + 1 + 1 + \u2026..+ <i>i<\/i><sub>n<\/sub> >> \u01a9<sub>n<\/sub>. \n<\/p><p>The communication problem, and consequently the HII problem, is that redefining the problem to be a simple problem fundamentally changes the problem itself. As a result, the communication itself changes and the relationships and interactions that are essential to understanding get lost. The communication-related literature seems to generally miss this important concept, although the computer science literature consistently contains quotes such as \u201cthe fundamental reason today\u2019s software engineering cannot effectively manage inherent complexity is that its basis is in developing individual programs rather than in interacting systems.\u201d<sup id=\"rdp-ebb-cite_ref-SommervilleLar12_24-1\" class=\"reference\"><a href=\"#cite_note-SommervilleLar12-24\" rel=\"external_link\">[24]<\/a><\/sup> (p. 73). In designs that strive to communicate complex information, an over-privileging of developing and\/or organizing content while discounting the interactions and relationships within and between content leads to results similar to those critiqued by Sommerville. \n<\/p><p>In the introduction, I described most simple information as being single information elements that can be looked up (Figure 5). Complex information on the other hand, has a multitude of information elements that a person must understand, some of which are outside of the bounds of a HII system. Factors such as asking a person down the hall, having heard of a bad experience with product <i>X<\/i>, <i>etc.<\/i><sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-3\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup> (p. 115) \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"3e15fa47463973960e327cfff064f68e\"><img alt=\"Fig5 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/d\/d4\/Fig5_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 5.<\/b> Relationships and understanding a situation<\/blockquote>\n<p>Figure 5 shows content as nice rectangles. But they could be better viewed as jigsaw pieces with the interlocks on the pieces representing the relationships (Figure 6), how the pieces interconnect with each other into a coherent understanding. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2e367319ac7c5bae39a5bc8a6e19c47e\"><img alt=\"Fig6 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/7\/7b\/Fig6_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 6.<\/b> Relationships and the complexities of assembling information into a coherent understanding.<br \/>The understanding comes from both the content and knowing how it fits together.<\/blockquote>\n<p>Unfortunately, the pieces in Figure 6 are not static jigsaw shapes. They are better viewed as a collection of puzzle pieces that change shape as they move away\/toward other pieces, and as they merge together (Figure 7). As a result, changes to one piece can ripple through the entire information web. A change to a piece in the upper left corner results in many of the pieces changing in some relevant manner. The information and HII appropriate and relevant for each point of a situation can be different. Situations are not static; they have a beginning, middle, and end. The puzzle pieces of Figure 7 are not just dynamic from a person\u2019s interaction, but also change over time and over the situation\u2019s evolution. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"db7345555aef81a75e400460d660210f\"><img alt=\"Fig7 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/9f\/Fig7_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 7.<\/b> Dynamic evolution of information. The relationships between information elements<br \/>can change over both time and by the act of connecting them.<\/blockquote>\n<p>Let\u2019s reexamine the six characteristics of complex information with respect to the puzzle metaphor.\n<\/p>\n<dl><dd><b>Multiple paths<\/b>: People have multiple paths through the information. The order in which they move between the puzzle pieces cannot be predefined. Each HII with a piece can change how it interconnects; thus, different paths through the information change how people build the relationships and, consequently, how the pieces fit together. <\/dd><\/dl>\n<dl><dd><b>Open-ended<\/b>: No clear point of enough information. The HII cannot move toward a predefined point of \u201cnow you have all the information.\u201d With a goal of communicating concepts and ideas, how many pieces a person needs, cannot be defined. Of course, the issue of information needs versus information wants also comes into play. Coupled with this is the fact that information search and problem-solving are sufficing processing.<sup id=\"rdp-ebb-cite_ref-KleinSou99_25-0\" class=\"reference\"><a href=\"#cite_note-KleinSou99-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SimonMod79_26-0\" class=\"reference\"><a href=\"#cite_note-SimonMod79-26\" rel=\"external_link\">[26]<\/a><\/sup> People stop once they are comfortable with their understanding of the found information. Unfortunately, people are poor judges of knowing they have found adequate information.<\/dd><\/dl>\n<dl><dd><b>Cannot be predefined<\/b>: How many pieces a person will interact with remains unknowable to a design team. Interestingly, the size (content) of individual pieces is dynamic; some people need smaller\/larger pieces to effectively comprehend the overall situation and tend to make choices in terms of immediate, rather than long-term, efficiency and effort of the HII.<sup id=\"rdp-ebb-cite_ref-FennemaAnt95_27-0\" class=\"reference\"><a href=\"#cite_note-FennemaAnt95-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PayneAda01_28-0\" class=\"reference\"><a href=\"#cite_note-PayneAda01-28\" rel=\"external_link\">[28]<\/a><\/sup><\/dd><\/dl>\n<dl><dd><b>History<\/b>: Past history of a situation affects how the pieces will evolve and the past history of the people interacting with the information affects how they interpret it. Two situations may appear to be the same but the past strongly influences how the piece will change. For example, six months ago sales in the southeast region were down by 30%. A set of decisions was made and sales have steadily increased since them. They are still 10% below desired levels, but are increasing at an acceptable rate. This situation is much different than if sales are 10% below the desired level and not conforming to predictions.<\/dd><\/dl>\n<dl><dd><b>Non-linear<\/b>: The relationships between pieces can show non-linear response to changes. Minor differences in past history or the HII path can result in widely different \u2014 but appropriate \u2014 end points. In individual piece can morph into very different final shapes, even though they started from similar initial conditions.<\/dd><\/dl>\n<dl><dd><b>Open system<\/b>: As the system evolves, the overall content within the system changes. Some new information gets introduced and some information drops out. The number of pieces, their content, and their shape can change.<\/dd><\/dl>\n<p>When the information is viewed as dynamic puzzle pieces, how those pieces change shape helps to emphasize how many solutions to problems introduce other problems, sometimes worse than the one they were intended to solve. For example, many environmental solutions inevitably end up doing more harm than good. The future changes to relationships are not understood when decisions are made, so the evolution of the situation is poorly predicted (poor contextual awareness). Worse, too many people make assumptions that they can change one piece with nothing else changing. In complex ecosystems \u2014 biological, educational or industrial \u2014 actions always have consequences that are hard to predict; a change to one piece ripples through all the others. The \u201chard to predict\u201d is a hallmark of a complex system. HII that supports interacting with the information must consider those ramifications.\n<\/p><p>Beyond the need for the HII to handle dynamic puzzle pieces, it must also allow for easy trimming of the information space. All of the available information is not relevant to any specific situation. Instead, a person needs a select collection of information (Figure 8).\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"79fadafb2918308d2d80aa78f461f323\"><img alt=\"Fig8 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/e\/e5\/Fig8_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 8.<\/b> Information relevant to a situation. Each situation (or class of situations) only needs<br \/>a subset of the information. High quality HII helps to keep the focus on only the relevant subset.<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"HII_for_decision-making\">HII for decision-making<\/span><\/h2>\n<p>The overall scope of the information flow for the HII of complex information moves to prominence when the interaction goal is decision-making. People need to collect information, analyze it, and make a decision; a process that depends on the flow of information both within and outside the control of an HII design team.\n<\/p><p>Decision-making occurs as a result of comparing what is perceived in the environment and what is known by the decision-maker.<sup id=\"rdp-ebb-cite_ref-MasonProg73_29-0\" class=\"reference\"><a href=\"#cite_note-MasonProg73-29\" rel=\"external_link\">[29]<\/a><\/sup> Essentially all of our current literature discusses considering the needs of people when designing information. However, current practice does not typically consider how people\u2019s questions depend on how they make decisions and interact with the system.<sup id=\"rdp-ebb-cite_ref-GerlachUnd91_30-0\" class=\"reference\"><a href=\"#cite_note-GerlachUnd91-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WoodsGoc88_31-0\" class=\"reference\"><a href=\"#cite_note-WoodsGoc88-31\" rel=\"external_link\">[31]<\/a><\/sup> HII attempts to take the entire cycle of information interpretation and decision-making and place it within people\u2019s current situation.\n<\/p><p>Mirel (1998)<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-1\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup> follows the same line of thinking as Conklin\u2019s (2003)<sup id=\"rdp-ebb-cite_ref-ConklinWicked_32-0\" class=\"reference\"><a href=\"#cite_note-ConklinWicked-32\" rel=\"external_link\">[32]<\/a><\/sup> wicked problems when she points out that analyzing complex tasks requires seeing more than a single path:\n<\/p>\n<blockquote>This broader view is necessary to capture the following traits of complex tasks: paths of action that are unpredictable, paths that are never completely visible from any one vantage point, and nuance judgments and interpretations that involve multiple factors and that yield many solutions.<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-2\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup> (p. 14)<\/blockquote>\n<p>Mirel may have focused on the analysis aspects, but that is an early and vital step in any information interaction, especially for HII of complex information. Unless a design team clearly understands the people\u2019s goals and information needs, there is minimal chance for the HII to support it.\n<\/p><p>The issue of wicked problems and unpredictable paths that both Conklin and Mirel discuss brings to light an interesting point. As design teams work toward HII of complex information, they encounter an interesting and confounding recursion: the HII of complex information is itself a complex problem and, as such, does not lend itself to easy answers. Yet, many design teams try to address those issues with the same process that works for simple information, which leads to an easy-answer mentality. As a result, they reduce the scope and avoid confronting the complex information HII issues head-on. Design teams who try to reduce all problems to a collection of individual parts or who assume they should consider only the information they control fall into this trap. They have redefined the complex HII problem into a simple problem. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Decision-making_strategies\">Decision-making strategies<\/span><\/h3>\n<p>People use the available information to make a decision about how to proceed in their current situation; information use is always based upon the context within which it is used.<sup id=\"rdp-ebb-cite_ref-FodelHII12_16-1\" class=\"reference\"><a href=\"#cite_note-FodelHII12-16\" rel=\"external_link\">[16]<\/a><\/sup> This hard connection between use and context greatly complicates a design team\u2019s task since they must understand both factors. And, depending on audience, the base information may exist within multiple contexts.\n<\/p><p>Based on the results of the decision, people take action and then need to re-analyze the situation. Closely related to decision-making is monitoring the situation to ensure it continues along the predicted path. If the information fits the expected progress, then a decision of \u201cdo nothing\u201d is correct. However, if the information indicates that it is deviating, then people cycle back into decision-making.\n<\/p><p>There are two broad decision-making strategies that people use. \n<\/p>\n<h4><span class=\"mw-headline\" id=\"Experience-based\">Experience-based<\/span><\/h4>\n<p>Experience-based decision-making is used the most and is a person\u2019s default strategy. It is a rule-based strategy: \u201cif <i>X<\/i> happens, then do <i>Y<\/i>.\u201d Most decision-making, at least in the corporate world, is\nexperienced-based.<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-3\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup> The reader is skilled in the overall area (such as a CFO making financial decisions), but does need specific information for the current situation. They need to know how this month\u2019s production or sales numbers differ from last month\u2019s or between different regions, but they do have a set of mental rules on how to interpret those numbers.\n<\/p><p>Based on past experience, people develop a rule-based strategy so they can reliably expect that \u201cwhen <i>X<\/i> happens, then do <i>Y<\/i>.\u201d Rule-based decisions reduce cognitive effort and work well within normal situations because the rules themselves evolved from past experience. However, that same rule-based strategy can cause problems if a person jumps to a decision based on incomplete information. <i>X<\/i> can happen in multiple situations but other factors differ\u2014the relationships of what\/how <i>X<\/i> is interconnected within the situation\u2014which cause solution <i>Y<\/i> to only be an appropriate response in some situations.\n<\/p><p>Klein\u2019s (1999)<sup id=\"rdp-ebb-cite_ref-KleinSou99_25-1\" class=\"reference\"><a href=\"#cite_note-KleinSou99-25\" rel=\"external_link\">[25]<\/a><\/sup> recognition-primed model of decision-making posits that experienced people do little problem solving, especially in routine situations. Instead, once they recognize a situation, they form a possible intention very rapidly, mentally evaluate it, and, if no major problems are evident, take action on it. Alternatives are not considered. In other words, they tend to use their prior knowledge and initial assessment to immediately pick a solution, and if it seems workable, then that is considered the answer. (Concerns about whether or not the solution is optimal are not part of the mental evaluation.)\n<\/p><p>Post-failure analysis tends to highlight that the decision-makers had the proper information, but improperly interpreted it. In many cases, it is a result of using experience-based decision-making and not realizing the \u201cif x, then y\u201d rule did not apply to the situation. \n<\/p>\n<h4><span class=\"mw-headline\" id=\"Knowledge-based\">Knowledge-based<\/span><\/h4>\n<p>Experience-based decision-making can break down, typically when the situation contains unexpected factors or relationships. When people recognize their rules no longer apply, they have to shift to knowledge-based decision-making. Unfortunately, the shift to knowledge-based decision-making often happens after the monitoring phase of an experienced-based strategy reveals the situation is not developing as predicted.\n<\/p><p>In knowledge-based decision, a person knows their experience-based rule set has broken down and they are no longer working within a normal situation. Something is not right in the current situation and they are trying to figure out what to change to fix the problem. In other words, the expected relationships are not being found and a person needs to understand why. It tends to happen when people do not understand what caused the base problem. Unlike decision-making based on rules of \u201cwhen <i>X<\/i> happens, then do <i>Y<\/i>,\u201d a person has to fully develop their contextual awareness so they can make a decision.\n<\/p><p>The HII of complex information applies more for knowledge-based strategies than it does for experience-based ones. Knowledge-based strategies require people to engage in a systematic interaction with the information and work to build up their contextual awareness. \n<\/p><p>Someone working on a global warming problem is working at the knowledge-based and not experienced-based level. Allowing people to manipulate data as if it was a simple information \u2014 a collection of individual components \u2014 can misguide them into thinking it\u2019s an experience-based problem and they can draw on their prior knowledge. Worse, defining the problem as \u201cif they can work this sea level rise model, then they will understand the problem\u201d redefines the problem from a complex problem to a simple problem. Unfortunately, but inevitable with the history component of complex information, prior knowledge and biases strongly influence how information gets interpreted. Of course, with global warming\u2014and many other situations\u2014the political aspects, factors outside of an HII design team\u2019s control, come to the forefront. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Providing people with any collection of information, especially complex information, can easily result in information overload. With the large amounts of information and its dynamic nature, high quality HII is essential for people to comprehend that information. \n<\/p><p>Rather than a lack of information, the failure to anticipate people\u2019s needs forms the basis of most information problems and poor decision-making; an issue design teams must explicitly address. Managers typically suffer from a shortage of decision-relevant information and a simultaneous overabundance of irrelevant information.<sup id=\"rdp-ebb-cite_ref-AckoffMan67_33-0\" class=\"reference\"><a href=\"#cite_note-AckoffMan67-33\" rel=\"external_link\">[33]<\/a><\/sup> People are forced to make decisions based on an unfiltered avalanche of information that may or may not be relevant to their needs.\n<\/p><p>Understanding a complex situation comes from understanding the relationships within the information. Building that understanding results not in knowing the individual pieces of information but in knowing how those pieces of information interconnect. That, in turn, requires looking at the system as not just a large collection of information, but as a highly dynamic integration of information and relationships between information. As a design team works on developing the system HII, they must ensure they capture those relationships and interconnections so they can be explicitly made to the people using the system. As such, a design team must:\n<\/p>\n<ul><li> Define the situations that must be understood<\/li>\n<li> Determine the information people need to understand the situation<\/li>\n<li> Determine how that information is connected and how people see those connections as they build their contextual awareness<\/li>\n<li> Understand how the information and relationships change as the situation evolves<\/li>\n<li> Understand the biases, interaction expectations, and decision-making styles of the audience<\/li>\n<li> Determine the best HII for presenting the information to achieve the best information communication while allowing for those biases, interaction expectations, and decision-making styles<\/li><\/ul>\n<p>HII plays a fundamental part in ensuring the overall information content is usable and that people can form the relationships needed to build their contextual awareness. As that understanding develops, people develop their contextual awareness of the situation and can make informed decisions. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflict_of_interests\">Conflict of interests<\/span><\/h2>\n<p>The author declares there is no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<ol class=\"references\">\n<li id=\"cite_note-AlbersCCI04-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlbersCCI04_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersCCI04_1-1\" rel=\"external_link\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersCCI04_1-2\" rel=\"external_link\">1.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Albers, M. (2004). <i>Communication of Complex Information: User Goals and Information Needs for Dynamic Web Information<\/i>. Erlbaum.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Communication+of+Complex+Information%3A+User+Goals+and+Information+Needs+for+Dynamic+Web+Information&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=2004&rft.pub=Erlbaum&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersHII12-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlbersHII12_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersHII12_2-1\" rel=\"external_link\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersHII12_2-2\" rel=\"external_link\">2.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Albers, M. (2012). <i>Human-Information Interaction and Technical Communication: Concepts and Frameworks<\/i>. IGI Global.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Human-Information+Interaction+and+Technical+Communication%3A+Concepts+and+Frameworks&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=2012&rft.pub=IGI+Global&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MirelCPS03-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MirelCPS03_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-MirelCPS03_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Mirel, B. (2003). <i>Interaction Design for Complex Problem Solving: Developing Useful and Usable Software<\/i>. Morgan Kaufmann.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Interaction+Design+for+Complex+Problem+Solving%3A+Developing+Useful+and+Usable+Software&rft.aulast=Mirel%2C+B.&rft.au=Mirel%2C+B.&rft.date=2003&rft.pub=Morgan+Kaufmann&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersHII11-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlbersHII11_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Albers, M. (July 2011). \"Human-information interactions with complex software\". <i>Proceedings of the HCI International<\/i>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Human-information+interactions+with+complex+software&rft.jtitle=Proceedings+of+the+HCI+International&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=July+2011&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SedigID13-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SedigID13_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sedig, K.; Parsons, P. (2013). \"Interaction design for complex cognitive activities with visual representations: A pattern-based approach\". <i>AIS Transactions on Human-Computer Interaction<\/i> <b>5<\/b>: 84\u2013133.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interaction+design+for+complex+cognitive+activities+with+visual+representations%3A+A+pattern-based+approach&rft.jtitle=AIS+Transactions+on+Human-Computer+Interaction&rft.aulast=Sedig%2C+K.%3B+Parsons%2C+P.&rft.au=Sedig%2C+K.%3B+Parsons%2C+P.&rft.date=2013&rft.volume=5&rft.pages=84%E2%80%93133&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-6\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cole, C. (2014). \"Google, tear down this wall to exploratory search!\". <i>Bulletin of the Association for Information Science and Technology<\/i> <b>40<\/b>: 50\u201354.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Google%2C+tear+down+this+wall+to+exploratory+search%21&rft.jtitle=Bulletin+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Cole%2C+C.&rft.au=Cole%2C+C.&rft.date=2014&rft.volume=40&rft.pages=50%E2%80%9354&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IngwersenTurn05-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IngwersenTurn05_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ingwersen, P.; Jarvelin, K. (2005). <i>The Turn: Integration of Information Seeking and Retrieval in Context<\/i>. Springer.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Turn%3A+Integration+of+Information+Seeking+and+Retrieval+in+Context&rft.aulast=Ingwersen%2C+P.%3B+Jarvelin%2C+K.&rft.au=Ingwersen%2C+P.%3B+Jarvelin%2C+K.&rft.date=2005&rft.pub=Springer&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ColeIN12-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ColeIN12_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-ColeIN12_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Cole, C. (2012). <i>Information Need: A Theory Connecting Information Search to Knowledge Formation<\/i>. Information Today.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Information+Need%3A+A+Theory+Connecting+Information+Search+to+Knowledge+Formation&rft.aulast=Cole%2C+C.&rft.au=Cole%2C+C.&rft.date=2012&rft.pub=Information+Today&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ParsonsAdj14-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ParsonsAdj14_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Parsons, P.; Sedig, K. (2014). \"Adjustable properties of visual representations: Improving the quality of human-information interaction\". <i>Journal of the Association for Information Science and Technology<\/i> <b>65<\/b>: 455\u2013482.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adjustable+properties+of+visual+representations%3A+Improving+the+quality+of+human-information+interaction&rft.jtitle=Journal+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Parsons%2C+P.%3B+Sedig%2C+K.&rft.au=Parsons%2C+P.%3B+Sedig%2C+K.&rft.date=2014&rft.volume=65&rft.pages=455%E2%80%93482&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersUsa10-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlbersUsa10_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersUsa10_10-1\" rel=\"external_link\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersUsa10_10-2\" rel=\"external_link\">10.2<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersUsa10_10-3\" rel=\"external_link\">10.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Albers, M.; Still, B. (Eds.) (2010). \"Usability and information relationships: Considering content relationships when testing complex information\". <i>Usability of Complex Information Systems: Evaluation of User Interaction<\/i>. CRC Press. pp. 109\u2013132.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Usability+and+information+relationships%3A+Considering+content+relationships+when+testing+complex+information&rft.atitle=Usability+of+Complex+Information+Systems%3A+Evaluation+of+User+Interaction&rft.aulast=Albers%2C+M.%3B+Still%2C+B.+%28Eds.%29&rft.au=Albers%2C+M.%3B+Still%2C+B.+%28Eds.%29&rft.date=2010&rft.pages=pp.%26nbsp%3B109%E2%80%93132&rft.pub=CRC+Press&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorQN68-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorQN68_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, R.S. (1968). \"Question-negotiation and information seeking in libraries\". <i>College & Research Libraries<\/i> <b>29<\/b>: 178\u2013194.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Question-negotiation+and+information+seeking+in+libraries&rft.jtitle=College+%26+Research+Libraries&rft.aulast=Taylor%2C+R.S.&rft.au=Taylor%2C+R.S.&rft.date=1968&rft.volume=29&rft.pages=178%E2%80%93194&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MirelApp98-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MirelApp98_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-MirelApp98_12-1\" rel=\"external_link\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-MirelApp98_12-2\" rel=\"external_link\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-MirelApp98_12-3\" rel=\"external_link\">12.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">\"Applied constructivism for user documentation\". <i>Journal of Business and Technical Communication<\/i> <b>12<\/b>: 7\u201349. 1998.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applied+constructivism+for+user+documentation&rft.jtitle=Journal+of+Business+and+Technical+Communication&rft.date=1998&rft.volume=12&rft.pages=7%E2%80%9349&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MirelDes03-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MirelDes03_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-MirelDes03_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Mirel, B.; Albers, M. (Ed.); Mazur, B. (Ed.) (2003). \"Dynamic usability: Designing usefulness into systems for complex tasks\". <i>Content and Complexity: Information Design in Software Development and Documentation<\/i>. Erlbaum. pp. 233\u2013261.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Dynamic+usability%3A+Designing+usefulness+into+systems+for+complex+tasks&rft.atitle=Content+and+Complexity%3A+Information+Design+in+Software+Development+and+Documentation&rft.aulast=Mirel%2C+B.%3B+Albers%2C+M.+%28Ed.%29%3B+Mazur%2C+B.+%28Ed.%29&rft.au=Mirel%2C+B.%3B+Albers%2C+M.+%28Ed.%29%3B+Mazur%2C+B.+%28Ed.%29&rft.date=2003&rft.pages=pp.%26nbsp%3B233%E2%80%93261&rft.pub=Erlbaum&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoodsCan02-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WoodsCan02_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">\"Can we ever escape from data overload? A cognitive systems diagnosis\". <i>Cognition, Technology & Work<\/i> <b>4<\/b>: 22\u201336. 2002.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Can+we+ever+escape+from+data+overload%3F+A+cognitive+systems+diagnosis&rft.jtitle=Cognition%2C+Technology+%26+Work&rft.date=2002&rft.volume=4&rft.pages=22%E2%80%9336&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersIR09-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlbersIR09_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Albers, M. (October 2009). \"Information relationships: The source of useful and usable content\". <i>Proceedings of the 29th Annual International Conference on Computer Documentation<\/i>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+relationships%3A+The+source+of+useful+and+usable+content&rft.jtitle=Proceedings+of+the+29th+Annual+International+Conference+on+Computer+Documentation&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=October+2009&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FodelHII12-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FodelHII12_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-FodelHII12_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Fidel, R. (2012). <i>Human Information Interaction: An Ecological Approach to Information Behavior<\/i>. MIT.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Human+Information+Interaction%3A+An+Ecological+Approach+to+Information+Behavior&rft.aulast=Fidel%2C+R.&rft.au=Fidel%2C+R.&rft.date=2012&rft.pub=MIT&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaracevicRel07-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SaracevicRel07_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-SaracevicRel07_17-1\" rel=\"external_link\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saracevic, T. (2007). \"Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II. Nature and manifestations of relevance\". <i>Journal of the Association for Information Science and Technology<\/i> <b>58<\/b>: 1915\u20131933.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Relevance%3A+A+review+of+the+literature+and+a+framework+for+thinking+on+the+notion+in+information+science.+Part+II.+Nature+and+manifestations+of+relevance&rft.jtitle=Journal+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Saracevic%2C+T.&rft.au=Saracevic%2C+T.&rft.date=2007&rft.volume=58&rft.pages=1915%E2%80%931933&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ThuringHyp95-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ThuringHyp95_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thuring, M.; Hannemann, J.; Haake, J. (1995). \"Hypermedia and cognition: Designing for comprehension\". <i>Communications of the ACM<\/i> <b>38<\/b>: 57\u201366.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hypermedia+and+cognition%3A+Designing+for+comprehension&rft.jtitle=Communications+of+the+ACM&rft.aulast=Thuring%2C+M.%3B+Hannemann%2C+J.%3B+Haake%2C+J.&rft.au=Thuring%2C+M.%3B+Hannemann%2C+J.%3B+Haake%2C+J.&rft.date=1995&rft.volume=38&rft.pages=57%E2%80%9366&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EndsleyTow95-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EndsleyTow95_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Endsley, M. (1995). \"Toward a theory of situation awareness in dynamic systems\". <i>Human Factors<\/i> <b>37<\/b>: 32\u201364.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+theory+of+situation+awareness+in+dynamic+systems&rft.jtitle=Human+Factors&rft.aulast=Endsley%2C+M.&rft.au=Endsley%2C+M.&rft.date=1995&rft.volume=37&rft.pages=32%E2%80%9364&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KainIU10-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KainIU10_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Kain, K. de Jong, M.; Smith, C.; Albers, M. (Ed.); Still, B. (Ed.) (2010). \"Information Usability testing as audience and context analysis for risk communication\". <i>Usability of Complex Information Systems: Evaluation of User Interaction<\/i>. CRC Press. pp. 305\u2013332.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Information+Usability+testing+as+audience+and+context+analysis+for+risk+communication&rft.atitle=Usability+of+Complex+Information+Systems%3A+Evaluation+of+User+Interaction&rft.aulast=Kain%2C+K.+de+Jong%2C+M.%3B+Smith%2C+C.%3B+Albers%2C+M.+%28Ed.%29%3B+Still%2C+B.+%28Ed.%29&rft.au=Kain%2C+K.+de+Jong%2C+M.%3B+Smith%2C+C.%3B+Albers%2C+M.+%28Ed.%29%3B+Still%2C+B.+%28Ed.%29&rft.date=2010&rft.pages=pp.%26nbsp%3B305%E2%80%93332&rft.pub=CRC+Press&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RichardsUsa10-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RichardsUsa10_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Richards, D. (2015). \"Testing the waters: Local Users, sea level rise, and the productive usability of interactive geovisualizations\". <i>Communication Design Quarterly<\/i> <b>3<\/b>: 24\u201332.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Testing+the+waters%3A+Local+Users%2C+sea+level+rise%2C+and+the+productive+usability+of+interactive+geovisualizations&rft.jtitle=Communication+Design+Quarterly&rft.aulast=Richards%2C+D.&rft.au=Richards%2C+D.&rft.date=2015&rft.volume=3&rft.pages=24%E2%80%9332&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannonMath48-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShannonMath48_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shannon, C. (1948). \"A mathematical theory of communication\". <i>Bell System Technical Journal<\/i> <b>27<\/b>: 379\u2013423.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+mathematical+theory+of+communication&rft.jtitle=Bell+System+Technical+Journal&rft.aulast=Shannon%2C+C.&rft.au=Shannon%2C+C.&rft.date=1948&rft.volume=27&rft.pages=379%E2%80%93423&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RedishExp07-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RedishExp07_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Redish, J. (2007). \"Expanding usability testing to evaluate complex systems\". <i>The Journal of Usability Studies<\/i> <b>2<\/b>: 102\u2013111.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Expanding+usability+testing+to+evaluate+complex+systems&rft.jtitle=The+Journal+of+Usability+Studies&rft.aulast=Redish%2C+J.&rft.au=Redish%2C+J.&rft.date=2007&rft.volume=2&rft.pages=102%E2%80%93111&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SommervilleLar12-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SommervilleLar12_24-0\" rel=\"external_link\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-SommervilleLar12_24-1\" rel=\"external_link\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sommerville, O.; Cliff, D.; Calinescu, R.; Keen, J.; Kelly, T.; Kwiatkowska, M.; Mcdermid, J.; Paige, R. (2012). \"Large scale complex IT systems\". <i>Communications of the ACM<\/i> <b>55<\/b>: 71\u201377.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Large+scale+complex+IT+systems&rft.jtitle=Communications+of+the+ACM&rft.aulast=Sommerville%2C+O.%3B+Cliff%2C+D.%3B+Calinescu%2C+R.%3B+Keen%2C+J.%3B+Kelly%2C+T.%3B+Kwiatkowska%2C+M.%3B+Mcdermid%2C+J.%3B+Paige%2C+R.&rft.au=Sommerville%2C+O.%3B+Cliff%2C+D.%3B+Calinescu%2C+R.%3B+Keen%2C+J.%3B+Kelly%2C+T.%3B+Kwiatkowska%2C+M.%3B+Mcdermid%2C+J.%3B+Paige%2C+R.&rft.date=2012&rft.volume=55&rft.pages=71%E2%80%9377&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KleinSou99-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KleinSou99_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-KleinSou99_25-1\" rel=\"external_link\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Klein, G. (1999). <i>Sources of Power: How People make Decisions<\/i>. MIT.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Sources+of+Power%3A+How+People+make+Decisions&rft.aulast=Klein%2C+G.&rft.au=Klein%2C+G.&rft.date=1999&rft.pub=MIT&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SimonMod79-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SimonMod79_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Simon, H. (1979). <i>Models of Thought<\/i>. Yale University Press.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Models+of+Thought&rft.aulast=Simon%2C+H.&rft.au=Simon%2C+H.&rft.date=1979&rft.pub=Yale+University+Press&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FennemaAnt95-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FennemaAnt95_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fennema, M.; Kleinmuntz, D. (1995). \"Anticipation of effort and accuracy in multi-attribute choice\". <i>Organizational Behavior and Human Decision Processes<\/i> <b>63<\/b>: 21\u201332.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Anticipation+of+effort+and+accuracy+in+multi-attribute+choice&rft.jtitle=Organizational+Behavior+and+Human+Decision+Processes&rft.aulast=Fennema%2C+M.%3B+Kleinmuntz%2C+D.&rft.au=Fennema%2C+M.%3B+Kleinmuntz%2C+D.&rft.date=1995&rft.volume=63&rft.pages=21%E2%80%9332&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PayneAda01-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PayneAda01_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Payne, S.; Howes. A.; Reader, W. (2001). \"Adaptively distributing cognition: A decision making perspective on human-computer interaction\". <i>Behaviour & Information Technology<\/i> <b>20<\/b>: 339\u2013346.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adaptively+distributing+cognition%3A+A+decision+making+perspective+on+human-computer+interaction&rft.jtitle=Behaviour+%26+Information+Technology&rft.aulast=Payne%2C+S.%3B+Howes.+A.%3B+Reader%2C+W.&rft.au=Payne%2C+S.%3B+Howes.+A.%3B+Reader%2C+W.&rft.date=2001&rft.volume=20&rft.pages=339%E2%80%93346&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MasonProg73-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MasonProg73_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mason, R.; Mitroff, I. (1973). \"A program for research on management information systems\". <i>Management Science<\/i> <b>19<\/b>: 475\u2013487.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+program+for+research+on+management+information+systems&rft.jtitle=Management+Science&rft.aulast=Mason%2C+R.%3B+Mitroff%2C+I.&rft.au=Mason%2C+R.%3B+Mitroff%2C+I.&rft.date=1973&rft.volume=19&rft.pages=475%E2%80%93487&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GerlachUnd91-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GerlachUnd91_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gerlach, K.; Kuo, F. (1991). \"Understanding human-computer interaction for information system design\". <i>MIS Quarterly<\/i> <b>15<\/b>: 527\u2013550.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+human-computer+interaction+for+information+system+design&rft.jtitle=MIS+Quarterly&rft.aulast=Gerlach%2C+K.%3B+Kuo%2C+F.&rft.au=Gerlach%2C+K.%3B+Kuo%2C+F.&rft.date=1991&rft.volume=15&rft.pages=527%E2%80%93550&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoodsGoc88-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WoodsGoc88_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Woods, D.; Roth, E. (1988). \"Cognitive engineering: Human problem solving with tools\". <i>Human Factors<\/i> <b>30<\/b>: 415\u2013430.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cognitive+engineering%3A+Human+problem+solving+with+tools&rft.jtitle=Human+Factors&rft.aulast=Woods%2C+D.%3B+Roth%2C+E.&rft.au=Woods%2C+D.%3B+Roth%2C+E.&rft.date=1988&rft.volume=30&rft.pages=415%E2%80%93430&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ConklinWicked-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ConklinWicked_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Conklin, J.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cognexus.org\/id26.htm\" target=\"_blank\">\"Wicked problems and fragmentation\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cognexus.org\/id26.htm\" target=\"_blank\">http:\/\/www.cognexus.org\/id26.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 24 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Wicked+problems+and+fragmentation&rft.atitle=&rft.aulast=Conklin%2C+J.&rft.au=Conklin%2C+J.&rft_id=http%3A%2F%2Fwww.cognexus.org%2Fid26.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AckoffMan67-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AckoffMan67_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ackoff, R. (1967). \"Management misinformation systems\". <i>Management Science<\/i> <b>14<\/b>: 147\u2013156.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Management+misinformation+systems&rft.jtitle=Management+Science&rft.aulast=Ackoff%2C+R.&rft.au=Ackoff%2C+R.&rft.date=1967&rft.volume=14&rft.pages=147%E2%80%93156&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation and content. The two most notable changes involve citations. Albers lists the following citation in his references as citation 20:\n<\/p><p>\"20. Albers, M. Contextual awareness as measure of human-information interaction in usability and design. In Proceedings of the HCI International, Orlando, FL, USA, 12\u201314 July 2011.\" \n<\/p><p>However, he never referenced that citation number in the paper. We have thus omitted it. Additionally, under the subsection \"Experience-based\", he references \"Klein\u2019s (1999)\" but seemingly uses the incorrect citation number of 12, for Mirel 1998. We've corrected the citation number.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210241\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.694 seconds\nReal time usage: 0.738 seconds\nPreprocessor visited node count: 23416\/1000000\nPreprocessor generated node count: 35444\/1000000\nPost\u2010expand include size: 112730\/2097152 bytes\nTemplate argument size: 35500\/2097152 bytes\nHighest expansion depth: 14\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 645.442 1 - -total\n 75.14% 484.986 33 - Template:Citation\/core\n 48.64% 313.930 21 - Template:Cite_journal\n 29.02% 187.328 11 - Template:Cite_book\n 10.31% 66.526 1 - Template:Infobox_journal_article\n 9.89% 63.842 1 - Template:Infobox\n 5.92% 38.236 80 - Template:Infobox\/row\n 4.04% 26.061 36 - Template:Citation\/make_link\n 2.79% 18.005 1 - Template:Cite_web\n 0.32% 2.077 8 - Template:Clear\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7605-0!*!0!!en!5!* and timestamp 20181213210240 and revision id 23410\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\">https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","be30bf8a40599ea6d6b707ba00d0baa1_images":["https:\/\/www.limswiki.org\/images\/f\/f8\/Fig1_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/7\/71\/Fig2_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/8\/83\/Fig3_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/b\/be\/Fig4_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/d\/d4\/Fig5_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/7\/7b\/Fig6_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/9\/9f\/Fig7_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/e\/e5\/Fig8_Albers_Informatics2015_2-2.jpg"],"be30bf8a40599ea6d6b707ba00d0baa1_timestamp":1544734960,"13b3b6c47f60a3b75f007d8cd017d140_type":"article","13b3b6c47f60a3b75f007d8cd017d140_title":"Generalized procedure for screening free software and open-source software applications (Joyce 2015)","13b3b6c47f60a3b75f007d8cd017d140_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version","13b3b6c47f60a3b75f007d8cd017d140_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Generalized procedure for screening free software and open-source software applications\/Print version\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nGeneralized Procedure for Screening Free Software and Open Source Software ApplicationsAuthor(s)\n \nJoyce, JohnAuthor affiliation(s)\n \nArcana Informatica; Scientific ComputingPrimary contact\n \nEmail: jrjoyce@gmail.comYear published\n \n2015Distribution license\n \nCreative Commons Attribution-ShareAlike 4.0 InternationalWebsite\n \nOriginal versionDownload\n \nPDF (Note: Inline references fail to load in PDF version)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Results \n\n3.1 Literature review \n3.2 Initial evaluation and selection recommendations \n\n3.2.1 Identify needs \n3.2.2 Identify candidates \n\n\n3.3 In-depth evaluation \n\n3.3.1 System functionality \n3.3.2 Community \n3.3.3 System cost \n3.3.4 Popularity \n3.3.5 Product support \n3.3.6 Maintenance \n3.3.7 Reliability \n3.3.8 Performance \n3.3.9 Scalability \n3.3.10 Usability \n3.3.11 Security \n3.3.12 Flexibility\/Customizability \n3.3.13 Interoperability\/Integration \n3.3.14 Legal\/License issues \n\n\n3.4 Completing the evaluation \n\n3.4.1 Caveats \n3.4.2 Screening tabulation \n3.4.3 Demonstration surveys \n\n\n\n\n4 Summary \n5 Glossary \n6 References \n7 Notes \n\n\n\nAbstract \nFree software and open-source software projects have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process, we have initiated a project to generate a library of in-depth reviews of free software and open-source software applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure is executed against a selected class of applications.\n\nIntroduction \nThere is much confusion regarding free software and open-source software, and many people use these terms interchangeably. However, the connotations associated with the terms are highly significant. So perhaps we should start with an examination of the terms to clarify what we are attempting to screen. While there are many groups and organizations involved with open-source software, two of the main ones are the Free Software Foundation (FSF) and the Open Source Initiative (OSI).\nWhen discussing free software, we are not explicitly discussing software for which no fee is charged; rather, we are referring to \"free\" in terms of liberty. To quote the Free Software Foundation (FSF)[1]:\n\nA program is free software if the program's users have the four essential freedoms:\n The freedom to run the program as you wish, for any purpose (freedom 0).\n The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.\n The freedom to redistribute copies so you can help your neighbor (freedom 2).\n The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.<\/blockquote>\nThis does not mean that a program is provided at no cost, or gratis, though some of these rights imply that it would be. In the FSF's analysis, any application that does not conform to these freedoms is unethical. While there is also \"free software\" or \"freeware\" that is given away at no charge but without the source code, this would not be considered free software under the FSF definition.\nThe Open Source Initiative (OSI), originally formed to promote free software, refers to it as open-source software (OSS) to make it sound more business friendly. The OSI defines open-source software as any application that meets the following 10 criteria, which they based on the Debian Free Software Guidelines[2]:\n\n Free redistribution\n Source code included\n Must allow derived works\n Must preserve the integrity of the author's source code\n License must not discriminate against persons or groups\n License must not discriminate against fields of endeavor\n Distribution of licenses\n License must not be specific to a production\n License must not restrict other software\n License must be technology-neutral\nOpen-source software adherents take what they consider the more pragmatic view of looking more at the license requirements and put significant effort into convincing commercial enterprises of the practical benefits of open source, meaning the free availability of application source code.\nIn an attempt to placate both groups when discussing the same software application, the term free\/open-source software (F\/OSS) was developed. Since the term \"free\" was still tending to confuse some people, the term \"libre,\" which connotes freedom, was added resulting in the term free\/libre open-source software (FLOSS). If you perform a detailed analysis on the full specifications, you will find that all free software fits the open-source software definition, while not all open-source software fits the free software definition. However, any open-source software that is not also free software is the exception rather than the rule. As a result, you will find these acronyms used almost interchangeably, but there are subtle differences in meaning, so stay alert. In the final analysis, the software license that accompanies the software is what you legally have to follow.\nThe reality is that since both groups trace their history back to the same origins, the practical differences between an application being free software or open-source are generally negligible. Keep in mind that the above descriptions are to some degree generalizations, as both organizations are involved in multiple activities. There are many additional groups interested in open source for a wide variety of reasons. However, this diversity is also a strong point, resulting in a vibrant and dynamic community. You should not allow the difference in terminology to be divisive. The fact that all of these terms can be traced back to the same origin should unite us.[3] In practice, many of the organization members will use the terms interchangeably, depending on the point that they are trying to get across. With an excess of 300,000 FLOSS applications currently registered in SourceForge.net[4] and over 10 million repositories on GitHub[5], there are generally multiple options accessible for any class of application, be it a laboratory information management system (LIMS), an office suite, a data base, or a document management system. Presumably you have gone through the assessment of the various challenges to using an open-source application[6] and have decided to move ahead with selecting an appropriate application. The difficulty now becomes selecting which application to use. While there are multiple indexes of FOSS projects, these are normally just listings of the applications with a brief description provided by the developers, with no indication of the vitality or independent evaluation of the project.\nWhat is missing is a catalog of in-depth reviews of these applications, eliminating the need for each group to go through the process of developing a list of potential applications, screening all available applications, and performing in-depth reviews of the most promising candidates. While true that once an organization has made a tentative selection it will need to perform its own testing to confirm that the selected application meets its specific needs, there is no reason for everyone to go through the tedious process of identifying projects and weeding out the untenable ones.\n\n\n\n\n\n\n\n\n\n Illustration 1.: This diagram, originally by Chao-Kuei and updated by several others since, explains the different categories of software. It's available as a scalable vector graphic and as an XFig document, under the terms of any of the GNU GPL v2 or later, the GNU FDL v1.2 or later, or the Creative Commons Attribution-Share Alike v2.0 or later\n\n\n\nThe primary goal of this document is to describe a general procedure capable of being used to screen any selected class of software applications. The immediate concern is with screening FLOSS applications, though allowances can be made to the process to allow at least rough cross-comparison of both FOSS and commercial applications. To that end, we start with an examination of published survey procedures. We then combine a subset of standard software evaluation procedures with recommendations for evaluating FLOSS applications. Because it is designed to screen such a diverse range of applications, the procedure is by necessity very general. However, as we move through the steps of the procedure, we will describe how to tune the process for the class of software that you are interested in.\nYou can also ignore any arguments regarding selecting between FLOSS and commercial applications. In this context, \"commercial\" refers to the marketing approach, not to the quality of the software. Many FLOSS applications have comparable, if not superior, quality to products that are traditionally marketed and licensed. Wheeler discusses this issue in more detail, showing that by many definitions FLOSS is commercial software.[7]\nThe final objective of this process is to document a procedure that can then be applied to any class of FOSS applications to determine which projects in the class are the most promising to pursue, allowing us to expend our limited resources most effectively. As the information available for evaluating FOSS projects is generally quite different from that available for commercially licensed applications, this evaluation procedure has been optimized to best take advantage of this additional information.\n\nResults \nLiterature review \nA search of the literature returns thousands of papers related to open-source software, but most are of limited value in regards to the scope of this project. The need for a process to assist in selecting between open-source projects is mentioned in a number of these papers, and there appear to be over a score of different published procedures. Regrettably, none of these methodologies appear to have gained large-scale support in the industry. \nStol and Babar have published a framework for comparing evaluation methods targeting open-source software and include a comparison of 20 of them.[8] They noted that web sites that simply consisted of a suggestion list for selecting an open-source application were not included in this comparison. This selection difficulty is nothing new with FLOSS applications. In their 1994 paper, Fritz and Carter review over a dozen existing selection methodologies, covering their strengths, weaknesses, the mathematics used, as well as other factors involved.[9]\n\n\n\n\n\n\n\n No.\n\n Name\n\n Year\n\n Orig\n\n Method\n\n\n1\n\nCapgemini Open Source Maturity Model\n\n2003\n\nI\n\nYes\n\n\n2\n\nEvaluation Framework for Open Source Software\n\n2004\n\nR\n\nNo\n\n\n3\n\nA Model for Comparative Assessment of Open Source Products\n\n2004\n\nR\n\nYes\n\n\n4\n\nNavica Open Source Maturity Model\n\n2004\n\nI\n\nYes\n\n\n5\n\nWoods and Guliani's OSMM\n\n2005\n\nI\n\nNo\n\n\n6\n\nOpen Business Readiness Rating (OpenBRR)[10][11]\n\n2005\n\nR\/I\n\nYes\n\n\n7\n\nAtos Origin Method for Qualification and Selection of Open Source Software (QSOS)\n\n2006\n\nI\n\nYes\n\n\n8\n\nEvaluation Criteria for Free\/Open Source Software Products\n\n2006\n\nR\n\nNo\n\n\n9\n\nA Quality Model for OSS Selection\n\n2007\n\nR\n\nNo\n\n\n10\n\nSelection Process of Open Source Software\n\n2007\n\nR\n\nYes\n\n\n11\n\nObservatory for Innovation and Technological transfer on Open Source software (OITOS)\n\n2007\n\nR\n\nYes\n\n\n12\n\nFramework for OS Critical Systems Evaluation (FOCSE)\n\n2007\n\nR\n\nNo\n\n\n13\n\nBalanced Scorecards for OSS\n\n2007\n\nR\n\nNo\n\n\n14\n\nOpen Business Quality Rating (OpenBQR)\n\n2007\n\nR\n\nYes\n\n\n15\n\nEvaluating OSS through Prototyping\n\n2007\n\nR\n\nYes\n\n\n16\n\nA Comprehensive Approach for Assessing Open Source Projects\n\n2008\n\nR\n\nNo\n\n\n17\n\nSoftware Quality Observatory for Open Source Software (SQO-OSS)\n\n2008\n\nR\n\nYes\n\n\n18\n\nAn operational approach for selecting open source components in a software development project[12]\n\n2008\n\nR\n\nNo\n\n\n19\n\nQualiPSo trustworthiness model\n\n2008\n\nR\n\nNo\n\n\n20\n\nOpenSource Maturity Model (OMM)[13]\n\n2009\n\nR\n\nNo\n\n\n\nTable 1.: Comparison frameworks and methodologies for examination of FLOSS applications extracted from Stol and Babar.[8] The selection\r\nprocedure is described in Stol's and Barbar's paper, however, 'Year' indicates the date of publication, 'Orig.' indicates whether the described\r\n process originated in industry (I) or research (R), while 'Method' indicates whether the paper describes a formal analysis method and procedure (Yes)\r\nor just a list of evaluation criteria (No).\nExtensive comparisons between some of these methods have also been published, such as Deprez's and Alexandre's comparative assessment of the OpenBRR and QSOS techniques.[14] Wasserman and Pal have also published a paper under the title of \"Evaluating Open Source Software,\" which appears to be more of an updated announcement and in-depth description of the Business Readiness Rating (BRR) framework.[15] Jadhav and Sonar have also examined the issue of both evaluating and selecting software packages. They include a helpful analysis of the strengths and weaknesses of the various techniques.[16] Perhaps more importantly, they clearly point out that there is no common list of evaluation criteria. While the majority of the articles they reviewed listed the criteria used, Jadhav and Sonar indicated that these criteria frequently did not include a detailed definition, which required each evaluator to use their own, sometimes conflicting, interpretation.\nSince the publication of Stol and Babar's paper, additional evaluation methods have been published. Of particular interest are a series of papers by Pani et al. describing their proposed FAME (Filter, Analyze, Measure and Evaluate) methodology.[17][18][19][20] In their \"Transferring FAME\" paper, they emphasized that all of the evaluation frameworks previously described in the published literature were frequently not easy to apply to real environments, as they were developed using an analytic research approach which incorporated a multitude of factors.[20]\nTheir stated design objective with FAME is to reduce the complexity of performing the application evaluation, particularly for small organizations. They specify \"[t]he goals of FAME methodology are to aid the choice of high-quality F\/OSS products, with high probability to be sustainable in the long term, and to be as simple and user friendly as possible.\" They further state that \"[t]he main idea behind FAME is that the users should evaluate which solution amongst those available is more suitable to their needs by comparing technical and economical factors, and also taking into account the total cost of individual solutions and cash outflows. It is necessary to consider the investment in its totality and not in separate parts that are independent of one another.\"[20]\nThis paper breaks the FAME methodology into four activities:\n\n Identify the constraints and risks of the projects\n Identify user requirements and rank\n Identify and rank all key objectives of the project\n Generate a priority framework to allow comparison of needs and features\nTheir paper includes a formula for generating a score from the information collected. The evaluated system with the highest \"major score,\" Pjtot, indicates the system selected. While it is a common practice to define an analysis process which condenses all of the information gathered into a single score, I highly caution against blindly accepting such a score. FAME, as well as a number or the other assessment methodologies, is designed for iterative use. The logical purpose of this is to allow the addition of factors initially overlooked into your assessment, as well as to change the weighting of existing factors as you reevaluate their importance. However, this feature means that it is also very easy to unconsciously, or consciously, skew the results of the evaluation to select any system you wish. Condensing everything down into a single value also strips out much of the information that you have worked so hard to gather. Note that you can generate the same result score using significantly different input values. While of value, selecting a system based on just the highest score could potentially leave you with a totally unworkable system.\nPani et al. also describe a FAMEtool to assist in this data gathering and evaluation.[19] However, a general web search as well as a review of their FAME papers revealed no indication of how to obtain this resource. While this paper includes additional comparisons with other FLOSS analysis methodologies, and there are some hints suggesting that the FAMEtool is being provided as a web service, I have found no URL specified for it. As of now, I have received no responses from the research team via either e-mail or Skype regarding FAME, the FAMEtool, or feedback on its use.\nDuring this same time frame Soto and Ciolkowski also published papers describing the QualOSS Open Source Assessment Model and compared it to a number of the procedures in Stol and Barbar's table.[21][22] Their focus was primarily on three process perspectives: product quality, process maturity, and sustainability of the development community. Due to the lack of anything more than a rudimentary process perspective examination, they felt that the following OSS project assessment models were unsatisfactory: QSOS, CapGemni OSMM, Navica OSMM, and OpenBRR. They position QualOSS as an extension of the tralatitious CMMI and SPICE process maturity models. While there are multiple items in the second paper that are worth incorporating into an in-depth evaluation process, they do not seem suitable for what is intended as a quick survey.\nAnother paper, published by Haaland and Groven, also compared a number of open source quality models. To this paper's credit, the authors devoted a significant amount of space to discussing the different definitions of quality and how the target audience of a tool might affect which definition was used.[23] Like Stohl and Babar, they listed a number of the quality assessment models to choose from, including OSMM, QSOS, OpenBRR, and others. For their comparison, they selected OpenBRR and QualOSS. They appear to have classified OpenBRR as a first generation tool with a \"[us]ser view on quality\" and QualOSS as a second generation tool with a \"business point of view.\" An additional variation is that OpenBRR is primarily a manual tool, while QualOSS is primarily an automated tool. Their analysis in this article clearly demonstrates the steps involve in using these tools and in highlighting where they are objective and subjective. While they were unable to answer their original question as to whether the first- or second-generation tools did a better job of evaluation, to me they answered the following even more important but unasked question. As they proceeded through their evaluation, it became apparent as to how much the questions defined in the methods could affect the results of the evaluations. Even though the authors might have considered the questions to be objective, I could readily see how some of these questions could be interpreted in alternate ways. My takeaway is an awareness of the potential danger of using rigid tools, as they can skew the accuracy of the evaluation results depending on exactly what you want the evaluated application to do and how you plan to use it. These models can be very useful guides, but they should not be used to replace a carefully considered evaluation as there will always be factors influencing the selection decision which did not occur to anyone when the specifications were being written.\nHauge et al. have noted that despite the development of several normative methods of assessment, empirical studies have failed to show wide spread adoption of these methods.[24] From their survey of a number of Norwegian software companies, they have noticed a tendency for selectors to skip the in-depth search for what they call the \"best fit\" application and fall back on what they refer to as a '\"first fit.\" This is an iterative procedure with the knowledge gained from the failure of one set of component tests being incorporated into the evaluation of the next one. Their recommendation is for researchers to stop attempting to develop either general evaluation schemas or normative selection methods which would be applicable to any software application and instead focus on identifying situationally sensitive factors which can be used as evaluation criteria. This is a very rational approach as all situations, even if evaluating the same set of applications is going to be different, as each user's needs are different.\nAyalal et al. have performed a study to try to more accurately determine why more people don't take advantage of the various published selection methodologies.[25] While they looked at a number of factors and identified several possible problems, one of the biggest factors was the difficulty in obtaining the needed information for the evaluation. Based on the projects they studied, many did not provide a number of the basic pieces of information required for the evaluation, or perhaps worse, required extensive examination of the project web site and documentation to retrieve the required information. From her paper, it sounds as if this issue was more of a communication breakdown than an attempt to hide any of the information, not that this had any impact on the inaccessibility of the information.\nIn addition to the low engagement rates for the various published evaluation methods, another concern is the viability of the sponsoring organizations. One of the assessment papers indicated that the published methods with the smallest footprint, or the easiest to use, appeared to be FAME and the OpenBRR. I have already mentioned my difficulty obtaining additional information regarding FAME, and OpenBRR appears to be even more problematic. BRR was first registered on SourceForge in September of 2005[26], and an extensive Request For Comments from the founding members of the BRR consortium (SpikeSource, the Center for Open Source Investigation at Carnegie Mellon West, and Intel Corporation) was released.[10] In 2006, in contrast to typical Open Source development groups, the OpenBRR group announced the formation of an OpenBRR Corporate Community group. Peter Galli's story indicates that \"the current plan is that membership will not be open to all.\"[27] He quotes Murugan Pal saying \"membership will be on an invitation-only basis to ensure that only trusted participants are coming into the system.\" However, for some reason, at least some in the group \"expressed concern and unhappiness about the idea of the information discussed not being shared with the broader open-source community.\"[27]\nWhile the original Business Readiness Rating web site still exists, it is currently little more than a static web page.[28] It appears that some of the original information posted on the site is still there, you just have to know what its URL is to access it, as the original links on the web site have been removed. Otherwise, you may have to turn to the Internet Archive to retrieve some of their documentation. The lack of any visible activity regarding OpenBRR prompted a blog post from one graduate student in 2012 asking \"What happened to OpenBRR (Business Readiness Rating for Open Source)?\"[29]\nIt appears that at some point any development activity regarding OpenBRR was morphed into OSSpal.[30] However, background information on this project is sparse as well. While the site briefly mentions that OSSpal incorporates a number of lessons learned from BRR, there is very little additional information regarding the group or the methods procedures. Their \"All Projects\" tab provides a list of over 30 open-source projects, but the majority simply show \"No votes yet\" under the various headings. In fact, as of now, the only projects showing any input at all are for Ubuntu and Mozilla Firefox.\n\nInitial evaluation and selection recommendations \nAt this point, we'll take a step back from the evaluation methodologies papers and examine some of the more general recommendations regarding evaluating and selecting FLOSS applications. The consistency of their recommendations may provide a more useful guide for an initial survey of FLOSS applications.\nIn TechRepublic, de Silva recommends 10 questions to ask when selecting a FLOSS application.[31] While he provides a brief discourse on each question in his paper to ensure you understand the point of his question, I've collected the 10 questions from his article into the following list. Once we see what overlap, if any, are amongst our general recommendations, we'll address some of the consolidated questions in more detail.\n\n Are the open source license terms compatible with my business requirements?\n What is the strength of the community?\n How well is the product adopted by users?\n Can I get a warranty or commercial support if I need it?\n What quality assurance processes exist?\n How good is the documentation?\n How easily can the system be customized to my exact requirements?\n How is this project governed and how easily can I influence the road map?\n Will the product scale to my enterprise's requirements?\n Are there regular security patches?\nSimilarly, in InfoWorld Phipps lists seven questions you should have answered before even starting to select a software package.[32] His list of questions, pulled directly from his article are:\n\n Am I granted copyright permission?\n Am I free to use my chosen business model?\n Am I unlikely to suffer patent attack?\n Am I free to compete with other community members?\n Am I free to contribute my improvements?\n Am I treated as a development peer?\n Am I inclusive of all people and skills?\nThis list of questions shows a moderately different point of view, as it is not only just about someone selecting an open-source system, but also it's about getting involved in its direct development. Padin, of 8th Light, Inc., takes the viewpoint of a developer who might incorporate open-source software into their projects.[33] The list of criteria pulled directly from his blog includes:\n\n Does it do what I need it to do?\n How much more do I need it to do?\n Documentation\n Easy to review source code\n Popularity\n Tests and specs\n Licensing\n Community\nMetcalfe of OSS Watch lists his top tips as[34]:\n\n Reputation\n Ongoing effort\n Standards and interoperability\n Support (Community)\n Support (Commercial)\n Version\n Version 1.0\n Documentation\n Skill setting\n Project Development Development Model\n License\nIn his LIMSexpert blog, Joel Limardo of ForwardPhase Technologies, LLC lists the following as components to check when evaluating an open-source application[35]:\n\n Check licensing\n Check code quality\n Test setup time\n Verify extensibility\n Check for separation of concerns\n Check for last updated date\n Check for dependence on outdated toolkits\/frameworks\nPerhaps the most referenced of the general articles on selecting FLOSS applications is David Wheeler's \"How to Evaluate Open Source Software \/ Free Software (OSS\/FS) Programs.\"[36] The detailed functionality to consider will vary with the types of applications being compared, but there are a number of general features that are relevant to almost any type of application. While we will cover them in more detail later, Wheeler categorizes the features to consider as the following:\n\n System functionality\n System cost \u2013 direct and in-direct\n Popularity of application, i.e. its market share for that type of application\n Varieties of product support available\n Maintenance of application, i.e, is development still taking place\n Reliability of application\n Performance of application\n Scalability of application\n Usability of application\n Security of application\n Adaptability\/customizability of application\n Interoperability of application\n Licensing and other legal issues\nWhile a hurried glance might suggest a lot of diversity in the features these various resources suggest, a closer look at the meaning of what they are saying shows a repetitive series of concerns. The primary significant differences between the functionality lists suggested is actually due more to how wide a breadth of the analysis process the authors are considering, as well as the underlying features that they are concerned with.\nWith a few additions, the high-level screening template described in the rest of this communication is based on Wheeler's previously mentioned document describing his recommended process for evaluating open-source software and free software programs. Structuring the items thus will make it easier to locate the corresponding sections in his document, which includes many useful specific recommendations as well as a great deal of background information to help you understand the why of the topic. I highly recommend reading it and following up on some of the links he provides. I will also include evaluation suggestions from several of the previously mentioned procedures where appropriate.\nWheeler defines four basic steps to this evaluation process, as listed below:\n\n Identify candidate applications.\n Read existing product reviews.\n Compare attributes of these applications to your needs.\n Analyze the applications best matching your needs in more depth.\nWheeler categorizes this process with the acronym IRCA. In this paper we will be focusing on the IRC components of this process. To confirm the efficacy of this protocol we will later apply it to several classes of open-source applications and examine the output of the protocol.\n\nIdentify needs \nRealistically, before you can perform a survey of applications to determine which ones best match your needs, you must determine what your needs actually are. The product of determining these needs is frequently referred to as the user requirements specification (URS).[37][38] This document can be generated in several ways, including having all of the potential users submit a list of the functions and capabilities that they feel is important. While the requirements document can be created by a single person, it is generally best to make it a group effort with multiple reviews of the draft document, including all of the users who will be working with the application. The reason for this is to ensure that an important requirement is not missed. When a requirement is missed, it is frequently due to the requirement being so basic that it never occurs to anyone that it specifically needed to be included in the requirements document. Admittedly, a detailed URS is not required at the survey level, but it is worth having if only to identify, by their implications, other features that might be significant.\nNeeds will, of course, vary with the type of application you are looking for and what you are planning to do with it. Keep in mind that the URS is a living document, subject to change through this whole process. Developing a URS is generally an iterative process, since as you explore systems, you may well see features that you hadn't considered that you find desirable. This process will also be impacted by whether the application to be selected will be used in a regulated environment. If it is, there will be existing documents that describe the minimum functionality that must be present in the system. Even if it is not to be used in a regulated environment, documents exist for many types of systems that describe the recommended functional requirements that would be expected for that type of system.\nFor a clarifying example, if you were attempting to select a laboratory information management system (LIMS), you can download checklists and standards of typical system requirements from a variety of sources.[39][40][41][42] These will provide you with examples of the questions to ask, but you will have to determine which ones are important to, or required for, your particular effort.\nDepending on the use to which this application is to be applied, you may be subject to other specific regulatory requirements as well. Which regulations may vary, since the same types of analysis performed for different industries fall under different regulatory organizations. This aspect is further complicated by the fact that you may be affected by more than one country's regulations if your analysis is applicable to products being shipped to other countries. While some specific regulations may have changed since its publication, an excellent resource to orient you to the diverse factors that must be considered is Siri Segalstad's book International IT Regulations and Compliance.[43] My understanding is that an updated version of this book is currently in preparation. Keep in mind that while regulatory requirements that you must meet will vary, these regulations by and large also describe best practices, or at least the minimal allowed practices. These requirements are not put in place arbitrarily (generally) or to make things difficult for you but to ensure the quality of the data produced. As such, any deviations should be carefully considered, whether working in a regulated environment or not. Proper due diligence would be to determine which regulations and standards would apply to your operation.\nFor a LIMS, an example of following best practices is to ensure that the application has a full and detailed audit trail. An audit trail allows you to follow the processing of items through your system, determining who did what and when. In any field where it might become important to identify the actions taken during a processing step, an audit trail should be mandatory. While your organization's operations may not fall under the FDA's 21 CFR Part 11 regulations, which address data access and security (including audit trails), it is still extremely prudent that the application you select complies with them. If it does not, then almost anyone could walk up to your system and modify data, either deliberately or accidentally, and you would have no idea of who made the changes or what changes they made. For that matter, you might not even be able to tell a change was made at all, which likely will raise concerns both inside and outside of your organization. This would obviously cause major problems if they became a hinge issue for any type of liability law suit.\nFor this screening procedure, you do not have to have a fully detailed URS, but it is expedient to have a list of your make-or-break issues. This list will be used later for comparing systems and determining which ones justify a more in-depth evaluation.\n\nIdentify candidates \nTo evaluate potential applications against your functional criteria, you must initially generate a list of potential systems. While this might sound easy, generating a comprehensive list frequently proves to be a challenge. When initiating the process, you must first determine the type of system that you are looking for, be it a LIMS, a hospital management system, a database, etc. At this point, you should be fairly open in building your list of candidates. By that, I mean that you should be careful not to select applications based solely on the utilization label applied to them. The same piece of software can frequently be applied to solve multiple problems, so you should cast a wide net and not automatically reject a system because the label you were looking for hadn't been applied to it. While the label may give you a convenient place to start searching, it is much more important to look for the functionality that you need, not what the system is called. In any case, many times the applied labels are vague and mean very different things to different people.\nThere are a variety of ways to generate your candidate list. A good place to start is simply talking with colleagues in your field. Have they heard of or used a FLOSS application of the appropriate type that they like? Another way is to just flip through journals and trade magazines that cover your field. Any sufficiently promising applications are likely to be mentioned there. Many of the trade magazines will have a special annual issue that covers equipment and software applicable to their field. It is difficult to generate a list of all potential resources, as many of these trade publications are little-known outside of their field. Also keep in mind that with the continued evolution of the World Wide Web, many of these trade publications also have associated web sites that you can scan or search. The table below includes just a minor fraction of these sites that are available. (We would welcome the suggestion of any additional resource sites that you are aware of. Please e-mail the fields covered, the resource name, and either its general URL or the URL of the specific resource section to the corresponding editor.)\n\n\n\n\n\n\n\n Field\n\n Resource Name\n\n URL\n\n\nAstronomy\n\nTech Support Alert\n\nhttp:\/\/www.techsupportalert.com\/best-free-astronomy-software.htm\n\n\nBusiness intelligence reporting\n\nTechnology Innovation Management Review\n\nhttp:\/\/timreview.ca\/article\/288\n\n\nCommunity radio\n\nPrometheus Radio Project\n\nhttp:\/\/prometheusradio.org\/Free_Open_Source_Tools_C_R\n\n\nComprehensive range of apps\n\nBlack Duck KnowledgeBase\n\nhttps:\/\/www.blackducksoftware.com\/products\/knowledgebase\n\n\nComprehensive range of apps\n\nThe Directory of Open Access Repositories - OpenDOAR\n\nhttp:\/\/www.opendoar.org\/\n\n\nData storage\n\nInfoStor\n\nhttp:\/\/www.infostor.com\/nas\/58-top-open-source-storage-project-1.html\n\n\nDigital audio editing\n\n25 Free Digital Audio Editors You Should Know\n\nhttp:\/\/www.hongkiat.com\/blog\/25-free-digital-audio-editors\/\n\n\nDigital video editing\n\nBest free video editing software: 20 top programs in 2015\n\nhttp:\/\/www.techradar.com\/us\/news\/software\/applications\/best-free-video-editing-software-9-top-programs-you-should-download-1136264\n\n\nDiverse range of Android apps\n\nWikipedia \u2013 Android applications\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_Android_applications\n\n\nDiverse range of apps\n\nLaunchpad, The Canonical Group\n\nhttp:\/\/launchpad.net\/\n\n\nDiverse range of apps\n\nMicrosoft CodePlex\n\nhttps:\/\/www.codeplex.com\/\n\n\nDiverse range of apps\n\nOpen Hub, Black Duck Software, Inc.\n\nhttps:\/\/www.openhub.net\/\n\n\nDiverse range of apps (but site is deprecated)\n\nGoogle Code\n\nhttps:\/\/code.google.com\/p\/support\/\n\n\nDiverse range of apps (focus on infrastructure)\n\nOpen Source Guide by Smile\n\nhttp:\/\/www.open-source-guide.com\/en\n\n\nDiverse range of apps (primarily targeting Samsung products)\n\nSamsung Open Source Release Center\n\nhttp:\/\/opensource.samsung.com\/reception.do\n\n\nDiverse range of commercial open source apps and services\n\nWikipedia \u2013 Commercial open source applications\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_commercial_open-source_applications_and_services\n\n\nDiverse range of GNU apps\n\nSavannah GNU\n\nhttp:\/\/savannah.gnu.org\/\n\n\nDiverse range of non-GNU apps\n\nSavannah Non-GNU\n\nhttp:\/\/savannah.nongnu.org\n\n\nDiverse range of top apps\n\nProjects and Applications\n\nhttp:\/\/opensource.com\/resources\/projects-and-applications\n\n\nDrug discovery\n\nDrug Discovery Today\n\nhttp:\/\/www.sciencedirect.com\/science\/article\/pii\/S1359644605036925\n\n\nElectronic engineering\n\nEducation Engineering (EDUCON), 2010 IEEE\n\nhttp:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=5492430\n\n\nEmbroidery\n\nMachine Embroidery Portal\n\nhttp:\/\/www.k2g2.org\/portal:machine_embroidery\n\n\nEnterprise resource planning (ERP)\n\nA Comparison of Open Source ERP Systems\n\nhttp:\/\/www.big.tuwien.ac.at\/system\/theses\/20\/papers.pdf?1298476232\n\n\nEnterprise resource planning (ERP)\n\nOpen Source ERP Site\n\nhttp:\/\/www.open-source-erp-site.com\/list-of-open-source-erps.html\n\n\nFashion\n\nEthical Fashion Forum - 9 Open-Source & Low Cost Digital Fashion Business Tools\n\nhttp:\/\/source.ethicalfashionforum.com\/article\/9-open-source-low-cost-digital-fashion-business-tools\n\n\nGeographic information systems (GIS)\n\nOpen Source GIS\n\nhttp:\/\/opensourcegis.org\/\n\n\nGeophysics\n\nGeopsy project\n\nhttp:\/\/www.geopsy.org\/\n\n\nGeophysics\n\nWikipedia - Geophysics software\n\nhttps:\/\/en.wikipedia.org\/wiki\/Comparison_of_free_geophysics_software\n\n\nGIS for libraries\n\nLibrary Hi Tech\n\nhttp:\/\/www.emeraldinsight.com\/doi\/abs\/10.1108\/07378831011026742\n\n\nHelp desk\n\nCIO\n\nhttp:\/\/www.cio.com.au\/article\/320110\/5_open_source_help_desk_apps_watch\/\n\n\nHighly diverse range of apps\n\nCode NASA\n\nhttps:\/\/code.nasa.gov\/#\/\n\n\nHighly diverse range of apps\n\nNASA Open Source Software\n\nhttp:\/\/ti.arc.nasa.gov\/opensource\/\n\n\nLaboratory informatics\n\nLIMSwiki - Laboratory informatics software (open source)\n\nhttp:\/\/www.limswiki.org\/index.php\/Category:Laboratory_informatics_software_%28open_source%29\n\n\nLearning management systems (LMS)\n\nOpen Source LMS \u2013 10 Alternatives to Moodle\n\nhttp:\/\/barrysampson.com\/2009\/04\/08\/open-source-lms-10-alternatives-to-moodle\/\n\n\nLibrary science\n\nFOSS4LIB\n\nhttps:\/\/foss4lib.org\/packages\n\n\nLimited range of iOS apps\n\nWikipedia - Open-source iOS applications\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_iOS_applications\n\n\nMedical and laboratory informatics\n\nHealthcare Freeware\n\nhttp:\/\/www.healthcarefreeware.com\/lim.htm\n\n\nMedical office records\n\nZDNet\n\nhttp:\/\/www.zdnet.com\/article\/free-and-open-source-healthcare-software-for-your-practice\/\n\n\nMedicine\n\n'Open' Health IT Solutions for the Public & Private Sectors\n\nhttps:\/\/drive.google.com\/file\/d\/0B_TuX9zE68eWcVM1dkdDQTVCdnM\/edit\n\n\nMedicine\n\n50 Successful Open Source Projects That Are Changing Medicine\n\nhttp:\/\/nursingassistantguides.com\/2009\/50-successful-open-source-projects-that-are-changing-medicine\/\n\n\nMedicine\n\nDatamation\n\nhttp:\/\/www.datamation.com\/open-source\/50-open-source-replacements-for-health-care-software-1.html\n\n\nMedicine\n\nMedfloss.org\n\nhttp:\/\/www.medfloss.org\/\n\n\nMIDI controllers\n\nOpen Source MIDI Controllers\n\nhttp:\/\/punkmanufacturing.com\/wiki\/open-source-midi-controllers\n\n\nMusic\n\n20 great free and open source music making programs\n\nhttp:\/\/www.musicradar.com\/tuition\/tech\/20-great-free-and-open-source-music-making-programs-582934\n\n\nPattern management for textiles\n\nTools for computer generated patterns\n\nhttps:\/\/xxxclairewilliamsxxx.wordpress.com\/tools-to-create-and-explore-digital-patterns\/\n\n\nPersonal information management\n\nWikipedia - Personal information managers\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_personal_information_managers#Open_source_applications\n\n\nPhotography\n\nOpen Source Photography\n\nhttp:\/\/opensourcephotography.org\/software-list\n\n\nProject management\n\nOpen Source Photography\n\nhttp:\/\/opensourcephotography.org\/software-list\n\n\nRepository management\n\nFree and open-source repository software\n\nhttp:\/\/oad.simmons.edu\/oadwiki\/Free_and_open-source_repository_software\n\n\nSoftware test management\n\nSoftware Testing Help - 15 Best Test Management Tools for Software Testers\n\nhttp:\/\/www.softwaretestinghelp.com\/15-best-test-management-tools-for-software-testers\/\n\n\nSurvey and feedback\n\nCapterra - Top survey software products\n\nhttp:\/\/www.capterra.com\/survey-software\/\n\n\n\nTable 2.: Examples of focused FLOSS resource sites available on the web\n Illustration 2.: Result page displayed on Black Duck Software's Open Hub site for the Bika Open Source LIMS 3 project.\nI also recommend checking some of the general open source project lists, such as the ones generated by Cynthia Harvey at Datamation[44], which has been covering the computer and data-processing industry since 1957. In particular, you might find their article \"Open Source Software List: 2015 Ultimate List\" useful. It itemizes over 1,200 open source applications, including some in categories that I didn't even know existed.[45]\nIt would also be prudent to search the major open source repositories such as SourceForge[46] and GitHub.[47] Wikipedia includes a comparison of source code hosting facilities that would be worth reviewing as well.[48] Keep in mind that you will need to be flexible with your search terms, as the developers might be looking at the application differently than you are. While they were created for a different purpose, an examination of the books in The Architecture of Open Source Applications might prove useful as well.[49] Other sites where you might find interesting information regarding new open-source applications, are the various OpenSource award sites, such as the InfoWorld Best of Open Source Software Awards, colloquially known as the Bossies.[50] \nWhen searching the web, don't rely on just Google or Bing. Don't forget to checkout all of the journal web sites such as SpringerLink, Wiley, ScienceDirect, PubMed, and others as they contain a surprising amount of information on FLOSS. If you don't wish to search each of them individually, there are other search engines out there which can give you an alternate view of the research resources available. To name just two, be sure to try both Google Scholar[51] and Microsoft Academic Search.[52] These tools can also be used to search for masters theses and doctoral dissertations, which likewise contain a significant amount of information regarding open source.\nWhile working on creating your candidate list, be sure to pull any application reviews that you come across.[53] If done well, these reviews can save you a significant amount of time in screening a potential system. However, unless you're familiar with the credentials of the author, be cautious of relying on them for all of your information. While not common, people have been known to post fake reviews online, sometimes when it is not even April 1! Another great resource, both for identifying projects and obtaining information about them, is Open HUB[54], a web site dedicated to open source and maintained by Black Duck Software, Inc. Open Hub allows you to search by project, person, organization, forums, and code. For example, if I searched for Bika LIMS, it would currently return the results for Bika Open Source LIMS 3 along with some basic information regarding the system. If I were to click on the project's name, a much more detailed page regarding this project is displayed. Moving your mouse cursor over the graphs displays the corresponding information for that date.\nOnce a list of candidate applications has been generated, the list of entries must be compared. Some of this comparison can be performed objectively, but it also requires subjective analysis of some components. As Stol and Babar have shown, there is no single recognized procedure for either the survey or detailed comparison of FLOSS applications that has shown a marked popularity above the others.\nThe importance of any one specific aspect of the evaluation will vary with the needs of the organization. General system functionality will be an important consideration, but specific aspects of the contained functionality will have different values to different groups. For instance, interoperability may be very important to some groups, while others may be using this application as their only data system and they have no interest in exchanging data files with others, so interoperability is not a concern to them. While you can develop a weighting system for different aspects of the system, this can easily skew selections, resulting in a system that has a very good rating yet is unable to perform the required function. Keep in mind that though this is a high-level survey, we are asking broad critical questions, not attempting to compare detailed minutia. Also keep in mind that a particular requirement might potentially fall under multiple headings. For example, compliance with 21 CFR Part 11 regulations might be included under functionality or security.\n\nIn-depth evaluation \nWhile in-depth analysis of the screened systems will require a more detailed examination and comparison, for the purpose of this initial survey a much simpler assessment protocol will serve. While there is no single \"correct\" evaluation protocol, something in the nature of the three-leaf scoring criteria described for QSOS[14] should be suitable. Keep in mind that for this quick assessment we are using broad criteria, so both the criteria and the scoring will both be more ambiguous than that required for an in-depth assessment. Do not be afraid to split any of these criteria up into multiple finer classes of criteria if the survey requires it. This need would be most likely appear under \"system functionality,\" as that is where most people's requirements greatly diverge.\nIn this process, we will assign one of three numeric values to each of the listed criteria. A score of zero indicates that the system does not meet the specified criteria. A score of one indicates that the system marginally meets the specified criteria. You can look on this as the feature is present and workable, but not to the degree you'd like it. Finally, a score of two indicates that the system fully meets or exceeds the specified requirement. In the sections below I will list some possible criteria for this table. However, you can adjust these descriptions or add a weighting factor, as many other protocols do, to adjust for the criticality of a given requirement.\nRealistically, when it comes down to some of your potential evaluation criteria, the actuality is that for some of them, you can compensate for the missing factor in some other way. For other criteria, their presence or absence can be a drop-dead issue. That is, if the particular criteria or feature isn't present, then it doesn't matter how well any of the other criteria are ranked: that particular system is out of consideration. Deciding which, if any, criteria are drop-dead items should ideally be determined before you start your survey. This will not only be more efficient, in that it will allow you to cut off data collection at any failure point, but it will also help dampen the psychological temptation to fudge your criteria, retroactively deciding that a given criteria was not that important after all.\nAt this stage we are just wanting to reduce the number of systems for in-depth evaluation from potentially dozens, to perhaps three or four. As such, we will be refining our review criteria later, so if something isn't really a drop-dead criteria, don't mark it so. It's amazing the variety of feature tradeoffs people tend to make further down the line.\n\nSystem functionality \nWhile functionality is a key aspect of selecting a system, its assessment must be used with care. Depending on how a system is designed, a key function that you are looking for might not have been implemented, but in one system it can easily be added, while in another it would take a complete redesign of the application. Also consider the possibility of whether this function must be intrinsic to the application or if you can pair the application being evaluated with another application to cover the gap.\nIn most cases, you can obtain much of the functionality information from the project's web site or, occasionally, web sites. Some projects have multiple web sites, usually with one focused on project development and another targeting general users or application support. There are two different types of functionality to be tested. The first might be termed \"general functionality\" that would apply to almost any system. Examples of this could include the following:\n\n User authentication\n Audit trail (I'm big on detailed audit trails, as they can make the difference between between being in regulatory compliance and having a plant shut down. Even if you aren't required to have them, they are generally a good idea, as the records they maintain may be the only way for you to identify and correct a problem.)\n Sufficient system status display that the user can understand the state of the system\n Ability to store data in a secure fashion\nWe might term the second as \"application functionality.\" This is functionality specifically required to be able to perform your job. As the subject matter expert, you will be the one to create this list. Items might be as diverse as the following:\nFor a laboratory information management system...\n\n Can it print bar coded labels?\n Can it track the location of samples through the laboratory, as well as maintain the chain-of-custody for the sample?\n Can it track the certification of analysts for different types of equipment, as well as monitor the preventive maintenance on the instruments?\n Can it generate and track aliquots of the original sample?\nFor a geographic information system (GIS)[55]...\n\n What range of map projections is supported?\n Does the system allow you to create custom parameters?\n Does it allow import of alternate system formats?\n Can it directly interface with geographic positioning devices (GPS)?\nFor a library management software system (LMSS) (or integrated library system [ILS], if you prefer, or even library information management system [LIMS]; have you ever noticed how scientists love to reuse acronyms, even within the same field?)...\n\n Can you identify the location of any item in the collection at a given time?\n Can you identify any items that were sent out for repair and when?\n Can it identify between multiple copies of an item and between same named items on different types of media?\n Can it handle input from RFID tags?\n Can it handle and differentiate different clients residing at the same address?\n If needed, can it correlate the clients age with the particular item they are requesting, in case you have to deal with any type of age appropriate restrictions? \n If so, can it be overridden where necessary (maintaining the appropriate records in the audit trail as to why the rule was overridden)?\nFor an archival record manager (This classification can cover a lot of ground due to all of the different ways that \"archival\" and \"record\" are interpreted.)...\n\n In some operations, a record can be any information about a sample, including the sample itself. By regulation, some types of information must be maintained essentially for ever. In others, you might have to keep information for five years, while you have to maintain data for another type for 50 years.\n Can the application handle tracking records for different amounts of time?\n Does the system automatically delete these records at the end of the retention period or does it ask for confirmation from a person?\n Can overrides on a particular sample be applied so that records are not allowed to be deleted, either manually of because they are past their holding date, such as those that might be related to any litigation, while again maintaining all information about the override and who applied the override in the audit trail?\n In other operations, an archival record manager may actually refer to the management of archival records, be these business plans, architectural plans, memos, art work, etc.\n Does the system keep track of the type of each record?\n Does the system support appropriate meta data on different types of records?\n Does it just record the items location and who is responsible for it, such as a work of art?\n If a document, does it just maintain an electronic representation of a document, such as a PDF file, or does it record the location of the original physical document, or can it do both?\n Can it manage both permanently archived items, such as a work of art or a historically significant document, and more transitory items, where your record retention rules say to save it for five, 10, 15 years, etc., and then destroy it?\n In the latter case, does the system require human approval before destroying any electronic documents or flagging a physical item for disposal? Does it require a single human sign-off or must a chain of people sign-off on it, just to confirm that it is something to be discarded by business rules and not an attempt to hide anything?\nFor medical records[56][57]...\n\n This is a challenging quagmire, with frequently changing regulations and requirements. Depending on how you want to break it down, this heading can be segmented into two classes: electronic medical records (EMR) which can constitute an electronic version of the tracking of the patients health and electronic health records (EHR) which contains extensive information on the patient, including test results, diagnostic information, and other observations by the medical staff. For those who want to get picky, you can also subdivide the heading into imaging systems, such as X-rays and CAT scans and other specialized systems.\n Can all records be accessed quickly and completely under emergency situations?\n What functionality is in place to minimize the risk of a Health Insurance Portability and Accountability Act (HIPAA) violation?\n What functionality exists for automated data transfer from instruments or laboratory data systems to minimize transcription errors.\n How are these records integrated with any billing or other medical practice system?\n If integrated with the EMR and EHR record systems, does this application apply granular control over who can access these records and what information they are able to see?\nFor an enterprise resource planning (ERP) system...\n\n Davis has indicated that a generally accepted definition of an ERP system is a \"complex, modular software integrated across the company with capabilities for most functions in the organization.\"[58] I believe this translates as \"good luck,\" considering the complexity of the systems an ERP is designed to model and all of the functional requirements that go into that. It is perhaps for that reason that successful ERP implementations generally take several years.\n ERP systems generally must be integrated with other informatics systems within the organization. What types of interfaces does this system support?\n Is their definition of plug-and-play that you just have to configure the addresses and fields to exchange?\n Is their definition of interface that the system can read in information in a specified format and export the same, leaving you to write the middle ware program to translate the formats between the two systems?\nFrom the above, it is easy to see why researchers have encountered difficulty in developing a fixed method that can be used to evaluate anything. At this point, it is quite acceptable to group similar functions together \u2014 as this is a high-level survey to identify which systems will definitely not be suitable \u2014 so we can focus our researches on those that might be. Researchers such as Sarrab and Rehman summarize system functionality as \"achieving the user's requirements, correct output as user's expectations and verify that the software functions appropriately as needed.\"[59]\nSuggested ratings are:\n\n Zero \u2013 Application does not support required functionality.\n One \u2013 Application supports the majority of functionality to at least an useable extent.\n Two \u2013 Application meets or exceeds all functional requirements.\nCommunity \nMany of the researchers I've encountered have indicated that community is the most critical factor of a FLOSS project. There are a number of reasons for this. First, the health and sustainability of a FLOSS project is indicated by a large and diverse community that is both active and responsive.[31] Additionally, the core programmers in a FLOSS project are generally few, and it is the size of the project community that determines how well-reviewed the application is, ensuring quality control of the projects code. Finally, the size of the project community correlates with the lifetime of the project.\nSuggested ratings are:\n\n Zero \u2013 No community exists. No development activity is observable. Project is dead.\n One - Community is small and perhaps insular. May consist of just one or two programmers with perhaps a small number of satellite users.\n Two \u2013 Community is large and dynamic, with many contributors and active communication between the core developers and the rest of the community. Community is responsive to outside inquiries.\nSystem cost \nSince the main goal of this survey procedure is to evaluate FLOSS products, the base system cost for the software will normally be low, frequently $0.00. However, this is not the only cost you need to consider. Many of these costs could potentially be placed under multiple headings depending on how your organization is structured. No matter how it itemizes them, there will be additional costs. Typical items to consider include the following:\n\n Cost of supporting software \u2013 e.g. Does it require a commercial data base such as Oracle or some other specialized commercial software component?\n Cost of additional hardware \u2013 e.g. Does the system require the purchase of additional servers or storage systems? Custom hardware interfaces?\n Cost of training \u2013 e.g. How difficult or intuitive is the system to operate? This will impact the cost of training that users must receive. Keep in mind this cost will exist whether you are dealing with an Open Source or proprietary system. Are costs for system manuals and other required training material included? Some proprietary systems don't, or might perhaps send a single hard copy of the manuals. Who will perform the training? Whether you hire someone from outside or have some of your own people do it, there will be a cost, as you would be pulling people away from from their regular jobs.\n Cost of support \u2013 e.g. Is support through a commercial organization or the Open Source development group? If the former, what are their contract costs? While harder to evaluate, what is the turnaround time from when you request the support? Immediate? Days? Sometime? The amount of time you have to wait for a problem to be fixed, is definitely a \"cost,\" whether it means your system is dead in the water or just not as efficient and productive as it could be.\nThe primary issue here is to be realistic in your evaluation. It's hard to believe that anyone would assume that there were no associated costs with using FLOSS, or even proprietary software for that matter, but apparently there are. Foote does a good job of exploring and disproving this belief, showing all of the items that go into figuring the total cost of ownership (TCO), which should be representative of FLOSS applications.[60] I wouldn't call them hidden costs, at least not with the FLOSS systems, but rather costs that are overlooked, as are so many other things when people focus on a single central item. To quote Robert Heinlein, \"TANSTAAFL!\" (There ain't no such thing as a free lunch!)[61]\nThe important thing here is to pay attention to all of the interactions taking place. For example, if you wished to interface a piece of equipment to your FLOSS application, remember to factor in the cost of the interface. Despite what some advertisers think, data bits don't just disappear from one place and magically appear in another. It is very easy to lose track of where the costs are. Keep in mind that many items, such as training, you will have a cost either way you go. If a proprietary vendor says they will provide free training, you can be assured that the cost for it is included in the contract. But if you take that route, be very careful to read the contract thoroughly, as not all vendors include any training at all. It would be very easy to end up having to pay for \"optional\" training.\nSuggested ratings are:\n\n Zero \u2013 Installation and support costs are excessive and greatly exceed any available budget.\n One \u2013 System may require purchase of additional hardware or customization. These costs, along with training costs, are within potential budget.\n Two \u2013 Installation and support costs are relatively minor, with no additional hardware required. System design is relatively intuitive with in-depth documentation and active support from the community.\nPopularity \nThis heading can be somewhat confusing in terms of how it is interpreted even though most of the recommendations we looked at include it. Popularity is sometimes considered to be similar to market share. That is, of the number of people using a specific application in a given class of open-source applications, what percentage do they represent out of all people running applications in the class? If the majority of people are using a single system, this might indicate that it is the better system, or it might just indicate that other systems are newer and, even if potentially better, people haven't migrated over to them yet. An alternate approach to examining it is to ask how many times the application has been downloaded. In general, the larger the market share or the number of downloads, the more likely that a given product is to be usable. This is not an absolute, as people may have downloaded the application for testing and then rejected it or downloaded it simply to game the system, but it is a place to start. The point of this question usually isn't to determine how popular a particular application is, but rather to ensure that it is being used and it is a living (as opposed to an abandoned) project. Be leery of those applications with just a few downloads. If there is a large group of people using the application, there is a higher probability that the application works as claimed.\nAt the same time, learning who some of the other users of this application are can give you some insight of how well it actually works. As Silva reminds us, \"the best insight you can get into a product is from another user who has been using it for a while.\"[31]\nSuggested ratings are:\n\n Zero \u2013 No other discernible users or reality of listed users is questionable.\n One \u2013 Application is being used by multiple groups, but represents only a small fraction of its \"market.\" Appears to be little 'buzz' about the application.\n Two \u2013 Application appears to be widely used and represents a significant fraction of its \"market.\" This rating is enhanced if listed users include major commercial organizations.\nProduct support \nProduct support can be a critically important topic for any application. Whether you are selecting a proprietary application or a FLOSS one, it is vitally important to ensure that you will have reliable support available. Just because you purchased a proprietary program will not ensure that you have the support you need. Some vendors include at least limited support in their contracts, others don't. However, over the years I've found that even purchasing a separate support contract doesn't ensure that the people who answer the phone will be able to help you. When making the final decision, don't make assumptions: research!\nSupport can be broken down into several different sub-categories:\n\n User manuals \u2013 Do they exist? What is their quality?\n System managers manuals \u2013 Do they exist? What is their quality?\n Application developers manuals \u2013 Do they exist? What is their quality?\n System design manuals \u2013 Do they exist? What degree of detail do they provide?\n For database-related projects, is an accurate and detailed entity relationship diagram (ERD) included?\n Have any third-party books been written about this application? Are they readily available, readable, and easy to interpret?\n Is product support provided directly by the application development community?\n Is product support provided by an independent user group, separate from the development group?\n Is commercial product support available?\n If so, what are their rates?\n Are on-site classes available?\n Are online training classes available?\n A frequently overlooked type of product support is how well documented the program code is. Are embedded comments sparse or frequent and meaningful? Are the names of program variables and functions arbitrary or meaningful?\nAnother factor in evaluating product support is whether you have anyone on your team with the expertise to understand it. That is not a derogatory statement: depending on the issue, someone might have to modify an associated data base, the application code, or the code in a required library, whether to correct an error or add functionality. Do any of your people have expertise in that language? Would someone have to start from scratch or do you have the budget to hire an outside consultant? Even if you have no desire to modify the code, having someone on the project that understands the language used can be a big help in discerning how the program works, as well as determining how meaningful the program comments are.\nSuggested ratings are:\n\n Zero \u2013 Limited or no support available. Documentation essentially non-existent, source code minimally documented, no user group support, and erratic response from the developers.\n One \u2013 Documentation scattered and of poor quality. Support from user group discouraged and no commercial support options exist.\n Two - Excellent documentation, including user, system manager, and developer documentation. Enthusiastic support from the user community and developers. Commercial third-party support available for those desiring it. Third-party books may also have been released documenting the use of this product.\nMaintenance \nThis is another item that can be interpreted in multiple ways. One way to look at it is how quickly the developers respond to any bug report. Depending on the particular FLOSS project, you may actually be able to review the problem logs for the system and see what the average response time was between a bug being reported and the problem resolved. In some cases this might be hours or days, in others it is never resolved. To be fair, don't base your decision on a single instance in the log file, as some bugs are much easier to find and fix than others. However, a constant stream of open bugs or bugs that have only been closed after months or years should make you leery.\nTo others this question is to determine whether development is still taking place on the project or if it is dead. Alternately, it is like asking if anybody is maintaining the system and correcting bugs when they are discovered. There are several ways of addressing this issue. Examining the problem logs described above is one way of checking for project activity, while another is the check the release dates for different versions of the application. Are releases random or on a temporal schedule? How long has it been since the application was last updated? If the last release date was over a year or two ago, this is cause for concern and should trigger a closer look. Just because there hasn't been a recent release does not mean that the project has been abandoned. If the development of the app has advanced to the point where it is stable and no other changes need to be made you may not see a recent release because none is needed. However, the latter is very rare, both because bugs can be so insidious and because a lot of programmers can't resist just tweaking things, to make them a wee bit better. If a project is inactive, but everything else regarding the project looks good, it might be possible to work with the developers to revitalize it. While this course of action is feasible, it is important to realize that it is taking on a great deal of responsibility and an unknown amount of expense. The latter is particularly true as you may be having to assign one or more developers to work on the project full time.\nSuggested ratings are:\n\n Zero \u2013 No releases, change log activity, or active development discussion in message forums in over two years.\n One \u2013 No releases, change log activity or active development discussions in message forums for between one and two years.\n Two \u2013 A new version has been released within the year, change logs show recent development activity, and there is active development discussion in the message forums.\nReliability \nReliability is the degree to which you can rely on the application to function properly. Of course, the exact definition becomes somewhat more involved. The reliability of a system is defined as the ability of an application to operate properly under a specified set of conditions for a specified period of time. Fleming states that \"[o]ne aspect of this characteristic is fault tolerance that is the ability of a system to withstand component failure. For example if the network goes down for 20 seconds then comes back the system should be able to recover and continue functioning.\"[62]\nBecause of its nature, the reliability of a system is hard to measure, as you are basically waiting to see how frequently it goes down. While we'd like to aim for never, one should probably be satisfied if the system recovered properly after the failure. In most instances, unless you are actually testing a system under load, the best that you can hope for is to observe indicators from which you can infer its reliability. As a generality, the more mature a given code base is, the more reliable it is, but keep in mind that this is a generality; there are always incidents that can occur to destabilize every thing. Unfortunately, it frequently feels as if the problem turns out to be something that you would swear was totally unrelated. Face it, Murphy is just cleverer than you.[63]\nWheeler also reminds us that \"[p]roblem reports are not necessarily a sign of poor reliability - people often complain about highly reliable programs, because their high reliability often leads both customers and engineers to extremely high expectations.\"[36] One thing that can be very reassuring is to see that the community takes reliability seriously by continually testing the system during development.\nSuggested ratings are:\n\n Zero \u2013 Error or bug tracking logs show a high incident of serious system problems. Perhaps worse, no logs of reported problems are kept at all, particularly for systems that have been in release for less than a year.\n One \u2013 Error logs show relatively few repeating or serious problems, particularly if these entries correlate with entries in the change logs indicating that a particular problem has been corrected. System has been in release for over a year.\n Two - Error logs are maintained, but show relatively few bug reports, with the majority of them being minor. A version of the system, using the same code base, has been in release for over two years. Developers both distribute and run a test suite to confirm proper system operation.\nPerformance \nPerformance of an application is always a concern. Depending on what the application is trying to do and how the developers coded the functions, you may encounter a program that works perfectly but is just too unresponsive to use. Sometimes this is a matter of hardware, other times it is just inefficient coding, such as making sequential calls to a database to return part of a block of data rather than making a single call to return all of the block at once. Performance and scalability are usually closely linked.\nYou might be able to obtain some information on the system's actual performance from the project web site, but it is hard to tell if this is for representative or selected data. Reviewing the project mailing list may provide a more accurate indication of the system's performance or any performance problems encountered. Testing the system under your working conditions is the only way to make certain what the system's actual performance is. Unfortunately, the steps involved in setting up such a test system require much more effort than a high-level survey will allow. If any user reviews exist, they may give an insight into the system's performance. Locating other users through the project message board might be a very useful resource as well, particularly if they handle the same projected work loads that you are expecting.\nIt is difficult to define performance ratings without having knowledge of what the application is supposed to do. However, for systems that interact with a human operator, the time lapse between when a function is initiated and when the system responds can be suggestive. If the project maintains a test suite, particularly one containing sample data, reviewing its processing time can give an insight to the system's performance as well. Response delays of even a few seconds in frequently executed functions will not only kill the overall process performance but also result in users resistive to using the system. \nSuggested ratings are:\n\n Zero \u2013 A system designed to be interactive fails to respond in an acceptable time frame. For many types of applications it is reasonable to expect an almost instantaneous response, particularly for screen management functions. It is not reasonable for a system to take over a minute, or even five seconds to switch screens or acknowledge an input, particularly in regards to frequently executed functions such as results entry, modification, or review.. A system that batch processes data maxes out under data loads below that of your current system.\n One \u2013 A system designed to be interactive appears to lag behind human entry for peripheral functions, but frequently accessed functions, such as results entry, modification, or review appear to respond almost instantaneously. A system that batch processes data maxes out under your existing data loads.\n Two \u2013 System is highly responsive, showing no annoying delayed responses. A system that batch processes data can process several times your current data load before maxing out.\nScalability \nScalability ensures that the application will operate over the data scale range you will be working with. In general, it means that if you test the system functionality with a low data load, the application will \"scale up\" to handle larger data loads. This may be handled by expanding from a single processor to a larger cluster or parallel processing system. Note that for a given application, throwing more hardware at it may not resolve the problem, as the application needs to be designed to take advantage of that additional hardware. Another caveat is to carefully examine the flow of data through your system. The processor is not the only place you can encounter roadblocks limiting scalability. Other possibilities include how quickly the system can access the needed data. If the system is processing the data faster than it can access it, adding more computer power will not resolve the problem. The limiting issues might be the bandwidth of your communication lines, the access speed of the devices that the data is stored on, or contention for needed resources with other applications. As with many aspects of selecting a system and getting it up and running, making assumptions is the real killer.\nSilva indicates that many open-source applications are built on the LAMP (Linux, Apache, MySQL, PHP\/Perl\/Python) technology stack and that this is one of the most scalable configurations available.[31] However, you should ensure that there is evidence that the application has been successfully tested that way; the performance survey and test phases are never a good time to start making assumptions. A look at the application's user base will likely identify someone who can provide this feedback.\nSuggested ratings are:\n\n Zero \u2013 System does not support scaling, whether due to application design or restriction of critical resources, such as rate of data access.\n One \u2013 System supports limited scaling, but overhead or resource contention, such as a data bottleneck, results in a quick performance fall off.\n Two - System is balanced and scales well, supporting large processing clusters or cloud operations without any restrictive resource pinch-points.\nUsability \nUsability means pretty much what it says. The concern here is not how well the program works but rather how easy is it to learn and use. The interface should be clear, intuitive, and help guide the user through the programs operation. Despite Steve Jobs, there is a limit to how intuitive an interface can be, thus the operation of the interface should be clearly documented in the user manual. Ideally the system will support a good context-sensitive help system as well. The best help systems may also provide multimedia support so that the system can actually show you how something should be done, rather than trying to tell you. I've found that frequently a good video can be worth well more than a thousand words! No matter how much time is spent writing the text for a manual or help system, it will always be unclear to somebody, if only because of the diversity of the backgrounds of people using it.\nThe interface between the operator and the computer may vary with the purpose of the application. While with new applications you are more likely to encounter a graphic user interface (GUI), there are still instances where you may encounter a command-line interface. Both types of interfaces have their advantages, and there are many times when something is actually easier to do with a command-line interface. The important thing to remember is that it is your interface with the system. It should be easy to submit commands to the system and interpret its response without having to hunt through a lot of extraneous information. This is normally best done by keeping the interface as clean and uncluttered as possible. As Abran et al. have pointed out, the usability of a given interface varies with \"the nature of the user, the task and the environment.\"[64]\nIf you would prefer a somewhat drier set of definitions, Abran et al. also extracted the definitions of usability from a variety of ISO standards, and they are included in the following table:\n\n\n\n\n\n\n\n ISO Usability Definitions\n\n\n\"The capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions.\" (ISO\/IEC9126-1, 2000)\n\n\n\"The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.\" (ISO 9241-11, 1998)\n\n\n\"The ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component.\" (IEEE Std. 610.12-1990)\n\n\n\nTable 3.: ISO Usability Definitions[64]\nFleming translates this into a somewhat more colloquial statement: \u201cUsability only exists with regard to functionality and refers to the ease of use for a given function.\u201d For those interested in learning more about the usability debate, I suggest that you check out Andreasen et al.[65] and especially Saxena and Dubey.[66]\nIn addition to usability, it has been highly recommended to me that, if budget exists, it is also helpful to have a user experience (UX) specialist on the review team as well.[67][68]\nSuggested ratings are:\n\n Zero \u2013 System is difficult to use, frequently requiring switching between multiple screens or menus to perform a simple function. Operation of system discourages use and can actively antagonize users.\n One \u2013 System is useable but relatively unintuitive regarding how to perform a function. Both control and output displays tend to be cluttered, increasing the effort required to operate the system and interpret its output.\n Two \u2013 System is relatively intuitive and designed to help guide the user through its operation. Ideally, this is complemented with a context-sensitive help system to minimize any uncertainties in operation, particularly for any rarely used functions.\nSecurity \nThis heading overlaps with functionality and is usually difficult to assess from a high-level evaluation. While it is unlikely that you will observe any obvious security issues during a survey of this type, there are indicators that can provide a hint as to how much the applications designers and developers were concerned with security.\nThe simplest approach is to simply take a look at whether they've done anything that shows a concern for possible security vulnerabilities. The following are a few potential indicators that you can look for, but just being observant when seeing a demonstration can also tell you a lot.\n\n Is there any mention of security in the systems documentation? Does it describe any potential holes that you need to guard against or configuration changes you might make to your system environment to reduce any risks.\n Do the manuals describe any type of procedure for reporting bugs or observed system issues?\n If you have access to the developers, discuss any existing process for reporting and tracking security issues.\n Check their error logs and see if any security related issues are listed. If they are, what was the turnaround time to have them repaired, or were they repaired?\n If the developers are security-conscious, they will almost certainly want to prove to whomever received the program that it hadn't been modified by a third party. The basic way of doing this is by separately sending you what is known as an MD5 hash. This is a distinctive number generated by another program from your applications code. If you generate a new MD5 hash from the code you receive, these numbers should match. If they don't match, that means that something in the code has been altered. For developers with more concern, they might generate a cryptographic signature incorporating the code. This will tell you who sent the code as well as indicate whether the program was altered.\n Depending on the type of program, does it allow a 21 CFR Part 11-compliant implementation or conform to a similar standard?\n Depending on the type of application, does it include a detailed audit trail and security logs?\nSuggested ratings are:\n\n Zero \u2013 System shows no concern with security or operator tracking. Anyone can walk up to it and execute a function without having to log in. System doesn't support even a minimal audit trail. Any intermediate files are easily accessible and modifiable outside of the system.\n One - System shows some attempt at user control but supports only a minimal audit trail. It may support a user table, but it fails to follow best practices by allowing user records to be deleted. Audit trail is modifiable by power users.\n Two \u2013 Maintaining system security is emphasized in the user documentation. A detailed audit trail is maintained that logs all system changes and user activities. Application is distributed along with an MD5 hash or incorporated into an electronic signature by the developer.\nFlexibility\/Customizability \nThe goal of this topic is to identify how easily the functionality of this application can be altered or how capable it is of handling situations outside of its design parameters. Systems are generally designed to be either configurable or customizable, sometimes with a combination of both.\n\n Configurability - This refers to how much or how easily the functionality of the system can be altered by changing configuration settings. Configurable changes do not require any changes to the application code and generally simplify future application upgrades.\n Customizability - This refers to whether the functionality of the system must be altered by modifying the applications code. As we are targeting open-source systems, the initial assumption might be that they are all customizable; however, this can be affected by the type of license that the application is released under. More practically, how easily an application can be customized depends on how well it is designed and documented. While in theory you might be able to customize a system, if it is a mass of spaghetti code and poorly compartmentalized, it might be a nightmare to do. In any case, if you customize the system code, you may not be able to take advantage of any system upgrades without having to recreate the customizations in them.\n Extendability \u2013 While you won't find this term in most definitions, it is a hybrid system that is both configurable and customizable. It is normally configured using the same approaches as a standard configurable system. However, the ability to be upgraded remains by feeding any code customizations through an aApplication program interface (API). As long as this API is maintained between upgrades, any extension modules should continue to work.\nIn addition, a well designed application is usually modular, which makes program changes easier. In an ideal world, any application that you may have to customize will be specifically designed to make customization simple. There are a variety of ways of doing this. Perhaps the easiest, for an application that is designed to be modular, would be to support optional software plug-in modules that added extra functionality. Unless these were \"off-the-shelf\" modules, you would need to confirm that there was appropriate documentation regarding their design and use. This would most likely be done via an API, as discussed above. Depending on the system, you could transfer data through the API or have one system control another, the caveat again being that you need to have thorough documentation of the API and its capabilities.\nIn the majority of situations, I strongly encourage you to stick with a configurable system, assuming you can find one that meets your needs. Customizing a system is rarely justified unless you are working situationally. While almost everyone feels that their needs are unique, the reality is that a well-designed configurable system can generally meet your needs.\nSuggested ratings are:\n\n Zero \u2013 Application does not support configuration and shows evidence of being difficult to customize, usually indicated by use of spaghetti code rather than modular design, poorly named variables and functions, along with cryptic or no embedded comments. In a worst-case situations, the source code has been deliberately obfuscated to make the system even less customizable.\n One \u2013 Application supports minor configuration capabilities or is moderately difficult to modify. The latter might be due to minimal application documentation or poor programming practices, but not deliberate obfuscation.\n Two \u2013 Application is highly configurable and accompanied by detailed documentation guiding the user through its configuration. Code is clearly documented and commented. It also follows good programming practices with highly modularized functionality, simplifying customization of the programs source code, ideally via an API.\nInteroperability\/Integration \nDetermine whether this software will work with the rest of the systems that you plan to use. Exactly what to check for is up to you, as you are the only one who has any idea what you will be doing with it. The following is a list of possible items that might conceivably fall under this heading:\n\n Does it understand the data and control protocols to talk to and control external equipment such as a drill press, telescope, or sewing machine (as appropriate)? \n Is it designed to conform to both electronic and physical standards to avoid being locked into a single supplier?\n Does it handle localization to avoid conflicts with local systems?\nSuggested ratings are:\n\n Zero \u2013 System provides no support for integration with other applications. File formats and communication protocols used are not documented.\n One \u2013 System is not optimized for either interoperability or integration with other systems. However, it does use standard protocols so that other applications can interpret its activities. Application likely does not include an API or any existing API is undocumented.\n Two \u2013 Application is optimized for interoperability and integration with other systems. All interfaces and protocols, particularly for any existing API, are clearly documented and accompanied with sample code.\nLegal\/License issues \nThis section refers to the type of license that the application was released under and the associated legal and functional implications. With proprietary software, many people never bother to read the software license either because they don't care or think they have no choice but to accept them. Be that as it may, when selecting a FLOSS application, it is wise to take the time to read the accompanying license: it can make a big difference in what you can do with the software. First, if the software has no license, legally you have no right to even download the software, let alone run it.[69][70]\nWhile you have no control over which license the application was released under, you definitely control whether you wish to use it under the terms of the license. Which types of licenses are acceptable strongly depends on what you plan to do with the application. Do you intend to use the application as is or do you plan to modify it? If the latter, what do you plan to do with the modified code? Do you want to integrate this FLOSS application with another, either proprietary or open-source? Does this license clash with theirs? Your right to do any of these things is controlled by the license, so it must be considered very carefully, both in the light of what you want to do now and what you might want to do in the future.\nOne of the first things to do is to confirm that the application is even open-source; just being able to see the source code is insufficient. To qualify as open-source the license must comply with the 10 points listed in the Open Source Definition maintained by the Open Source Initiative (OSI).[2] Pulling just the headers, their web site lists these as the required criteria:\n\n Free redistribution\n Source code\n Derived works\n Integrity of the author's source code\n No discrimination against persons or groups\n No discrimination against fields of endeavor\n Distribution of licenses\n License must not be specific to a product\n License must not restrict other software\n License must be technology-neutral\nWhile somewhat cryptic to look at cold, each of the headings is associated with a longer definition, which primarily boils down to the freedom to use, modify, and redistribute the software. For those wanting to know the justification for each item, there is also an annotated version of this definition.[71] At present, OSI recognizes 71 distinct Open Source licenses, not counting the WTFPL.[72]\nOne of the functions of the OSI is to review prospective licenses to determine whether they meet these criteria and are indeed open-source. The OSI web site maintains a list of popular licenses, along with links to all approved licenses[73], sorted by name or category. These lists include full copies of the licenses. While clearer than most legal documents, they can still be somewhat confusing, particularly if you are trying to select one. If the definitions seem to blur, you might want to check out the Software Licenses Explained in Plain English web page maintained by TL;DRLegal.[74] As long as the license is classified as an open-source license and you aren't planning to modify it yourself or integrate it into other systems, you probably won't have any problems. However, if you have any uncertainty at all, it might be worth making the investment to discuss the license with an intellectual property lawyer who is familiar with OSS\/FS before you inadvertently commit your organization to terms that conflict with their plans.\nIf your plans include the possible creation of complementary software, I suggest a quick read of Wheeler's essay on selecting a license.[75] The potential problem here is that most open-source and proprietary applications contain multiple libraries or sub-applications, each with their own license. Depending on which licenses the original developers used, they may be compatible with other applications you wish to use, or they may be incompatible. The following figure illustrates some of these complications:\n\n\n\n\n\n\n\n\n\n Illustration 3.: License compatibility between common FOSS software licenses according to David A. Wheeler (2007): the vector arrows denote an one directional compatibility, therefore better compatibility on the left side than on the right side.[76]\n\n\n\nJust to be absolutely clear, the license of FLOSS and proprietary applications generally disavows any type of warranty that the program will work and disclaims any liability for any damage or injuries that result from the programs use. Now, having said that, you may be able to purchase a warranty separately; just don't anticipate any legal recourse in the event of a system failure.\nSuggested ratings are:\n\n Zero \u2013 Application does not include a license or license terms are unacceptable or incompatible with those of other applications being used.\n One \u2013 Application includes a license, but it contains potential conflicts with other licenses or allowable use that will need to be carefully reviewed.\n Two \u2013 License is fully open, allowing you to freely use the software.\nCompleting the evaluation \nCaveats \nOther than independent reviews, one of the best ways to obtain some of the above information is direct testing of a system. Usually that is impractical in a survey situation because of the time it would take to install and configure an instance of the application. However, a new factor has entered the picture which may change this. Docker is a new service that combines an application, along with all of its dependencies, into a single distributable package, which they call a container. Because everything required is in the container, it is guaranteed to run the same on any system. Docker specifically states that their containers \"are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.\"[77] Using only a fraction of the resources that a virtual machine (VM) would require, you can easily run multiple containers on a laptop and switch back and forth while testing.[78] \nWhile the Docker web site contains repositories for a number of different containers, there are multiple web sites that also host containers configured with a variety of applications. If you can locate one that contains the application you are surveying, this makes it a simple matter to try it out. Probably the easiest way to check is to run a web search containing the terms \"docker\", \"container\", and the name of the application that you are seeking.\n\nScreening tabulation \nWhile you can perform this application survey in many ways, to keep your defined scoring criteria in front of the evaluators and to simplify scoring the survey, it might be prudent to generate a document such as the following table with columns representing the criteria being evaluated, your rating definitions, and the numeric rating that your evaluators actually assign to the system. If you have decided to take the approach of using weighting factors instead of adjusting your definitions, you will also need to include columns for your weighting factors and a column containing the results of the evaluation after applying the weighting factor.\n\n\n\n\n\n\n\n Criteria\n\n Rating = 0\n\n Rating = 1\n\n Rating = 2\n\n Score\n\n\nSystem functionality\n\n                \n\n                \n\n                \n\n                \n\n\nCommunity\n\n                \n\n                \n\n                \n\n                \n\n\nSystem cost\n\n                \n\n                \n\n                \n\n                \n\n\nPopularity\n\n                \n\n                \n\n                \n\n                \n\n\nProduct support\n\n                \n\n                \n\n                \n\n                \n\n\nMaintenance\n\n                \n\n                \n\n                \n\n                \n\n\nReliability\n\n                \n\n                \n\n                \n\n                \n\n\nPerformance\n\n                \n\n                \n\n                \n\n                \n\n\nScalability\n\n                \n\n                \n\n                \n\n                \n\n\nUsability\n\n                \n\n                \n\n                \n\n                \n\n\nSecurity\n\n                \n\n                \n\n                \n\n                \n\n\nFlexibility\/Customizability\n\n                \n\n                \n\n                \n\n                \n\n\nInteroperability\/Integration\n\n                \n\n                \n\n                \n\n                \n\n\nLegal\/License issues\n\n                \n\n                \n\n                \n\n                \n\n\nSummary Score\n\n                \n\n\n\n\n\n\n\n\n\n Table 4.: Potential screening criteria to filter prospective FLOSS applications. Rows can be added or dropped as required by your needs. For example, if it is important that the application be coded in a specific programming language or combination of languages, you could add a row for this. The most likely anticipated change would be to subdivide the \"system functionality\" row so that specific information on the functionality of various system components can be displayed. To minimize the risk of confusion, you can transfer your selected scoring criteria for each rating into the cell corresponding to that rating and the item being evaluated, though ths would admittedly generate a long check list.\n\n\n\nDemonstration surveys \nTo demonstrate how this survey procedure is intended to be used, this section will apply the defined protocol to a block of open-source LIMS applications suitable for use in a chemical laboratory. Note that in this instance, we are using LIMS to refer to a laboratory information management system, not a labor information management system, a legislative information management system, or any other permutation that matches the LIMS acronym. The specific criteria used will quite likely be different from the ones used in your screening as the types of systems or the specific functionality required will be different.\nFor the purpose of this demonstration, we will not attempt to screen all of the open-source LIMS available. Instead, we will select a subset of the systems that have been announced and attempt to apply our protocol to them, hopefully screening out the systems not meeting our requirements so that the number of in-depth evaluations can be reduced. Part of the difficulty here is the broad range of fields that the term LIMS covers. There are general purpose LIMS, which can be configured to handle a wide range of samples, as well as targeted LIMS, which are designed to fill a particular niche. As such, you can find LIMS targeting specific areas as drinking water\/waste water analysis, mining, radioisotopes, proteomics, and genetic analysis. Whether you make it a formal part of the screening process or isolate them while collecting the applications to be surveyed, you will need to filter out the systems which will not handle the type of samples you are dealing with. This issue is particularly prominent with LIMS, but it will likely be encountered when screening other types of software as well. The following are the systems that we will include in this attempt:\n\n Bika LIMS\n eyeLIMS\n Open-LIMS\nFor simplicity in comparing results, I've reordered the screening table so that the first column contains the criteria being evaluated and the other columns correspond to the evaluation results for the LIMS being evaluated, with the bottom row reserved for the corresponding score summary. I have also added two subdivisions under \"system functionality\" for the programming language and the operating system used. Depending on the types of systems you are surveying, you will likely be including additional subdivisions. Scoring can be handled several ways. In this example, you might list the programming language in the appropriate cell. Depending on whether you have a team member competent in that language, you can use the results to set the value for the main criteria, e.g. system functionality. Alternately, while you will record the information for programming language in your survey notebook, you can insert an actual numerical result into the corresponding cell so that the sub-criteria can be evaluated separately or used to generate a value for the main criteria field.\nNormally, the best starting approach is to check for existing reviews of systems, but here we have a problem in finding any. While references to Bika LIMS were common, and it was referenced in a number of scientific papers, actual reviews of the product were hard to come by and were usually for older versions.[79] No reviews were found for eyeLIMS and for Open-LIMS; the closest thing I found to an impartial review has been two postings on Joel Limardo's LIMSExpert blog.[80][81]\n\n\n\n\n\n\n\n Criteria\n\n Bika LIMS 3.1.8\n\n eyeLIMS\n\n Open-LIMS*\n\n\nSystem functionality\n\n2\n\n0 (Project appears dead, no activity since 2008)\n\n1 (System functional for a very specific application only)\n\n\nProgramming language\n\nPython and PLONE CMS\n\neyeOS\n\nPHP\n\n\nOperating system\n\nplatform-independent\n\neyeOS\n\nAny OS supporting PostgreSQL and PHP\n\n\nCommunity\n\n2\n\n0\n\n1\n\n\nSystem cost\n\n\n\n0\n\n1\n\n\nPopularity\n\n2\n\n0\n\n0\n\n\nProduct support\n\n2\n\n0\n\n1\n\n\nMaintenance\n\n2\n\n0\n\n1\n\n\nReliability\n\n\n\n0\n\n\n\n\nPerformance\n\n\n\n0\n\n\n\n\nScalability\n\n\n\n0\n\n\n\n\nUsability\n\n2\n\n0\n\n1\n\n\nSecurity\n\n2\n\n0\n\n1\n\n\nFlexibility\/Customizability\n\n2\n\n0\n\n1\n\n\nInteroperability\/Integration\n\n2? (Supports import and export of CSV files)\n\n0\n\n0\n\n\nLegal\/License issues\n\n2 (AGPL 3.0 and GPL1)\n\n2 (GNU Affero GPL v3)\n\n2 (GNU GPL 3.0)\n\n\nSummary Score\n\n22\n\n2\n\n10\n\n\n\nBased on the above summary scores, we would definitely filter out both eyeLIMS and Open-LIMS, while Bika LIMS would justify a more in-depth evaluation.\n* Note: The rating for Open-LIMS may need to be revisited, as it appears that major development work on this system has been taken over by Joel Limardo of ForwardPhase Technologies, so several of the rated parameters may be subject to major shifts. It is currently unclear whether this is a joint project with the original developer or a fork.\n\nSummary \nIn support of a project to prepare published evaluations of various FLOSS applications, we have reviewed the FLOSS literature, focusing particularly on any assessment or evaluation documents. While many described proposed evaluation methods, none of them appear to be particularly popular or have developed an active community around them. Several review papers on the topic, while identifying multiple methods and their advantages, found flaws in all of them, particularly in terms of being able to perform a quality assessment on any FLOSS application. Many of the described systems were explicitly focused on a single class of FLOSS applications, such as library management systems.\nBy consolidating suggestions and procedures from a number of these papers, we synthesized a general survey process to allow us to quickly assess the status of any given type of FLOSS applications, allowing us to triage them and identify the most promising candidates for in-depth evaluation. Note that this process is designed for performing high-level surveys, it is not designed to perform the in-depth evaluations required for product selection.\nAs a minor aside, in the course of researching this article I was surprised by the high percentage of the published papers on FLOSS which were published in classic subscription journals, as opposed to any of the various open-source journals available.[82] Somehow it seems like a curious disconnect not to publish articles on open-source Software in open-source journals. Whether this is just habit of submission or due to more considered reasons would be interesting to know.\n\nGlossary \n\n\n\n\n\n\n Term\n\n Explanation\n\n\n.ogg\n\nFile extension for Ogg Vorbis, an open-source patent-free audio compression format\n\n\n21 CFR Part 11\n\nUnited States Food and Drug Administration electronic records and electronic signatures rule\n\n\nAcademia.edu\n\nA web site that allows academics to share research papers and exchange information\n\n\naccuracy\n\nDefines how well the results of an analysis or measurement conforms to the actual or \"correct\" value\n\n\nAEQ\n\nAnalytical equipment qualification\n\n\nagnostic\n\nA term applied to both hardware and software that indicates that the item is interoperable with different systems (The correct term should probably be \"technology-independent\" or \"technology-neutral,\" but the use of the term \"agnostic\" appears to be well entrenched.)\n\n\nAHP\n\nAnalytic hierarchy process (AHP), a structured technique developed by Thomas Saaty in the 1970s for organizing and analyzing complex decisions\n\n\nAIQ\n\nAnalytical instrument qualification, a term used in the pharmaceutical industry for the process of ensuring that an instrument meets the requirements for its intended application\n\n\nApache Hadoop YARN\n\nYet Another Resource Negotiator (YARN), a technology for managing resources on a cluster of computers\n\n\nApache Lucene\n\nA software library for information retrieval from fields of text contained within document files\n\n\nASE\n\nAdaptive Server Enterprise\n\n\naudit trail\n\nA log of records documenting the sequence of activities that have been performed on a system\n\n\nbinary package distribution\n\nA compilation of the compiled version of a program and all related documentation designed for release to the end user\n\n\nBREW\n\nBinary Runtime Environment for Wireless, a runtime and application development environment from Qualcomm that isolates portable applications from the hardware interface of mobile phones employing code division multiplex access (CDMA)\n\n\nBRR\n\nBusiness readiness rating\n\n\nCDMS\n\nChromatography data management system\n\n\nCFR\n\nUnited States Code of Federal Regulations\n\n\nchange log\n\nA log documenting the changes made to a software product, which may include a list of new features, changes to behavior, or elimination of software bugs\n\n\nCOC\n\nChain of custody, a paper or electronic documentation trail that documents responsibility of a sample (This is required for legal reasons under various regulatory programs, and it also provides information used to track faulty or contaminated items back to their source.)\n\n\ncommitter\n\nThe person that represents the quality control of the community, controlling what changes are included in the originally licensed version, though users are free to make any changes they want in their own copies of the program\n\n\nCompiere\n\nA suite of open-source applications (for small- to medium-sized businesses) that provides a number of business support applications\n\n\ncopyleft\n\nPer Wikipedia:\nCopyleft (a play on the word \"copyright\") is the practice of offering people the right to freely distribute copies and modified versions of a work with the stipulation that the same rights be preserved in derivative works down the line.\nCopyleft is a form of licensing, and can be used to maintain copyright conditions for works ranging from computer software, to documents, to art. In general, copyright law is used by an author to prohibit recipients from reproducing, adapting, or distributing copies of their work. In contrast, under copyleft, an author may give every person who receives a copy of the work permission to reproduce, adapt, or distribute it, with the accompanying requirement that any resulting copies or adaptations are also bound by the same licensing agreement.\nCopyleft licenses (for software) require that information necessary for reproducing and modifying the work must be made available to recipients of the binaries. The source code files will usually contain a copy of the license terms and acknowledge the author(s).\n\nCopyleft type licenses are a novel use of existing copyright law to ensure a work remains freely available. The GNU General Public License, originally written by Richard Stallman, was the first software copyleft license to see extensive use, and continues to dominate in that area. Creative Commons, a non-profit organization founded by Lawrence Lessig, provides a similar license provision condition called ShareAlike.\n\n\nCosmos\n\nC# Open Source Managed Operating System\n\n\nCOTS\n\nCommercial, off-the-shelf\n\n\nCROMERR\n\nCross-Media Electronic Reporting Rule (or typically 40 CFR Part 3), an Environmental Protection Agency (EPA) rule specifying how electronic reporting should be performed for the EPA's various regulatory programs\n\n\nCygwin\n\nA collection of tools that emulate a Linux environment, allowing Linux applications to be compiled for and executed in a MS Windows environment\n\n\ndata loading\n\nThe loading of data into static tables, including test definitions, sample container descriptions, location information, etc.\n\n\ndatabase-agnostic\n\nIndicates an application capable of running with database systems from any vendor\n\n\ndigital commons\n\nName given to a collaboratively developed online resource that is managed by a community of people\n\n\nDMOZ\n\nAlso known as the Open Directory Project (ODP), an attempt to create the largest human curated open-content directory of web links (Originates from one of its earlier domain names, directory.mozilla.org)\n\n\nDocker\n\nA system designed to package an application with all of its dependencies into a standardized software container using an alternate architectural approach than a virtual machine, while providing similar resource isolation in a smaller footprint\n\n\ndocumentation, administrator\n\nDocumentation for a system designed to be issued to a system administrator, providing information on configuring and operating the application\n\n\ndocumentation, developer\n\nDocumentation for a system designed to be issued to a system developer, providing detailed information on how the system operates and is structured as well as how to alter the system code and extract data from the system in an ad hoc manner\n\n\ndocumentation, user\n\nDocumentation for a system designed to be issued to a user, helping to guide the user through using the application\n\n\nDOI\n\nDigital object identifier, a character string identifier used to uniquely identify an electronic document (In many instances a scientific\/technical publication will have a DOI printed on it, allowing you to access or purchase the article on-line; standardized under ISO 26324.)\n\n\nDrizzle\n\nA MySQL 6.0-derived database optimized for cloud computing\n\n\nDrupal\n\nAn open-source application designed for creating and managing a variety of web sites\n\n\nEclipse\n\nA Java-based integrated development environment that can be customized via use of community-developed plug-ins\n\n\nEDD\n\nElectronic data delivery or electronic data deliverable\n\n\nEHR\n\nElectronic health record\n\n\nElasticsearch\n\nAn open-source search engine employing a RESTful (see \"REST\") interface, built on Apache Lucene\n\n\nELN\n\nElectronic laboratory notebook\n\n\nEMR\n\nElectronic medical record\n\n\nEPA\n\nUnited States Environmental Protection Agency\n\n\nERP\n\nEnterprise resource planning, typically constituting a variety of integrated applications, with a shared database, that integrates critical business functions such as accounting, human resources, customer relationship management, inventory and order tracking, etc. into a single system\n\n\nF\/OSS\n\nFree\/open-source software (See \"FOSS\")\n\n\nFAME\n\nFilter, Analyze, Measure, and Evaluate (FAME) methodology for evaluating open-source applications\n\n\nFDA\n\nUnited States Food and Drug Administration\n\n\nFedora\n\nA version of the Linux operating system sponsored by Red Hat\n\n\nFirefox\n\nOne of a number of open-source web browsers that can be highly customized via the use of member-developed \"plug-ins\"\n\n\nFLOSS\n\nPer Wikipedia:\nThe acronym FLOSS was coined in 2001 by Rishab Aiyer Ghosh for \"free\/libre and open-source software.\" Later that year, the European Commission (EC) used the phrase when they funded a study on the topic.\nUnlike \"libre software,\" which aimed to solve the ambiguity problem, FLOSS aimed to avoid taking sides in the debate over whether it was better to say \"free software\" or to say \"open-source software\"\n\n\nFLOSShub\n\nPer FLOSS Research Group:\nFLOSShub is a portal for free\/libre and open source software (FLOSS) research resources and discussion. FLOSShub's goal is to provide a central location for connecting researchers and FLOSS community members to research papers, data, tools, and most importantly, community.\n\n\nForge\n\nPer Wikipedia:\nIn FLOSS development communities, a forge is a web-based collaborative software platform for both developing and sharing computer applications. (The word derives from the metalworking forge, used for shaping metal parts.) A forge platform is generally able to host multiple independent projects.\nFor software developers it is a place to host, among others, source code (often version-controlled), bug database and documentation for their projects. For users, a forge is a repository of computer applications.\nSoftware forges have become popular, and have proven successful as a software development model for a large number of software projects.\n\nThe term forge refers to a common prefix or suffix adopted by various platforms created after the example of SourceForge (such as GForge and FusionForge).\n\n\nforking\n\nThe process where a group takes an open-source application's source code and starts developing it in an independent direction from the original program\n\n\nFOSS\n\nPer Wikipedia:\nFree and open-source software (FOSS) is computer software that can be classified as both free software and open-source software. That is, anyone is freely licensed to use, copy, study, and change the software in any way, and the source code is openly shared so that people are encouraged to voluntarily improve the design of the software. This is in contrast to proprietary software, where the software is under restrictive copyright and the source code is usually hidden from the users.\n\n\nfree software\n\nSee \"software, free\"\n\n\nFree Software Foundation\n\nPer Wikipedia:\nThe Free Software Foundation (FSF) is a 501(c)(3) non-profit organization founded by Richard Stallman on 4 October 1985 to support the free software movement, which promotes the universal freedom to study, distribute, create, and modify computer software, with the organization's preference for software being distributed under copyleft (\"share alike\") terms, such as with its own GNU General Public License. The FSF was incorporated in Massachusetts, USA, where it is also based. \nFrom its founding until the mid-1990s, FSF's funds were mostly used to employ software developers to write free software for the GNU Project. Since the mid-1990s, the FSF's employees and volunteers have mostly worked on legal and structural issues for the free software movement and the free software community. \n\nConsistent with its goals, only free software is used on the FSF's computers.\n\n\nfunctional requirement\n\nPer Wikipedia:\nIn software engineering (and systems engineering), a functional requirement defines a function of a system and its components. A function is described as a set of inputs, the behavior, and outputs (see also software).\nFunctional requirements may be calculations, technical details, data manipulation and processing and other specific functionality that define what a system is supposed to accomplish. Behavioral requirements describing all the cases where the system uses the functional requirements are captured in use cases. Functional requirements are supported by non-functional requirements (also known as quality requirements), which impose constraints on the design or implementation (such as performance requirements, security, or reliability). Generally, functional requirements are expressed in the form \"system must do <requirement>\", while non-functional requirements are \"system shall be <requirement>\". The plan for implementing functional requirements is detailed in the system design. The plan for implementing non-functional requirements is detailed in the system architecture.\n\n\nGALP\n\nGood Automated Laboratory Practices\n\n\nGCP\n\nGood Clinical Practice\n\n\nGCP\n\nGood Laboratory Practice\n\n\nGMP\n\nGood Manufacturing Practice\n\n\nGNU\n\nGNU\u2019s Not Unix, a project to create a FLOSS operating system\n\n\nGNU General Public License\n\nPer Wikipedia:\nThe GNU General Public License (GNU GPL or GPL) is the most widely used free software license, which guarantees end users (individuals, organizations, companies) the freedoms to run, study, share (copy), and modify the software. Software that allows these rights is called free software and, if the software is copylefted, requires those rights to be retained. The GPL demands both. The license was originally written by Richard Stallman of the Free Software Foundation (FSF) for the GNU project.\nIn other words, the GPL grants the recipients of a computer program the rights of the Free Software Definition and uses copyleft to ensure the freedoms are preserved whenever the work is distributed, even when the work is changed or added to. The GPL is a copyleft license, which means that derived works can only be distributed under the same license terms. This is in distinction to permissive free software licenses, of which the BSD licenses and the MIT License are the standard examples. GPL was the first copyleft license for general use.\n\n\nGNU Project\n\nPer Wikipedia:\nThe GNU Project is a free software, mass collaboration project, announced on 27 September 1983, by Richard Stallman at MIT. Its aim is to give computer users freedom and control in their use of their computers and computing devices, by collaboratively developing and providing software that is based on the following freedom rights: users are free to run the software, share it (copy, distribute), study it and modify it. GNU software guarantees these freedom-rights legally (via its license), and is therefore free software; the use of the word \"free\" always being taken to refer to freedom.\r\n \r\n\nIn order to ensure that the entire software of a computer grants its users all freedom rights (use, share, study, modify), even the most fundamental and important part, the operating system (including all its numerous utility programs), needed to be written. The founding goal of the project was, in the words of its initial announcement, to develop \"a sufficient body of free software [...] to get along without any software that is not free.\" Stallman decided to call this operating system GNU (a recursive acronym meaning \"GNU's not Unix\"), basing its design on that of Unix; however, in contrast to Unix which was proprietary software, GNU was to be freedom-respecting software (free software) that users can use, share, study and modify. Development was initiated in January 1984. The goal of making a completely free software operating system was achieved in 1992 when the third-party Linux kernel was released as free software, under version 2 of the GNU General Public License, to be used with the GNU software stack.\n\n\nGoogle Scholar\n\nA version of Google search optimized for searching and retrieving scholarly papers\n\n\nGPL\n\nSee \"GNU General Public License\"\n\n\nHDL\n\nhardware description language\n\n\nHL7\n\nA non-profit organization working to define standard methods for the exchange and retrieval of medical information\n\n\nIEC\n\nInternational Electrotechnical Commission\n\n\nillegal operation\n\nTerm for an operating system command that is unknown to the operating system or processor\n\n\nISO\n\nInternational Organization for Standardization, an independent, non-governmental membership organization and the world's largest developer of voluntary International Standards\n\n\nJabber\n\nAn XML based instant messaging platform\n\n\nLIMS\n\nLaboratory information management system, an informatics system designed to track samples and analytical results through a laboratory\n\n\nLIS\n\nLaboratory information system, an informatics system designed to track samples and test results through a clinical laboratory\n\n\nLMS\n\nLibrary management system\n\n\nMicrosoft Academic Search\n\nAn experimental Microsoft search project focusing on scientific and technical information\n\n\nmodular\n\nA design process where an application is broken down into smaller parts, frequently grouped by function\n\n\nMongoDB\n\nAn open-source NoSQL document-oriented database\n\n\nnative code\n\nA program that is designed to run with a specific computer's hardware program codes, on a different computer\/processor via the use of an emulator\n\n\nnon-functional requirements\n\nRequirements describing how the system works and how it should behave\n\n\nOALib\n\nOpen Access Library, an open-access search engine and journal\n\n\nODBC\n\nOpen Database Connectivity, a middleware API that allows an application to connect with any database\n\n\nODF\n\nOpen Document Format, an XML based open-source document format developed for office suite use, prominent in the OpenOffice and LibreOffice projects\n\n\nODP\n\nOpen Directory Project; see \"DMOZ\"\n\n\nOSI\n\nOpen Source Initiative, a global non-profit organization promoting open source and preventing abuse of the open source concept; also the steward of the Open Source Definition (OSD)\n\n\nOpenDOAR\n\nDirectory of Open Access Repositories\n\n\nOpenLogic Exchange\n\nSoftware-as-a-Service (SaaS) governance platform for comprehensive governance and provisioning of open-source software\n\n\nOpenSSL\n\nA general-purpose encryption library used in many web servers that provides an implementation of the Internet's Secure Sockets Layer (SSL) and Transport Layer Security (TLS) protocols\n\n\nOpenVPN\n\nOpen-source virtual private network (VPN) software; see \"VPN\"\n\n\nOPML\n\nOutline Processor Markup Language, an XML-based file format used for creating outlines\n\n\nOS\n\nOperating system\n\n\nOSDL\n\nOpen Source Development Labs, conceived as a facility to allow open source developers to collaborate to create standardized implementations of Linux and Linux applications by IBM, Intel, and Computer Associates\n\n\nOSS\n\nPer Wikipedia:\nOpen-source software (OSS) is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change, and distribute the software to anyone and for any purpose. Open-source software may be developed in a collaborative public manner. Open-source software is the most prominent example of open-source development and often compared to (technically defined) user-generated content or (legally defined) open-content movements.\n\n\nOSS Watch\n\nAn independent non-advocacy service on free and open-source software\n\n\nPDA\n\nParenteral Drug Association\n\n\nPDF\n\nAdobe Portable Document Format\n\n\nPerl\n\nAn interpreted script programming language invented by Larry Wall that is popular for creating common gateway interface (CGI) programs\n\n\nprecision\n\nHow closely a set of measurements agree with their average\n\n\nproprietary software\n\nSee \"software, proprietary\"\n\n\nPROSE\n\nPromoting Open Source in Europe\n\n\nQA\n\nQuality assurance, responsible for auditing the QC process and ensuring that all SOPs and standards are being followed\n\n\nQC\n\nQuality control, responsible for ensuring all parts of a process are within designed specifications (designated testing and inspection)\n\n\nQMS\n\nQuality management system\n\n\nQSOS\n\nQualification and Selection Open Source, a methodology for evaluating open-source applications backed by Atos Origin\n\n\nREST\n\nRepresentational state transfer (occasionally ReST), an architectural style commonly used for APIs that is defined by six constraints: uniform interface, stateless, cacheable, client-server, layered system, and code on demand\n\n\nResearchGate\n\nA professional network allowing scientists and researchers to exchange papers and collaborate\n\n\nrevenue trigger\n\nA method of bringing in money to support program development, appearing as ad sales to training, consulting contracts, and\/or support subscriptions\n\n\nRFI\n\nRequest for information, a document sent by potential customers to vendors requesting detailed information regarding their system, usually in relation to selecting a hardware or software system to acquire\n\n\nRFID\n\nRadio frequency identification device\n\n\nRFP\n\nRequest for proposal, a document sent to vendors specifying application requirements and requesting a proposal of what they can supply\n\n\nRuby\n\nAn open-source object-oriented scripting\/programming language created by Yukihiro Matsumoto\n\n\nSamba\n\nAn implementation of Server Message Block (SMB) and Common Internet File System (CIFS) client\/server protocols that allows shared access of resources over a network\n\n\nshareware\n\nPer Wikipedia:\nShareware is a type of proprietary software which is provided (initially) free of charge to users, who are allowed and encouraged to make and share copies of the program, which helps to distribute it. The word \"shareware\" is a portmanteau combining the words \"share\" and \"software\". Shareware is often offered as a download from an Internet website or as a compact disc included with a magazine.\nThere are many types of shareware, and while they may not require an initial up-front payment, all are intended to generate revenue in one way or another. Some limit use to personal non-commercial purposes only, with purchase of a license required for use in a business enterprise. The software itself may be limited in functionality or be time-limited. Or it may remind you that payment would be appreciated.\nShareware is available on all major personal computer platforms. Titles cover a very wide range of categories including: business, software development, education, home, multimedia, design, drivers, games, and utilities. Because of its minimal overhead and low cost, the shareware model is often the only one practical for distributing non-free software for abandoned or orphaned platforms such as the Atari ST and Amiga.\n\nThe term shareware is used in contrast to open-source software, in which the source code is available for anyone to inspect and alter, and freeware, which is software distributed at no cost to the user but without source code being made available. Note that two types of shareware, donationware and freemiums, are also types of freeware.\n\n\nSmalltalk\n\nAn experimental language developed at Xerox in the 1970s to investigate the concept of object-oriented programming\n\n\nSME\n\nSubject matter expert, a person who is an authority on the particular subject area or topic that a software application addresses\n\n\nsoftware, free\n\nPer Wikipedia:\nFree software, software libre, or libre software is computer software that gives users the freedom to run the software for any purpose as well as to study, modify, and distribute the original software and the adapted versions. The rights to study and modify free software imply unfettered access to its source code. For computer programs which are covered by copyright law this is achieved with a software license where the author grants users the aforementioned freedoms. Software which is not covered by copyright law, such as software in the public domain can also be free if the source code is in the public domain (or otherwise available without restrictions). Other legal and technical aspects such as software patents and DRM may impede users from exercising these rights, and thus prevent software from being free. Free software may be developed collaboratively by volunteer computer programmers or by corporations; as part of a commercial activity or not.\r\n \r\n\nFree software is primarily a matter of liberty, not price: users, individually or collectively, are free to do whatever they want with it \u2013 this includes the freedom to redistribute the software free of charge, or to sell it (or related services such as support or warranty) for profit. Free software thus differs from proprietary software (such as Microsoft Windows), which to varying degrees prevents users from studying, modifying and sharing the software. Free software is also distinct from freeware, which is simply a category of proprietary software which does not require payment for use. Proprietary software (including freeware) uses restrictive software licences or user agreements and usually does not provide access to the source code. Users are thus prevented from modifying the software, and this results in the user becoming dependent on software companies to provide updates and support (vendor lock-in). Users can also not necessarily reverse engineer, modify, or redistribute proprietary software.\n\n\nsoftware, open-source\n\nPer Wikipedia:\nOpen-source software (OSS) is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change, and distribute the software to anyone and for any purpose. Open-source software may be developed in a collaborative public manner. Open-source software is the most prominent example of open-source development and often compared to (technically defined) user-generated content or (legally defined) open-content movements.\n\n\nsoftware, proprietary\n\nPer Wikipedia:\nProprietary software, non-free software (in the sense of missing freedoms), or closed-source software is software, where the developers or distributors reserve all freedoms and rights.\nAmong the freedoms and rights that proprietary software deprives (to end-users), are:\n\n the freedom to analyze the software, and to change it (often deprived through intentional non-availability of sourcecode, or through Non-disclosure agreements (NDA))\n the freedom to share the software (often deprived through copy prohibition via EULA (End User License Agreement) or NDA)\n the freedom to run the software for any purpose (often deprived through user-restrictions via EULA).\nIn contrast to proprietary software, free software, is software that grants a user all these freedoms, on reception of the software.\nProprietary software is licensed under legal right of the copyright holder, with the intent that the licensee is given the right to use the software only under certain conditions, and restricted from other uses, such as modification, sharing, studying, redistribution, or reverse engineering. Usually the source code of proprietary software is not made available.\nComplementary terms include free software, licensed by the owner under more permissive terms, and public domain software, which is not subject to copyright and can be used for any purpose. Proponents of free and open-source software use proprietary or non-free to describe software that is not free or open-source.\n\nA related, but distinct categorization in the software industry is commercial software, which refers to software produced for sale, but without meaning it is closed-source.\n\n\nSOP\n\nStandard operating procedure\n\n\nsource code\n\nA version of the program in human-readable form\n\n\nsource package distribution\n\nA consolidation of all of the source code and documentation relevant to a specific software release intended for the application developer and\/or maintainer\n\n\nSourceForge\n\nA software repository on the web providing support for open source developers and for distribution of their application\n\n\nspaghetti code\n\nA contemptuous phrase applied to unstructured applications, particularly those with GOTO statements or other control structures bouncing control around different portions of the program\n\n\nSQA\n\nSoftware quality assurance, responsible for auditing the SQC process and ensuring that all SOPs and standards are being followed\n\n\nSQC\n\nSoftware quality control, responsible for ensuring all parts of a process were within designed specifications (designated testing and inspection)\n\n\nSQL\n\nStructured Query Language\n\n\nSubversion\n\nAn open-source program designed by Karl Fogel and Ben Collins-Sussman to serve as a version control system that tracks changes made to files and folders\n\n\nsystem suitability testing\n\nApplied to analytical procedures, the idea that everything involved in the analysis (analytical equipment, data capture electronics, analytical procedures, and analytical samples) constitutes a system and can be evaluated as such\n\n\ntcpdump\n\nA command line tool for monitoring network traffic, also useful in troubleshooting network problems\n\n\nTsunami UDP\n\nA file transfer protocol enabling high-speed data transfers over networks with large end-to-end delays, often used for bulk data transfers\n\n\nURI\n\nPer Wikipedia:\nIn computing, a uniform resource identifier (URI) is a string of characters used to identify the name of a resource. Such identification enables interaction with representations of the resource over a network, typically the World Wide Web, using specific protocols. Schemes specifying a concrete syntax and associated protocols define each URI. The most common form of URI is the uniform resource locator (URL), frequently referred to informally as a web address. More rarely seen in usage is the uniform resource name (URN), which was designed to complement URLs by providing a mechanism for the identification of resources in particular namespaces.\n\n\nURL\n\nUniversal resource locator, a specific type of uniform resource identifier that identifies objects on the World Wide Web, including their address\n\n\nURS\n\nUser requirements specification, a document that lists all of the requirements that the users require the system to support, frequently considered the key document in regards to the development life cycle of the application\n\n\nuser experience\n\nThe overall aspects that an end-user experiences when interacting with the application, distinct from the functionality of the user interface\n\n\nUSP\n\nPer the USP:\nThe U.S. Pharmacopeial Convention (USP) is a scientific nonprofit organization that sets standards for the identity, strength, quality, and purity of medicines, food ingredients, and dietary supplements manufactured, distributed and consumed worldwide. USP\u2019s drug standards are enforceable in the United States by the Food and Drug Administration, and these standards are used in more than 140 countries.\n\n\nvalidation\n\nA process of confirming that the application works as designed\n\n\nvendor-neutral\n\nA business approach designed to avoid lock-in with a particular supplier and ensure broad compatibility and interchangeability of products and technologies\n\n\nversioning\n\nThe assignment of a unique name or number to a given compilation of software, frequently in terms of a major and minor version number\n\n\nVM\n\nVirtual machine, a program running on another computer that emulates a full computer system (Originally developed to allow more efficient use of idle time on existing hardware servers)\n\n\nVPN\n\nPer Wikipedia:\nA virtual private network (VPN) extends a private network across a public network, such as the Internet. It enables users to send and receive data across shared or public networks as if their computing devices were directly connected to the private network, and thus are benefiting from the functionality, security and management policies of the private network. A VPN is created by establishing a virtual point-to-point connection through the use of dedicated connections, virtual tunneling protocols, or traffic encryption.\n\n\nWAN\n\nWide area network, a computer network connecting multiple locations, as opposed to a local area network (LAN) that services one location\n\n\nwiki\n\nA server program designed to promote collaborating development and used in a diverse variety of ways, including system documentation\n\n\nWLAN\n\nWireless local area network\n\n\nXBRL\n\nExtensible Business Reporting Language\n\n\nXML\n\nExtensible Markup Language\n\n\n\nReferences \n\n\n\u2191 \"What is free software?\". GNU Project. Free Software Foundation, Inc. 2015. http:\/\/www.gnu.org\/philosophy\/free-sw.html . Retrieved 17 June 2015 .   \n\n\u2191 2.0 2.1 \"The Open Source Definition\". Open Source Initiative. 2015. http:\/\/opensource.org\/osd . Retrieved 17 June 2015 .   \n\n\u2191 Schie\u00dfle, Bj\u00f6rn (12 August 2012). \"Free Software, Open Source, FOSS, FLOSS - same same but different\". Free Software Foundation Europe. https:\/\/fsfe.org\/freesoftware\/basics\/comparison.en.html . Retrieved 5 June 2015 .   \n\n\u2191 \"RepOSS: A Flexible OSS Assessment Repository\" (PDF). Northeast Asia OSS Promotion Forum WG3. 05 November 2012. http:\/\/events.linuxfoundation.org\/images\/stories\/pdf\/lceu2012_date.pdf . Retrieved 05 May 2015 .   \n\n\u2191 Doll, Brian (23 December 2013). \"10 Million Repositories\". GitHub, Inc. https:\/\/github.com\/blog\/1724-10-millionrepositories . Retrieved 08 August 2015 .   \n\n\u2191 Sarrab, Mohamed; Elsabir, Mahmoud; Elgamel, Laila (March 2013). \"The Technical, Non-technical Issues and the Challenges of Migration to Free and Open Source Software\" (PDF). IJCSI International Journal of Computer Science Issues 10 (2.3). http:\/\/ijcsi.org\/papers\/IJCSI-10-2-3-464-469.pdf .   \n\n\u2191 Wheeler, David A. (14 June 2011). \"Free-Libre \/ Open Source Software (FLOSS) is Commercial Software\". dwheeler.com. http:\/\/www.dwheeler.com\/essays\/commercial-floss.html . Retrieved 28 May 2015 .   \n\n\u2191 8.0 8.1 Stol, Klaas-Jan; Ali Babar, Muhammad (2010). \"A Comparison Framework for Open Source Software Evaluation Methods\". In \u00c5gerfalk, P.J.; Boldyreff, C.; Gonz\u00e1lez-Barahona, J.M.; Madey, G.R.; Noll, J. Open Source Software: New Horizons. Springer. pp. 389\u2013394. doi:10.1007\/978-3-642-13244-5_36. ISBN 9783642132445.   \n\n\u2191 Fritz, Catherine A.; Carter, Bradley D. (23 August 1994). A Classification And Summary Of Software Evaluation And Selection Methodologies. Mississippi State, MS: Department of Computer Science, Mississippi State University. http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.55.4470 .   \n\n\u2191 10.0 10.1 \"OpenBRR, Business Readiness Rating for Open Source: A Proposed Open Standard to Facilitate Assessment and Adoption of Open Source Software\" (PDF). OpenBRR. 2005. http:\/\/docencia.etsit.urjc.es\/moodle\/file.php\/125\/OpenBRR_Whitepaper.pdf . Retrieved 13 April 2015 .   \n\n\u2191 Wasserman, A.I.; Pal, M.; Chan, C. (10 June 2006). \"The Business Readiness Rating: a Framework for Evaluating Open Source\" (PDF). Proceedings of the Workshop on Evaluation Frameworks for Open Source Software (EFOSS) at the Second International Conference on Open Source Systems. Lake Como, Italy. pp. 1\u20135. Archived from the original on 11 January 2007. http:\/\/web.archive.org\/web\/20070111113722\/http:\/\/www.openbrr.org\/comoworkshop\/papers\/WassermanPalChan_EFOSS06.pdf . Retrieved 15 April 2015 .   \n\n\u2191 Majchrowski, Annick; Deprez, Jean-Christophe (2008). \"An Operational Approach for Selecting Open Source Components in a Software Development Project\". In O'Connor, R.; Baddoo, N.; Smolander, K.; Messnarz, R.. Software Process Improvement. Springer. pp. 176\u2013188. doi:10.1007\/978-3-540-85936-9_16. ISBN 9783540859369.   \n\n\u2191 Petrinja, E.; Nambakam, R.; Sillitti, A. (2009). \"Introducing the Open Source Maturity Model\". ICSE Workshop on Emerging Trends in Free\/Libre\/Open Source Software Research and Development, 2009. IEEE. pp. 37\u201341. doi:10.1109\/FLOSS.2009.5071358. ISBN 9781424437207.   \n\n\u2191 14.0 14.1 Deprez,Jean-Christophe; Alexandre, Simon (2008). \"Comparing Assessment Methodologies for Free\/Open Source Software: OpenBRR and QSOS\". In Jedlitschka, Andreas; Salo, Outi. Product-Focused Software Process Improvement. Springer. pp. 189-203. doi:10.1007\/978-3-540-69566-0_17. ISBN 9783540695660.   \n\n\u2191 Wasserman, Anthony I.; Pal, Murugan (2010). \"Evaluating Open Source Software\" (PDF). Carnegie Mellon University - Silicon Valley. Archived from the original on 18 February 2015. https:\/\/web.archive.org\/web\/20150218173146\/http:\/\/oss.sv.cmu.edu\/readings\/EvaluatingOSS_Wasserman.pdf . Retrieved 31 May 2015 .   \n\n\u2191 Jadhav, Anil S.; Sonar, Rajendra M. (March 2009). \"Evaluating and selecting software packages: A review\". Information and Software Technology 51 (3): 555\u2013563. doi:10.1016\/j.infsof.2008.09.003.   \n\n\u2191 Pani, F.E.; Sanna, D. (11 June 2010). \"FAME, A Methodology for Assessing Software Maturity\". Atti della IV Conferenza Italiana sul Software Libero. Cagliari, Italy.   \n\n\u2191 Pani, F.E.; Concas, G.; Sanna, D.; Carrogu, L. (2010). \"The FAME Approach: An Assessing Methodology\". In Niola, V.; Quartieri, J.; Neri, F.; Caballero, A.A.; Rivas-Echeverria, F.; Mastorakis, N. (PDF). Proceedings of the 9th WSEAS International Conference on Telecommunications and Informatics. Stevens Point, WI: WSEAS. ISBN 9789549260021. http:\/\/www.wseas.us\/e-library\/conferences\/2010\/Catania\/TELE-INFO\/TELE-INFO-10.pdf .   \n\n\u2191 19.0 19.1 Pani, F.E.; Concas, G.; Sanna, S.; Carrogu, L. (August 2010). \"The FAMEtool: an automated supporting tool for assessing methodology\" (PDF). WSEAS Transactions on Information Science and Applications 7 (8): 1078\u20131089. http:\/\/www.wseas.us\/e-library\/transactions\/information\/2010\/88-137.pdf .   \n\n\u2191 20.0 20.1 20.2 Pani, F.E.; Sanna, D.; Marchesi, M.; Concas, G. (2010). \"Transferring FAME, a Methodology for Assessing Open Source Solutions, from University to SMEs\". In D'Atri, A.; De Marco, M.; Braccini, A.M.; Cabiddu, F.. Management of the Interconnected World. Springer. pp. 495\u2013502. doi:10.1007\/978-3-7908-2404-9_57. ISBN 9783790824049.   \n\n\u2191 Soto, M.; Ciolkowski, M. (2009). \"The QualOSS open source assessment model measuring the performance of open source communities\". 3rd International Symposium on Empirical Software Engineering and Measurement, 2009. IEEE. pp. 498-501. doi:10.1109\/ESEM.2009.5314237. ISBN 9781424448425.   \n\n\u2191 Soto, M.; Ciolkowski, M. (2009). \"The QualOSS Process Evaluation: Initial Experiences with Assessing Open Source Processes\". In O'Connor, R.; Baddoo, N.; Cuadrado-Gallego, J.J.; Rejas Muslera, R.; Smolander, K.; Messnarz, R.. Software Process Improvement. Springer. pp. 105\u2013116. doi:10.1007\/978-3-642-04133-4_9. ISBN 9783642041334.   \n\n\u2191 Haaland, Kirsten; Groven, Arne-Kristian; Glott, Ruediger; Tannenberg, Anna (01 July 2010). \"Free\/Libre Open Source Quality Models - a comparison between two approaches\" (PDF). 4th FLOSS International Workshop on Free\/Libre Open Source Software. Jena, Germany. pp. 1\u201317. http:\/\/publications.nr.no\/directdownload\/publications.nr.no\/5444\/Haaland_-_Free_Libre_Open_Source_Quality_Models-_a_compariso.pdf . Retrieved 15 April 2015 .   \n\n\u2191 Hauge, O.; Osterlie, T.; Sorensen, C.-F.; Gerea, M. (2009). \"An empirical study on selection of Open Source Software - Preliminary results\". ICSE Workshop on Emerging Trends in Free\/Libre\/Open Source Software Research and Development, 2009. IEEE. pp. 42-47. doi:10.1109\/FLOSS.2009.5071359. ISBN 9781424437207.   \n\n\u2191 Ayala, Claudia; Cruzes, Daniela S.; Franch, Xavier; Conradi, Reidar (2011). \"Towards Improving OSS Products Selection \u2013 Matching Selectors and OSS Communities Perspectives\". In Hissam, S.; Russo, B.; de Mendon\u00e7a Neto, M.G.; Kon, F.. Open Source Systems: Grounding Research. Springer. pp. 244\u2013258. doi:10.1007\/978-3-642-24418-6_17. ISBN 9783642244186.   \n\n\u2191 Chan, C.; enugroho; Wasserman, T. (17 April 2013). \"Business Readiness Rating (BRR)\". SourceForge. https:\/\/sourceforge.net\/projects\/openbrr\/ . Retrieved 21 April 2015 .   \n\n\u2191 27.0 27.1 Galli, Peter (24 April 2006). \"OpenBRR Launches Closed Open-Source Group\". eWeek. QuinStreet, Inc. http:\/\/www.eweek.com\/c\/a\/Linux-and-Open-Source\/OpenBRR-Launches-Closed-OpenSource-Group . Retrieved 13 April 2015 .   \n\n\u2191 \"Welcome to Business Readiness Rating: A FrameWork for Evaluating OpenSource Software\". OpenBRR. Archived from the original on 24 December 2014. https:\/\/web.archive.org\/web\/20141224233009\/http:\/\/www.openbrr.org\/ . Retrieved 14 April 2015 .   \n\n\u2191 Arjona, Laura (06 January 2012). \"What happened to OpenBRR (Business Readiness Rating for Open Source)?\". The Bright Side. https:\/\/larjona.wordpress.com\/2012\/01\/06\/what-happened-to-openbrr-business-readiness-rating-for-open-source\/ . Retrieved 13 April 2015 .   \n\n\u2191 \"Welcome to OSSpal\". OSSpal. http:\/\/osspal.org\/ . Retrieved 18 April 2015 .   \n\n\u2191 31.0 31.1 31.2 31.3 Silva, Chamindra de (20 December 2009). \"10 questions to ask when selecting open source products for your enterprise\". TechRepublic. CBS Interactive. http:\/\/www.techrepublic.com\/blog\/10-things\/10-questions-to-ask-when-selecting-open-source-products-for-your-enterprise\/ . Retrieved 13 April 2015 .   \n\n\u2191 Phipps, Simon (21 January 2015). \"7 questions to ask any open source project\". InfoWorld. InfoWorld, Inc. http:\/\/www.infoworld.com\/article\/2872094\/open-source-software\/seven-questions-to-ask-any-open-source-project.html . Retrieved 10 April 2015 .   \n\n\u2191 Padin, Sandro (03 January 2014). \"How I Evaluate Open-Source Software\". 8th Light, Inc. https:\/\/blog.8thlight.com\/sandro-padin\/2014\/01\/03\/how-i-evaluate-open-source-software.html . Retrieved 01 June 2015 .   \n\n\u2191 Metcalfe, Randy (01 February 2004). \"Top tips for selecting open source software\". OSSWatch. University of Oxford. http:\/\/oss-watch.ac.uk\/resources\/tips . Retrieved 23 March 2015 .   \n\n\u2191 Limardo, J. (2013). \"DIY Evaluation Process\". LIMSExpert.com. ForwardPhase Technologies, LLC. http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1363288315&title=DIY%20Evaluation%20Process . Retrieved 07 February 2015 .   \n\n\u2191 36.0 36.1 Wheeler, David A. (05 August 2011). \"How to Evaluate Open Source Software \/ Free Software (OSS\/FS) Programs\". dwheeler.com. http:\/\/www.dwheeler.com\/oss_fs_eval.html . Retrieved 19 March 2015 .   \n\n\u2191 \"User Requirements Specification (URS)\". validation-online.net. Validation Online. http:\/\/www.validation-online.net\/user-requirements-specification.html . Retrieved 08 August 2015 .   \n\n\u2191 O'Keefe, Graham (01 March 2015). \"How to Create a Bullet-Proof User Requirement Specification (URS)\". askaboutgmp. http:\/\/www.askaboutgmp.com\/296-how-to-create-a-bullet-proof-urs . Retrieved 08 August 2015 .   \n\n\u2191 \"ASTM E1578-13, Standard Guide for Laboratory Informatics\". West Conshohocken, PA: ASTM International. 2013. doi:10.1520\/E1578. http:\/\/www.astm.org\/Standards\/E1578.htm . Retrieved 14 March 2015 .   \n\n\u2191 \"User Requirements Checklist\". Autoscribe Informatics. http:\/\/www.autoscribeinformatics.com\/services\/user-requirements . Retrieved 10 April 2015 .   \n\n\u2191 Laboratory Informatics Institute, ed. (2015). \"The Complete Guide to LIMS & Laboratory Informatics \u2013 2015 Edition\". LabLynx, Inc. http:\/\/www.limsbook.com\/the-complete-guide-to-lims-laboratory-informatics-2015-edition\/ . Retrieved 10 April 2015 .   \n\n\u2191 \"Part 11, Electronic Records; Electronic Signatures \u2014 Scope and Application\". U.S. Food and Drug Administration. 26 August 2015. http:\/\/www.fda.gov\/regulatoryinformation\/guidances\/ucm125067.htm . Retrieved 10 June 2015 .   \n\n\u2191 Segalstad, Siri H. (2008). International IT Regulations and Compliance: Quality Standards in the Pharmaceutical and Regulated Industries. John Wiley & Sons, Inc. pp. 338. ISBN 9780470758823. http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0470758821.html .   \n\n\u2191 \"More articles by Cynthia Harvey\". Datamation. QuinStreet, Inc. 2015. http:\/\/www.datamation.com\/author\/Cynthia-Harvey-6460.html . Retrieved 12 April 2015 .   \n\n\u2191 Harvey, Cynthia (05 January 2015). \"Open Source Software List: 2015 Ultimate List\". Datamation. QuinStreet, Inc. http:\/\/www.datamation.com\/open-source\/open-source-software-list-2015-ultimate-list-1.html . Retrieved 12 April 2015 .   \n\n\u2191 \"SourceForge - Download, Develop and Publish Free Open Source Software\". SourceForge. Slashdot Media. 2015. https:\/\/sourceforge.net\/ . Retrieved 14 June 2015 .   \n\n\u2191 \"GitHub: Where software is built\". GitHub. GitHub, Inc. 2015. https:\/\/github.com\/ . Retrieved 14 June 2015 .   \n\n\u2191 \"Comparison of source code hosting facilities\". Wikipedia. Wikimedia Foundation. 21 September 2015. https:\/\/en.wikipedia.org\/w\/index.php?title=Comparison_of_source_code_hosting_facilities&oldid=682090863 . Retrieved 28 September 2015 .   \n\n\u2191 \"The Architecture of Open Source Applications\". AOSA. AOSA Editors. 2015. http:\/\/aosabook.org\/en\/index.html . Retrieved 08 October 2015 .   \n\n\u2191 Knorr, Eric (28 September 2015). \"5 key trends in open source\". InfoWorld. InfoWorld, Inc. http:\/\/www.infoworld.com\/article\/2986769\/open-source-tools\/5-key-trends-in-open-source.html . Retrieved 28 September 2015 .   \n\n\u2191 \"Google Scholar\". Google, Inc. 2015. https:\/\/scholar.google.com\/ . Retrieved 08 August 2015 .   \n\n\u2191 \"Microsoft Academic Search\". Microsoft Corporation. 2015. http:\/\/academic.research.microsoft.com\/ . Retrieved 08 August 2015 .   \n\n\u2191 Wasike, Sylvia Nasambu (October 2010). \"Selection Process of Open Source Software Component\". http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.227.5951 . Retrieved 10 August 2015 .   \n\n\u2191 \"Open HUB, the open source network\". Open HUB. Black Duck Software, Inc. 2015. https:\/\/www.openhub.net\/ . Retrieved 10 August 2015 .   \n\n\u2191 Netelera, Markus; Bowmanb, M. Hamish; Landac, Martin; Metz, Markus (May 2012). \"GRASS GIS: A multi-purpose open source GIS\". Environmental Modelling & Software 31: 124\u2013130. doi:10.1016\/j.envsoft.2011.11.014.   \n\n\u2191 Khanine, Dmitri (07 May 2015). \"\"Meaningful Use\" Regulations of Medical Information in Health IT\". Toad World - Oracle Community. Dell Software, Inc. http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/07\/quot-meaningful-use-quot-regulations-of-medical-information-in-health-it . Retrieved 12 June 2015 .   \n\n\u2191 Khanine, Dmitri (06 May 2015). \"Open-Source Medical Record Systems of 2015\". Toad World - Oracle Community. Dell Software, Inc. http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/06\/open-source-medical-record-systems-of-2015 . Retrieved 28 May 2015 .   \n\n\u2191 Davis, Ashley (2008). \"Enterprise Resource Planning Under Open Source Software\". In Ferran, Carlos; Salim, Ricardo. Enterprise Resource Planning for Global Economies: Managerial Issues and Challenges. Hershey, PA: IGI Global. doi:10.4018\/978-1-59904-531-3.ch004. ISBN 9781599045313.   \n\n\u2191 Sarrah, Mohamed; Rehman, Osama M. Hussain (2013). \"Selection Criteria of Open Source Software: First Stage for Adoption\". International Journal of Information Processing and Management 4 (4): 51\u201358. doi:10.4156\/ijipm.vol4.issue4.6.   \n\n\u2191 Foote, Amanda (2010). \"The Myth of Free: The Hidden Costs of Open Source Software\". Dalhousie Journal of Interdisciplinary Management 6 (Spring 2010): 1\u20139. doi:10.5931\/djim.v6i1.31.   \n\n\u2191 Heinlein, Robert A. (1997) [1966]. The Moon Is a Harsh Mistress. New York, NY: Tom Doherty Associates. pp. 8\u20139. ISBN 9780312863555.   \n\n\u2191 Fleming, Ian (2014). \"ISO 9126 Software Quality Characteristics\". SQA Definition. http:\/\/www.sqa.net\/iso9126.html . Retrieved 18 June 2015 .   \n\n\u2191 Taylor, Dave (2015). \"Murphy's Laws\". Dave Taylor's Educational & Guidance Counseling Services. http:\/\/davetgc.com\/Murphys_Law.html . Retrieved 08 August 2015 .   \n\n\u2191 64.0 64.1 Abran, Alain; Khelifi, Adel; Suryn, Witold; Seffah, Ahmed (2003). \"Usability Meanings and Interpretations in ISO Standards\". Software Quality Journal 11 (4): 325\u2013338. doi:10.1023\/A:1025869312943.   \n\n\u2191 Andreasen, M.S.; Nielsen, H.V.; Schr\u00f8der, S.O.; Stage, J. (2006). \"Usability in open source software development: Opinions and practice\" (PDF). Information Technology and Control 35 (3A): 303-312. http:\/\/itc.ktu.lt\/itc353\/Stage353.pdf .   \n\n\u2191 Saxena, S.; Dubey, S.K. (January 2013). \"Impact of Software Design Aspects on Usability\". International Journal of Computer Applications 61 (22): 48-53. doi:10.5120\/10233-5043. http:\/\/www.ijcaonline.org\/archives\/volume61\/number22\/10233-5043 .   \n\n\u2191 AllAboutUX.org volunteers (08 October 2010). \"User experience definitions\". All About UX. http:\/\/www.allaboutux.org\/ux-definitions . Retrieved 08 August 2015 .   \n\n\u2191 Gube, Jacob (05 October 2010). \"What Is User Experience Design? Overview, Tools And Resources\". Smashing Magazine. Smashing Magazine GmbH. http:\/\/www.smashingmagazine.com\/2010\/10\/what-is-user-experience-design-overview-tools-and-resources\/ . Retrieved 08 August 2105 .   \n\n\u2191 Reitz, Kenneth (2014). \"Choosing a License\". The Hitchhiker's Guide to Python!. http:\/\/docs.python-guide.org\/en\/latest\/writing\/license\/ . Retrieved 13 May 2015 .   \n\n\u2191 Atwood, Jeff (03 April 2007). \"Pick a License, Any License\". Coding Horror: Programming and Human Factors. http:\/\/blog.codinghorror.com\/pick-a-license-any-license\/ . Retrieved 13 May 2015 .   \n\n\u2191 \"The Open Source Definition (Annotated)\". Open Source Initiative. 2015. http:\/\/opensource.org\/docs\/definition.php . Retrieved 28 May 2015 .   \n\n\u2191 Hocevar, Sam (2015). \"WTFPL \u2013 Do What the Fuck You Want to Public License\". WTFPL.net. http:\/\/www.wtfpl.net\/ . Retrieved 16 June 2015 .   \n\n\u2191 \"Licenses & Standards\". Open Source Initiative. 2015. http:\/\/opensource.org\/licenses . Retrieved 13 May 2015 .   \n\n\u2191 \"TL;DRLegal - Software Licenses Explained in Plain English\". FOSSA, Inc. 2015. https:\/\/tldrlegal.com\/ . Retrieved 16 June 2015 .   \n\n\u2191 Wheeler, David A. (16 February 2014). \"Make Your Open Source Software GPL-Compatible. Or Else\". dwheeler.com. http:\/\/www.dwheeler.com\/essays\/gpl-compatible.html . Retrieved 20 March 2015 .   \n\n\u2191 Wheeler, David A. (27 September 2007). \"The Free-Libre \/ Open Source Software (FLOSS) License Slide\". dwheeler.com. http:\/\/www.dwheeler.com\/essays\/floss-license-slide.html . Retrieved 28 May 2015 .   \n\n\u2191 \"What Is Docker?\". Docker.com. Docker, Inc. https:\/\/www.docker.com\/whatisdocker . Retrieved 18 June 2015 .   \n\n\u2191 Knorr, Eric (23 March 2015). \"When will we see Docker in production?\". InfoWorld. InfoWorld, Inc. http:\/\/www.infoworld.com\/article\/2900333\/cloud-computing\/scenes-from-the-docker-revolution.html . Retrieved 18 June 2015 .   \n\n\u2191 Emadeen, Hamza (01 August 2010). \"Bika : Free , Open source LIMS \u2013 Laboratory Information Management System for Windows , Linux and Mac OSX\". Goomedic.com. http:\/\/www.goomedic.com\/bika-free-open-source-lims-laboratory-information-management-system-for-windows-linux-and-mac-osx.html . Retrieved 21 July 2015 .   \n\n\u2191 Limardo, J. (2011). \"Open-LIMS Analysis Results\". LIMSExpert.com. ForwardPhase Technologies, LLC. http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1304895280&title=Open-LIMS%20Analysis%20Results . Retrieved 07 February 2015 .   \n\n\u2191 Limardo, J. (2011). \"Scoring Open-LIMS\". LIMSExpert.com. ForwardPhase Technologies, LLC. http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1305075855&title=Scoring%20Open-LIMS . Retrieved 07 February 2015 .   \n\n\u2191 Glover, H.D. (2013). \"Academia Should Embrace Open Access Scholarly Publishing\". Open Journal of Accounting 2 (4): 95\u201396. doi:10.4236\/ojacct.2013.24012.   \n\n\nNotes \nThis article has not officially been published in a journal. However, this presentation is largely faithful to the original paper. The content has been edited for grammar, punctuation, and spelling. Additional error correction of a few reference URLs and types as well as cleaning up of the glossary also occurred. Redundancies and references to entities that don't offer open-source software were removed from the FLOSS examples in Table 2. DOIs and other identifiers have been added to the references to make them more useful. This article is being made available for the first time under the Creative Commons Attribution-ShareAlike 4.0 International license, the same license used on this wiki.\n\r\n\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\">https:\/\/www.limswiki.org\/index.php\/Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version<\/a>\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 6 November 2015, at 19:01.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 3,885 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","13b3b6c47f60a3b75f007d8cd017d140_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Generalized_procedure_for_screening_free_software_and_open-source_software_applications_Print_version skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Generalized procedure for screening free software and open-source software applications\/Print version<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Free software and <a href=\"https:\/\/www.limswiki.org\/index.php\/Category:Open-source_software\" title=\"Category:Open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"1aa6da5776c348224e3820317e698a07\">open-source software projects<\/a> have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process, we have initiated a project to generate a library of in-depth reviews of <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"688dbf2b7fb58c69c6ad476828ee2c6a\">free software and open-source software<\/a> applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure is executed against a selected class of applications.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>There is much confusion regarding free software and open-source software, and many people use these terms interchangeably. However, the connotations associated with the terms are highly significant. So perhaps we should start with an examination of the terms to clarify what we are attempting to screen. While there are many groups and organizations involved with open-source software, two of the main ones are the Free Software Foundation (FSF) and the Open Source Initiative (OSI).\n<\/p><p>When discussing free software, we are not explicitly discussing software for which no fee is charged; rather, we are referring to \"free\" in terms of liberty. To quote the Free Software Foundation (FSF)<sup id=\"rdp-ebb-cite_ref-FSFWhat15_1-0\" class=\"reference\"><a href=\"#cite_note-FSFWhat15-1\" rel=\"external_link\">[1]<\/a><\/sup>:\n<\/p>\n<blockquote>A program is free software if the program's users have the four essential freedoms:\n<ul><li> The freedom to run the program as you wish, for any purpose (freedom 0).<\/li>\n<li> The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.<\/li>\n<li> The freedom to redistribute copies so you can help your neighbor (freedom 2).<\/li>\n<li> The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.<\/blockquote><\/li><\/ul>\n<p>This does not mean that a program is provided at no cost, or gratis, though some of these rights imply that it would be. In the FSF's analysis, any application that does not conform to these freedoms is unethical. While there is also \"free software\" or \"freeware\" that is given away at no charge but without the source code, this would not be considered free software under the FSF definition.\n<\/p><p>The Open Source Initiative (OSI), originally formed to promote free software, refers to it as open-source software (OSS) to make it sound more business friendly. The OSI defines open-source software as any application that meets the following 10 criteria, which they based on the Debian Free Software Guidelines<sup id=\"rdp-ebb-cite_ref-OSIDef15_2-0\" class=\"reference\"><a href=\"#cite_note-OSIDef15-2\" rel=\"external_link\">[2]<\/a><\/sup>:\n<\/p>\n<ul><li> Free redistribution<\/li>\n<li> Source code included<\/li>\n<li> Must allow derived works<\/li>\n<li> Must preserve the integrity of the author's source code<\/li>\n<li> License must not discriminate against persons or groups<\/li>\n<li> License must not discriminate against fields of endeavor<\/li>\n<li> Distribution of licenses<\/li>\n<li> License must not be specific to a production<\/li>\n<li> License must not restrict other software<\/li>\n<li> License must be technology-neutral<\/li><\/ul>\n<p>Open-source software adherents take what they consider the more pragmatic view of looking more at the license requirements and put significant effort into convincing commercial enterprises of the practical benefits of open source, meaning the free availability of application source code.\n<\/p><p>In an attempt to placate both groups when discussing the same software application, the term <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"688dbf2b7fb58c69c6ad476828ee2c6a\">free\/open-source software<\/a> (F\/OSS) was developed. Since the term \"free\" was still tending to confuse some people, the term \"libre,\" which connotes freedom, was added resulting in the term <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software#FLOSS\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"7ebfaa4419be758c507232ec468fbdfc\">free\/libre open-source software<\/a> (FLOSS). If you perform a detailed analysis on the full specifications, you will find that all free software fits the open-source software definition, while not all open-source software fits the free software definition. However, any open-source software that is not also free software is the exception rather than the rule. As a result, you will find these acronyms used almost interchangeably, but there are subtle differences in meaning, so stay alert. In the final analysis, the software license that accompanies the software is what you legally have to follow.\n<\/p><p>The reality is that since both groups trace their history back to the same origins, the practical differences between an application being free software or open-source are generally negligible. Keep in mind that the above descriptions are to some degree generalizations, as both organizations are involved in multiple activities. There are many additional groups interested in open source for a wide variety of reasons. However, this diversity is also a strong point, resulting in a vibrant and dynamic community. You should not allow the difference in terminology to be divisive. The fact that all of these terms can be traced back to the same origin should unite us.<sup id=\"rdp-ebb-cite_ref-Schie.C3.9FleFree12_3-0\" class=\"reference\"><a href=\"#cite_note-Schie.C3.9FleFree12-3\" rel=\"external_link\">[3]<\/a><\/sup> In practice, many of the organization members will use the terms interchangeably, depending on the point that they are trying to get across. With an excess of 300,000 FLOSS applications currently registered in SourceForge.net<sup id=\"rdp-ebb-cite_ref-DateRepOSS12_4-0\" class=\"reference\"><a href=\"#cite_note-DateRepOSS12-4\" rel=\"external_link\">[4]<\/a><\/sup> and over 10 million repositories on GitHub<sup id=\"rdp-ebb-cite_ref-Doll10Mil13_5-0\" class=\"reference\"><a href=\"#cite_note-Doll10Mil13-5\" rel=\"external_link\">[5]<\/a><\/sup>, there are generally multiple options accessible for any class of application, be it a <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management system<\/a> (LIMS), an office suite, a data base, or a document management system. Presumably you have gone through the assessment of the various challenges to using an open-source application<sup id=\"rdp-ebb-cite_ref-6\" class=\"reference\"><a href=\"#cite_note-6\" rel=\"external_link\">[6]<\/a><\/sup> and have decided to move ahead with selecting an appropriate application. The difficulty now becomes selecting which application to use. While there are multiple indexes of FOSS projects, these are normally just listings of the applications with a brief description provided by the developers, with no indication of the vitality or independent evaluation of the project.\n<\/p><p>What is missing is a catalog of in-depth reviews of these applications, eliminating the need for each group to go through the process of developing a list of potential applications, screening all available applications, and performing in-depth reviews of the most promising candidates. While true that once an organization has made a tentative selection it will need to perform its own testing to confirm that the selected application meets its specific needs, there is no reason for everyone to go through the tedious process of identifying projects and weeding out the untenable ones.\n<\/p>\n<dl><dd><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Joyce_2015.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"37d066ebcf584eb9b883df94b8c647a6\"><img alt=\"Fig1 Joyce 2015.png\" src=\"https:\/\/www.limswiki.org\/images\/3\/3c\/Fig1_Joyce_2015.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a><\/dd><\/dl>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Illustration 1.<\/b>: This diagram, originally by Chao-Kuei and updated by several others since, explains the different categories of software. It's available as a scalable vector graphic and as an XFig document, under the terms of any of the GNU GPL v2 or later, the GNU FDL v1.2 or later, or the Creative Commons Attribution-Share Alike v2.0 or later<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The primary goal of this document is to describe a general procedure capable of being used to screen any selected class of software applications. The immediate concern is with screening FLOSS applications, though allowances can be made to the process to allow at least rough cross-comparison of both FOSS and commercial applications. To that end, we start with an examination of published survey procedures. We then combine a subset of standard software evaluation procedures with recommendations for evaluating FLOSS applications. Because it is designed to screen such a diverse range of applications, the procedure is by necessity very general. However, as we move through the steps of the procedure, we will describe how to tune the process for the class of software that you are interested in.\n<\/p><p>You can also ignore any arguments regarding selecting between FLOSS and commercial applications. In this context, \"commercial\" refers to the marketing approach, not to the quality of the software. Many FLOSS applications have comparable, if not superior, quality to products that are traditionally marketed and licensed. Wheeler discusses this issue in more detail, showing that by many definitions FLOSS <i>is<\/i> commercial software.<sup id=\"rdp-ebb-cite_ref-WheelerFree11_7-0\" class=\"reference\"><a href=\"#cite_note-WheelerFree11-7\" rel=\"external_link\">[7]<\/a><\/sup>\n<\/p><p>The final objective of this process is to document a procedure that can then be applied to any class of FOSS applications to determine which projects in the class are the most promising to pursue, allowing us to expend our limited resources most effectively. As the information available for evaluating FOSS projects is generally quite different from that available for commercially licensed applications, this evaluation procedure has been optimized to best take advantage of this additional information.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Literature_review\">Literature review<\/span><\/h3>\n<p>A search of the literature returns thousands of papers related to open-source software, but most are of limited value in regards to the scope of this project. The need for a process to assist in selecting between open-source projects is mentioned in a number of these papers, and there appear to be over a score of different published procedures. Regrettably, none of these methodologies appear to have gained large-scale support in the industry. \n<\/p><p>Stol and Babar have published a framework for comparing evaluation methods targeting open-source software and include a comparison of 20 of them.<sup id=\"rdp-ebb-cite_ref-StolAComp10_8-0\" class=\"reference\"><a href=\"#cite_note-StolAComp10-8\" rel=\"external_link\">[8]<\/a><\/sup> They noted that web sites that simply consisted of a suggestion list for selecting an open-source application were not included in this comparison. This selection difficulty is nothing new with FLOSS applications. In their 1994 paper, Fritz and Carter review over a dozen existing selection methodologies, covering their strengths, weaknesses, the mathematics used, as well as other factors involved.<sup id=\"rdp-ebb-cite_ref-FritzAClass94_9-0\" class=\"reference\"><a href=\"#cite_note-FritzAClass94-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> No.\n<\/th>\n<th> Name\n<\/th>\n<th> Year\n<\/th>\n<th> Orig\n<\/th>\n<th> Method\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Capgemini Open Source Maturity Model\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2003\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluation Framework for Open Source Software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2004\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A Model for Comparative Assessment of Open Source Products\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2004\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Navica Open Source Maturity Model\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2004\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Woods and Guliani's OSMM\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2005\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Business Readiness Rating (OpenBRR)<sup id=\"rdp-ebb-cite_ref-OpenBRROpen2005_10-0\" class=\"reference\"><a href=\"#cite_note-OpenBRROpen2005-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WassermanTheBus06_11-0\" class=\"reference\"><a href=\"#cite_note-WassermanTheBus06-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2005\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\/I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Atos Origin Method for Qualification and Selection of Open Source Software (QSOS)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">I\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluation Criteria for Free\/Open Source Software Products\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A Quality Model for OSS Selection\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Selection Process of Open Source Software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Observatory for Innovation and Technological transfer on Open Source software (OITOS)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Framework for OS Critical Systems Evaluation (FOCSE)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Balanced Scorecards for OSS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Business Quality Rating (OpenBQR)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Evaluating OSS through Prototyping\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2007\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A Comprehensive Approach for Assessing Open Source Projects\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2008\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Software Quality Observatory for Open Source Software (SQO-OSS)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2008\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">An operational approach for selecting open source components in a software development project<sup id=\"rdp-ebb-cite_ref-MajchrowskiAnOp08_12-0\" class=\"reference\"><a href=\"#cite_note-MajchrowskiAnOp08-12\" rel=\"external_link\">[12]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2008\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">QualiPSo trustworthiness model\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2008\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">OpenSource Maturity Model (OMM)<sup id=\"rdp-ebb-cite_ref-PetrinjaIntro09_13-0\" class=\"reference\"><a href=\"#cite_note-PetrinjaIntro09-13\" rel=\"external_link\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2009\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<blockquote><b>Table 1.<\/b>: Comparison frameworks and methodologies for examination of FLOSS applications extracted from Stol and Babar.<sup id=\"rdp-ebb-cite_ref-StolAComp10_8-1\" class=\"reference\"><a href=\"#cite_note-StolAComp10-8\" rel=\"external_link\">[8]<\/a><\/sup> The selection<br \/>procedure is described in Stol's and Barbar's paper, however, 'Year' indicates the date of publication, 'Orig.' indicates whether the described<br \/> process originated in industry (I) or research (R), while 'Method' indicates whether the paper describes a formal analysis method and procedure (Yes)<br \/>or just a list of evaluation criteria (No).<\/blockquote>\n<p>Extensive comparisons between some of these methods have also been published, such as Deprez's and Alexandre's comparative assessment of the OpenBRR and QSOS techniques.<sup id=\"rdp-ebb-cite_ref-Deprezcomp08_14-0\" class=\"reference\"><a href=\"#cite_note-Deprezcomp08-14\" rel=\"external_link\">[14]<\/a><\/sup> Wasserman and Pal have also published a paper under the title of \"Evaluating Open Source Software,\" which appears to be more of an updated announcement and in-depth description of the Business Readiness Rating (BRR) framework.<sup id=\"rdp-ebb-cite_ref-WassermanEval10_15-0\" class=\"reference\"><a href=\"#cite_note-WassermanEval10-15\" rel=\"external_link\">[15]<\/a><\/sup> Jadhav and Sonar have also examined the issue of both evaluating and selecting software packages. They include a helpful analysis of the strengths and weaknesses of the various techniques.<sup id=\"rdp-ebb-cite_ref-JadhavEval09_16-0\" class=\"reference\"><a href=\"#cite_note-JadhavEval09-16\" rel=\"external_link\">[16]<\/a><\/sup> Perhaps more importantly, they clearly point out that there is no common list of evaluation criteria. While the majority of the articles they reviewed listed the criteria used, Jadhav and Sonar indicated that these criteria frequently did not include a detailed definition, which required each evaluator to use their own, sometimes conflicting, interpretation.\n<\/p><p>Since the publication of Stol and Babar's paper, additional evaluation methods have been published. Of particular interest are a series of papers by Pani et al. describing their proposed FAME (Filter, Analyze, Measure and Evaluate) methodology.<sup id=\"rdp-ebb-cite_ref-PaniAMeth10_17-0\" class=\"reference\"><a href=\"#cite_note-PaniAMeth10-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PaniTheFAMEApp10_18-0\" class=\"reference\"><a href=\"#cite_note-PaniTheFAMEApp10-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PaniTheFAMEtool10_19-0\" class=\"reference\"><a href=\"#cite_note-PaniTheFAMEtool10-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PaniTrans10_20-0\" class=\"reference\"><a href=\"#cite_note-PaniTrans10-20\" rel=\"external_link\">[20]<\/a><\/sup> In their \"Transferring FAME\" paper, they emphasized that all of the evaluation frameworks previously described in the published literature were frequently not easy to apply to real environments, as they were developed using an analytic research approach which incorporated a multitude of factors.<sup id=\"rdp-ebb-cite_ref-PaniTrans10_20-1\" class=\"reference\"><a href=\"#cite_note-PaniTrans10-20\" rel=\"external_link\">[20]<\/a><\/sup>\n<\/p><p>Their stated design objective with FAME is to reduce the complexity of performing the application evaluation, particularly for small organizations. They specify \"[t]he goals of FAME methodology are to aid the choice of high-quality F\/OSS products, with high probability to be sustainable in the long term, and to be as simple and user friendly as possible.\" They further state that \"[t]he main idea behind FAME is that the users should evaluate which solution amongst those available is more suitable to their needs by comparing technical and economical factors, and also taking into account the total cost of individual solutions and cash outflows. It is necessary to consider the investment in its totality and not in separate parts that are independent of one another.\"<sup id=\"rdp-ebb-cite_ref-PaniTrans10_20-2\" class=\"reference\"><a href=\"#cite_note-PaniTrans10-20\" rel=\"external_link\">[20]<\/a><\/sup>\n<\/p><p>This paper breaks the FAME methodology into four activities:\n<\/p>\n<ol><li> Identify the constraints and risks of the projects<\/li>\n<li> Identify user requirements and rank<\/li>\n<li> Identify and rank all key objectives of the project<\/li>\n<li> Generate a priority framework to allow comparison of needs and features<\/li><\/ol>\n<p>Their paper includes a formula for generating a score from the information collected. The evaluated system with the highest \"major score,\" <i>Pjtot<\/i>, indicates the system selected. While it is a common practice to define an analysis process which condenses all of the information gathered into a single score, I highly caution against blindly accepting such a score. FAME, as well as a number or the other assessment methodologies, is designed for iterative use. The logical purpose of this is to allow the addition of factors initially overlooked into your assessment, as well as to change the weighting of existing factors as you reevaluate their importance. However, this feature means that it is also very easy to unconsciously, or consciously, skew the results of the evaluation to select any system you wish. Condensing everything down into a single value also strips out much of the information that you have worked so hard to gather. Note that you can generate the same result score using significantly different input values. While of value, selecting a system based on just the highest score could potentially leave you with a totally unworkable system.\n<\/p><p>Pani et al. also describe a FAMEtool to assist in this data gathering and evaluation.<sup id=\"rdp-ebb-cite_ref-PaniTheFAMEtool10_19-1\" class=\"reference\"><a href=\"#cite_note-PaniTheFAMEtool10-19\" rel=\"external_link\">[19]<\/a><\/sup> However, a general web search as well as a review of their FAME papers revealed no indication of how to obtain this resource. While this paper includes additional comparisons with other FLOSS analysis methodologies, and there are some hints suggesting that the FAMEtool is being provided as a web service, I have found no URL specified for it. As of now, I have received no responses from the research team via either e-mail or Skype regarding FAME, the FAMEtool, or feedback on its use.\n<\/p><p>During this same time frame Soto and Ciolkowski also published papers describing the QualOSS Open Source Assessment Model and compared it to a number of the procedures in Stol and Barbar's table.<sup id=\"rdp-ebb-cite_ref-SotoTheQual09_21-0\" class=\"reference\"><a href=\"#cite_note-SotoTheQual09-21\" rel=\"external_link\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SotoTheQualProc09_22-0\" class=\"reference\"><a href=\"#cite_note-SotoTheQualProc09-22\" rel=\"external_link\">[22]<\/a><\/sup> Their focus was primarily on three process perspectives: product quality, process maturity, and sustainability of the development community. Due to the lack of anything more than a rudimentary process perspective examination, they felt that the following OSS project assessment models were unsatisfactory: QSOS, CapGemni OSMM, Navica OSMM, and OpenBRR. They position QualOSS as an extension of the tralatitious CMMI and SPICE process maturity models. While there are multiple items in the second paper that are worth incorporating into an in-depth evaluation process, they do not seem suitable for what is intended as a quick survey.\n<\/p><p>Another paper, published by Haaland and Groven, also compared a number of open source quality models. To this paper's credit, the authors devoted a significant amount of space to discussing the different definitions of quality and how the target audience of a tool might affect which definition was used.<sup id=\"rdp-ebb-cite_ref-HaalandFree10_23-0\" class=\"reference\"><a href=\"#cite_note-HaalandFree10-23\" rel=\"external_link\">[23]<\/a><\/sup> Like Stohl and Babar, they listed a number of the quality assessment models to choose from, including OSMM, QSOS, OpenBRR, and others. For their comparison, they selected OpenBRR and QualOSS. They appear to have classified OpenBRR as a first generation tool with a \"[us]ser view on quality\" and QualOSS as a second generation tool with a \"business point of view.\" An additional variation is that OpenBRR is primarily a manual tool, while QualOSS is primarily an automated tool. Their analysis in this article clearly demonstrates the steps involve in using these tools and in highlighting where they are objective and subjective. While they were unable to answer their original question as to whether the first- or second-generation tools did a better job of evaluation, to me they answered the following even more important but unasked question. As they proceeded through their evaluation, it became apparent as to how much the questions defined in the methods could affect the results of the evaluations. Even though the authors might have considered the questions to be objective, I could readily see how some of these questions could be interpreted in alternate ways. My takeaway is an awareness of the potential danger of using rigid tools, as they can skew the accuracy of the evaluation results depending on exactly what you want the evaluated application to do and how you plan to use it. These models can be very useful guides, but they should not be used to replace a carefully considered evaluation as there will always be factors influencing the selection decision which did not occur to anyone when the specifications were being written.\n<\/p><p>Hauge et al. have noted that despite the development of several normative methods of assessment, empirical studies have failed to show wide spread adoption of these methods.<sup id=\"rdp-ebb-cite_ref-HaugeAnEmp09_24-0\" class=\"reference\"><a href=\"#cite_note-HaugeAnEmp09-24\" rel=\"external_link\">[24]<\/a><\/sup> From their survey of a number of Norwegian software companies, they have noticed a tendency for selectors to skip the in-depth search for what they call the \"best fit\" application and fall back on what they refer to as a '\"first fit.\" This is an iterative procedure with the knowledge gained from the failure of one set of component tests being incorporated into the evaluation of the next one. Their recommendation is for researchers to stop attempting to develop either general evaluation schemas or normative selection methods which would be applicable to any software application and instead focus on identifying situationally sensitive factors which can be used as evaluation criteria. This is a very rational approach as all situations, even if evaluating the same set of applications is going to be different, as each user's needs are different.\n<\/p><p>Ayalal et al. have performed a study to try to more accurately determine why more people don't take advantage of the various published selection methodologies.<sup id=\"rdp-ebb-cite_ref-AyalaTowards11_25-0\" class=\"reference\"><a href=\"#cite_note-AyalaTowards11-25\" rel=\"external_link\">[25]<\/a><\/sup> While they looked at a number of factors and identified several possible problems, one of the biggest factors was the difficulty in obtaining the needed information for the evaluation. Based on the projects they studied, many did not provide a number of the basic pieces of information required for the evaluation, or perhaps worse, required extensive examination of the project web site and documentation to retrieve the required information. From her paper, it sounds as if this issue was more of a communication breakdown than an attempt to hide any of the information, not that this had any impact on the inaccessibility of the information.\n<\/p><p>In addition to the low engagement rates for the various published evaluation methods, another concern is the viability of the sponsoring organizations. One of the assessment papers indicated that the published methods with the smallest footprint, or the easiest to use, appeared to be FAME and the OpenBRR. I have already mentioned my difficulty obtaining additional information regarding FAME, and OpenBRR appears to be even more problematic. BRR was first registered on SourceForge in September of 2005<sup id=\"rdp-ebb-cite_ref-OBRRSource_26-0\" class=\"reference\"><a href=\"#cite_note-OBRRSource-26\" rel=\"external_link\">[26]<\/a><\/sup>, and an extensive Request For Comments from the founding members of the BRR consortium (SpikeSource, the Center for Open Source Investigation at Carnegie Mellon West, and Intel Corporation) was released.<sup id=\"rdp-ebb-cite_ref-OpenBRROpen2005_10-1\" class=\"reference\"><a href=\"#cite_note-OpenBRROpen2005-10\" rel=\"external_link\">[10]<\/a><\/sup> In 2006, in contrast to typical Open Source development groups, the OpenBRR group announced the formation of an OpenBRR Corporate Community group. Peter Galli's story indicates that \"the current plan is that membership will not be open to all.\"<sup id=\"rdp-ebb-cite_ref-GalliOpenBRR06_27-0\" class=\"reference\"><a href=\"#cite_note-GalliOpenBRR06-27\" rel=\"external_link\">[27]<\/a><\/sup> He quotes Murugan Pal saying \"membership will be on an invitation-only basis to ensure that only trusted participants are coming into the system.\" However, for some reason, at least some in the group \"expressed concern and unhappiness about the idea of the information discussed not being shared with the broader open-source community.\"<sup id=\"rdp-ebb-cite_ref-GalliOpenBRR06_27-1\" class=\"reference\"><a href=\"#cite_note-GalliOpenBRR06-27\" rel=\"external_link\">[27]<\/a><\/sup>\n<\/p><p>While the original Business Readiness Rating web site still exists, it is currently little more than a static web page.<sup id=\"rdp-ebb-cite_ref-OBRRArch_28-0\" class=\"reference\"><a href=\"#cite_note-OBRRArch-28\" rel=\"external_link\">[28]<\/a><\/sup> It appears that some of the original information posted on the site is still there, you just have to know what its URL is to access it, as the original links on the web site have been removed. Otherwise, you may have to turn to the Internet Archive to retrieve some of their documentation. The lack of any visible activity regarding OpenBRR prompted a blog post from one graduate student in 2012 asking \"What happened to OpenBRR (Business Readiness Rating for Open Source)?\"<sup id=\"rdp-ebb-cite_ref-ArjonaWhat12_29-0\" class=\"reference\"><a href=\"#cite_note-ArjonaWhat12-29\" rel=\"external_link\">[29]<\/a><\/sup>\n<\/p><p>It appears that at some point any development activity regarding OpenBRR was morphed into OSSpal.<sup id=\"rdp-ebb-cite_ref-OSSpalHome_30-0\" class=\"reference\"><a href=\"#cite_note-OSSpalHome-30\" rel=\"external_link\">[30]<\/a><\/sup> However, background information on this project is sparse as well. While the site briefly mentions that OSSpal incorporates a number of lessons learned from BRR, there is very little additional information regarding the group or the methods procedures. Their \"All Projects\" tab provides a list of over 30 open-source projects, but the majority simply show \"No votes yet\" under the various headings. In fact, as of now, the only projects showing any input at all are for Ubuntu and Mozilla Firefox.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Initial_evaluation_and_selection_recommendations\">Initial evaluation and selection recommendations<\/span><\/h3>\n<p>At this point, we'll take a step back from the evaluation methodologies papers and examine some of the more general recommendations regarding evaluating and selecting <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software#FLOSS\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"7ebfaa4419be758c507232ec468fbdfc\">FLOSS<\/a> applications. The consistency of their recommendations may provide a more useful guide for an initial survey of FLOSS applications.\n<\/p><p>In <i>TechRepublic<\/i>, de Silva recommends 10 questions to ask when selecting a FLOSS application.<sup id=\"rdp-ebb-cite_ref-Silva10Quest09_31-0\" class=\"reference\"><a href=\"#cite_note-Silva10Quest09-31\" rel=\"external_link\">[31]<\/a><\/sup> While he provides a brief discourse on each question in his paper to ensure you understand the point of his question, I've collected the 10 questions from his article into the following list. Once we see what overlap, if any, are amongst our general recommendations, we'll address some of the consolidated questions in more detail.\n<\/p>\n<ol><li> Are the open source license terms compatible with my business requirements?<\/li>\n<li> What is the strength of the community?<\/li>\n<li> How well is the product adopted by users?<\/li>\n<li> Can I get a warranty or commercial support if I need it?<\/li>\n<li> What quality assurance processes exist?<\/li>\n<li> How good is the documentation?<\/li>\n<li> How easily can the system be customized to my exact requirements?<\/li>\n<li> How is this project governed and how easily can I influence the road map?<\/li>\n<li> Will the product scale to my enterprise's requirements?<\/li>\n<li> Are there regular security patches?<\/li><\/ol>\n<p>Similarly, in <i>InfoWorld<\/i> Phipps lists seven questions you should have answered before even starting to select a software package.<sup id=\"rdp-ebb-cite_ref-Phipps7Quest15_32-0\" class=\"reference\"><a href=\"#cite_note-Phipps7Quest15-32\" rel=\"external_link\">[32]<\/a><\/sup> His list of questions, pulled directly from his article are:\n<\/p>\n<ol><li> Am I granted copyright permission?<\/li>\n<li> Am I free to use my chosen business model?<\/li>\n<li> Am I unlikely to suffer patent attack?<\/li>\n<li> Am I free to compete with other community members?<\/li>\n<li> Am I free to contribute my improvements?<\/li>\n<li> Am I treated as a development peer?<\/li>\n<li> Am I inclusive of all people and skills?<\/li><\/ol>\n<p>This list of questions shows a moderately different point of view, as it is not only just about someone selecting an open-source system, but also it's about getting involved in its direct development. Padin, of 8th Light, Inc., takes the viewpoint of a developer who might incorporate open-source software into their projects.<sup id=\"rdp-ebb-cite_ref-PadinHow14_33-0\" class=\"reference\"><a href=\"#cite_note-PadinHow14-33\" rel=\"external_link\">[33]<\/a><\/sup> The list of criteria pulled directly from his blog includes:\n<\/p>\n<ol><li> Does it do what I need it to do?<\/li>\n<li> How much more do I need it to do?<\/li>\n<li> Documentation<\/li>\n<li> Easy to review source code<\/li>\n<li> Popularity<\/li>\n<li> Tests and specs<\/li>\n<li> Licensing<\/li>\n<li> Community<\/li><\/ol>\n<p>Metcalfe of OSS Watch lists his top tips as<sup id=\"rdp-ebb-cite_ref-MetcalfeTop13_34-0\" class=\"reference\"><a href=\"#cite_note-MetcalfeTop13-34\" rel=\"external_link\">[34]<\/a><\/sup>:\n<\/p>\n<ol><li> Reputation<\/li>\n<li> Ongoing effort<\/li>\n<li> Standards and interoperability<\/li>\n<li> Support (Community)<\/li>\n<li> Support (Commercial)<\/li>\n<li> Version<\/li>\n<li> Version 1.0<\/li>\n<li> Documentation<\/li>\n<li> Skill setting<\/li>\n<li> Project Development Development Model<\/li>\n<li> License<\/li><\/ol>\n<p>In his LIMSexpert blog, Joel Limardo of ForwardPhase Technologies, LLC lists the following as components to check when evaluating an open-source application<sup id=\"rdp-ebb-cite_ref-LimardoDIY13_35-0\" class=\"reference\"><a href=\"#cite_note-LimardoDIY13-35\" rel=\"external_link\">[35]<\/a><\/sup>:\n<\/p>\n<ul><li> Check licensing<\/li>\n<li> Check code quality<\/li>\n<li> Test setup time<\/li>\n<li> Verify extensibility<\/li>\n<li> Check for separation of concerns<\/li>\n<li> Check for last updated date<\/li>\n<li> Check for dependence on outdated toolkits\/frameworks<\/li><\/ul>\n<p>Perhaps the most referenced of the general articles on selecting FLOSS applications is David Wheeler's \"How to Evaluate Open Source Software \/ Free Software (OSS\/FS) Programs.\"<sup id=\"rdp-ebb-cite_ref-WheelerHow11_36-0\" class=\"reference\"><a href=\"#cite_note-WheelerHow11-36\" rel=\"external_link\">[36]<\/a><\/sup> The detailed functionality to consider will vary with the types of applications being compared, but there are a number of general features that are relevant to almost any type of application. While we will cover them in more detail later, Wheeler categorizes the features to consider as the following:\n<\/p>\n<ul><li> System functionality<\/li>\n<li> System cost \u2013 direct and in-direct<\/li>\n<li> Popularity of application, i.e. its market share for that type of application<\/li>\n<li> Varieties of product support available<\/li>\n<li> Maintenance of application, i.e, is development still taking place<\/li>\n<li> Reliability of application<\/li>\n<li> Performance of application<\/li>\n<li> Scalability of application<\/li>\n<li> Usability of application<\/li>\n<li> Security of application<\/li>\n<li> Adaptability\/customizability of application<\/li>\n<li> Interoperability of application<\/li>\n<li> Licensing and other legal issues<\/li><\/ul>\n<p>While a hurried glance might suggest a lot of diversity in the features these various resources suggest, a closer look at the meaning of what they are saying shows a repetitive series of concerns. The primary significant differences between the functionality lists suggested is actually due more to how wide a breadth of the analysis process the authors are considering, as well as the underlying features that they are concerned with.\n<\/p><p>With a few additions, the high-level screening template described in the rest of this communication is based on Wheeler's previously mentioned document describing his recommended process for evaluating open-source software and free software programs. Structuring the items thus will make it easier to locate the corresponding sections in his document, which includes many useful specific recommendations as well as a great deal of background information to help you understand the why of the topic. I highly recommend reading it and following up on some of the links he provides. I will also include evaluation suggestions from several of the previously mentioned procedures where appropriate.\n<\/p><p>Wheeler defines four basic steps to this evaluation process, as listed below:\n<\/p>\n<ul><li> Identify candidate applications.<\/li>\n<li> Read existing product reviews.<\/li>\n<li> Compare attributes of these applications to your needs.<\/li>\n<li> Analyze the applications best matching your needs in more depth.<\/li><\/ul>\n<p>Wheeler categorizes this process with the acronym IRCA. In this paper we will be focusing on the IRC components of this process. To confirm the efficacy of this protocol we will later apply it to several classes of open-source applications and examine the output of the protocol.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Identify_needs\">Identify needs<\/span><\/h4>\n<p>Realistically, before you can perform a survey of applications to determine which ones best match your needs, you must determine what your needs actually are. The product of determining these needs is frequently referred to as the user requirements specification (URS).<sup id=\"rdp-ebb-cite_ref-VOURS15_37-0\" class=\"reference\"><a href=\"#cite_note-VOURS15-37\" rel=\"external_link\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OKeeffeHow15_38-0\" class=\"reference\"><a href=\"#cite_note-OKeeffeHow15-38\" rel=\"external_link\">[38]<\/a><\/sup> This document can be generated in several ways, including having all of the potential users submit a list of the functions and capabilities that they feel is important. While the requirements document can be created by a single person, it is generally best to make it a group effort with multiple reviews of the draft document, including all of the users who will be working with the application. The reason for this is to ensure that an important requirement is not missed. When a requirement is missed, it is frequently due to the requirement being so basic that it never occurs to anyone that it specifically needed to be included in the requirements document. Admittedly, a detailed URS is not required at the survey level, but it is worth having if only to identify, by their implications, other features that might be significant.\n<\/p><p>Needs will, of course, vary with the type of application you are looking for and what you are planning to do with it. Keep in mind that the URS is a living document, subject to change through this whole process. Developing a URS is generally an iterative process, since as you explore systems, you may well see features that you hadn't considered that you find desirable. This process will also be impacted by whether the application to be selected will be used in a regulated environment. If it is, there will be existing documents that describe the minimum functionality that must be present in the system. Even if it is not to be used in a regulated environment, documents exist for many types of systems that describe the recommended functional requirements that would be expected for that type of system.\n<\/p><p>For a clarifying example, if you were attempting to select a laboratory information management system (LIMS), you can download checklists and standards of typical system requirements from a variety of sources.<sup id=\"rdp-ebb-cite_ref-ASTMStandsLIMS_39-0\" class=\"reference\"><a href=\"#cite_note-ASTMStandsLIMS-39\" rel=\"external_link\">[39]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ASURCheck_40-0\" class=\"reference\"><a href=\"#cite_note-ASURCheck-40\" rel=\"external_link\">[40]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LIIComplete15_41-0\" class=\"reference\"><a href=\"#cite_note-LIIComplete15-41\" rel=\"external_link\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FDAP1115_42-0\" class=\"reference\"><a href=\"#cite_note-FDAP1115-42\" rel=\"external_link\">[42]<\/a><\/sup> These will provide you with examples of the questions to ask, but you will have to determine which ones are important to, or required for, your particular effort.\n<\/p><p>Depending on the use to which this application is to be applied, you may be subject to other specific regulatory requirements as well. Which regulations may vary, since the same types of analysis performed for different industries fall under different regulatory organizations. This aspect is further complicated by the fact that you may be affected by more than one country's regulations if your analysis is applicable to products being shipped to other countries. While some specific regulations may have changed since its publication, an excellent resource to orient you to the diverse factors that must be considered is Siri Segalstad's book <i>International IT Regulations and Compliance<\/i>.<sup id=\"rdp-ebb-cite_ref-SegalstadInt08_43-0\" class=\"reference\"><a href=\"#cite_note-SegalstadInt08-43\" rel=\"external_link\">[43]<\/a><\/sup> My understanding is that an updated version of this book is currently in preparation. Keep in mind that while regulatory requirements that you must meet will vary, these regulations by and large also describe best practices, or at least the minimal allowed practices. These requirements are not put in place arbitrarily (generally) or to make things difficult for you but to ensure the quality of the data produced. As such, any deviations should be carefully considered, whether working in a regulated environment or not. Proper due diligence would be to determine which regulations and standards would apply to your operation.\n<\/p><p>For a LIMS, an example of following best practices is to ensure that the application has a full and detailed <a href=\"https:\/\/www.limswiki.org\/index.php\/Audit_trail\" title=\"Audit trail\" target=\"_blank\" class=\"wiki-link\" data-key=\"96a617b543c5b2f26617288ba923c0f0\">audit trail<\/a>. An audit trail allows you to follow the processing of items through your system, determining who did what and when. In any field where it might become important to identify the actions taken during a processing step, an audit trail should be mandatory. While your organization's operations may not fall under the FDA's <a href=\"https:\/\/www.limswiki.org\/index.php\/21_CFR_Part_11\" title=\"21 CFR Part 11\" target=\"_blank\" class=\"wiki-link\" data-key=\"0ec495c20a8f17894bd0c1388eda8e7a\">21 CFR Part 11<\/a> regulations, which address data access and security (including audit trails), it is still extremely prudent that the application you select complies with them. If it does not, then almost anyone could walk up to your system and modify data, either deliberately or accidentally, and you would have no idea of who made the changes or what changes they made. For that matter, you might not even be able to tell a change was made at all, which likely will raise concerns both inside and outside of your organization. This would obviously cause major problems if they became a hinge issue for any type of liability law suit.\n<\/p><p>For this screening procedure, you do not have to have a fully detailed URS, but it is expedient to have a list of your make-or-break issues. This list will be used later for comparing systems and determining which ones justify a more in-depth evaluation.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Identify_candidates\">Identify candidates<\/span><\/h4>\n<p>To evaluate potential applications against your functional criteria, you must initially generate a list of potential systems. While this might sound easy, generating a comprehensive list frequently proves to be a challenge. When initiating the process, you must first determine the type of system that you are looking for, be it a LIMS, a hospital management system, a database, etc. At this point, you should be fairly open in building your list of candidates. By that, I mean that you should be careful not to select applications based solely on the utilization label applied to them. The same piece of software can frequently be applied to solve multiple problems, so you should cast a wide net and not automatically reject a system because the label you were looking for hadn't been applied to it. While the label may give you a convenient place to start searching, it is much more important to look for the functionality that you need, not what the system is called. In any case, many times the applied labels are vague and mean very different things to different people.\n<\/p><p>There are a variety of ways to generate your candidate list. A good place to start is simply talking with colleagues in your field. Have they heard of or used a FLOSS application of the appropriate type that they like? Another way is to just flip through journals and trade magazines that cover your field. Any sufficiently promising applications are likely to be mentioned there. Many of the trade magazines will have a special annual issue that covers equipment and software applicable to their field. It is difficult to generate a list of all potential resources, as many of these trade publications are little-known outside of their field. Also keep in mind that with the continued evolution of the World Wide Web, many of these trade publications also have associated web sites that you can scan or search. The table below includes just a minor fraction of these sites that are available. (We would welcome the suggestion of any additional resource sites that you are aware of. Please e-mail the fields covered, the resource name, and either its general URL or the URL of the specific resource section to the corresponding editor.)\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Field\n<\/th>\n<th> Resource Name\n<\/th>\n<th> URL\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Astronomy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tech Support Alert\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.techsupportalert.com\/best-free-astronomy-software.htm\" target=\"_blank\">http:\/\/www.techsupportalert.com\/best-free-astronomy-software.htm<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Business intelligence reporting\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Technology Innovation Management Review\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/timreview.ca\/article\/288\" target=\"_blank\">http:\/\/timreview.ca\/article\/288<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Community radio\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Prometheus Radio Project\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/prometheusradio.org\/Free_Open_Source_Tools_C_R\" target=\"_blank\">http:\/\/prometheusradio.org\/Free_Open_Source_Tools_C_R<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Comprehensive range of apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Black Duck KnowledgeBase\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.blackducksoftware.com\/products\/knowledgebase\" target=\"_blank\">https:\/\/www.blackducksoftware.com\/products\/knowledgebase<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Comprehensive range of apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The Directory of Open Access Repositories - OpenDOAR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.opendoar.org\/\" target=\"_blank\">http:\/\/www.opendoar.org\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data storage\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">InfoStor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.infostor.com\/nas\/58-top-open-source-storage-project-1.html\" target=\"_blank\">http:\/\/www.infostor.com\/nas\/58-top-open-source-storage-project-1.html<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Digital audio editing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25 Free Digital Audio Editors You Should Know\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.hongkiat.com\/blog\/25-free-digital-audio-editors\/\" target=\"_blank\">http:\/\/www.hongkiat.com\/blog\/25-free-digital-audio-editors\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Digital video editing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Best free video editing software: 20 top programs in 2015\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.techradar.com\/us\/news\/software\/applications\/best-free-video-editing-software-9-top-programs-you-should-download-1136264\" target=\"_blank\">http:\/\/www.techradar.com\/us\/news\/software\/applications\/best-free-video-editing-software-9-top-programs-you-should-download-1136264<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of Android apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wikipedia \u2013 Android applications\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_Android_applications\" target=\"_blank\">https:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_Android_applications<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Launchpad, The Canonical Group\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/launchpad.net\/\" target=\"_blank\">http:\/\/launchpad.net\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Microsoft CodePlex\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.codeplex.com\/\" target=\"_blank\">https:\/\/www.codeplex.com\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Hub, Black Duck Software, Inc.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.openhub.net\/\" target=\"_blank\">https:\/\/www.openhub.net\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of apps (but site is deprecated)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Google Code\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/code.google.com\/p\/support\/\" target=\"_blank\">https:\/\/code.google.com\/p\/support\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of apps (focus on infrastructure)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Source Guide by Smile\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.open-source-guide.com\/en\" target=\"_blank\">http:\/\/www.open-source-guide.com\/en<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of apps (primarily targeting Samsung products)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Samsung Open Source Release Center\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensource.samsung.com\/reception.do\" target=\"_blank\">http:\/\/opensource.samsung.com\/reception.do<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of commercial open source apps and services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wikipedia \u2013 Commercial open source applications\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_commercial_open-source_applications_and_services\" target=\"_blank\">https:\/\/en.wikipedia.org\/wiki\/List_of_commercial_open-source_applications_and_services<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of GNU apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Savannah GNU\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/savannah.gnu.org\/\" target=\"_blank\">http:\/\/savannah.gnu.org\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of non-GNU apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Savannah Non-GNU\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/savannah.nongnu.org\" target=\"_blank\">http:\/\/savannah.nongnu.org<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Diverse range of top apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Projects and Applications\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensource.com\/resources\/projects-and-applications\" target=\"_blank\">http:\/\/opensource.com\/resources\/projects-and-applications<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Drug discovery\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Drug Discovery Today\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sciencedirect.com\/science\/article\/pii\/S1359644605036925\" target=\"_blank\">http:\/\/www.sciencedirect.com\/science\/article\/pii\/S1359644605036925<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Electronic engineering\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Education Engineering (EDUCON), 2010 IEEE\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=5492430\" target=\"_blank\">http:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=5492430<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Embroidery\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Machine Embroidery Portal\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.k2g2.org\/portal:machine_embroidery\" target=\"_blank\">http:\/\/www.k2g2.org\/portal:machine_embroidery<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Enterprise resource planning (ERP)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A Comparison of Open Source ERP Systems\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.big.tuwien.ac.at\/system\/theses\/20\/papers.pdf?1298476232\" target=\"_blank\">http:\/\/www.big.tuwien.ac.at\/system\/theses\/20\/papers.pdf?1298476232<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Enterprise resource planning (ERP)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Source ERP Site\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.open-source-erp-site.com\/list-of-open-source-erps.html\" target=\"_blank\">http:\/\/www.open-source-erp-site.com\/list-of-open-source-erps.html<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Fashion\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ethical Fashion Forum - 9 Open-Source & Low Cost Digital Fashion Business Tools\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/source.ethicalfashionforum.com\/article\/9-open-source-low-cost-digital-fashion-business-tools\" target=\"_blank\">http:\/\/source.ethicalfashionforum.com\/article\/9-open-source-low-cost-digital-fashion-business-tools<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Geographic information systems (GIS)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Source GIS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensourcegis.org\/\" target=\"_blank\">http:\/\/opensourcegis.org\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Geophysics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Geopsy project\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.geopsy.org\/\" target=\"_blank\">http:\/\/www.geopsy.org\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Geophysics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wikipedia - Geophysics software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/en.wikipedia.org\/wiki\/Comparison_of_free_geophysics_software\" target=\"_blank\">https:\/\/en.wikipedia.org\/wiki\/Comparison_of_free_geophysics_software<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GIS for libraries\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Library Hi Tech\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.emeraldinsight.com\/doi\/abs\/10.1108\/07378831011026742\" target=\"_blank\">http:\/\/www.emeraldinsight.com\/doi\/abs\/10.1108\/07378831011026742<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Help desk\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CIO\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cio.com.au\/article\/320110\/5_open_source_help_desk_apps_watch\/\" target=\"_blank\">http:\/\/www.cio.com.au\/article\/320110\/5_open_source_help_desk_apps_watch\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Highly diverse range of apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Code NASA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/code.nasa.gov\/#\/\" target=\"_blank\">https:\/\/code.nasa.gov\/#\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Highly diverse range of apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NASA Open Source Software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ti.arc.nasa.gov\/opensource\/\" target=\"_blank\">http:\/\/ti.arc.nasa.gov\/opensource\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Laboratory informatics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">LIMSwiki - Laboratory informatics software (open source)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.limswiki.org\/index.php\/Category:Laboratory_informatics_software_%28open_source%29\" target=\"_blank\">http:\/\/www.limswiki.org\/index.php\/Category:Laboratory_informatics_software_%28open_source%29<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Learning management systems (LMS)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Source LMS \u2013 10 Alternatives to Moodle\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/barrysampson.com\/2009\/04\/08\/open-source-lms-10-alternatives-to-moodle\/\" target=\"_blank\">http:\/\/barrysampson.com\/2009\/04\/08\/open-source-lms-10-alternatives-to-moodle\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Library science\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">FOSS4LIB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/foss4lib.org\/packages\" target=\"_blank\">https:\/\/foss4lib.org\/packages<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Limited range of iOS apps\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wikipedia - Open-source iOS applications\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_iOS_applications\" target=\"_blank\">https:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_iOS_applications<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medical and laboratory informatics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Healthcare Freeware\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.healthcarefreeware.com\/lim.htm\" target=\"_blank\">http:\/\/www.healthcarefreeware.com\/lim.htm<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medical office records\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ZDNet\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.zdnet.com\/article\/free-and-open-source-healthcare-software-for-your-practice\/\" target=\"_blank\">http:\/\/www.zdnet.com\/article\/free-and-open-source-healthcare-software-for-your-practice\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medicine\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">'Open' Health IT Solutions for the Public & Private Sectors\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/drive.google.com\/file\/d\/0B_TuX9zE68eWcVM1dkdDQTVCdnM\/edit\" target=\"_blank\">https:\/\/drive.google.com\/file\/d\/0B_TuX9zE68eWcVM1dkdDQTVCdnM\/edit<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medicine\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50 Successful Open Source Projects That Are Changing Medicine\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/nursingassistantguides.com\/2009\/50-successful-open-source-projects-that-are-changing-medicine\/\" target=\"_blank\">http:\/\/nursingassistantguides.com\/2009\/50-successful-open-source-projects-that-are-changing-medicine\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medicine\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Datamation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.datamation.com\/open-source\/50-open-source-replacements-for-health-care-software-1.html\" target=\"_blank\">http:\/\/www.datamation.com\/open-source\/50-open-source-replacements-for-health-care-software-1.html<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medicine\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Medfloss.org\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.medfloss.org\/\" target=\"_blank\">http:\/\/www.medfloss.org\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">MIDI controllers\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Source MIDI Controllers\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/punkmanufacturing.com\/wiki\/open-source-midi-controllers\" target=\"_blank\">http:\/\/punkmanufacturing.com\/wiki\/open-source-midi-controllers<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Music\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20 great free and open source music making programs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.musicradar.com\/tuition\/tech\/20-great-free-and-open-source-music-making-programs-582934\" target=\"_blank\">http:\/\/www.musicradar.com\/tuition\/tech\/20-great-free-and-open-source-music-making-programs-582934<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pattern management for textiles\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tools for computer generated patterns\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/xxxclairewilliamsxxx.wordpress.com\/tools-to-create-and-explore-digital-patterns\/\" target=\"_blank\">https:\/\/xxxclairewilliamsxxx.wordpress.com\/tools-to-create-and-explore-digital-patterns\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Personal information management\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wikipedia - Personal information managers\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"https:\/\/en.wikipedia.org\/wiki\/List_of_personal_information_managers#Open_source_applications\" target=\"_blank\">https:\/\/en.wikipedia.org\/wiki\/List_of_personal_information_managers#Open_source_applications<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Photography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Source Photography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensourcephotography.org\/software-list\" target=\"_blank\">http:\/\/opensourcephotography.org\/software-list<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Project management\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Open Source Photography\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensourcephotography.org\/software-list\" target=\"_blank\">http:\/\/opensourcephotography.org\/software-list<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Repository management\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Free and open-source repository software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/oad.simmons.edu\/oadwiki\/Free_and_open-source_repository_software\" target=\"_blank\">http:\/\/oad.simmons.edu\/oadwiki\/Free_and_open-source_repository_software<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Software test management\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Software Testing Help - 15 Best Test Management Tools for Software Testers\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.softwaretestinghelp.com\/15-best-test-management-tools-for-software-testers\/\" target=\"_blank\">http:\/\/www.softwaretestinghelp.com\/15-best-test-management-tools-for-software-testers\/<\/a>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Survey and feedback\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Capterra - Top survey software products\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.capterra.com\/survey-software\/\" target=\"_blank\">http:\/\/www.capterra.com\/survey-software\/<\/a>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<blockquote><b>Table 2.<\/b>: Examples of focused FLOSS resource sites available on the web<\/blockquote>\n<div class=\"thumb tright\"><div class=\"thumbinner\" style=\"width:491px;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Joyce_2015.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b9001be696dcccd2e77948f12f5091f1\"><img alt=\"\" src=\"https:\/\/www.limswiki.org\/images\/9\/9f\/Fig2_Joyce_2015.png\" class=\"thumbimage\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a> <div class=\"thumbcaption\"><div class=\"magnify\"><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Joyce_2015.png\" class=\"internal wiki-link\" title=\"Enlarge\" target=\"_blank\" data-key=\"b9001be696dcccd2e77948f12f5091f1\"><\/a><\/div><b>Illustration 2.<\/b>: Result page displayed on Black Duck Software's Open Hub site for the Bika Open Source LIMS 3 project.<\/div><\/div><\/div>\n<p>I also recommend checking some of the general open source project lists, such as the ones generated by Cynthia Harvey at Datamation<sup id=\"rdp-ebb-cite_ref-HarveyData15_44-0\" class=\"reference\"><a href=\"#cite_note-HarveyData15-44\" rel=\"external_link\">[44]<\/a><\/sup>, which has been covering the computer and data-processing industry since 1957. In particular, you might find their article \"Open Source Software List: 2015 Ultimate List\" useful. It itemizes over 1,200 open source applications, including some in categories that I didn't even know existed.<sup id=\"rdp-ebb-cite_ref-HarveyOpen15_45-0\" class=\"reference\"><a href=\"#cite_note-HarveyOpen15-45\" rel=\"external_link\">[45]<\/a><\/sup>\n<\/p><p>It would also be prudent to search the major open source repositories such as SourceForge<sup id=\"rdp-ebb-cite_ref-SFHome_46-0\" class=\"reference\"><a href=\"#cite_note-SFHome-46\" rel=\"external_link\">[46]<\/a><\/sup> and GitHub.<sup id=\"rdp-ebb-cite_ref-GHHome_47-0\" class=\"reference\"><a href=\"#cite_note-GHHome-47\" rel=\"external_link\">[47]<\/a><\/sup> <i>Wikipedia<\/i> includes a comparison of source code hosting facilities that would be worth reviewing as well.<sup id=\"rdp-ebb-cite_ref-WPSourceRepos_48-0\" class=\"reference\"><a href=\"#cite_note-WPSourceRepos-48\" rel=\"external_link\">[48]<\/a><\/sup> Keep in mind that you will need to be flexible with your search terms, as the developers might be looking at the application differently than you are. While they were created for a different purpose, an examination of the books in <i>The Architecture of Open Source Applications<\/i> might prove useful as well.<sup id=\"rdp-ebb-cite_ref-AOSAHome_49-0\" class=\"reference\"><a href=\"#cite_note-AOSAHome-49\" rel=\"external_link\">[49]<\/a><\/sup> Other sites where you might find interesting information regarding new open-source applications, are the various OpenSource award sites, such as the <i>InfoWorld<\/i> Best of Open Source Software Awards, colloquially known as the Bossies.<sup id=\"rdp-ebb-cite_ref-IW5Key15_50-0\" class=\"reference\"><a href=\"#cite_note-IW5Key15-50\" rel=\"external_link\">[50]<\/a><\/sup> \n<\/p><p>When searching the web, don't rely on just Google or Bing. Don't forget to checkout all of the journal web sites such as SpringerLink, Wiley, ScienceDirect, PubMed, and others as they contain a surprising amount of information on FLOSS. If you don't wish to search each of them individually, there are other search engines out there which can give you an alternate view of the research resources available. To name just two, be sure to try both Google Scholar<sup id=\"rdp-ebb-cite_ref-GSHome_51-0\" class=\"reference\"><a href=\"#cite_note-GSHome-51\" rel=\"external_link\">[51]<\/a><\/sup> and Microsoft Academic Search.<sup id=\"rdp-ebb-cite_ref-MASHome_52-0\" class=\"reference\"><a href=\"#cite_note-MASHome-52\" rel=\"external_link\">[52]<\/a><\/sup> These tools can also be used to search for masters theses and doctoral dissertations, which likewise contain a significant amount of information regarding open source.\n<\/p><p>While working on creating your candidate list, be sure to pull any application reviews that you come across.<sup id=\"rdp-ebb-cite_ref-WasikeSelect10_53-0\" class=\"reference\"><a href=\"#cite_note-WasikeSelect10-53\" rel=\"external_link\">[53]<\/a><\/sup> If done well, these reviews can save you a significant amount of time in screening a potential system. However, unless you're familiar with the credentials of the author, be cautious of relying on them for all of your information. While not common, people have been known to post fake reviews online, sometimes when it is not even April 1! Another great resource, both for identifying projects and obtaining information about them, is Open HUB<sup id=\"rdp-ebb-cite_ref-OHHome_54-0\" class=\"reference\"><a href=\"#cite_note-OHHome-54\" rel=\"external_link\">[54]<\/a><\/sup>, a web site dedicated to open source and maintained by Black Duck Software, Inc. Open Hub allows you to search by project, person, organization, forums, and code. For example, if I searched for <a href=\"https:\/\/www.limswiki.org\/index.php\/Bika_LIMS\" title=\"Bika LIMS\" target=\"_blank\" class=\"wiki-link\" data-key=\"daf3e2e28f594ad55c814ae802d43fcb\">Bika LIMS<\/a>, it would currently return the results for Bika Open Source LIMS 3 along with some basic information regarding the system. If I were to click on the project's name, a much more detailed page regarding this project is displayed. Moving your mouse cursor over the graphs displays the corresponding information for that date.\n<\/p><p>Once a list of candidate applications has been generated, the list of entries must be compared. Some of this comparison can be performed objectively, but it also requires subjective analysis of some components. As Stol and Babar have shown, there is no single recognized procedure for either the survey or detailed comparison of FLOSS applications that has shown a marked popularity above the others.\n<\/p><p>The importance of any one specific aspect of the evaluation will vary with the needs of the organization. General system functionality will be an important consideration, but specific aspects of the contained functionality will have different values to different groups. For instance, interoperability may be very important to some groups, while others may be using this application as their only data system and they have no interest in exchanging data files with others, so interoperability is not a concern to them. While you can develop a weighting system for different aspects of the system, this can easily skew selections, resulting in a system that has a very good rating yet is unable to perform the required function. Keep in mind that though this is a high-level survey, we are asking broad critical questions, not attempting to compare detailed minutia. Also keep in mind that a particular requirement might potentially fall under multiple headings. For example, compliance with 21 CFR Part 11 regulations might be included under functionality or security.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"In-depth_evaluation\">In-depth evaluation<\/span><\/h3>\n<p>While in-depth analysis of the screened systems will require a more detailed examination and comparison, for the purpose of this initial survey a much simpler assessment protocol will serve. While there is no single \"correct\" evaluation protocol, something in the nature of the three-leaf scoring criteria described for QSOS<sup id=\"rdp-ebb-cite_ref-Deprezcomp08_14-1\" class=\"reference\"><a href=\"#cite_note-Deprezcomp08-14\" rel=\"external_link\">[14]<\/a><\/sup> should be suitable. Keep in mind that for this quick assessment we are using broad criteria, so both the criteria and the scoring will both be more ambiguous than that required for an in-depth assessment. Do not be afraid to split any of these criteria up into multiple finer classes of criteria if the survey requires it. This need would be most likely appear under \"system functionality,\" as that is where most people's requirements greatly diverge.\n<\/p><p>In this process, we will assign one of three numeric values to each of the listed criteria. A score of zero indicates that the system does not meet the specified criteria. A score of one indicates that the system marginally meets the specified criteria. You can look on this as the feature is present and workable, but not to the degree you'd like it. Finally, a score of two indicates that the system fully meets or exceeds the specified requirement. In the sections below I will list some possible criteria for this table. However, you can adjust these descriptions or add a weighting factor, as many other protocols do, to adjust for the criticality of a given requirement.\n<\/p><p>Realistically, when it comes down to some of your potential evaluation criteria, the actuality is that for some of them, you can compensate for the missing factor in some other way. For other criteria, their presence or absence can be a drop-dead issue. That is, if the particular criteria or feature isn't present, then it doesn't matter how well any of the other criteria are ranked: that particular system is out of consideration. Deciding which, if any, criteria are drop-dead items should ideally be determined before you start your survey. This will not only be more efficient, in that it will allow you to cut off data collection at any failure point, but it will also help dampen the psychological temptation to fudge your criteria, retroactively deciding that a given criteria was not that important after all.\n<\/p><p>At this stage we are just wanting to reduce the number of systems for in-depth evaluation from potentially dozens, to perhaps three or four. As such, we will be refining our review criteria later, so if something isn't really a drop-dead criteria, don't mark it so. It's amazing the variety of feature tradeoffs people tend to make further down the line.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"System_functionality\">System functionality<\/span><\/h4>\n<p>While functionality is a key aspect of selecting a system, its assessment must be used with care. Depending on how a system is designed, a key function that you are looking for might not have been implemented, but in one system it can easily be added, while in another it would take a complete redesign of the application. Also consider the possibility of whether this function must be intrinsic to the application or if you can pair the application being evaluated with another application to cover the gap.\n<\/p><p>In most cases, you can obtain much of the functionality information from the project's web site or, occasionally, web sites. Some projects have multiple web sites, usually with one focused on project development and another targeting general users or application support. There are two different types of functionality to be tested. The first might be termed \"general functionality\" that would apply to almost any system. Examples of this could include the following:\n<\/p>\n<ul><li> User authentication<\/li>\n<li> Audit trail (I'm big on detailed audit trails, as they can make the difference between between being in regulatory compliance and having a plant shut down. Even if you aren't required to have them, they are generally a good idea, as the records they maintain may be the only way for you to identify and correct a problem.)<\/li>\n<li> Sufficient system status display that the user can understand the state of the system<\/li>\n<li> Ability to store data in a secure fashion<\/li><\/ul>\n<p>We might term the second as \"application functionality.\" This is functionality specifically required to be able to perform your job. As the subject matter expert, <i>you<\/i> will be the one to create this list. Items might be as diverse as the following:\n<\/p><p>For a laboratory information management system...\n<\/p>\n<ul><li> Can it print bar coded labels?<\/li>\n<li> Can it track the location of samples through the laboratory, as well as maintain the chain-of-custody for the sample?<\/li>\n<li> Can it track the certification of analysts for different types of equipment, as well as monitor the preventive maintenance on the instruments?<\/li>\n<li> Can it generate and track aliquots of the original sample?<\/li><\/ul>\n<p>For a <a href=\"https:\/\/www.limswiki.org\/index.php\/Geographic_information_system\" title=\"Geographic information system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8981ab93f8ebf0730c3b38949b39ad99\">geographic information system<\/a> (GIS)<sup id=\"rdp-ebb-cite_ref-NetelerGRASS12_55-0\" class=\"reference\"><a href=\"#cite_note-NetelerGRASS12-55\" rel=\"external_link\">[55]<\/a><\/sup>...\n<\/p>\n<ul><li> What range of map projections is supported?<\/li>\n<li> Does the system allow you to create custom parameters?<\/li>\n<li> Does it allow import of alternate system formats?<\/li>\n<li> Can it directly interface with geographic positioning devices (GPS)?<\/li><\/ul>\n<p>For a library management software system (LMSS) (or integrated library system [ILS], if you prefer, or even library information management system [LIMS]; have you ever noticed how scientists love to reuse acronyms, even within the same field?)...\n<\/p>\n<ul><li> Can you identify the location of any item in the collection at a given time?<\/li>\n<li> Can you identify any items that were sent out for repair and when?<\/li>\n<li> Can it identify between multiple copies of an item and between same named items on different types of media?<\/li>\n<li> Can it handle input from RFID tags?<\/li>\n<li> Can it handle and differentiate different clients residing at the same address?<\/li>\n<li> If needed, can it correlate the clients age with the particular item they are requesting, in case you have to deal with any type of age appropriate restrictions? \n<ul><li> If so, can it be overridden where necessary (maintaining the appropriate records in the audit trail as to why the rule was overridden)?<\/li><\/ul><\/li><\/ul>\n<p>For an archival record manager (This classification can cover a lot of ground due to all of the different ways that \"archival\" and \"record\" are interpreted.)...\n<\/p>\n<ul><li> In some operations, a record can be any information about a sample, including the sample itself. By regulation, some types of information must be maintained essentially for ever. In others, you might have to keep information for five years, while you have to maintain data for another type for 50 years.\n<ul><li> Can the application handle tracking records for different amounts of time?<\/li>\n<li> Does the system automatically delete these records at the end of the retention period or does it ask for confirmation from a person?<\/li>\n<li> Can overrides on a particular sample be applied so that records are not allowed to be deleted, either manually of because they are past their holding date, such as those that might be related to any litigation, while again maintaining all information about the override and who applied the override in the audit trail?<\/li><\/ul><\/li>\n<li> In other operations, an archival record manager may actually refer to the management of archival records, be these business plans, architectural plans, memos, art work, etc.\n<ul><li> Does the system keep track of the type of each record?<\/li>\n<li> Does the system support appropriate meta data on different types of records?\n<ul><li> Does it just record the items location and who is responsible for it, such as a work of art?<\/li>\n<li> If a document, does it just maintain an electronic representation of a document, such as a PDF file, or does it record the location of the original physical document, or can it do both?<\/li>\n<li> Can it manage both permanently archived items, such as a work of art or a historically significant document, and more transitory items, where your record retention rules say to save it for five, 10, 15 years, etc., and then destroy it?\n<ul><li> In the latter case, does the system require human approval before destroying any electronic documents or flagging a physical item for disposal? Does it require a single human sign-off or must a chain of people sign-off on it, just to confirm that it is something to be discarded by business rules and not an attempt to hide anything?<\/li><\/ul><\/li><\/ul><\/li><\/ul><\/li><\/ul>\n<p>For medical records<sup id=\"rdp-ebb-cite_ref-KhanineMean15_56-0\" class=\"reference\"><a href=\"#cite_note-KhanineMean15-56\" rel=\"external_link\">[56]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KhanineOpen15_57-0\" class=\"reference\"><a href=\"#cite_note-KhanineOpen15-57\" rel=\"external_link\">[57]<\/a><\/sup>...\n<\/p>\n<ul><li> This is a challenging quagmire, with frequently changing regulations and requirements. Depending on how you want to break it down, this heading can be segmented into two classes: <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical records<\/a> (EMR) which can constitute an electronic version of the tracking of the patients health and <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> (EHR) which contains extensive information on the patient, including test results, diagnostic information, and other observations by the medical staff. For those who want to get picky, you can also subdivide the heading into imaging systems, such as X-rays and CAT scans and other specialized systems.\n<ul><li> Can all records be accessed quickly and completely under emergency situations?<\/li>\n<li> What functionality is in place to minimize the risk of a <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Insurance_Portability_and_Accountability_Act\" title=\"Health Insurance Portability and Accountability Act\" target=\"_blank\" class=\"wiki-link\" data-key=\"b70673a0117c21576016cb7498867153\">Health Insurance Portability and Accountability Act<\/a> (HIPAA) violation?<\/li>\n<li> What functionality exists for automated data transfer from instruments or laboratory data systems to minimize transcription errors.<\/li>\n<li> How are these records integrated with any billing or other medical practice system?\n<ul><li> If integrated with the EMR and EHR record systems, does this application apply granular control over who can access these records and what information they are able to see?<\/li><\/ul><\/li><\/ul><\/li><\/ul>\n<p>For an enterprise resource planning (ERP) system...\n<\/p>\n<ul><li> Davis has indicated that a generally accepted definition of an ERP system is a \"complex, modular software integrated across the company with capabilities for most functions in the organization.\"<sup id=\"rdp-ebb-cite_ref-DavisEnt08_58-0\" class=\"reference\"><a href=\"#cite_note-DavisEnt08-58\" rel=\"external_link\">[58]<\/a><\/sup> I believe this translates as \"good luck,\" considering the complexity of the systems an ERP is designed to model and all of the functional requirements that go into that. It is perhaps for that reason that successful ERP implementations generally take several years.\n<ul><li> ERP systems generally must be integrated with other informatics systems within the organization. What types of interfaces does this system support?\n<ul><li> Is their definition of plug-and-play that you just have to configure the addresses and fields to exchange?<\/li>\n<li> Is their definition of interface that the system can read in information in a specified format and export the same, leaving you to write the middle ware program to translate the formats between the two systems?<\/li><\/ul><\/li><\/ul><\/li><\/ul>\n<p>From the above, it is easy to see why researchers have encountered difficulty in developing a fixed method that can be used to evaluate anything. At this point, it is quite acceptable to group similar functions together \u2014 as this is a high-level survey to identify which systems will definitely <i>not<\/i> be suitable \u2014 so we can focus our researches on those that might be. Researchers such as Sarrab and Rehman summarize system functionality as \"achieving the user's requirements, correct output as user's expectations and verify that the software functions appropriately as needed.\"<sup id=\"rdp-ebb-cite_ref-SarrabSel13_59-0\" class=\"reference\"><a href=\"#cite_note-SarrabSel13-59\" rel=\"external_link\">[59]<\/a><\/sup>\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 Application does not support required functionality.<\/li>\n<li> One \u2013 Application supports the majority of functionality to at least an useable extent.<\/li>\n<li> Two \u2013 Application meets or exceeds all functional requirements.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Community\">Community<\/span><\/h4>\n<p>Many of the researchers I've encountered have indicated that community is the most critical factor of a <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software#FLOSS\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"7ebfaa4419be758c507232ec468fbdfc\">FLOSS<\/a> project. There are a number of reasons for this. First, the health and sustainability of a FLOSS project is indicated by a large and diverse community that is both active and responsive.<sup id=\"rdp-ebb-cite_ref-Silva10Quest09_31-1\" class=\"reference\"><a href=\"#cite_note-Silva10Quest09-31\" rel=\"external_link\">[31]<\/a><\/sup> Additionally, the core programmers in a FLOSS project are generally few, and it is the size of the project community that determines how well-reviewed the application is, ensuring quality control of the projects code. Finally, the size of the project community correlates with the lifetime of the project.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 No community exists. No development activity is observable. Project is dead.<\/li>\n<li> One - Community is small and perhaps insular. May consist of just one or two programmers with perhaps a small number of satellite users.<\/li>\n<li> Two \u2013 Community is large and dynamic, with many contributors and active communication between the core developers and the rest of the community. Community is responsive to outside inquiries.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"System_cost\">System cost<\/span><\/h4>\n<p>Since the main goal of this survey procedure is to evaluate FLOSS products, the base system cost for the software will normally be low, frequently $0.00. However, this is not the only cost you need to consider. Many of these costs could potentially be placed under multiple headings depending on how your organization is structured. No matter how it itemizes them, there will be additional costs. Typical items to consider include the following:\n<\/p>\n<ul><li> Cost of supporting software \u2013 e.g. Does it require a commercial data base such as Oracle or some other specialized commercial software component?<\/li>\n<li> Cost of additional hardware \u2013 e.g. Does the system require the purchase of additional servers or storage systems? Custom hardware interfaces?<\/li>\n<li> Cost of training \u2013 e.g. How difficult or intuitive is the system to operate? This will impact the cost of training that users must receive. Keep in mind this cost will exist whether you are dealing with an Open Source or proprietary system. Are costs for system manuals and other required training material included? Some proprietary systems don't, or might perhaps send a single hard copy of the manuals. Who will perform the training? Whether you hire someone from outside or have some of your own people do it, there will be a cost, as you would be pulling people away from from their regular jobs.<\/li>\n<li> Cost of support \u2013 e.g. Is support through a commercial organization or the Open Source development group? If the former, what are their contract costs? While harder to evaluate, what is the turnaround time from when you request the support? Immediate? Days? Sometime? The amount of time you have to wait for a problem to be fixed, is definitely a \"cost,\" whether it means your system is dead in the water or just not as efficient and productive as it could be.<\/li><\/ul>\n<p>The primary issue here is to be realistic in your evaluation. It's hard to believe that anyone would assume that there were no associated costs with using FLOSS, or even proprietary software for that matter, but apparently there are. Foote does a good job of exploring and disproving this belief, showing all of the items that go into figuring the total cost of ownership (TCO), which should be representative of FLOSS applications.<sup id=\"rdp-ebb-cite_ref-FooteTheMyth10_60-0\" class=\"reference\"><a href=\"#cite_note-FooteTheMyth10-60\" rel=\"external_link\">[60]<\/a><\/sup> I wouldn't call them hidden costs, at least not with the FLOSS systems, but rather costs that are overlooked, as are so many other things when people focus on a single central item. To quote Robert Heinlein, \"TANSTAAFL!\" (There ain't no such thing as a free lunch!)<sup id=\"rdp-ebb-cite_ref-HeinleinMoon66_61-0\" class=\"reference\"><a href=\"#cite_note-HeinleinMoon66-61\" rel=\"external_link\">[61]<\/a><\/sup>\n<\/p><p>The important thing here is to pay attention to all of the interactions taking place. For example, if you wished to interface a piece of equipment to your FLOSS application, remember to factor in the cost of the interface. Despite what some advertisers think, data bits don't just disappear from one place and magically appear in another. It is very easy to lose track of where the costs are. Keep in mind that many items, such as training, you will have a cost either way you go. If a proprietary vendor says they will provide free training, you can be assured that the cost for it is included in the contract. But if you take that route, be very careful to read the contract thoroughly, as not all vendors include any training at all. It would be very easy to end up having to pay for \"optional\" training.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 Installation and support costs are excessive and greatly exceed any available budget.<\/li>\n<li> One \u2013 System may require purchase of additional hardware or customization. These costs, along with training costs, are within potential budget.<\/li>\n<li> Two \u2013 Installation and support costs are relatively minor, with no additional hardware required. System design is relatively intuitive with in-depth documentation and active support from the community.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Popularity\">Popularity<\/span><\/h4>\n<p>This heading can be somewhat confusing in terms of how it is interpreted even though most of the recommendations we looked at include it. Popularity is sometimes considered to be similar to market share. That is, of the number of people using a specific application in a given class of open-source applications, what percentage do they represent out of all people running applications in the class? If the majority of people are using a single system, this might indicate that it is the better system, or it might just indicate that other systems are newer and, even if potentially better, people haven't migrated over to them yet. An alternate approach to examining it is to ask how many times the application has been downloaded. In general, the larger the market share or the number of downloads, the more likely that a given product is to be usable. This is not an absolute, as people may have downloaded the application for testing and then rejected it or downloaded it simply to game the system, but it is a place to start. The point of this question usually isn't to determine how popular a particular application is, but rather to ensure that it is being used and it is a living (as opposed to an abandoned) project. Be leery of those applications with just a few downloads. If there is a large group of people using the application, there is a higher probability that the application works as claimed.\n<\/p><p>At the same time, learning who some of the other users of this application are can give you some insight of how well it actually works. As Silva reminds us, \"the best insight you can get into a product is from another user who has been using it for a while.\"<sup id=\"rdp-ebb-cite_ref-Silva10Quest09_31-2\" class=\"reference\"><a href=\"#cite_note-Silva10Quest09-31\" rel=\"external_link\">[31]<\/a><\/sup>\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 No other discernible users or reality of listed users is questionable.<\/li>\n<li> One \u2013 Application is being used by multiple groups, but represents only a small fraction of its \"market.\" Appears to be little 'buzz' about the application.<\/li>\n<li> Two \u2013 Application appears to be widely used and represents a significant fraction of its \"market.\" This rating is enhanced if listed users include major commercial organizations.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Product_support\">Product support<\/span><\/h4>\n<p>Product support can be a critically important topic for any application. Whether you are selecting a proprietary application or a FLOSS one, it is vitally important to ensure that you will have reliable support available. Just because you purchased a proprietary program will not ensure that you have the support you need. Some vendors include at least limited support in their contracts, others don't. However, over the years I've found that even purchasing a separate support contract doesn't ensure that the people who answer the phone will be able to help you. When making the final decision, don't make assumptions: research!\n<\/p><p>Support can be broken down into several different sub-categories:\n<\/p>\n<ul><li> User manuals \u2013 Do they exist? What is their quality?<\/li>\n<li> System managers manuals \u2013 Do they exist? What is their quality?<\/li>\n<li> Application developers manuals \u2013 Do they exist? What is their quality?<\/li>\n<li> System design manuals \u2013 Do they exist? What degree of detail do they provide?<\/li>\n<li> For database-related projects, is an accurate and detailed entity relationship diagram (ERD) included?<\/li>\n<li> Have any third-party books been written about this application? Are they readily available, readable, and easy to interpret?<\/li>\n<li> Is product support provided directly by the application development community?<\/li>\n<li> Is product support provided by an independent user group, separate from the development group?<\/li>\n<li> Is commercial product support available?\n<ul><li> If so, what are their rates?<\/li>\n<li> Are on-site classes available?<\/li>\n<li> Are online training classes available?<\/li><\/ul><\/li>\n<li> A frequently overlooked type of product support is how well documented the program code is. Are embedded comments sparse or frequent and meaningful? Are the names of program variables and functions arbitrary or meaningful?<\/li><\/ul>\n<p>Another factor in evaluating product support is whether you have anyone on your team with the expertise to understand it. That is not a derogatory statement: depending on the issue, someone might have to modify an associated data base, the application code, or the code in a required library, whether to correct an error or add functionality. Do any of your people have expertise in that language? Would someone have to start from scratch or do you have the budget to hire an outside consultant? Even if you have no desire to modify the code, having someone on the project that understands the language used can be a big help in discerning how the program works, as well as determining how meaningful the program comments are.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 Limited or no support available. Documentation essentially non-existent, source code minimally documented, no user group support, and erratic response from the developers.<\/li>\n<li> One \u2013 Documentation scattered and of poor quality. Support from user group discouraged and no commercial support options exist.<\/li>\n<li> Two - Excellent documentation, including user, system manager, and developer documentation. Enthusiastic support from the user community and developers. Commercial third-party support available for those desiring it. Third-party books may also have been released documenting the use of this product.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Maintenance\">Maintenance<\/span><\/h4>\n<p>This is another item that can be interpreted in multiple ways. One way to look at it is how quickly the developers respond to any bug report. Depending on the particular FLOSS project, you may actually be able to review the problem logs for the system and see what the average response time was between a bug being reported and the problem resolved. In some cases this might be hours or days, in others it is never resolved. To be fair, don't base your decision on a single instance in the log file, as some bugs are much easier to find and fix than others. However, a constant stream of open bugs or bugs that have only been closed after months or years should make you leery.\n<\/p><p>To others this question is to determine whether development is still taking place on the project or if it is dead. Alternately, it is like asking if anybody is maintaining the system and correcting bugs when they are discovered. There are several ways of addressing this issue. Examining the problem logs described above is one way of checking for project activity, while another is the check the release dates for different versions of the application. Are releases random or on a temporal schedule? How long has it been since the application was last updated? If the last release date was over a year or two ago, this is cause for concern and should trigger a closer look. Just because there hasn't been a recent release does not mean that the project has been abandoned. If the development of the app has advanced to the point where it is stable and no other changes need to be made you may not see a recent release because none is needed. However, the latter is very rare, both because bugs can be so insidious and because a lot of programmers can't resist just tweaking things, to make them a wee bit better. If a project is inactive, but everything else regarding the project looks good, it might be possible to work with the developers to revitalize it. While this course of action is feasible, it is important to realize that it is taking on a great deal of responsibility and an unknown amount of expense. The latter is particularly true as you may be having to assign one or more developers to work on the project full time.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 No releases, change log activity, or active development discussion in message forums in over two years.<\/li>\n<li> One \u2013 No releases, change log activity or active development discussions in message forums for between one and two years.<\/li>\n<li> Two \u2013 A new version has been released within the year, change logs show recent development activity, and there is active development discussion in the message forums.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Reliability\">Reliability<\/span><\/h4>\n<p>Reliability is the degree to which you can rely on the application to function properly. Of course, the exact definition becomes somewhat more involved. The reliability of a system is defined as the ability of an application to operate properly under a specified set of conditions for a specified period of time. Fleming states that \"[o]ne aspect of this characteristic is fault tolerance that is the ability of a system to withstand component failure. For example if the network goes down for 20 seconds then comes back the system should be able to recover and continue functioning.\"<sup id=\"rdp-ebb-cite_ref-FlemingISO14_62-0\" class=\"reference\"><a href=\"#cite_note-FlemingISO14-62\" rel=\"external_link\">[62]<\/a><\/sup>\n<\/p><p>Because of its nature, the reliability of a system is hard to measure, as you are basically waiting to see how frequently it goes down. While we'd like to aim for never, one should probably be satisfied if the system recovered properly after the failure. In most instances, unless you are actually testing a system under load, the best that you can hope for is to observe indicators from which you can infer its reliability. As a generality, the more mature a given code base is, the more reliable it is, but keep in mind that this is a generality; there are always incidents that can occur to destabilize every thing. Unfortunately, it frequently feels as if the problem turns out to be something that you would swear was totally unrelated. Face it, Murphy is just cleverer than you.<sup id=\"rdp-ebb-cite_ref-TaylorMurphy_63-0\" class=\"reference\"><a href=\"#cite_note-TaylorMurphy-63\" rel=\"external_link\">[63]<\/a><\/sup>\n<\/p><p>Wheeler also reminds us that \"[p]roblem reports are not necessarily a sign of poor reliability - people often complain about highly reliable programs, because their high reliability often leads both customers and engineers to extremely high expectations.\"<sup id=\"rdp-ebb-cite_ref-WheelerHow11_36-1\" class=\"reference\"><a href=\"#cite_note-WheelerHow11-36\" rel=\"external_link\">[36]<\/a><\/sup> One thing that can be very reassuring is to see that the community takes reliability seriously by continually testing the system during development.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 Error or bug tracking logs show a high incident of serious system problems. Perhaps worse, no logs of reported problems are kept at all, particularly for systems that have been in release for less than a year.<\/li>\n<li> One \u2013 Error logs show relatively few repeating or serious problems, particularly if these entries correlate with entries in the change logs indicating that a particular problem has been corrected. System has been in release for over a year.<\/li>\n<li> Two - Error logs are maintained, but show relatively few bug reports, with the majority of them being minor. A version of the system, using the same code base, has been in release for over two years. Developers both distribute and run a test suite to confirm proper system operation.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Performance\">Performance<\/span><\/h4>\n<p>Performance of an application is always a concern. Depending on what the application is trying to do and how the developers coded the functions, you may encounter a program that works perfectly but is just too unresponsive to use. Sometimes this is a matter of hardware, other times it is just inefficient coding, such as making sequential calls to a database to return part of a block of data rather than making a single call to return all of the block at once. Performance and scalability are usually closely linked.\n<\/p><p>You might be able to obtain some information on the system's actual performance from the project web site, but it is hard to tell if this is for representative or selected data. Reviewing the project mailing list may provide a more accurate indication of the system's performance or any performance problems encountered. Testing the system under your working conditions is the only way to make certain what the system's actual performance is. Unfortunately, the steps involved in setting up such a test system require much more effort than a high-level survey will allow. If any user reviews exist, they may give an insight into the system's performance. Locating other users through the project message board might be a very useful resource as well, particularly if they handle the same projected work loads that you are expecting.\n<\/p><p>It is difficult to define performance ratings without having knowledge of what the application is supposed to do. However, for systems that interact with a human operator, the time lapse between when a function is initiated and when the system responds can be suggestive. If the project maintains a test suite, particularly one containing sample data, reviewing its processing time can give an insight to the system's performance as well. Response delays of even a few seconds in frequently executed functions will not only kill the overall process performance but also result in users resistive to using the system. \n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 A system designed to be interactive fails to respond in an acceptable time frame. For many types of applications it is reasonable to expect an almost instantaneous response, particularly for screen management functions. It is not reasonable for a system to take over a minute, or even five seconds to switch screens or acknowledge an input, particularly in regards to frequently executed functions such as results entry, modification, or review.. A system that batch processes data maxes out under data loads below that of your current system.<\/li>\n<li> One \u2013 A system designed to be interactive appears to lag behind human entry for peripheral functions, but frequently accessed functions, such as results entry, modification, or review appear to respond almost instantaneously. A system that batch processes data maxes out under your existing data loads.<\/li>\n<li> Two \u2013 System is highly responsive, showing no annoying delayed responses. A system that batch processes data can process several times your current data load before maxing out.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Scalability\">Scalability<\/span><\/h4>\n<p>Scalability ensures that the application will operate over the data scale range you will be working with. In general, it means that if you test the system functionality with a low data load, the application will \"scale up\" to handle larger data loads. This may be handled by expanding from a single processor to a larger cluster or parallel processing system. Note that for a given application, throwing more hardware at it may not resolve the problem, as the application needs to be designed to take advantage of that additional hardware. Another caveat is to carefully examine the flow of data through your system. The processor is not the only place you can encounter roadblocks limiting scalability. Other possibilities include how quickly the system can access the needed data. If the system is processing the data faster than it can access it, adding more computer power will not resolve the problem. The limiting issues might be the bandwidth of your communication lines, the access speed of the devices that the data is stored on, or contention for needed resources with other applications. As with many aspects of selecting a system and getting it up and running, making assumptions is the real killer.\n<\/p><p>Silva indicates that many open-source applications are built on the LAMP (Linux, Apache, MySQL, PHP\/Perl\/Python) technology stack and that this is one of the most scalable configurations available.<sup id=\"rdp-ebb-cite_ref-Silva10Quest09_31-3\" class=\"reference\"><a href=\"#cite_note-Silva10Quest09-31\" rel=\"external_link\">[31]<\/a><\/sup> However, you should ensure that there is evidence that the application has been successfully tested that way; the performance survey and test phases are never a good time to start making assumptions. A look at the application's user base will likely identify someone who can provide this feedback.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 System does not support scaling, whether due to application design or restriction of critical resources, such as rate of data access.<\/li>\n<li> One \u2013 System supports limited scaling, but overhead or resource contention, such as a data bottleneck, results in a quick performance fall off.<\/li>\n<li> Two - System is balanced and scales well, supporting large processing clusters or cloud operations without any restrictive resource pinch-points.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Usability\">Usability<\/span><\/h4>\n<p>Usability means pretty much what it says. The concern here is not how well the program works but rather how easy is it to learn and use. The interface should be clear, intuitive, and help guide the user through the programs operation. Despite Steve Jobs, there is a limit to how intuitive an interface can be, thus the operation of the interface should be clearly documented in the user manual. Ideally the system will support a good context-sensitive help system as well. The best help systems may also provide multimedia support so that the system can actually show you how something should be done, rather than trying to tell you. I've found that frequently a good video can be worth well more than a thousand words! No matter how much time is spent writing the text for a manual or help system, it will always be unclear to somebody, if only because of the diversity of the backgrounds of people using it.\n<\/p><p>The interface between the operator and the computer may vary with the purpose of the application. While with new applications you are more likely to encounter a graphic user interface (GUI), there are still instances where you may encounter a command-line interface. Both types of interfaces have their advantages, and there are many times when something is actually easier to do with a command-line interface. The important thing to remember is that it <i>is<\/i> your interface with the system. It should be easy to submit commands to the system and interpret its response without having to hunt through a lot of extraneous information. This is normally best done by keeping the interface as clean and uncluttered as possible. As Abran et al. have pointed out, the usability of a given interface varies with \"the nature of the user, the task and the environment.\"<sup id=\"rdp-ebb-cite_ref-AbranUsa03_64-0\" class=\"reference\"><a href=\"#cite_note-AbranUsa03-64\" rel=\"external_link\">[64]<\/a><\/sup>\n<\/p><p>If you would prefer a somewhat drier set of definitions, Abran et al. also extracted the definitions of usability from a variety of ISO standards, and they are included in the following table:\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> ISO Usability Definitions\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"The capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions.\" (ISO\/IEC9126-1, 2000)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.\" (ISO 9241-11, 1998)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"The ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component.\" (IEEE Std. 610.12-1990)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<blockquote><b>Table 3.<\/b>: ISO Usability Definitions<sup id=\"rdp-ebb-cite_ref-AbranUsa03_64-1\" class=\"reference\"><a href=\"#cite_note-AbranUsa03-64\" rel=\"external_link\">[64]<\/a><\/sup><\/blockquote>\n<p>Fleming translates this into a somewhat more colloquial statement: \u201cUsability only exists with regard to functionality and refers to the ease of use for a given function.\u201d For those interested in learning more about the usability debate, I suggest that you check out Andreasen et al.<sup id=\"rdp-ebb-cite_ref-AndreasenUsa06_65-0\" class=\"reference\"><a href=\"#cite_note-AndreasenUsa06-65\" rel=\"external_link\">[65]<\/a><\/sup> and especially Saxena and Dubey.<sup id=\"rdp-ebb-cite_ref-SaxenaImpact13_66-0\" class=\"reference\"><a href=\"#cite_note-SaxenaImpact13-66\" rel=\"external_link\">[66]<\/a><\/sup>\n<\/p><p>In addition to usability, it has been highly recommended to me that, if budget exists, it is also helpful to have a user experience (UX) specialist on the review team as well.<sup id=\"rdp-ebb-cite_ref-AllUXDefs_67-0\" class=\"reference\"><a href=\"#cite_note-AllUXDefs-67\" rel=\"external_link\">[67]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GubeWhat10_68-0\" class=\"reference\"><a href=\"#cite_note-GubeWhat10-68\" rel=\"external_link\">[68]<\/a><\/sup>\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 System is difficult to use, frequently requiring switching between multiple screens or menus to perform a simple function. Operation of system discourages use and can actively antagonize users.<\/li>\n<li> One \u2013 System is useable but relatively unintuitive regarding how to perform a function. Both control and output displays tend to be cluttered, increasing the effort required to operate the system and interpret its output.<\/li>\n<li> Two \u2013 System is relatively intuitive and designed to help guide the user through its operation. Ideally, this is complemented with a context-sensitive help system to minimize any uncertainties in operation, particularly for any rarely used functions.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Security\">Security<\/span><\/h4>\n<p>This heading overlaps with functionality and is usually difficult to assess from a high-level evaluation. While it is unlikely that you will observe any obvious security issues during a survey of this type, there are indicators that can provide a hint as to how much the applications designers and developers were concerned with security.\n<\/p><p>The simplest approach is to simply take a look at whether they've done anything that shows a concern for possible security vulnerabilities. The following are a few potential indicators that you can look for, but just being observant when seeing a demonstration can also tell you a lot.\n<\/p>\n<ul><li> Is there any mention of security in the systems documentation? Does it describe any potential holes that you need to guard against or configuration changes you might make to your system environment to reduce any risks.<\/li>\n<li> Do the manuals describe any type of procedure for reporting bugs or observed system issues?<\/li>\n<li> If you have access to the developers, discuss any existing process for reporting and tracking security issues.<\/li>\n<li> Check their error logs and see if any security related issues are listed. If they are, what was the turnaround time to have them repaired, or were they repaired?<\/li>\n<li> If the developers are security-conscious, they will almost certainly want to prove to whomever received the program that it hadn't been modified by a third party. The basic way of doing this is by separately sending you what is known as an MD5 hash. This is a distinctive number generated by another program from your applications code. If you generate a new MD5 hash from the code you receive, these numbers should match. If they don't match, that means that something in the code has been altered. For developers with more concern, they might generate a cryptographic signature incorporating the code. This will tell you who sent the code as well as indicate whether the program was altered.<\/li>\n<li> Depending on the type of program, does it allow a 21 CFR Part 11-compliant implementation or conform to a similar standard?<\/li>\n<li> Depending on the type of application, does it include a detailed audit trail and security logs?<\/li><\/ul>\n<p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 System shows no concern with security or operator tracking. Anyone can walk up to it and execute a function without having to log in. System doesn't support even a minimal audit trail. Any intermediate files are easily accessible and modifiable outside of the system.<\/li>\n<li> One - System shows some attempt at user control but supports only a minimal audit trail. It may support a user table, but it fails to follow best practices by allowing user records to be deleted. Audit trail is modifiable by power users.<\/li>\n<li> Two \u2013 Maintaining system security is emphasized in the user documentation. A detailed audit trail is maintained that logs all system changes and user activities. Application is distributed along with an MD5 hash or incorporated into an electronic signature by the developer.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Flexibility.2FCustomizability\">Flexibility\/Customizability<\/span><\/h4>\n<p>The goal of this topic is to identify how easily the functionality of this application can be altered or how capable it is of handling situations outside of its design parameters. Systems are generally designed to be either configurable or customizable, sometimes with a combination of both.\n<\/p>\n<ul><li> Configurability - This refers to how much or how easily the functionality of the system can be altered by changing configuration settings. Configurable changes do not require any changes to the application code and generally simplify future application upgrades.<\/li>\n<li> Customizability - This refers to whether the functionality of the system must be altered by modifying the applications code. As we are targeting open-source systems, the initial assumption might be that they are all customizable; however, this can be affected by the type of license that the application is released under. More practically, how easily an application can be customized depends on how well it is designed and documented. While in theory you might be able to customize a system, if it is a mass of spaghetti code and poorly compartmentalized, it might be a nightmare to do. In any case, if you customize the system code, you may not be able to take advantage of any system upgrades without having to recreate the customizations in them.<\/li>\n<li> Extendability \u2013 While you won't find this term in most definitions, it is a hybrid system that is both configurable and customizable. It is normally configured using the same approaches as a standard configurable system. However, the ability to be upgraded remains by feeding any code customizations through an aApplication program interface (API). As long as this API is maintained between upgrades, any extension modules should continue to work.<\/li><\/ul>\n<p>In addition, a well designed application is usually modular, which makes program changes easier. In an ideal world, any application that you may have to customize will be specifically designed to make customization simple. There are a variety of ways of doing this. Perhaps the easiest, for an application that is designed to be modular, would be to support optional software plug-in modules that added extra functionality. Unless these were \"off-the-shelf\" modules, you would need to confirm that there was appropriate documentation regarding their design and use. This would most likely be done via an API, as discussed above. Depending on the system, you could transfer data through the API or have one system control another, the caveat again being that you need to have thorough documentation of the API and its capabilities.\n<\/p><p>In the majority of situations, I strongly encourage you to stick with a configurable system, assuming you can find one that meets your needs. Customizing a system is rarely justified unless you are working situationally. While almost everyone feels that their needs are unique, the reality is that a well-designed configurable system can generally meet your needs.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 Application does not support configuration and shows evidence of being difficult to customize, usually indicated by use of spaghetti code rather than modular design, poorly named variables and functions, along with cryptic or no embedded comments. In a worst-case situations, the source code has been deliberately obfuscated to make the system even less customizable.<\/li>\n<li> One \u2013 Application supports minor configuration capabilities or is moderately difficult to modify. The latter might be due to minimal application documentation or poor programming practices, but not deliberate obfuscation.<\/li>\n<li> Two \u2013 Application is highly configurable and accompanied by detailed documentation guiding the user through its configuration. Code is clearly documented and commented. It also follows good programming practices with highly modularized functionality, simplifying customization of the programs source code, ideally via an API.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Interoperability.2FIntegration\">Interoperability\/Integration<\/span><\/h4>\n<p>Determine whether this software will work with the rest of the systems that you plan to use. Exactly what to check for is up to you, as you are the only one who has any idea what you will be doing with it. The following is a list of possible items that might conceivably fall under this heading:\n<\/p>\n<ul><li> Does it understand the data and control protocols to talk to and control external equipment such as a drill press, telescope, or sewing machine (as appropriate)? <\/li>\n<li> Is it designed to conform to both electronic and physical standards to avoid being locked into a single supplier?<\/li>\n<li> Does it handle localization to avoid conflicts with local systems?<\/li><\/ul>\n<p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 System provides no support for integration with other applications. File formats and communication protocols used are not documented.<\/li>\n<li> One \u2013 System is not optimized for either interoperability or integration with other systems. However, it does use standard protocols so that other applications can interpret its activities. Application likely does not include an API or any existing API is undocumented.<\/li>\n<li> Two \u2013 Application is optimized for interoperability and integration with other systems. All interfaces and protocols, particularly for any existing API, are clearly documented and accompanied with sample code.<\/li><\/ul>\n<h4><span class=\"mw-headline\" id=\"Legal.2FLicense_issues\">Legal\/License issues<\/span><\/h4>\n<p>This section refers to the type of license that the application was released under and the associated legal and functional implications. With proprietary software, many people never bother to read the software license either because they don't care or think they have no choice but to accept them. Be that as it may, when selecting a FLOSS application, it is wise to take the time to read the accompanying license: it can make a big difference in what you can do with the software. First, if the software has no license, legally you have no right to even download the software, let alone run it.<sup id=\"rdp-ebb-cite_ref-ReitzChoosing14_69-0\" class=\"reference\"><a href=\"#cite_note-ReitzChoosing14-69\" rel=\"external_link\">[69]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AtwoodPick07_70-0\" class=\"reference\"><a href=\"#cite_note-AtwoodPick07-70\" rel=\"external_link\">[70]<\/a><\/sup>\n<\/p><p>While you have no control over which license the application was released under, you definitely control whether you wish to use it under the terms of the license. Which types of licenses are acceptable strongly depends on what you plan to do with the application. Do you intend to use the application as is or do you plan to modify it? If the latter, what do you plan to do with the modified code? Do you want to integrate this FLOSS application with another, either proprietary or open-source? Does this license clash with theirs? Your right to do any of these things is controlled by the license, so it must be considered very carefully, both in the light of what you want to do now and what you might want to do in the future.\n<\/p><p>One of the first things to do is to confirm that the application is even open-source; just being able to see the source code is insufficient. To qualify as open-source the license must comply with the 10 points listed in the Open Source Definition maintained by the Open Source Initiative (OSI).<sup id=\"rdp-ebb-cite_ref-OSIDef15_2-1\" class=\"reference\"><a href=\"#cite_note-OSIDef15-2\" rel=\"external_link\">[2]<\/a><\/sup> Pulling just the headers, their web site lists these as the required criteria:\n<\/p>\n<ul><li> Free redistribution<\/li>\n<li> Source code<\/li>\n<li> Derived works<\/li>\n<li> Integrity of the author's source code<\/li>\n<li> No discrimination against persons or groups<\/li>\n<li> No discrimination against fields of endeavor<\/li>\n<li> Distribution of licenses<\/li>\n<li> License must not be specific to a product<\/li>\n<li> License must not restrict other software<\/li>\n<li> License must be technology-neutral<\/li><\/ul>\n<p>While somewhat cryptic to look at cold, each of the headings is associated with a longer definition, which primarily boils down to the freedom to use, modify, and redistribute the software. For those wanting to know the justification for each item, there is also an annotated version of this definition.<sup id=\"rdp-ebb-cite_ref-OSIAnnoDef15_71-0\" class=\"reference\"><a href=\"#cite_note-OSIAnnoDef15-71\" rel=\"external_link\">[71]<\/a><\/sup> At present, OSI recognizes 71 distinct Open Source licenses, not counting the WTFPL.<sup id=\"rdp-ebb-cite_ref-WTFPLHome_72-0\" class=\"reference\"><a href=\"#cite_note-WTFPLHome-72\" rel=\"external_link\">[72]<\/a><\/sup>\n<\/p><p>One of the functions of the OSI is to review prospective licenses to determine whether they meet these criteria and are indeed open-source. The OSI web site maintains a list of popular licenses, along with links to all approved licenses<sup id=\"rdp-ebb-cite_ref-OSILic15_73-0\" class=\"reference\"><a href=\"#cite_note-OSILic15-73\" rel=\"external_link\">[73]<\/a><\/sup>, sorted by name or category. These lists include full copies of the licenses. While clearer than most legal documents, they can still be somewhat confusing, particularly if you are trying to select one. If the definitions seem to blur, you might want to check out the Software Licenses Explained in Plain English web page maintained by TL;DRLegal.<sup id=\"rdp-ebb-cite_ref-TLDRLegalHome_74-0\" class=\"reference\"><a href=\"#cite_note-TLDRLegalHome-74\" rel=\"external_link\">[74]<\/a><\/sup> As long as the license is classified as an open-source license and you aren't planning to modify it yourself or integrate it into other systems, you probably won't have any problems. However, if you have any uncertainty at all, it might be worth making the investment to discuss the license with an intellectual property lawyer who is familiar with OSS\/FS before you inadvertently commit your organization to terms that conflict with their plans.\n<\/p><p>If your plans include the possible creation of complementary software, I suggest a quick read of Wheeler's essay on selecting a license.<sup id=\"rdp-ebb-cite_ref-WheelerMake14_75-0\" class=\"reference\"><a href=\"#cite_note-WheelerMake14-75\" rel=\"external_link\">[75]<\/a><\/sup> The potential problem here is that most open-source and proprietary applications contain multiple libraries or sub-applications, each with their own license. Depending on which licenses the original developers used, they may be compatible with other applications you wish to use, or they may be incompatible. The following figure illustrates some of these complications:\n<\/p>\n<dl><dd><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Joyce_2015.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"be5bfd38b5192a0d1bd1240353c6b68f\"><img alt=\"Fig3 Joyce 2015.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/84\/Fig3_Joyce_2015.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a><\/dd><\/dl>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Illustration 3.<\/b>: License compatibility between common FOSS software licenses according to David A. Wheeler (2007): the vector arrows denote an one directional compatibility, therefore better compatibility on the left side than on the right side.<sup id=\"rdp-ebb-cite_ref-WheelerTheFree07_76-0\" class=\"reference\"><a href=\"#cite_note-WheelerTheFree07-76\" rel=\"external_link\">[76]<\/a><\/sup><\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Just to be absolutely clear, the license of FLOSS and proprietary applications generally disavows any type of warranty that the program will work and disclaims any liability for any damage or injuries that result from the programs use. Now, having said that, you may be able to purchase a warranty separately; just don't anticipate any legal recourse in the event of a system failure.\n<\/p><p>Suggested ratings are:\n<\/p>\n<ul><li> Zero \u2013 Application does not include a license or license terms are unacceptable or incompatible with those of other applications being used.<\/li>\n<li> One \u2013 Application includes a license, but it contains potential conflicts with other licenses or allowable use that will need to be carefully reviewed.<\/li>\n<li> Two \u2013 License is fully open, allowing you to freely use the software.<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Completing_the_evaluation\">Completing the evaluation<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Caveats\">Caveats<\/span><\/h4>\n<p>Other than independent reviews, one of the best ways to obtain some of the above information is direct testing of a system. Usually that is impractical in a survey situation because of the time it would take to install and configure an instance of the application. However, a new factor has entered the picture which may change this. Docker is a new service that combines an application, along with all of its dependencies, into a single distributable package, which they call a container. Because everything required is in the container, it is guaranteed to run the same on any system. Docker specifically states that their containers \"are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.\"<sup id=\"rdp-ebb-cite_ref-DockerWhatIs_77-0\" class=\"reference\"><a href=\"#cite_note-DockerWhatIs-77\" rel=\"external_link\">[77]<\/a><\/sup> Using only a fraction of the resources that a virtual machine (VM) would require, you can easily run multiple containers on a laptop and switch back and forth while testing.<sup id=\"rdp-ebb-cite_ref-KnorrWhen15_78-0\" class=\"reference\"><a href=\"#cite_note-KnorrWhen15-78\" rel=\"external_link\">[78]<\/a><\/sup> \n<\/p><p>While the Docker web site contains repositories for a number of different containers, there are multiple web sites that also host containers configured with a variety of applications. If you can locate one that contains the application you are surveying, this makes it a simple matter to try it out. Probably the easiest way to check is to run a web search containing the terms \"docker\", \"container\", and the name of the application that you are seeking.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Screening_tabulation\">Screening tabulation<\/span><\/h4>\n<p>While you can perform this application survey in many ways, to keep your defined scoring criteria in front of the evaluators and to simplify scoring the survey, it might be prudent to generate a document such as the following table with columns representing the criteria being evaluated, your rating definitions, and the numeric rating that your evaluators actually assign to the system. If you have decided to take the approach of using weighting factors instead of adjusting your definitions, you will also need to include columns for your weighting factors and a column containing the results of the evaluation after applying the weighting factor.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Criteria\n<\/th>\n<th> Rating = 0\n<\/th>\n<th> Rating = 1\n<\/th>\n<th> Rating = 2\n<\/th>\n<th> Score\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System functionality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Community\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System cost\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Popularity\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Product support\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Maintenance\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reliability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Performance\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Scalability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Usability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Flexibility\/Customizability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Interoperability\/Integration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Legal\/License issues\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<tr>\n<td style=\"background-color:#EEEEEE; padding-left:10px; padding-right:10px;\" colspan=\"4\">Summary Score\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">                \n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Table 4.<\/b>: Potential screening criteria to filter prospective <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software#FLOSS\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"7ebfaa4419be758c507232ec468fbdfc\">FLOSS<\/a> applications. Rows can be added or dropped as required by your needs. For example, if it is important that the application be coded in a specific programming language or combination of languages, you could add a row for this. The most likely anticipated change would be to subdivide the \"system functionality\" row so that specific information on the functionality of various system components can be displayed. To minimize the risk of confusion, you can transfer your selected scoring criteria for each rating into the cell corresponding to that rating and the item being evaluated, though ths would admittedly generate a long check list.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h4><span class=\"mw-headline\" id=\"Demonstration_surveys\">Demonstration surveys<\/span><\/h4>\n<p>To demonstrate how this survey procedure is intended to be used, this section will apply the defined protocol to a block of open-source LIMS applications suitable for use in a chemical <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a>. Note that in this instance, we are using LIMS to refer to a laboratory information management system, not a labor information management system, a legislative information management system, or any other permutation that matches the LIMS acronym. The specific criteria used will quite likely be different from the ones used in your screening as the types of systems or the specific functionality required will be different.\n<\/p><p>For the purpose of this demonstration, we will not attempt to screen all of the open-source LIMS available. Instead, we will select a subset of the systems that have been announced and attempt to apply our protocol to them, hopefully screening out the systems not meeting our requirements so that the number of in-depth evaluations can be reduced. Part of the difficulty here is the broad range of fields that the term LIMS covers. There are general purpose LIMS, which can be configured to handle a wide range of samples, as well as targeted LIMS, which are designed to fill a particular niche. As such, you can find LIMS targeting specific areas as drinking water\/waste water analysis, mining, radioisotopes, proteomics, and genetic analysis. Whether you make it a formal part of the screening process or isolate them while collecting the applications to be surveyed, you will need to filter out the systems which will not handle the type of samples you are dealing with. This issue is particularly prominent with LIMS, but it will likely be encountered when screening other types of software as well. The following are the systems that we will include in this attempt:\n<\/p>\n<ul><li> <a href=\"https:\/\/www.limswiki.org\/index.php\/Bika_LIMS\" title=\"Bika LIMS\" target=\"_blank\" class=\"wiki-link\" data-key=\"daf3e2e28f594ad55c814ae802d43fcb\">Bika LIMS<\/a><\/li>\n<li> eyeLIMS<\/li>\n<li> <a href=\"https:\/\/www.limswiki.org\/index.php\/Open-LIMS\" title=\"Open-LIMS\" target=\"_blank\" class=\"wiki-link\" data-key=\"a32e943ce983cd067b7f6208fa9dc3f5\">Open-LIMS<\/a><\/li><\/ul>\n<p>For simplicity in comparing results, I've reordered the screening table so that the first column contains the criteria being evaluated and the other columns correspond to the evaluation results for the LIMS being evaluated, with the bottom row reserved for the corresponding score summary. I have also added two subdivisions under \"system functionality\" for the programming language and the operating system used. Depending on the types of systems you are surveying, you will likely be including additional subdivisions. Scoring can be handled several ways. In this example, you might list the programming language in the appropriate cell. Depending on whether you have a team member competent in that language, you can use the results to set the value for the main criteria, e.g. system functionality. Alternately, while you will record the information for programming language in your survey notebook, you can insert an actual numerical result into the corresponding cell so that the sub-criteria can be evaluated separately or used to generate a value for the main criteria field.\n<\/p><p>Normally, the best starting approach is to check for existing reviews of systems, but here we have a problem in finding any. While references to Bika LIMS were common, and it was referenced in a number of scientific papers, actual reviews of the product were hard to come by and were usually for older versions.<sup id=\"rdp-ebb-cite_ref-EmadeenBika10_79-0\" class=\"reference\"><a href=\"#cite_note-EmadeenBika10-79\" rel=\"external_link\">[79]<\/a><\/sup> No reviews were found for eyeLIMS and for Open-LIMS; the closest thing I found to an impartial review has been two postings on Joel Limardo's LIMSExpert blog.<sup id=\"rdp-ebb-cite_ref-LimardoOpen11_80-0\" class=\"reference\"><a href=\"#cite_note-LimardoOpen11-80\" rel=\"external_link\">[80]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LimardoScoring11_81-0\" class=\"reference\"><a href=\"#cite_note-LimardoScoring11-81\" rel=\"external_link\">[81]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Criteria\n<\/th>\n<th> Bika LIMS 3.1.8\n<\/th>\n<th> eyeLIMS\n<\/th>\n<th> Open-LIMS*\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System functionality\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0 (Project appears dead, no activity since 2008)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1 (System functional for a very specific application only)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Programming language\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Python and PLONE CMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">eyeOS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHP\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Operating system\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">platform-independent\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">eyeOS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Any OS supporting PostgreSQL and PHP\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Community\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">System cost\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Popularity\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Product support\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Maintenance\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reliability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Performance\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Scalability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Usability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Flexibility\/Customizability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Interoperability\/Integration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2? (Supports import and export of CSV files)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Legal\/License issues\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (AGPL 3.0 and GPL1)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (GNU Affero GPL v3)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 (GNU GPL 3.0)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:#EEEEEE; padding-left:10px; padding-right:10px;\">Summary Score\n<\/td>\n<td style=\"background-color:#EEEEEE; padding-left:10px; padding-right:10px;\">22\n<\/td>\n<td style=\"background-color:#EEEEEE; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:#EEEEEE; padding-left:10px; padding-right:10px;\">10\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Based on the above summary scores, we would definitely filter out both eyeLIMS and Open-LIMS, while Bika LIMS would justify a more in-depth evaluation.\n<\/p><p>* Note: The rating for Open-LIMS may need to be revisited, as it appears that major development work on this system has been taken over by Joel Limardo of ForwardPhase Technologies, so several of the rated parameters may be subject to major shifts. It is currently unclear whether this is a joint project with the original developer or a fork.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Summary\">Summary<\/span><\/h2>\n<p>In support of a project to prepare published evaluations of various <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software#FLOSS\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"7ebfaa4419be758c507232ec468fbdfc\">FLOSS<\/a> applications, we have reviewed the FLOSS literature, focusing particularly on any assessment or evaluation documents. While many described proposed evaluation methods, none of them appear to be particularly popular or have developed an active community around them. Several review papers on the topic, while identifying multiple methods and their advantages, found flaws in all of them, particularly in terms of being able to perform a quality assessment on any FLOSS application. Many of the described systems were explicitly focused on a single class of FLOSS applications, such as library management systems.\n<\/p><p>By consolidating suggestions and procedures from a number of these papers, we synthesized a general survey process to allow us to quickly assess the status of any given type of FLOSS applications, allowing us to triage them and identify the most promising candidates for in-depth evaluation. Note that this process is designed for performing high-level surveys, it is not designed to perform the in-depth evaluations required for product selection.\n<\/p><p>As a minor aside, in the course of researching this article I was surprised by the high percentage of the published papers on FLOSS which were published in classic subscription journals, as opposed to any of the various open-source journals available.<sup id=\"rdp-ebb-cite_ref-82\" class=\"reference\"><a href=\"#cite_note-82\" rel=\"external_link\">[82]<\/a><\/sup> Somehow it seems like a curious disconnect not to publish articles on open-source Software in open-source journals. Whether this is just habit of submission or due to more considered reasons would be interesting to know.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Glossary\">Glossary<\/span><\/h2>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Term\n<\/th>\n<th> Explanation\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">.ogg\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">File extension for Ogg Vorbis, an open-source patent-free audio compression format\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/21_CFR_Part_11\" title=\"21 CFR Part 11\" target=\"_blank\" class=\"wiki-link\" data-key=\"0ec495c20a8f17894bd0c1388eda8e7a\">21 CFR Part 11<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">United States Food and Drug Administration electronic records and electronic signatures rule\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.academia.edu\/\" target=\"_blank\">Academia.edu<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A web site that allows academics to share research papers and exchange information\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">accuracy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Defines how well the results of an analysis or measurement conforms to the actual or \"correct\" value\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">AEQ\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Analytical equipment qualification\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">agnostic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A term applied to both hardware and software that indicates that the item is interoperable with different systems (The correct term should probably be \"technology-independent\" or \"technology-neutral,\" but the use of the term \"agnostic\" appears to be well entrenched.)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">AHP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Analytic hierarchy process (AHP), a structured technique developed by Thomas Saaty in the 1970s for organizing and analyzing complex decisions\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">AIQ\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Analytical instrument qualification, a term used in the pharmaceutical industry for the process of ensuring that an instrument meets the requirements for its intended application\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Apache Hadoop YARN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Yet Another Resource Negotiator (YARN), a technology for managing resources on a cluster of computers\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Apache Lucene\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A software library for information retrieval from fields of text contained within document files\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">ASE\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Adaptive Server Enterprise\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Audit_trail\" title=\"Audit trail\" target=\"_blank\" class=\"wiki-link\" data-key=\"96a617b543c5b2f26617288ba923c0f0\">audit trail<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A log of records documenting the sequence of activities that have been performed on a system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">binary package distribution\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A compilation of the compiled version of a program and all related documentation designed for release to the end user\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">BREW\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Binary Runtime Environment for Wireless, a runtime and application development environment from Qualcomm that isolates portable applications from the hardware interface of mobile phones employing code division multiplex access (CDMA)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">BRR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Business readiness rating\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Chromatography_data_management_system\" title=\"Chromatography data management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"e0b59eaaca214b3b757b613af60743ad\">CDMS<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Chromatography data management system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">CFR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">United States Code of Federal Regulations\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">change log\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A log documenting the changes made to a software product, which may include a list of new features, changes to behavior, or elimination of software bugs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Chain_of_custody\" title=\"Chain of custody\" target=\"_blank\" class=\"wiki-link\" data-key=\"6ba04fe2bb1c8375e133455821aa3894\">COC<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Chain of custody, a paper or electronic documentation trail that documents responsibility of a sample (This is required for legal reasons under various regulatory programs, and it also provides information used to track faulty or contaminated items back to their source.)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">committer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">The person that represents the quality control of the community, controlling what changes are included in the originally licensed version, though users are free to make any changes they want in their own copies of the program\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Compiere\" title=\"Compiere\" target=\"_blank\" class=\"wiki-link\" data-key=\"5670d0e1f6e0c09a531c2a8b9b2cb982\">Compiere<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A suite of open-source applications (for small- to medium-sized businesses) that provides a number of business support applications\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">copyleft\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Copyleft&oldid=674443822\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>Copyleft (a play on the word \"copyright\") is the practice of offering people the right to freely distribute copies and modified versions of a work with the stipulation that the same rights be preserved in derivative works down the line.\n<p>Copyleft is a form of licensing, and can be used to maintain copyright conditions for works ranging from computer software, to documents, to art. In general, copyright law is used by an author to prohibit recipients from reproducing, adapting, or distributing copies of their work. In contrast, under copyleft, an author may give every person who receives a copy of the work permission to reproduce, adapt, or distribute it, with the accompanying requirement that any resulting copies or adaptations are also bound by the same licensing agreement.\n<\/p><p>Copyleft licenses (for software) require that information necessary for reproducing and modifying the work must be made available to recipients of the binaries. The source code files will usually contain a copy of the license terms and acknowledge the author(s).\n<\/p>\nCopyleft type licenses are a novel use of existing copyright law to ensure a work remains freely available. The GNU General Public License, originally written by Richard Stallman, was the first software copyleft license to see extensive use, and continues to dominate in that area. Creative Commons, a non-profit organization founded by Lawrence Lessig, provides a similar license provision condition called ShareAlike.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Cosmos\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">C# Open Source Managed Operating System\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">COTS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Commercial, off-the-shelf\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/40_CFR_Part_3\" title=\"40 CFR Part 3\" target=\"_blank\" class=\"wiki-link\" data-key=\"f6e16cb8df3738dee667b175cc486cef\">CROMERR<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Cross-Media Electronic Reporting Rule (or typically 40 CFR Part 3), an Environmental Protection Agency (EPA) rule specifying how electronic reporting should be performed for the EPA's various regulatory programs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Cygwin\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A collection of tools that emulate a Linux environment, allowing Linux applications to be compiled for and executed in a MS Windows environment\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">data loading\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">The loading of data into static tables, including test definitions, sample container descriptions, location information, etc.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">database-agnostic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Indicates an application capable of running with database systems from any vendor\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">digital commons\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Name given to a collaboratively developed online resource that is managed by a community of people\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">DMOZ\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Also known as the Open Directory Project (ODP), an attempt to create the largest human curated open-content directory of web links (Originates from one of its earlier domain names, directory.mozilla.org)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Docker\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A system designed to package an application with all of its dependencies into a standardized software container using an alternate architectural approach than a virtual machine, while providing similar resource isolation in a smaller footprint\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">documentation, administrator\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Documentation for a system designed to be issued to a system administrator, providing information on configuring and operating the application\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">documentation, developer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Documentation for a system designed to be issued to a system developer, providing detailed information on how the system operates and is structured as well as how to alter the system code and extract data from the system in an <i>ad hoc<\/i> manner\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">documentation, user\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Documentation for a system designed to be issued to a user, helping to guide the user through using the application\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">DOI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Digital object identifier, a character string identifier used to uniquely identify an electronic document (In many instances a scientific\/technical publication will have a DOI printed on it, allowing you to access or purchase the article on-line; standardized under ISO 26324.)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Drizzle\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A MySQL 6.0-derived database optimized for cloud computing\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Drupal\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An open-source application designed for creating and managing a variety of web sites\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Eclipse\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A Java-based integrated development environment that can be customized via use of community-developed plug-ins\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">EDD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Electronic data delivery or electronic data deliverable\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">EHR<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Electronic health record\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Elasticsearch\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An open-source search engine employing a RESTful (see \"REST\") interface, built on Apache Lucene\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" target=\"_blank\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">ELN<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Electronic laboratory notebook\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">EMR<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Electronic medical record\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.epa.gov\/\" target=\"_blank\">EPA<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">United States Environmental Protection Agency\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">ERP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Enterprise resource planning, typically constituting a variety of integrated applications, with a shared database, that integrates critical business functions such as accounting, human resources, customer relationship management, inventory and order tracking, etc. into a single system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">F\/OSS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Free\/open-source software (See \"FOSS\")\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">FAME\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Filter, Analyze, Measure, and Evaluate (FAME) methodology for evaluating open-source applications\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.fda.gov\/\" target=\"_blank\">FDA<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">United States Food and Drug Administration\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Fedora\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A version of the Linux operating system sponsored by Red Hat\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Firefox\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">One of a number of open-source web browsers that can be highly customized via the use of member-developed \"plug-ins\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">FLOSS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Alternative_terms_for_free_software&oldid=679497590\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>The acronym FLOSS was coined in 2001 by Rishab Aiyer Ghosh for \"free\/libre and open-source software.\" Later that year, the European Commission (EC) used the phrase when they funded a study on the topic.\nUnlike \"libre software,\" which aimed to solve the ambiguity problem, FLOSS aimed to avoid taking sides in the debate over whether it was better to say \"free software\" or to say \"open-source software\"<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">FLOSShub\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/flosshub.org\/node\/9\" target=\"_blank\">Per FLOSS Research Group<\/a>:\n<blockquote>FLOSShub is a portal for free\/libre and open source software (FLOSS) research resources and discussion. FLOSShub's goal is to provide a central location for connecting researchers and FLOSS community members to research papers, data, tools, and most importantly, community.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Forge\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Forge_%28software%29&oldid=674218812\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>In FLOSS development communities, a forge is a web-based collaborative software platform for both developing and sharing computer applications. (The word derives from the metalworking forge, used for shaping metal parts.) A forge platform is generally able to host multiple independent projects.\n<p>For software developers it is a place to host, among others, source code (often version-controlled), bug database and documentation for their projects. For users, a forge is a repository of computer applications.\n<\/p><p>Software forges have become popular, and have proven successful as a software development model for a large number of software projects.\n<\/p>\nThe term forge refers to a common prefix or suffix adopted by various platforms created after the example of SourceForge (such as GForge and FusionForge).<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">forking\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">The process where a group takes an open-source application's source code and starts developing it in an independent direction from the original program\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">FOSS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Free_and_open-source_software&oldid=673395320\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>Free and open-source software (FOSS) is computer software that can be classified as both free software and open-source software. That is, anyone is freely licensed to use, copy, study, and change the software in any way, and the source code is openly shared so that people are encouraged to voluntarily improve the design of the software. This is in contrast to proprietary software, where the software is under restrictive copyright and the source code is usually hidden from the users.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">free software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">See \"software, free\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Free Software Foundation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Free_Software_Foundation&oldid=672578474\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>The Free Software Foundation (FSF) is a 501(c)(3) non-profit organization founded by Richard Stallman on 4 October 1985 to support the free software movement, which promotes the universal freedom to study, distribute, create, and modify computer software, with the organization's preference for software being distributed under copyleft (\"share alike\") terms, such as with its own GNU General Public License. The FSF was incorporated in Massachusetts, USA, where it is also based. \n<p>From its founding until the mid-1990s, FSF's funds were mostly used to employ software developers to write free software for the GNU Project. Since the mid-1990s, the FSF's employees and volunteers have mostly worked on legal and structural issues for the free software movement and the free software community. \n<\/p>\nConsistent with its goals, only free software is used on the FSF's computers.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">functional requirement\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Functional_requirement&oldid=669297630\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>In software engineering (and systems engineering), a functional requirement defines a function of a system and its components. A function is described as a set of inputs, the behavior, and outputs (see also software).\nFunctional requirements may be calculations, technical details, data manipulation and processing and other specific functionality that define what a system is supposed to accomplish. Behavioral requirements describing all the cases where the system uses the functional requirements are captured in use cases. Functional requirements are supported by non-functional requirements (also known as quality requirements), which impose constraints on the design or implementation (such as performance requirements, security, or reliability). Generally, functional requirements are expressed in the form \"system must do <requirement>\", while non-functional requirements are \"system shall be <requirement>\". The plan for implementing functional requirements is detailed in the system design. The plan for implementing non-functional requirements is detailed in the system architecture.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Good_Automated_Laboratory_Practices\" title=\"Good Automated Laboratory Practices\" target=\"_blank\" class=\"wiki-link\" data-key=\"bef4ea1fa3e792ccf7471f9f09b804e6\">GALP<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Good Automated Laboratory Practices\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">GCP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Good Clinical Practice\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">GCP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Good Laboratory Practice\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">GMP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Good Manufacturing Practice\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">GNU\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">GNU\u2019s Not Unix, a project to create a FLOSS operating system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">GNU General Public License\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=GNU_General_Public_License&oldid=675202761\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>The GNU General Public License (GNU GPL or GPL) is the most widely used free software license, which guarantees end users (individuals, organizations, companies) the freedoms to run, study, share (copy), and modify the software. Software that allows these rights is called free software and, if the software is copylefted, requires those rights to be retained. The GPL demands both. The license was originally written by Richard Stallman of the Free Software Foundation (FSF) for the GNU project.\nIn other words, the GPL grants the recipients of a computer program the rights of the Free Software Definition and uses copyleft to ensure the freedoms are preserved whenever the work is distributed, even when the work is changed or added to. The GPL is a copyleft license, which means that derived works can only be distributed under the same license terms. This is in distinction to permissive free software licenses, of which the BSD licenses and the MIT License are the standard examples. GPL was the first copyleft license for general use.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">GNU Project\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=GNU_Project&oldid=670291710\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>The GNU Project is a free software, mass collaboration project, announced on 27 September 1983, by Richard Stallman at MIT. Its aim is to give computer users freedom and control in their use of their computers and computing devices, by collaboratively developing and providing software that is based on the following freedom rights: users are free to run the software, share it (copy, distribute), study it and modify it. GNU software guarantees these freedom-rights legally (via its license), and is therefore free software; the use of the word \"free\" always being taken to refer to freedom.<br \/> <br \/>\nIn order to ensure that the entire software of a computer grants its users all freedom rights (use, share, study, modify), even the most fundamental and important part, the operating system (including all its numerous utility programs), needed to be written. The founding goal of the project was, in the words of its initial announcement, to develop \"a sufficient body of free software [...] to get along without any software that is not free.\" Stallman decided to call this operating system GNU (a recursive acronym meaning \"GNU's not Unix\"), basing its design on that of Unix; however, in contrast to Unix which was proprietary software, GNU was to be freedom-respecting software (free software) that users can use, share, study and modify. Development was initiated in January 1984. The goal of making a completely free software operating system was achieved in 1992 when the third-party Linux kernel was released as free software, under version 2 of the GNU General Public License, to be used with the GNU software stack.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Google Scholar\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A version of Google search optimized for searching and retrieving scholarly papers\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">GPL\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">See \"GNU General Public License\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">HDL\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">hardware description language\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" target=\"_blank\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">HL7<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A non-profit organization working to define standard methods for the exchange and retrieval of medical information\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/International_Electrotechnical_Commission\" title=\"International Electrotechnical Commission\" target=\"_blank\" class=\"wiki-link\" data-key=\"25341e03365f68cab3f247b7508a23e5\">IEC<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">International Electrotechnical Commission\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Term for an operating system command that is unknown to the operating system or processor\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/International_Organization_for_Standardization\" title=\"International Organization for Standardization\" target=\"_blank\" class=\"wiki-link\" data-key=\"116defc5d89c8a55f5b7c1be0790b442\">ISO<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">International Organization for Standardization, an independent, non-governmental membership organization and the world's largest developer of voluntary International Standards\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Jabber\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An XML based instant messaging platform\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">LIMS<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Laboratory information management system, an informatics system designed to track samples and analytical results through a laboratory\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_system\" title=\"Laboratory information system\" target=\"_blank\" class=\"wiki-link\" data-key=\"37add65b4d1c678b382a7d4817a9cf64\">LIS<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Laboratory information system, an informatics system designed to track samples and test results through a clinical laboratory\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">LMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Library management system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Microsoft Academic Search\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An experimental Microsoft search project focusing on scientific and technical information\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">modular\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A design process where an application is broken down into smaller parts, frequently grouped by function\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">MongoDB\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An open-source NoSQL document-oriented database\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">native code\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A program that is designed to run with a specific computer's hardware program codes, on a different computer\/processor via the use of an emulator\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">non-functional requirements\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Requirements describing how the system works and how it should behave\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.oalib.com\/\" target=\"_blank\">OALib<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Open Access Library, an open-access search engine and journal\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">ODBC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Open Database Connectivity, a middleware API that allows an application to connect with any database\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">ODF\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Open Document Format, an XML based open-source document format developed for office suite use, prominent in the OpenOffice and LibreOffice projects\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">ODP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Open Directory Project; see \"DMOZ\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OSI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Open Source Initiative, a global non-profit organization promoting open source and preventing abuse of the open source concept; also the steward of the Open Source Definition (OSD)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OpenDOAR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Directory of Open Access Repositories\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OpenLogic Exchange\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Software-as-a-Service (SaaS) governance platform for comprehensive governance and provisioning of open-source software\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OpenSSL\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A general-purpose encryption library used in many web servers that provides an implementation of the Internet's Secure Sockets Layer (SSL) and Transport Layer Security (TLS) protocols\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/openvpn.net\/\" target=\"_blank\">OpenVPN<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Open-source virtual private network (VPN) software; see \"VPN\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OPML\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Outline Processor Markup Language, an XML-based file format used for creating outlines\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Operating system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OSDL\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Open Source Development Labs, conceived as a facility to allow open source developers to collaborate to create standardized implementations of Linux and Linux applications by IBM, Intel, and Computer Associates\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">OSS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Open-source_software&oldid=673820216\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>Open-source software (OSS) is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change, and distribute the software to anyone and for any purpose. Open-source software may be developed in a collaborative public manner. Open-source software is the most prominent example of open-source development and often compared to (technically defined) user-generated content or (legally defined) open-content movements.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/oss-watch.ac.uk\/\" target=\"_blank\">OSS Watch<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An independent non-advocacy service on free and open-source software\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">PDA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Parenteral Drug Association\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">PDF\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Adobe Portable Document Format\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Perl\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An interpreted script programming language invented by Larry Wall that is popular for creating common gateway interface (CGI) programs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">precision\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">How closely a set of measurements agree with their average\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">proprietary software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">See \"software, proprietary\"\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/opensourceprojects.eu\/\" target=\"_blank\">PROSE<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Promoting Open Source in Europe\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">QA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Quality assurance, responsible for auditing the QC process and ensuring that all SOPs and standards are being followed\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">QC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Quality control, responsible for ensuring all parts of a process are within designed specifications (designated testing and inspection)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">QMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Quality management system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">QSOS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Qualification and Selection Open Source, a methodology for evaluating open-source applications backed by Atos Origin\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">REST\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Representational state transfer (occasionally ReST), an architectural style commonly used for APIs that is defined by six constraints: uniform interface, stateless, cacheable, client-server, layered system, and code on demand\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.researchgate.net\/\" target=\"_blank\">ResearchGate<\/a>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A professional network allowing scientists and researchers to exchange papers and collaborate\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">revenue trigger\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A method of bringing in money to support program development, appearing as ad sales to training, consulting contracts, and\/or support subscriptions\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">RFI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Request for information, a document sent by potential customers to vendors requesting detailed information regarding their system, usually in relation to selecting a hardware or software system to acquire\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">RFID\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Radio frequency identification device\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">RFP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Request for proposal, a document sent to vendors specifying application requirements and requesting a proposal of what they can supply\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Ruby\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An open-source object-oriented scripting\/programming language created by Yukihiro Matsumoto\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Samba\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An implementation of Server Message Block (SMB) and Common Internet File System (CIFS) client\/server protocols that allows shared access of resources over a network\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">shareware\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Shareware&oldid=664844862\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>Shareware is a type of proprietary software which is provided (initially) free of charge to users, who are allowed and encouraged to make and share copies of the program, which helps to distribute it. The word \"shareware\" is a portmanteau combining the words \"share\" and \"software\". Shareware is often offered as a download from an Internet website or as a compact disc included with a magazine.\n<p>There are many types of shareware, and while they may not require an initial up-front payment, all are intended to generate revenue in one way or another. Some limit use to personal non-commercial purposes only, with purchase of a license required for use in a business enterprise. The software itself may be limited in functionality or be time-limited. Or it may remind you that payment would be appreciated.\n<\/p><p>Shareware is available on all major personal computer platforms. Titles cover a very wide range of categories including: business, software development, education, home, multimedia, design, drivers, games, and utilities. Because of its minimal overhead and low cost, the shareware model is often the only one practical for distributing non-free software for abandoned or orphaned platforms such as the Atari ST and Amiga.\n<\/p>\nThe term shareware is used in contrast to open-source software, in which the source code is available for anyone to inspect and alter, and freeware, which is software distributed at no cost to the user but without source code being made available. Note that two types of shareware, donationware and freemiums, are also types of freeware.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Smalltalk\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An experimental language developed at Xerox in the 1970s to investigate the concept of object-oriented programming\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">SME\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Subject matter expert, a person who is an authority on the particular subject area or topic that a software application addresses\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">software, free\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Free_software&oldid=677757698\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>Free software, software libre, or libre software is computer software that gives users the freedom to run the software for any purpose as well as to study, modify, and distribute the original software and the adapted versions. The rights to study and modify free software imply unfettered access to its source code. For computer programs which are covered by copyright law this is achieved with a software license where the author grants users the aforementioned freedoms. Software which is not covered by copyright law, such as software in the public domain can also be free if the source code is in the public domain (or otherwise available without restrictions). Other legal and technical aspects such as software patents and DRM may impede users from exercising these rights, and thus prevent software from being free. Free software may be developed collaboratively by volunteer computer programmers or by corporations; as part of a commercial activity or not.<br \/> <br \/>\nFree software is primarily a matter of liberty, not price: users, individually or collectively, are free to do whatever they want with it \u2013 this includes the freedom to redistribute the software free of charge, or to sell it (or related services such as support or warranty) for profit. Free software thus differs from proprietary software (such as Microsoft Windows), which to varying degrees prevents users from studying, modifying and sharing the software. Free software is also distinct from freeware, which is simply a category of proprietary software which does not require payment for use. Proprietary software (including freeware) uses restrictive software licences or user agreements and usually does not provide access to the source code. Users are thus prevented from modifying the software, and this results in the user becoming dependent on software companies to provide updates and support (vendor lock-in). Users can also not necessarily reverse engineer, modify, or redistribute proprietary software.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">software, open-source\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Open-source_software&oldid=678115562\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>Open-source software (OSS) is computer software with its source code made available with a license in which the copyright holder provides the rights to study, change, and distribute the software to anyone and for any purpose. Open-source software may be developed in a collaborative public manner. Open-source software is the most prominent example of open-source development and often compared to (technically defined) user-generated content or (legally defined) open-content movements.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">software, proprietary\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Proprietary_software&oldid=673807377\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>Proprietary software, non-free software (in the sense of missing freedoms), or closed-source software is software, where the developers or distributors reserve all freedoms and rights.\n<p>Among the freedoms and rights that proprietary software deprives (to end-users), are:\n<\/p>\n<ul><li> the freedom to analyze the software, and to change it (often deprived through intentional non-availability of sourcecode, or through Non-disclosure agreements (NDA))<\/li>\n<li> the freedom to share the software (often deprived through copy prohibition via EULA (End User License Agreement) or NDA)<\/li>\n<li> the freedom to run the software for any purpose (often deprived through user-restrictions via EULA).<\/li><\/ul>\n<p>In contrast to proprietary software, free software, is software that grants a user all these freedoms, on reception of the software.\n<\/p><p>Proprietary software is licensed under legal right of the copyright holder, with the intent that the licensee is given the right to use the software only under certain conditions, and restricted from other uses, such as modification, sharing, studying, redistribution, or reverse engineering. Usually the source code of proprietary software is not made available.\n<\/p><p>Complementary terms include free software, licensed by the owner under more permissive terms, and public domain software, which is not subject to copyright and can be used for any purpose. Proponents of free and open-source software use proprietary or non-free to describe software that is not free or open-source.\n<\/p>\nA related, but distinct categorization in the software industry is commercial software, which refers to software produced for sale, but without meaning it is closed-source.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">SOP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Standard operating procedure\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">source code\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A version of the program in human-readable form\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">source package distribution\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A consolidation of all of the source code and documentation relevant to a specific software release intended for the application developer and\/or maintainer\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">SourceForge\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A software repository on the web providing support for open source developers and for distribution of their application\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">spaghetti code\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A contemptuous phrase applied to unstructured applications, particularly those with GOTO statements or other control structures bouncing control around different portions of the program\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">SQA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Software quality assurance, responsible for auditing the SQC process and ensuring that all SOPs and standards are being followed\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">SQC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Software quality control, responsible for ensuring all parts of a process were within designed specifications (designated testing and inspection)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">SQL\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Structured Query Language\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Subversion\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">An open-source program designed by Karl Fogel and Ben Collins-Sussman to serve as a version control system that tracks changes made to files and folders\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">system suitability testing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Applied to analytical procedures, the idea that everything involved in the analysis (analytical equipment, data capture electronics, analytical procedures, and analytical samples) constitutes a system and can be evaluated as such\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">tcpdump\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A command line tool for monitoring network traffic, also useful in troubleshooting network problems\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">Tsunami UDP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A file transfer protocol enabling high-speed data transfers over networks with large end-to-end delays, often used for bulk data transfers\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">URI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Uniform_Resource_Identifier&oldid=679010464\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>In computing, a uniform resource identifier (URI) is a string of characters used to identify the name of a resource. Such identification enables interaction with representations of the resource over a network, typically the World Wide Web, using specific protocols. Schemes specifying a concrete syntax and associated protocols define each URI. The most common form of URI is the uniform resource locator (URL), frequently referred to informally as a web address. More rarely seen in usage is the uniform resource name (URN), which was designed to complement URLs by providing a mechanism for the identification of resources in particular namespaces.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">URL\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Universal resource locator, a specific type of uniform resource identifier that identifies objects on the World Wide Web, including their address\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">URS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">User requirements specification, a document that lists all of the requirements that the users require the system to support, frequently considered the key document in regards to the development life cycle of the application\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">user experience\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">The overall aspects that an end-user experiences when interacting with the application, distinct from the functionality of the user interface\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">USP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.usp.org\/about-usp\" target=\"_blank\">Per the USP<\/a>:\n<blockquote>The U.S. Pharmacopeial Convention (USP) is a scientific nonprofit organization that sets standards for the identity, strength, quality, and purity of medicines, food ingredients, and dietary supplements manufactured, distributed and consumed worldwide. USP\u2019s drug standards are enforceable in the United States by the Food and Drug Administration, and these standards are used in more than 140 countries.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">validation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A process of confirming that the application works as designed\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">vendor-neutral\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A business approach designed to avoid lock-in with a particular supplier and ensure broad compatibility and interchangeability of products and technologies\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">versioning\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">The assignment of a unique name or number to a given compilation of software, frequently in terms of a major and minor version number\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">VM\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Virtual machine, a program running on another computer that emulates a full computer system (Originally developed to allow more efficient use of idle time on existing hardware servers)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">VPN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Virtual_private_network&oldid=685070408\" target=\"_blank\">Per <i>Wikipedia<\/i><\/a>:\n<blockquote>A virtual private network (VPN) extends a private network across a public network, such as the Internet. It enables users to send and receive data across shared or public networks as if their computing devices were directly connected to the private network, and thus are benefiting from the functionality, security and management policies of the private network. A VPN is created by establishing a virtual point-to-point connection through the use of dedicated connections, virtual tunneling protocols, or traffic encryption.<\/blockquote>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">WAN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Wide area network, a computer network connecting multiple locations, as opposed to a local area network (LAN) that services one location\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">wiki\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">A server program designed to promote collaborating development and used in a diverse variety of ways, including system documentation\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">WLAN\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Wireless local area network\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">XBRL\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Extensible Business Reporting Language\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px; vertical-align: text-top;\">XML\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-bottom:10px;\">Extensible Markup Language\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-FSFWhat15-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FSFWhat15_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.gnu.org\/philosophy\/free-sw.html\" target=\"_blank\">\"What is free software?\"<\/a>. <i>GNU Project<\/i>. Free Software Foundation, Inc. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.gnu.org\/philosophy\/free-sw.html\" target=\"_blank\">http:\/\/www.gnu.org\/philosophy\/free-sw.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+is+free+software%3F&rft.atitle=GNU+Project&rft.date=2015&rft.pub=Free+Software+Foundation%2C+Inc&rft_id=http%3A%2F%2Fwww.gnu.org%2Fphilosophy%2Ffree-sw.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSIDef15-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OSIDef15_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-OSIDef15_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/opensource.org\/osd\" target=\"_blank\">\"The Open Source Definition\"<\/a>. Open Source Initiative. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensource.org\/osd\" target=\"_blank\">http:\/\/opensource.org\/osd<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Open+Source+Definition&rft.atitle=&rft.date=2015&rft.pub=Open+Source+Initiative&rft_id=http%3A%2F%2Fopensource.org%2Fosd&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Schie.C3.9FleFree12-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Schie.C3.9FleFree12_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Schie\u00dfle, Bj\u00f6rn (12 August 2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/fsfe.org\/freesoftware\/basics\/comparison.en.html\" target=\"_blank\">\"Free Software, Open Source, FOSS, FLOSS - same same but different\"<\/a>. Free Software Foundation Europe<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/fsfe.org\/freesoftware\/basics\/comparison.en.html\" target=\"_blank\">https:\/\/fsfe.org\/freesoftware\/basics\/comparison.en.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 5 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Free+Software%2C+Open+Source%2C+FOSS%2C+FLOSS+-+same+same+but+different&rft.atitle=&rft.aulast=Schie%C3%9Fle%2C+Bj%C3%B6rn&rft.au=Schie%C3%9Fle%2C+Bj%C3%B6rn&rft.date=12+August+2012&rft.pub=Free+Software+Foundation+Europe&rft_id=https%3A%2F%2Ffsfe.org%2Ffreesoftware%2Fbasics%2Fcomparison.en.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DateRepOSS12-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DateRepOSS12_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/events.linuxfoundation.org\/images\/stories\/pdf\/lceu2012_date.pdf\" target=\"_blank\">\"RepOSS: A Flexible OSS Assessment Repository\"<\/a> (PDF). Northeast Asia OSS Promotion Forum WG3. 05 November 2012<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/events.linuxfoundation.org\/images\/stories\/pdf\/lceu2012_date.pdf\" target=\"_blank\">http:\/\/events.linuxfoundation.org\/images\/stories\/pdf\/lceu2012_date.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 05 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=RepOSS%3A+A+Flexible+OSS+Assessment+Repository&rft.atitle=&rft.date=05+November+2012&rft.pub=Northeast+Asia+OSS+Promotion+Forum+WG3&rft_id=http%3A%2F%2Fevents.linuxfoundation.org%2Fimages%2Fstories%2Fpdf%2Flceu2012_date.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Doll10Mil13-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Doll10Mil13_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Doll, Brian (23 December 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/blog\/1724-10-millionrepositories\" target=\"_blank\">\"10 Million Repositories\"<\/a>. GitHub, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/blog\/1724-10-millionrepositories\" target=\"_blank\">https:\/\/github.com\/blog\/1724-10-millionrepositories<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=10+Million+Repositories&rft.atitle=&rft.aulast=Doll%2C+Brian&rft.au=Doll%2C+Brian&rft.date=23+December+2013&rft.pub=GitHub%2C+Inc&rft_id=https%3A%2F%2Fgithub.com%2Fblog%2F1724-10-millionrepositories&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-6\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sarrab, Mohamed; Elsabir, Mahmoud; Elgamel, Laila (March 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ijcsi.org\/papers\/IJCSI-10-2-3-464-469.pdf\" target=\"_blank\">\"The Technical, Non-technical Issues and the Challenges of Migration to Free and Open Source Software\"<\/a> (PDF). <i>IJCSI International Journal of Computer Science Issues<\/i> <b>10<\/b> (2.3)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ijcsi.org\/papers\/IJCSI-10-2-3-464-469.pdf\" target=\"_blank\">http:\/\/ijcsi.org\/papers\/IJCSI-10-2-3-464-469.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Technical%2C+Non-technical+Issues+and+the+Challenges+of+Migration+to+Free+and+Open+Source+Software&rft.jtitle=IJCSI+International+Journal+of+Computer+Science+Issues&rft.aulast=Sarrab%2C+Mohamed%3B+Elsabir%2C+Mahmoud%3B+Elgamel%2C+Laila&rft.au=Sarrab%2C+Mohamed%3B+Elsabir%2C+Mahmoud%3B+Elgamel%2C+Laila&rft.date=March+2013&rft.volume=10&rft.issue=2.3&rft_id=http%3A%2F%2Fijcsi.org%2Fpapers%2FIJCSI-10-2-3-464-469.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheelerFree11-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WheelerFree11_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wheeler, David A. (14 June 2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.dwheeler.com\/essays\/commercial-floss.html\" target=\"_blank\">\"Free-Libre \/ Open Source Software (FLOSS) is Commercial Software\"<\/a>. <i>dwheeler.com<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dwheeler.com\/essays\/commercial-floss.html\" target=\"_blank\">http:\/\/www.dwheeler.com\/essays\/commercial-floss.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Free-Libre+%2F+Open+Source+Software+%28FLOSS%29+is+Commercial+Software&rft.atitle=dwheeler.com&rft.aulast=Wheeler%2C+David+A.&rft.au=Wheeler%2C+David+A.&rft.date=14+June+2011&rft_id=http%3A%2F%2Fwww.dwheeler.com%2Fessays%2Fcommercial-floss.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StolAComp10-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-StolAComp10_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-StolAComp10_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Stol, Klaas-Jan; Ali Babar, Muhammad (2010). \"A Comparison Framework for Open Source Software Evaluation Methods\". In \u00c5gerfalk, P.J.; Boldyreff, C.; Gonz\u00e1lez-Barahona, J.M.; Madey, G.R.; Noll, J. <i>Open Source Software: New Horizons<\/i>. Springer. pp. 389\u2013394. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-13244-5_36\" target=\"_blank\">10.1007\/978-3-642-13244-5_36<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9783642132445.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+Comparison+Framework+for+Open+Source+Software+Evaluation+Methods&rft.atitle=Open+Source+Software%3A+New+Horizons&rft.aulast=Stol%2C+Klaas-Jan%3B+Ali+Babar%2C+Muhammad&rft.au=Stol%2C+Klaas-Jan%3B+Ali+Babar%2C+Muhammad&rft.date=2010&rft.pages=pp.%26nbsp%3B389%E2%80%93394&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-642-13244-5_36&rft.isbn=9783642132445&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FritzAClass94-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FritzAClass94_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Fritz, Catherine A.; Carter, Bradley D. (23 August 1994). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.55.4470\" target=\"_blank\"><i>A Classification And Summary Of Software Evaluation And Selection Methodologies<\/i><\/a>. Mississippi State, MS: Department of Computer Science, Mississippi State University<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.55.4470\" target=\"_blank\">http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.55.4470<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=A+Classification+And+Summary+Of+Software+Evaluation+And+Selection+Methodologies&rft.aulast=Fritz%2C+Catherine+A.%3B+Carter%2C+Bradley+D.&rft.au=Fritz%2C+Catherine+A.%3B+Carter%2C+Bradley+D.&rft.date=23+August+1994&rft.place=Mississippi+State%2C+MS&rft.pub=Department+of+Computer+Science%2C+Mississippi+State+University&rft_id=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fsummary%3Fdoi%3D10.1.1.55.4470&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OpenBRROpen2005-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OpenBRROpen2005_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-OpenBRROpen2005_10-1\" rel=\"external_link\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/docencia.etsit.urjc.es\/moodle\/file.php\/125\/OpenBRR_Whitepaper.pdf\" target=\"_blank\">\"OpenBRR, Business Readiness Rating for Open Source: A Proposed Open Standard to Facilitate Assessment and Adoption of Open Source Software\"<\/a> (PDF). OpenBRR. 2005<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/docencia.etsit.urjc.es\/moodle\/file.php\/125\/OpenBRR_Whitepaper.pdf\" target=\"_blank\">http:\/\/docencia.etsit.urjc.es\/moodle\/file.php\/125\/OpenBRR_Whitepaper.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OpenBRR%2C+Business+Readiness+Rating+for+Open+Source%3A+A+Proposed+Open+Standard+to+Facilitate+Assessment+and+Adoption+of+Open+Source+Software&rft.atitle=&rft.date=2005&rft.pub=OpenBRR&rft_id=http%3A%2F%2Fdocencia.etsit.urjc.es%2Fmoodle%2Ffile.php%2F125%2FOpenBRR_Whitepaper.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WassermanTheBus06-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WassermanTheBus06_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wasserman, A.I.; Pal, M.; Chan, C. (10 June 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/web.archive.org\/web\/20070111113722\/http:\/\/www.openbrr.org\/comoworkshop\/papers\/WassermanPalChan_EFOSS06.pdf\" target=\"_blank\">\"The Business Readiness Rating: a Framework for Evaluating Open Source\"<\/a> (PDF). <i>Proceedings of the Workshop on Evaluation Frameworks for Open Source Software (EFOSS) at the Second International Conference on Open Source Systems<\/i>. Lake Como, Italy. pp. 1\u20135. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.openbrr.org\/comoworkshop\/papers\/WassermanPalChan_EFOSS06.pdf\" target=\"_blank\">the original<\/a> on 11 January 2007<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/web.archive.org\/web\/20070111113722\/http:\/\/www.openbrr.org\/comoworkshop\/papers\/WassermanPalChan_EFOSS06.pdf\" target=\"_blank\">http:\/\/web.archive.org\/web\/20070111113722\/http:\/\/www.openbrr.org\/comoworkshop\/papers\/WassermanPalChan_EFOSS06.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Business+Readiness+Rating%3A+a+Framework+for+Evaluating+Open+Source&rft.atitle=Proceedings+of+the+Workshop+on+Evaluation+Frameworks+for+Open+Source+Software+%28EFOSS%29+at+the+Second+International+Conference+on+Open+Source+Systems&rft.aulast=Wasserman%2C+A.I.%3B+Pal%2C+M.%3B+Chan%2C+C.&rft.au=Wasserman%2C+A.I.%3B+Pal%2C+M.%3B+Chan%2C+C.&rft.date=10+June+2006&rft.pages=pp.+1%E2%80%935&rft.place=Lake+Como%2C+Italy&rft_id=http%3A%2F%2Fweb.archive.org%2Fweb%2F20070111113722%2Fhttp%3A%2F%2Fwww.openbrr.org%2Fcomoworkshop%2Fpapers%2FWassermanPalChan_EFOSS06.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MajchrowskiAnOp08-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MajchrowskiAnOp08_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Majchrowski, Annick; Deprez, Jean-Christophe (2008). \"An Operational Approach for Selecting Open Source Components in a Software Development Project\". In O'Connor, R.; Baddoo, N.; Smolander, K.; Messnarz, R.. <i>Software Process Improvement<\/i>. Springer. pp. 176\u2013188. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-540-85936-9_16\" target=\"_blank\">10.1007\/978-3-540-85936-9_16<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9783540859369.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=An+Operational+Approach+for+Selecting+Open+Source+Components+in+a+Software+Development+Project&rft.atitle=Software+Process+Improvement&rft.aulast=Majchrowski%2C+Annick%3B+Deprez%2C+Jean-Christophe&rft.au=Majchrowski%2C+Annick%3B+Deprez%2C+Jean-Christophe&rft.date=2008&rft.pages=pp.%26nbsp%3B176%E2%80%93188&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-540-85936-9_16&rft.isbn=9783540859369&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PetrinjaIntro09-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PetrinjaIntro09_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Petrinja, E.; Nambakam, R.; Sillitti, A. (2009). \"Introducing the Open Source Maturity Model\". <i>ICSE Workshop on Emerging Trends in Free\/Libre\/Open Source Software Research and Development, 2009<\/i>. IEEE. pp. 37\u201341. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FFLOSS.2009.5071358\" target=\"_blank\">10.1109\/FLOSS.2009.5071358<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781424437207.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Introducing+the+Open+Source+Maturity+Model&rft.atitle=ICSE+Workshop+on+Emerging+Trends+in+Free%2FLibre%2FOpen+Source+Software+Research+and+Development%2C+2009&rft.aulast=Petrinja%2C+E.%3B+Nambakam%2C+R.%3B+Sillitti%2C+A.&rft.au=Petrinja%2C+E.%3B+Nambakam%2C+R.%3B+Sillitti%2C+A.&rft.date=2009&rft.pages=pp.%26nbsp%3B37%E2%80%9341&rft.pub=IEEE&rft_id=info:doi\/10.1109%2FFLOSS.2009.5071358&rft.isbn=9781424437207&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Deprezcomp08-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Deprezcomp08_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-Deprezcomp08_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Deprez,Jean-Christophe; Alexandre, Simon (2008). \"Comparing Assessment Methodologies for Free\/Open Source Software: OpenBRR and QSOS\". In Jedlitschka, Andreas; Salo, Outi. <i>Product-Focused Software Process Improvement<\/i>. Springer. pp. 189-203. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-540-69566-0_17\" target=\"_blank\">10.1007\/978-3-540-69566-0_17<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9783540695660.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Comparing+Assessment+Methodologies+for+Free%2FOpen+Source+Software%3A+OpenBRR+and+QSOS&rft.atitle=Product-Focused+Software+Process+Improvement&rft.aulast=Deprez%2CJean-Christophe%3B+Alexandre%2C+Simon&rft.au=Deprez%2CJean-Christophe%3B+Alexandre%2C+Simon&rft.date=2008&rft.pages=pp.%26nbsp%3B189-203&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-540-69566-0_17&rft.isbn=9783540695660&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WassermanEval10-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WassermanEval10_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wasserman, Anthony I.; Pal, Murugan (2010). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20150218173146\/http:\/\/oss.sv.cmu.edu\/readings\/EvaluatingOSS_Wasserman.pdf\" target=\"_blank\">\"Evaluating Open Source Software\"<\/a> (PDF). Carnegie Mellon University - Silicon Valley. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/oss.sv.cmu.edu\/readings\/EvaluatingOSS_Wasserman.pdf\" target=\"_blank\">the original<\/a> on 18 February 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20150218173146\/http:\/\/oss.sv.cmu.edu\/readings\/EvaluatingOSS_Wasserman.pdf\" target=\"_blank\">https:\/\/web.archive.org\/web\/20150218173146\/http:\/\/oss.sv.cmu.edu\/readings\/EvaluatingOSS_Wasserman.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 31 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Evaluating+Open+Source+Software&rft.atitle=&rft.aulast=Wasserman%2C+Anthony+I.%3B+Pal%2C+Murugan&rft.au=Wasserman%2C+Anthony+I.%3B+Pal%2C+Murugan&rft.date=2010&rft.pub=Carnegie+Mellon+University+-+Silicon+Valley&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20150218173146%2Fhttp%3A%2F%2Foss.sv.cmu.edu%2Freadings%2FEvaluatingOSS_Wasserman.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JadhavEval09-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JadhavEval09_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jadhav, Anil S.; Sonar, Rajendra M. (March 2009). \"Evaluating and selecting software packages: A review\". <i>Information and Software Technology<\/i> <b>51<\/b> (3): 555\u2013563. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.infsof.2008.09.003\" target=\"_blank\">10.1016\/j.infsof.2008.09.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluating+and+selecting+software+packages%3A+A+review&rft.jtitle=Information+and+Software+Technology&rft.aulast=Jadhav%2C+Anil+S.%3B+Sonar%2C+Rajendra+M.&rft.au=Jadhav%2C+Anil+S.%3B+Sonar%2C+Rajendra+M.&rft.date=March+2009&rft.volume=51&rft.issue=3&rft.pages=555%E2%80%93563&rft_id=info:doi\/10.1016%2Fj.infsof.2008.09.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PaniAMeth10-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PaniAMeth10_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Pani, F.E.; Sanna, D. (11 June 2010). \"FAME, A Methodology for Assessing Software Maturity\". <i>Atti della IV Conferenza Italiana sul Software Libero<\/i>. Cagliari, Italy.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=FAME%2C+A+Methodology+for+Assessing+Software+Maturity&rft.atitle=Atti+della+IV+Conferenza+Italiana+sul+Software+Libero&rft.aulast=Pani%2C+F.E.%3B+Sanna%2C+D.&rft.au=Pani%2C+F.E.%3B+Sanna%2C+D.&rft.date=11+June+2010&rft.place=Cagliari%2C+Italy&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PaniTheFAMEApp10-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PaniTheFAMEApp10_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Pani, F.E.; Concas, G.; Sanna, D.; Carrogu, L. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wseas.us\/e-library\/conferences\/2010\/Catania\/TELE-INFO\/TELE-INFO-10.pdf\" target=\"_blank\">\"The FAME Approach: An Assessing Methodology\"<\/a>. In Niola, V.; Quartieri, J.; Neri, F.; Caballero, A.A.; Rivas-Echeverria, F.; Mastorakis, N. (PDF). <i>Proceedings of the 9th WSEAS International Conference on Telecommunications and Informatics<\/i>. Stevens Point, WI: WSEAS. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9789549260021<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.wseas.us\/e-library\/conferences\/2010\/Catania\/TELE-INFO\/TELE-INFO-10.pdf\" target=\"_blank\">http:\/\/www.wseas.us\/e-library\/conferences\/2010\/Catania\/TELE-INFO\/TELE-INFO-10.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+FAME+Approach%3A+An+Assessing+Methodology&rft.atitle=Proceedings+of+the+9th+WSEAS+International+Conference+on+Telecommunications+and+Informatics&rft.aulast=Pani%2C+F.E.%3B+Concas%2C+G.%3B+Sanna%2C+D.%3B+Carrogu%2C+L.&rft.au=Pani%2C+F.E.%3B+Concas%2C+G.%3B+Sanna%2C+D.%3B+Carrogu%2C+L.&rft.date=2010&rft.place=Stevens+Point%2C+WI&rft.pub=WSEAS&rft.isbn=9789549260021&rft_id=http%3A%2F%2Fwww.wseas.us%2Fe-library%2Fconferences%2F2010%2FCatania%2FTELE-INFO%2FTELE-INFO-10.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PaniTheFAMEtool10-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PaniTheFAMEtool10_19-0\" rel=\"external_link\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-PaniTheFAMEtool10_19-1\" rel=\"external_link\">19.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pani, F.E.; Concas, G.; Sanna, S.; Carrogu, L. (August 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wseas.us\/e-library\/transactions\/information\/2010\/88-137.pdf\" target=\"_blank\">\"The FAMEtool: an automated supporting tool for assessing methodology\"<\/a> (PDF). <i>WSEAS Transactions on Information Science and Applications<\/i> <b>7<\/b> (8): 1078\u20131089<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.wseas.us\/e-library\/transactions\/information\/2010\/88-137.pdf\" target=\"_blank\">http:\/\/www.wseas.us\/e-library\/transactions\/information\/2010\/88-137.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+FAMEtool%3A+an+automated+supporting+tool+for+assessing+methodology&rft.jtitle=WSEAS+Transactions+on+Information+Science+and+Applications&rft.aulast=Pani%2C+F.E.%3B+Concas%2C+G.%3B+Sanna%2C+S.%3B+Carrogu%2C+L.&rft.au=Pani%2C+F.E.%3B+Concas%2C+G.%3B+Sanna%2C+S.%3B+Carrogu%2C+L.&rft.date=August+2010&rft.volume=7&rft.issue=8&rft.pages=1078%E2%80%931089&rft_id=http%3A%2F%2Fwww.wseas.us%2Fe-library%2Ftransactions%2Finformation%2F2010%2F88-137.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PaniTrans10-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PaniTrans10_20-0\" rel=\"external_link\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-PaniTrans10_20-1\" rel=\"external_link\">20.1<\/a><\/sup> <sup><a href=\"#cite_ref-PaniTrans10_20-2\" rel=\"external_link\">20.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Pani, F.E.; Sanna, D.; Marchesi, M.; Concas, G. (2010). \"Transferring FAME, a Methodology for Assessing Open Source Solutions, from University to SMEs\". In D'Atri, A.; De Marco, M.; Braccini, A.M.; Cabiddu, F.. <i>Management of the Interconnected World<\/i>. Springer. pp. 495\u2013502. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-7908-2404-9_57\" target=\"_blank\">10.1007\/978-3-7908-2404-9_57<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9783790824049.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Transferring+FAME%2C+a+Methodology+for+Assessing+Open+Source+Solutions%2C+from+University+to+SMEs&rft.atitle=Management+of+the+Interconnected+World&rft.aulast=Pani%2C+F.E.%3B+Sanna%2C+D.%3B+Marchesi%2C+M.%3B+Concas%2C+G.&rft.au=Pani%2C+F.E.%3B+Sanna%2C+D.%3B+Marchesi%2C+M.%3B+Concas%2C+G.&rft.date=2010&rft.pages=pp.%26nbsp%3B495%E2%80%93502&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-7908-2404-9_57&rft.isbn=9783790824049&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SotoTheQual09-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SotoTheQual09_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Soto, M.; Ciolkowski, M. (2009). \"The QualOSS open source assessment model measuring the performance of open source communities\". <i>3rd International Symposium on Empirical Software Engineering and Measurement, 2009<\/i>. IEEE. pp. 498-501. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FESEM.2009.5314237\" target=\"_blank\">10.1109\/ESEM.2009.5314237<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781424448425.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+QualOSS+open+source+assessment+model+measuring+the+performance+of+open+source+communities&rft.atitle=3rd+International+Symposium+on+Empirical+Software+Engineering+and+Measurement%2C+2009&rft.aulast=Soto%2C+M.%3B+Ciolkowski%2C+M.&rft.au=Soto%2C+M.%3B+Ciolkowski%2C+M.&rft.date=2009&rft.pages=pp.%26nbsp%3B498-501&rft.pub=IEEE&rft_id=info:doi\/10.1109%2FESEM.2009.5314237&rft.isbn=9781424448425&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SotoTheQualProc09-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SotoTheQualProc09_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Soto, M.; Ciolkowski, M. (2009). \"The QualOSS Process Evaluation: Initial Experiences with Assessing Open Source Processes\". In O'Connor, R.; Baddoo, N.; Cuadrado-Gallego, J.J.; Rejas Muslera, R.; Smolander, K.; Messnarz, R.. <i>Software Process Improvement<\/i>. Springer. pp. 105\u2013116. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-04133-4_9\" target=\"_blank\">10.1007\/978-3-642-04133-4_9<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9783642041334.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+QualOSS+Process+Evaluation%3A+Initial+Experiences+with+Assessing+Open+Source+Processes&rft.atitle=Software+Process+Improvement&rft.aulast=Soto%2C+M.%3B+Ciolkowski%2C+M.&rft.au=Soto%2C+M.%3B+Ciolkowski%2C+M.&rft.date=2009&rft.pages=pp.%26nbsp%3B105%E2%80%93116&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-642-04133-4_9&rft.isbn=9783642041334&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaalandFree10-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HaalandFree10_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Haaland, Kirsten; Groven, Arne-Kristian; Glott, Ruediger; Tannenberg, Anna (01 July 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/publications.nr.no\/directdownload\/publications.nr.no\/5444\/Haaland_-_Free_Libre_Open_Source_Quality_Models-_a_compariso.pdf\" target=\"_blank\">\"Free\/Libre Open Source Quality Models - a comparison between two approaches\"<\/a> (PDF). <i>4th FLOSS International Workshop on Free\/Libre Open Source Software<\/i>. Jena, Germany. pp. 1\u201317<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/publications.nr.no\/directdownload\/publications.nr.no\/5444\/Haaland_-_Free_Libre_Open_Source_Quality_Models-_a_compariso.pdf\" target=\"_blank\">http:\/\/publications.nr.no\/directdownload\/publications.nr.no\/5444\/Haaland_-_Free_Libre_Open_Source_Quality_Models-_a_compariso.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 15 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Free%2FLibre+Open+Source+Quality+Models+-+a+comparison+between+two+approaches&rft.atitle=4th+FLOSS+International+Workshop+on+Free%2FLibre+Open+Source+Software&rft.aulast=Haaland%2C+Kirsten%3B+Groven%2C+Arne-Kristian%3B+Glott%2C+Ruediger%3B+Tannenberg%2C+Anna&rft.au=Haaland%2C+Kirsten%3B+Groven%2C+Arne-Kristian%3B+Glott%2C+Ruediger%3B+Tannenberg%2C+Anna&rft.date=01+July+2010&rft.pages=pp.+1%E2%80%9317&rft.place=Jena%2C+Germany&rft_id=http%3A%2F%2Fpublications.nr.no%2Fdirectdownload%2Fpublications.nr.no%2F5444%2FHaaland_-_Free_Libre_Open_Source_Quality_Models-_a_compariso.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaugeAnEmp09-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HaugeAnEmp09_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hauge, O.; Osterlie, T.; Sorensen, C.-F.; Gerea, M. (2009). \"An empirical study on selection of Open Source Software - Preliminary results\". <i>ICSE Workshop on Emerging Trends in Free\/Libre\/Open Source Software Research and Development, 2009<\/i>. IEEE. pp. 42-47. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FFLOSS.2009.5071359\" target=\"_blank\">10.1109\/FLOSS.2009.5071359<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781424437207.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=An+empirical+study+on+selection+of+Open+Source+Software+-+Preliminary+results&rft.atitle=ICSE+Workshop+on+Emerging+Trends+in+Free%2FLibre%2FOpen+Source+Software+Research+and+Development%2C+2009&rft.aulast=Hauge%2C+O.%3B+Osterlie%2C+T.%3B+Sorensen%2C+C.-F.%3B+Gerea%2C+M.&rft.au=Hauge%2C+O.%3B+Osterlie%2C+T.%3B+Sorensen%2C+C.-F.%3B+Gerea%2C+M.&rft.date=2009&rft.pages=pp.%26nbsp%3B42-47&rft.pub=IEEE&rft_id=info:doi\/10.1109%2FFLOSS.2009.5071359&rft.isbn=9781424437207&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AyalaTowards11-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AyalaTowards11_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ayala, Claudia; Cruzes, Daniela S.; Franch, Xavier; Conradi, Reidar (2011). \"Towards Improving OSS Products Selection \u2013 Matching Selectors and OSS Communities Perspectives\". In Hissam, S.; Russo, B.; de Mendon\u00e7a Neto, M.G.; Kon, F.. <i>Open Source Systems: Grounding Research<\/i>. Springer. pp. 244\u2013258. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-3-642-24418-6_17\" target=\"_blank\">10.1007\/978-3-642-24418-6_17<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9783642244186.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Towards+Improving+OSS+Products+Selection+%E2%80%93+Matching+Selectors+and+OSS+Communities+Perspectives&rft.atitle=Open+Source+Systems%3A+Grounding+Research&rft.aulast=Ayala%2C+Claudia%3B+Cruzes%2C+Daniela+S.%3B+Franch%2C+Xavier%3B+Conradi%2C+Reidar&rft.au=Ayala%2C+Claudia%3B+Cruzes%2C+Daniela+S.%3B+Franch%2C+Xavier%3B+Conradi%2C+Reidar&rft.date=2011&rft.pages=pp.%26nbsp%3B244%E2%80%93258&rft.pub=Springer&rft_id=info:doi\/10.1007%2F978-3-642-24418-6_17&rft.isbn=9783642244186&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OBRRSource-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OBRRSource_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Chan, C.; enugroho; Wasserman, T. (17 April 2013). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/sourceforge.net\/projects\/openbrr\/\" target=\"_blank\">\"Business Readiness Rating (BRR)\"<\/a>. SourceForge<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/sourceforge.net\/projects\/openbrr\/\" target=\"_blank\">https:\/\/sourceforge.net\/projects\/openbrr\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Business+Readiness+Rating+%28BRR%29&rft.atitle=&rft.aulast=Chan%2C+C.%3B+enugroho%3B+Wasserman%2C+T.&rft.au=Chan%2C+C.%3B+enugroho%3B+Wasserman%2C+T.&rft.date=17+April+2013&rft.pub=SourceForge&rft_id=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fopenbrr%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GalliOpenBRR06-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GalliOpenBRR06_27-0\" rel=\"external_link\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-GalliOpenBRR06_27-1\" rel=\"external_link\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Galli, Peter (24 April 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.eweek.com\/c\/a\/Linux-and-Open-Source\/OpenBRR-Launches-Closed-OpenSource-Group\" target=\"_blank\">\"OpenBRR Launches Closed Open-Source Group\"<\/a>. <i>eWeek<\/i>. QuinStreet, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.eweek.com\/c\/a\/Linux-and-Open-Source\/OpenBRR-Launches-Closed-OpenSource-Group\" target=\"_blank\">http:\/\/www.eweek.com\/c\/a\/Linux-and-Open-Source\/OpenBRR-Launches-Closed-OpenSource-Group<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OpenBRR+Launches+Closed+Open-Source+Group&rft.atitle=eWeek&rft.aulast=Galli%2C+Peter&rft.au=Galli%2C+Peter&rft.date=24+April+2006&rft.pub=QuinStreet%2C+Inc&rft_id=http%3A%2F%2Fwww.eweek.com%2Fc%2Fa%2FLinux-and-Open-Source%2FOpenBRR-Launches-Closed-OpenSource-Group&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OBRRArch-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OBRRArch_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20141224233009\/http:\/\/www.openbrr.org\/\" target=\"_blank\">\"Welcome to Business Readiness Rating: A FrameWork for Evaluating OpenSource Software\"<\/a>. OpenBRR. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.openbrr.org\/\" target=\"_blank\">the original<\/a> on 24 December 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20141224233009\/http:\/\/www.openbrr.org\/\" target=\"_blank\">https:\/\/web.archive.org\/web\/20141224233009\/http:\/\/www.openbrr.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Welcome+to+Business+Readiness+Rating%3A+A+FrameWork+for+Evaluating+OpenSource+Software&rft.atitle=&rft.pub=OpenBRR&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20141224233009%2Fhttp%3A%2F%2Fwww.openbrr.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArjonaWhat12-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArjonaWhat12_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Arjona, Laura (06 January 2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/larjona.wordpress.com\/2012\/01\/06\/what-happened-to-openbrr-business-readiness-rating-for-open-source\/\" target=\"_blank\">\"What happened to OpenBRR (Business Readiness Rating for Open Source)?\"<\/a>. <i>The Bright Side<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/larjona.wordpress.com\/2012\/01\/06\/what-happened-to-openbrr-business-readiness-rating-for-open-source\/\" target=\"_blank\">https:\/\/larjona.wordpress.com\/2012\/01\/06\/what-happened-to-openbrr-business-readiness-rating-for-open-source\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+happened+to+OpenBRR+%28Business+Readiness+Rating+for+Open+Source%29%3F&rft.atitle=The+Bright+Side&rft.aulast=Arjona%2C+Laura&rft.au=Arjona%2C+Laura&rft.date=06+January+2012&rft_id=https%3A%2F%2Flarjona.wordpress.com%2F2012%2F01%2F06%2Fwhat-happened-to-openbrr-business-readiness-rating-for-open-source%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSSpalHome-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSSpalHome_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/osspal.org\/\" target=\"_blank\">\"Welcome to OSSpal\"<\/a>. OSSpal<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/osspal.org\/\" target=\"_blank\">http:\/\/osspal.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Welcome+to+OSSpal&rft.atitle=&rft.pub=OSSpal&rft_id=http%3A%2F%2Fosspal.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Silva10Quest09-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Silva10Quest09_31-0\" rel=\"external_link\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-Silva10Quest09_31-1\" rel=\"external_link\">31.1<\/a><\/sup> <sup><a href=\"#cite_ref-Silva10Quest09_31-2\" rel=\"external_link\">31.2<\/a><\/sup> <sup><a href=\"#cite_ref-Silva10Quest09_31-3\" rel=\"external_link\">31.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Silva, Chamindra de (20 December 2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.techrepublic.com\/blog\/10-things\/10-questions-to-ask-when-selecting-open-source-products-for-your-enterprise\/\" target=\"_blank\">\"10 questions to ask when selecting open source products for your enterprise\"<\/a>. <i>TechRepublic<\/i>. CBS Interactive<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.techrepublic.com\/blog\/10-things\/10-questions-to-ask-when-selecting-open-source-products-for-your-enterprise\/\" target=\"_blank\">http:\/\/www.techrepublic.com\/blog\/10-things\/10-questions-to-ask-when-selecting-open-source-products-for-your-enterprise\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=10+questions+to+ask+when+selecting+open+source+products+for+your+enterprise&rft.atitle=TechRepublic&rft.aulast=Silva%2C+Chamindra+de&rft.au=Silva%2C+Chamindra+de&rft.date=20+December+2009&rft.pub=CBS+Interactive&rft_id=http%3A%2F%2Fwww.techrepublic.com%2Fblog%2F10-things%2F10-questions-to-ask-when-selecting-open-source-products-for-your-enterprise%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Phipps7Quest15-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Phipps7Quest15_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Phipps, Simon (21 January 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.infoworld.com\/article\/2872094\/open-source-software\/seven-questions-to-ask-any-open-source-project.html\" target=\"_blank\">\"7 questions to ask any open source project\"<\/a>. <i>InfoWorld<\/i>. InfoWorld, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.infoworld.com\/article\/2872094\/open-source-software\/seven-questions-to-ask-any-open-source-project.html\" target=\"_blank\">http:\/\/www.infoworld.com\/article\/2872094\/open-source-software\/seven-questions-to-ask-any-open-source-project.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=7+questions+to+ask+any+open+source+project&rft.atitle=InfoWorld&rft.aulast=Phipps%2C+Simon&rft.au=Phipps%2C+Simon&rft.date=21+January+2015&rft.pub=InfoWorld%2C+Inc&rft_id=http%3A%2F%2Fwww.infoworld.com%2Farticle%2F2872094%2Fopen-source-software%2Fseven-questions-to-ask-any-open-source-project.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PadinHow14-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PadinHow14_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Padin, Sandro (03 January 2014). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/blog.8thlight.com\/sandro-padin\/2014\/01\/03\/how-i-evaluate-open-source-software.html\" target=\"_blank\">\"How I Evaluate Open-Source Software\"<\/a>. 8th Light, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/blog.8thlight.com\/sandro-padin\/2014\/01\/03\/how-i-evaluate-open-source-software.html\" target=\"_blank\">https:\/\/blog.8thlight.com\/sandro-padin\/2014\/01\/03\/how-i-evaluate-open-source-software.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 01 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+I+Evaluate+Open-Source+Software&rft.atitle=&rft.aulast=Padin%2C+Sandro&rft.au=Padin%2C+Sandro&rft.date=03+January+2014&rft.pub=8th+Light%2C+Inc&rft_id=https%3A%2F%2Fblog.8thlight.com%2Fsandro-padin%2F2014%2F01%2F03%2Fhow-i-evaluate-open-source-software.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MetcalfeTop13-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MetcalfeTop13_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Metcalfe, Randy (01 February 2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/oss-watch.ac.uk\/resources\/tips\" target=\"_blank\">\"Top tips for selecting open source software\"<\/a>. <i>OSSWatch<\/i>. University of Oxford<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/oss-watch.ac.uk\/resources\/tips\" target=\"_blank\">http:\/\/oss-watch.ac.uk\/resources\/tips<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 23 March 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Top+tips+for+selecting+open+source+software&rft.atitle=OSSWatch&rft.aulast=Metcalfe%2C+Randy&rft.au=Metcalfe%2C+Randy&rft.date=01+February+2004&rft.pub=University+of+Oxford&rft_id=http%3A%2F%2Foss-watch.ac.uk%2Fresources%2Ftips&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LimardoDIY13-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LimardoDIY13_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Limardo, J. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1363288315&title=DIY%20Evaluation%20Process\" target=\"_blank\">\"DIY Evaluation Process\"<\/a>. <i>LIMSExpert.com<\/i>. ForwardPhase Technologies, LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1363288315&title=DIY%20Evaluation%20Process\" target=\"_blank\">http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1363288315&title=DIY%20Evaluation%20Process<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 February 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=DIY+Evaluation+Process&rft.atitle=LIMSExpert.com&rft.aulast=Limardo%2C+J.&rft.au=Limardo%2C+J.&rft.date=2013&rft.pub=ForwardPhase+Technologies%2C+LLC&rft_id=http%3A%2F%2Fwww.limsexpert.com%2Fcgi-bin%2Fbixchange%2Fbixchange.cgi%3Fpom%3Dlimsexpert3%26iid%3DreadMore%3Bgo%3D1363288315%26title%3DDIY%2520Evaluation%2520Process&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheelerHow11-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WheelerHow11_36-0\" rel=\"external_link\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-WheelerHow11_36-1\" rel=\"external_link\">36.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wheeler, David A. (05 August 2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.dwheeler.com\/oss_fs_eval.html\" target=\"_blank\">\"How to Evaluate Open Source Software \/ Free Software (OSS\/FS) Programs\"<\/a>. <i>dwheeler.com<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dwheeler.com\/oss_fs_eval.html\" target=\"_blank\">http:\/\/www.dwheeler.com\/oss_fs_eval.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 March 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+to+Evaluate+Open+Source+Software+%2F+Free+Software+%28OSS%2FFS%29+Programs&rft.atitle=dwheeler.com&rft.aulast=Wheeler%2C+David+A.&rft.au=Wheeler%2C+David+A.&rft.date=05+August+2011&rft_id=http%3A%2F%2Fwww.dwheeler.com%2Foss_fs_eval.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VOURS15-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VOURS15_37-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.validation-online.net\/user-requirements-specification.html\" target=\"_blank\">\"User Requirements Specification (URS)\"<\/a>. <i>validation-online.net<\/i>. Validation Online<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.validation-online.net\/user-requirements-specification.html\" target=\"_blank\">http:\/\/www.validation-online.net\/user-requirements-specification.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=User+Requirements+Specification+%28URS%29&rft.atitle=validation-online.net&rft.pub=Validation+Online&rft_id=http%3A%2F%2Fwww.validation-online.net%2Fuser-requirements-specification.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OKeeffeHow15-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OKeeffeHow15_38-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">O'Keefe, Graham (01 March 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.askaboutgmp.com\/296-how-to-create-a-bullet-proof-urs\" target=\"_blank\">\"How to Create a Bullet-Proof User Requirement Specification (URS)\"<\/a>. askaboutgmp<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.askaboutgmp.com\/296-how-to-create-a-bullet-proof-urs\" target=\"_blank\">http:\/\/www.askaboutgmp.com\/296-how-to-create-a-bullet-proof-urs<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=How+to+Create+a+Bullet-Proof+User+Requirement+Specification+%28URS%29&rft.atitle=&rft.aulast=O%27Keefe%2C+Graham&rft.au=O%27Keefe%2C+Graham&rft.date=01+March+2015&rft.pub=askaboutgmp&rft_id=http%3A%2F%2Fwww.askaboutgmp.com%2F296-how-to-create-a-bullet-proof-urs&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ASTMStandsLIMS-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ASTMStandsLIMS_39-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.astm.org\/Standards\/E1578.htm\" target=\"_blank\">\"ASTM E1578-13, Standard Guide for Laboratory Informatics\"<\/a>. West Conshohocken, PA: ASTM International. 2013. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1520%2FE1578\" target=\"_blank\">10.1520\/E1578<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.astm.org\/Standards\/E1578.htm\" target=\"_blank\">http:\/\/www.astm.org\/Standards\/E1578.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 March 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ASTM+E1578-13%2C+Standard+Guide+for+Laboratory+Informatics&rft.atitle=&rft.date=2013&rft.place=West+Conshohocken%2C+PA&rft.pub=ASTM+International&rft_id=info:doi\/10.1520%2FE1578&rft_id=http%3A%2F%2Fwww.astm.org%2FStandards%2FE1578.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ASURCheck-40\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ASURCheck_40-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.autoscribeinformatics.com\/services\/user-requirements\" target=\"_blank\">\"User Requirements Checklist\"<\/a>. Autoscribe Informatics<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.autoscribeinformatics.com\/services\/user-requirements\" target=\"_blank\">http:\/\/www.autoscribeinformatics.com\/services\/user-requirements<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=User+Requirements+Checklist&rft.atitle=&rft.pub=Autoscribe+Informatics&rft_id=http%3A%2F%2Fwww.autoscribeinformatics.com%2Fservices%2Fuser-requirements&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LIIComplete15-41\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LIIComplete15_41-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Laboratory Informatics Institute, ed. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.limsbook.com\/the-complete-guide-to-lims-laboratory-informatics-2015-edition\/\" target=\"_blank\">\"The Complete Guide to LIMS & Laboratory Informatics \u2013 2015 Edition\"<\/a>. LabLynx, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.limsbook.com\/the-complete-guide-to-lims-laboratory-informatics-2015-edition\/\" target=\"_blank\">http:\/\/www.limsbook.com\/the-complete-guide-to-lims-laboratory-informatics-2015-edition\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Complete+Guide+to+LIMS+%26+Laboratory+Informatics+%E2%80%93+2015+Edition&rft.atitle=&rft.date=2015&rft.pub=LabLynx%2C+Inc&rft_id=http%3A%2F%2Fwww.limsbook.com%2Fthe-complete-guide-to-lims-laboratory-informatics-2015-edition%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FDAP1115-42\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FDAP1115_42-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.fda.gov\/regulatoryinformation\/guidances\/ucm125067.htm\" target=\"_blank\">\"Part 11, Electronic Records; Electronic Signatures \u2014 Scope and Application\"<\/a>. U.S. Food and Drug Administration. 26 August 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.fda.gov\/regulatoryinformation\/guidances\/ucm125067.htm\" target=\"_blank\">http:\/\/www.fda.gov\/regulatoryinformation\/guidances\/ucm125067.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Part+11%2C+Electronic+Records%3B+Electronic+Signatures+%E2%80%94+Scope+and+Application&rft.atitle=&rft.date=26+August+2015&rft.pub=U.S.+Food+and+Drug+Administration&rft_id=http%3A%2F%2Fwww.fda.gov%2Fregulatoryinformation%2Fguidances%2Fucm125067.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SegalstadInt08-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SegalstadInt08_43-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Segalstad, Siri H. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0470758821.html\" target=\"_blank\"><i>International IT Regulations and Compliance: Quality Standards in the Pharmaceutical and Regulated Industries<\/i><\/a>. John Wiley & Sons, Inc. pp. 338. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780470758823<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0470758821.html\" target=\"_blank\">http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0470758821.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=International+IT+Regulations+and+Compliance%3A+Quality+Standards+in+the+Pharmaceutical+and+Regulated+Industries&rft.aulast=Segalstad%2C+Siri+H.&rft.au=Segalstad%2C+Siri+H.&rft.date=2008&rft.pages=pp.%26nbsp%3B338&rft.pub=John+Wiley+%26+Sons%2C+Inc&rft.isbn=9780470758823&rft_id=http%3A%2F%2Fwww.wiley.com%2FWileyCDA%2FWileyTitle%2FproductCd-0470758821.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarveyData15-44\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HarveyData15_44-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.datamation.com\/author\/Cynthia-Harvey-6460.html\" target=\"_blank\">\"More articles by Cynthia Harvey\"<\/a>. <i>Datamation<\/i>. QuinStreet, Inc. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.datamation.com\/author\/Cynthia-Harvey-6460.html\" target=\"_blank\">http:\/\/www.datamation.com\/author\/Cynthia-Harvey-6460.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=More+articles+by+Cynthia+Harvey&rft.atitle=Datamation&rft.date=2015&rft.pub=QuinStreet%2C+Inc&rft_id=http%3A%2F%2Fwww.datamation.com%2Fauthor%2FCynthia-Harvey-6460.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarveyOpen15-45\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HarveyOpen15_45-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Harvey, Cynthia (05 January 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.datamation.com\/open-source\/open-source-software-list-2015-ultimate-list-1.html\" target=\"_blank\">\"Open Source Software List: 2015 Ultimate List\"<\/a>. <i>Datamation<\/i>. QuinStreet, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.datamation.com\/open-source\/open-source-software-list-2015-ultimate-list-1.html\" target=\"_blank\">http:\/\/www.datamation.com\/open-source\/open-source-software-list-2015-ultimate-list-1.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open+Source+Software+List%3A+2015+Ultimate+List&rft.atitle=Datamation&rft.aulast=Harvey%2C+Cynthia&rft.au=Harvey%2C+Cynthia&rft.date=05+January+2015&rft.pub=QuinStreet%2C+Inc&rft_id=http%3A%2F%2Fwww.datamation.com%2Fopen-source%2Fopen-source-software-list-2015-ultimate-list-1.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SFHome-46\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SFHome_46-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/sourceforge.net\/\" target=\"_blank\">\"SourceForge - Download, Develop and Publish Free Open Source Software\"<\/a>. <i>SourceForge<\/i>. Slashdot Media. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/sourceforge.net\/\" target=\"_blank\">https:\/\/sourceforge.net\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SourceForge+-+Download%2C+Develop+and+Publish+Free+Open+Source+Software&rft.atitle=SourceForge&rft.date=2015&rft.pub=Slashdot+Media&rft_id=https%3A%2F%2Fsourceforge.net%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GHHome-47\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GHHome_47-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/\" target=\"_blank\">\"GitHub: Where software is built\"<\/a>. <i>GitHub<\/i>. GitHub, Inc. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/\" target=\"_blank\">https:\/\/github.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GitHub%3A+Where+software+is+built&rft.atitle=GitHub&rft.date=2015&rft.pub=GitHub%2C+Inc&rft_id=https%3A%2F%2Fgithub.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WPSourceRepos-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WPSourceRepos_48-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Comparison_of_source_code_hosting_facilities&oldid=682090863\" target=\"_blank\">\"Comparison of source code hosting facilities\"<\/a>. <i>Wikipedia<\/i>. Wikimedia Foundation. 21 September 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/en.wikipedia.org\/w\/index.php?title=Comparison_of_source_code_hosting_facilities&oldid=682090863\" target=\"_blank\">https:\/\/en.wikipedia.org\/w\/index.php?title=Comparison_of_source_code_hosting_facilities&oldid=682090863<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 September 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Comparison+of+source+code+hosting+facilities&rft.atitle=Wikipedia&rft.date=21+September+2015&rft.pub=Wikimedia+Foundation&rft_id=https%3A%2F%2Fen.wikipedia.org%2Fw%2Findex.php%3Ftitle%3DComparison_of_source_code_hosting_facilities%26oldid%3D682090863&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AOSAHome-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AOSAHome_49-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/aosabook.org\/en\/index.html\" target=\"_blank\">\"The Architecture of Open Source Applications\"<\/a>. <i>AOSA<\/i>. AOSA Editors. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/aosabook.org\/en\/index.html\" target=\"_blank\">http:\/\/aosabook.org\/en\/index.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 October 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Architecture+of+Open+Source+Applications&rft.atitle=AOSA&rft.date=2015&rft.pub=AOSA+Editors&rft_id=http%3A%2F%2Faosabook.org%2Fen%2Findex.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IW5Key15-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IW5Key15_50-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Knorr, Eric (28 September 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.infoworld.com\/article\/2986769\/open-source-tools\/5-key-trends-in-open-source.html\" target=\"_blank\">\"5 key trends in open source\"<\/a>. <i>InfoWorld<\/i>. InfoWorld, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.infoworld.com\/article\/2986769\/open-source-tools\/5-key-trends-in-open-source.html\" target=\"_blank\">http:\/\/www.infoworld.com\/article\/2986769\/open-source-tools\/5-key-trends-in-open-source.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 September 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=5+key+trends+in+open+source&rft.atitle=InfoWorld&rft.aulast=Knorr%2C+Eric&rft.au=Knorr%2C+Eric&rft.date=28+September+2015&rft.pub=InfoWorld%2C+Inc&rft_id=http%3A%2F%2Fwww.infoworld.com%2Farticle%2F2986769%2Fopen-source-tools%2F5-key-trends-in-open-source.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GSHome-51\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GSHome_51-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/scholar.google.com\/\" target=\"_blank\">\"Google Scholar\"<\/a>. Google, Inc. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/scholar.google.com\/\" target=\"_blank\">https:\/\/scholar.google.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Google+Scholar&rft.atitle=&rft.date=2015&rft.pub=Google%2C+Inc&rft_id=https%3A%2F%2Fscholar.google.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MASHome-52\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MASHome_52-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/academic.research.microsoft.com\/\" target=\"_blank\">\"Microsoft Academic Search\"<\/a>. Microsoft Corporation. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/academic.research.microsoft.com\/\" target=\"_blank\">http:\/\/academic.research.microsoft.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Microsoft+Academic+Search&rft.atitle=&rft.date=2015&rft.pub=Microsoft+Corporation&rft_id=http%3A%2F%2Facademic.research.microsoft.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WasikeSelect10-53\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WasikeSelect10_53-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wasike, Sylvia Nasambu (October 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.227.5951\" target=\"_blank\">\"Selection Process of Open Source Software Component\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.227.5951\" target=\"_blank\">http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.227.5951<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Selection+Process+of+Open+Source+Software+Component&rft.atitle=&rft.aulast=Wasike%2C+Sylvia+Nasambu&rft.au=Wasike%2C+Sylvia+Nasambu&rft.date=October+2010&rft_id=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fsummary%3Fdoi%3D10.1.1.227.5951&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OHHome-54\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OHHome_54-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.openhub.net\/\" target=\"_blank\">\"Open HUB, the open source network\"<\/a>. <i>Open HUB<\/i>. Black Duck Software, Inc. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.openhub.net\/\" target=\"_blank\">https:\/\/www.openhub.net\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 10 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open+HUB%2C+the+open+source+network&rft.atitle=Open+HUB&rft.date=2015&rft.pub=Black+Duck+Software%2C+Inc&rft_id=https%3A%2F%2Fwww.openhub.net%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NetelerGRASS12-55\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NetelerGRASS12_55-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Netelera, Markus; Bowmanb, M. Hamish; Landac, Martin; Metz, Markus (May 2012). \"GRASS GIS: A multi-purpose open source GIS\". <i>Environmental Modelling & Software<\/i> <b>31<\/b>: 124\u2013130. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.envsoft.2011.11.014\" target=\"_blank\">10.1016\/j.envsoft.2011.11.014<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GRASS+GIS%3A+A+multi-purpose+open+source+GIS&rft.jtitle=Environmental+Modelling+%26+Software&rft.aulast=Netelera%2C+Markus%3B+Bowmanb%2C+M.+Hamish%3B+Landac%2C+Martin%3B+Metz%2C+Markus&rft.au=Netelera%2C+Markus%3B+Bowmanb%2C+M.+Hamish%3B+Landac%2C+Martin%3B+Metz%2C+Markus&rft.date=May+2012&rft.volume=31&rft.pages=124%E2%80%93130&rft_id=info:doi\/10.1016%2Fj.envsoft.2011.11.014&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhanineMean15-56\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KhanineMean15_56-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Khanine, Dmitri (07 May 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/07\/quot-meaningful-use-quot-regulations-of-medical-information-in-health-it\" target=\"_blank\">\"\"Meaningful Use\" Regulations of Medical Information in Health IT\"<\/a>. <i>Toad World - Oracle Community<\/i>. Dell Software, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/07\/quot-meaningful-use-quot-regulations-of-medical-information-in-health-it\" target=\"_blank\">http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/07\/quot-meaningful-use-quot-regulations-of-medical-information-in-health-it<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=%22Meaningful+Use%22+Regulations+of+Medical+Information+in+Health+IT&rft.atitle=Toad+World+-+Oracle+Community&rft.aulast=Khanine%2C+Dmitri&rft.au=Khanine%2C+Dmitri&rft.date=07+May+2015&rft.pub=Dell+Software%2C+Inc&rft_id=http%3A%2F%2Fwww.toadworld.com%2Fplatforms%2Foracle%2Fb%2Fweblog%2Farchive%2F2015%2F05%2F07%2Fquot-meaningful-use-quot-regulations-of-medical-information-in-health-it&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhanineOpen15-57\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KhanineOpen15_57-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Khanine, Dmitri (06 May 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/06\/open-source-medical-record-systems-of-2015\" target=\"_blank\">\"Open-Source Medical Record Systems of 2015\"<\/a>. <i>Toad World - Oracle Community<\/i>. Dell Software, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/06\/open-source-medical-record-systems-of-2015\" target=\"_blank\">http:\/\/www.toadworld.com\/platforms\/oracle\/b\/weblog\/archive\/2015\/05\/06\/open-source-medical-record-systems-of-2015<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open-Source+Medical+Record+Systems+of+2015&rft.atitle=Toad+World+-+Oracle+Community&rft.aulast=Khanine%2C+Dmitri&rft.au=Khanine%2C+Dmitri&rft.date=06+May+2015&rft.pub=Dell+Software%2C+Inc&rft_id=http%3A%2F%2Fwww.toadworld.com%2Fplatforms%2Foracle%2Fb%2Fweblog%2Farchive%2F2015%2F05%2F06%2Fopen-source-medical-record-systems-of-2015&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DavisEnt08-58\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DavisEnt08_58-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Davis, Ashley (2008). \"Enterprise Resource Planning Under Open Source Software\". In Ferran, Carlos; Salim, Ricardo. <i>Enterprise Resource Planning for Global Economies: Managerial Issues and Challenges<\/i>. Hershey, PA: IGI Global. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4018%2F978-1-59904-531-3.ch004\" target=\"_blank\">10.4018\/978-1-59904-531-3.ch004<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781599045313.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Enterprise+Resource+Planning+Under+Open+Source+Software&rft.atitle=Enterprise+Resource+Planning+for+Global+Economies%3A+Managerial+Issues+and+Challenges&rft.aulast=Davis%2C+Ashley&rft.au=Davis%2C+Ashley&rft.date=2008&rft.place=Hershey%2C+PA&rft.pub=IGI+Global&rft_id=info:doi\/10.4018%2F978-1-59904-531-3.ch004&rft.isbn=9781599045313&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SarrabSel13-59\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SarrabSel13_59-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sarrah, Mohamed; Rehman, Osama M. Hussain (2013). \"Selection Criteria of Open Source Software: First Stage for Adoption\". <i>International Journal of Information Processing and Management<\/i> <b>4<\/b> (4): 51\u201358. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4156%2Fijipm.vol4.issue4.6\" target=\"_blank\">10.4156\/ijipm.vol4.issue4.6<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Selection+Criteria+of+Open+Source+Software%3A+First+Stage+for+Adoption&rft.jtitle=International+Journal+of+Information+Processing+and+Management&rft.aulast=Sarrah%2C+Mohamed%3B+Rehman%2C+Osama+M.+Hussain&rft.au=Sarrah%2C+Mohamed%3B+Rehman%2C+Osama+M.+Hussain&rft.date=2013&rft.volume=4&rft.issue=4&rft.pages=51%E2%80%9358&rft_id=info:doi\/10.4156%2Fijipm.vol4.issue4.6&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FooteTheMyth10-60\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FooteTheMyth10_60-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Foote, Amanda (2010). \"The Myth of Free: The Hidden Costs of Open Source Software\". <i>Dalhousie Journal of Interdisciplinary Management<\/i> <b>6<\/b> (Spring 2010): 1\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5931%2Fdjim.v6i1.31\" target=\"_blank\">10.5931\/djim.v6i1.31<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Myth+of+Free%3A+The+Hidden+Costs+of+Open+Source+Software&rft.jtitle=Dalhousie+Journal+of+Interdisciplinary+Management&rft.aulast=Foote%2C+Amanda&rft.au=Foote%2C+Amanda&rft.date=2010&rft.volume=6&rft.issue=Spring+2010&rft.pages=1%E2%80%939&rft_id=info:doi\/10.5931%2Fdjim.v6i1.31&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeinleinMoon66-61\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeinleinMoon66_61-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Heinlein, Robert A. (1997) [1966]. <i>The Moon Is a Harsh Mistress<\/i>. New York, NY: Tom Doherty Associates. pp. 8\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780312863555.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Moon+Is+a+Harsh+Mistress&rft.aulast=Heinlein%2C+Robert+A.&rft.au=Heinlein%2C+Robert+A.&rft.date=1997&rft.pages=pp.%26nbsp%3B8%E2%80%939&rft.place=New+York%2C+NY&rft.pub=Tom+Doherty+Associates&rft.isbn=9780312863555&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FlemingISO14-62\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FlemingISO14_62-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fleming, Ian (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.sqa.net\/iso9126.html\" target=\"_blank\">\"ISO 9126 Software Quality Characteristics\"<\/a>. <i>SQA Definition<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sqa.net\/iso9126.html\" target=\"_blank\">http:\/\/www.sqa.net\/iso9126.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ISO+9126+Software+Quality+Characteristics&rft.atitle=SQA+Definition&rft.aulast=Fleming%2C+Ian&rft.au=Fleming%2C+Ian&rft.date=2014&rft_id=http%3A%2F%2Fwww.sqa.net%2Fiso9126.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorMurphy-63\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorMurphy_63-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Taylor, Dave (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/davetgc.com\/Murphys_Law.html\" target=\"_blank\">\"Murphy's Laws\"<\/a>. <i>Dave Taylor's Educational & Guidance Counseling Services<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/davetgc.com\/Murphys_Law.html\" target=\"_blank\">http:\/\/davetgc.com\/Murphys_Law.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Murphy%27s+Laws&rft.atitle=Dave+Taylor%27s+Educational+%26+Guidance+Counseling+Services&rft.aulast=Taylor%2C+Dave&rft.au=Taylor%2C+Dave&rft.date=2015&rft_id=http%3A%2F%2Fdavetgc.com%2FMurphys_Law.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbranUsa03-64\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AbranUsa03_64-0\" rel=\"external_link\">64.0<\/a><\/sup> <sup><a href=\"#cite_ref-AbranUsa03_64-1\" rel=\"external_link\">64.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abran, Alain; Khelifi, Adel; Suryn, Witold; Seffah, Ahmed (2003). \"Usability Meanings and Interpretations in ISO Standards\". <i>Software Quality Journal<\/i> <b>11<\/b> (4): 325\u2013338. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1023%2FA%3A1025869312943\" target=\"_blank\">10.1023\/A:1025869312943<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Usability+Meanings+and+Interpretations+in+ISO+Standards&rft.jtitle=Software+Quality+Journal&rft.aulast=Abran%2C+Alain%3B+Khelifi%2C+Adel%3B+Suryn%2C+Witold%3B+Seffah%2C+Ahmed&rft.au=Abran%2C+Alain%3B+Khelifi%2C+Adel%3B+Suryn%2C+Witold%3B+Seffah%2C+Ahmed&rft.date=2003&rft.volume=11&rft.issue=4&rft.pages=325%E2%80%93338&rft_id=info:doi\/10.1023%2FA%3A1025869312943&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AndreasenUsa06-65\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AndreasenUsa06_65-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Andreasen, M.S.; Nielsen, H.V.; Schr\u00f8der, S.O.; Stage, J. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/itc.ktu.lt\/itc353\/Stage353.pdf\" target=\"_blank\">\"Usability in open source software development: Opinions and practice\"<\/a> (PDF). <i>Information Technology and Control<\/i> <b>35<\/b> (3A): 303-312<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/itc.ktu.lt\/itc353\/Stage353.pdf\" target=\"_blank\">http:\/\/itc.ktu.lt\/itc353\/Stage353.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Usability+in+open+source+software+development%3A+Opinions+and+practice&rft.jtitle=Information+Technology+and+Control&rft.aulast=Andreasen%2C+M.S.%3B+Nielsen%2C+H.V.%3B+Schr%C3%B8der%2C+S.O.%3B+Stage%2C+J.&rft.au=Andreasen%2C+M.S.%3B+Nielsen%2C+H.V.%3B+Schr%C3%B8der%2C+S.O.%3B+Stage%2C+J.&rft.date=2006&rft.volume=35&rft.issue=3A&rft.pages=303-312&rft_id=http%3A%2F%2Fitc.ktu.lt%2Fitc353%2FStage353.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaxenaImpact13-66\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SaxenaImpact13_66-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saxena, S.; Dubey, S.K. (January 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ijcaonline.org\/archives\/volume61\/number22\/10233-5043\" target=\"_blank\">\"Impact of Software Design Aspects on Usability\"<\/a>. <i>International Journal of Computer Applications<\/i> <b>61<\/b> (22): 48-53. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5120%2F10233-5043\" target=\"_blank\">10.5120\/10233-5043<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ijcaonline.org\/archives\/volume61\/number22\/10233-5043\" target=\"_blank\">http:\/\/www.ijcaonline.org\/archives\/volume61\/number22\/10233-5043<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Impact+of+Software+Design+Aspects+on+Usability&rft.jtitle=International+Journal+of+Computer+Applications&rft.aulast=Saxena%2C+S.%3B+Dubey%2C+S.K.&rft.au=Saxena%2C+S.%3B+Dubey%2C+S.K.&rft.date=January+2013&rft.volume=61&rft.issue=22&rft.pages=48-53&rft_id=info:doi\/10.5120%2F10233-5043&rft_id=http%3A%2F%2Fwww.ijcaonline.org%2Farchives%2Fvolume61%2Fnumber22%2F10233-5043&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AllUXDefs-67\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AllUXDefs_67-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">AllAboutUX.org volunteers (08 October 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.allaboutux.org\/ux-definitions\" target=\"_blank\">\"User experience definitions\"<\/a>. <i>All About UX<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.allaboutux.org\/ux-definitions\" target=\"_blank\">http:\/\/www.allaboutux.org\/ux-definitions<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=User+experience+definitions&rft.atitle=All+About+UX&rft.aulast=AllAboutUX.org+volunteers&rft.au=AllAboutUX.org+volunteers&rft.date=08+October+2010&rft_id=http%3A%2F%2Fwww.allaboutux.org%2Fux-definitions&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GubeWhat10-68\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GubeWhat10_68-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Gube, Jacob (05 October 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.smashingmagazine.com\/2010\/10\/what-is-user-experience-design-overview-tools-and-resources\/\" target=\"_blank\">\"What Is User Experience Design? Overview, Tools And Resources\"<\/a>. <i>Smashing Magazine<\/i>. Smashing Magazine GmbH<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.smashingmagazine.com\/2010\/10\/what-is-user-experience-design-overview-tools-and-resources\/\" target=\"_blank\">http:\/\/www.smashingmagazine.com\/2010\/10\/what-is-user-experience-design-overview-tools-and-resources\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 08 August 2105<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+Is+User+Experience+Design%3F+Overview%2C+Tools+And+Resources&rft.atitle=Smashing+Magazine&rft.aulast=Gube%2C+Jacob&rft.au=Gube%2C+Jacob&rft.date=05+October+2010&rft.pub=Smashing+Magazine+GmbH&rft_id=http%3A%2F%2Fwww.smashingmagazine.com%2F2010%2F10%2Fwhat-is-user-experience-design-overview-tools-and-resources%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReitzChoosing14-69\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReitzChoosing14_69-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Reitz, Kenneth (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/docs.python-guide.org\/en\/latest\/writing\/license\/\" target=\"_blank\">\"Choosing a License\"<\/a>. <i>The Hitchhiker's Guide to Python!<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/docs.python-guide.org\/en\/latest\/writing\/license\/\" target=\"_blank\">http:\/\/docs.python-guide.org\/en\/latest\/writing\/license\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Choosing+a+License&rft.atitle=The+Hitchhiker%27s+Guide+to+Python%21&rft.aulast=Reitz%2C+Kenneth&rft.au=Reitz%2C+Kenneth&rft.date=2014&rft_id=http%3A%2F%2Fdocs.python-guide.org%2Fen%2Flatest%2Fwriting%2Flicense%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AtwoodPick07-70\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AtwoodPick07_70-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Atwood, Jeff (03 April 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/blog.codinghorror.com\/pick-a-license-any-license\/\" target=\"_blank\">\"Pick a License, Any License\"<\/a>. <i>Coding Horror: Programming and Human Factors<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/blog.codinghorror.com\/pick-a-license-any-license\/\" target=\"_blank\">http:\/\/blog.codinghorror.com\/pick-a-license-any-license\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Pick+a+License%2C+Any+License&rft.atitle=Coding+Horror%3A+Programming+and+Human+Factors&rft.aulast=Atwood%2C+Jeff&rft.au=Atwood%2C+Jeff&rft.date=03+April+2007&rft_id=http%3A%2F%2Fblog.codinghorror.com%2Fpick-a-license-any-license%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSIAnnoDef15-71\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSIAnnoDef15_71-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/opensource.org\/docs\/definition.php\" target=\"_blank\">\"The Open Source Definition (Annotated)\"<\/a>. Open Source Initiative. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensource.org\/docs\/definition.php\" target=\"_blank\">http:\/\/opensource.org\/docs\/definition.php<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Open+Source+Definition+%28Annotated%29&rft.atitle=&rft.date=2015&rft.pub=Open+Source+Initiative&rft_id=http%3A%2F%2Fopensource.org%2Fdocs%2Fdefinition.php&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WTFPLHome-72\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WTFPLHome_72-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hocevar, Sam (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wtfpl.net\/\" target=\"_blank\">\"WTFPL \u2013 Do What the Fuck You Want to Public License\"<\/a>. <i>WTFPL.net<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.wtfpl.net\/\" target=\"_blank\">http:\/\/www.wtfpl.net\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=WTFPL+%E2%80%93+Do+What+the+Fuck+You+Want+to+Public+License&rft.atitle=WTFPL.net&rft.aulast=Hocevar%2C+Sam&rft.au=Hocevar%2C+Sam&rft.date=2015&rft_id=http%3A%2F%2Fwww.wtfpl.net%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OSILic15-73\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OSILic15_73-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/opensource.org\/licenses\" target=\"_blank\">\"Licenses & Standards\"<\/a>. Open Source Initiative. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/opensource.org\/licenses\" target=\"_blank\">http:\/\/opensource.org\/licenses<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 13 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Licenses+%26+Standards&rft.atitle=&rft.date=2015&rft.pub=Open+Source+Initiative&rft_id=http%3A%2F%2Fopensource.org%2Flicenses&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TLDRLegalHome-74\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TLDRLegalHome_74-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/tldrlegal.com\/\" target=\"_blank\">\"TL;DRLegal - Software Licenses Explained in Plain English\"<\/a>. FOSSA, Inc. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/tldrlegal.com\/\" target=\"_blank\">https:\/\/tldrlegal.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=TL%3BDRLegal+-+Software+Licenses+Explained+in+Plain+English&rft.atitle=&rft.date=2015&rft.pub=FOSSA%2C+Inc&rft_id=https%3A%2F%2Ftldrlegal.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheelerMake14-75\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WheelerMake14_75-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wheeler, David A. (16 February 2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.dwheeler.com\/essays\/gpl-compatible.html\" target=\"_blank\">\"Make Your Open Source Software GPL-Compatible. Or Else\"<\/a>. <i>dwheeler.com<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dwheeler.com\/essays\/gpl-compatible.html\" target=\"_blank\">http:\/\/www.dwheeler.com\/essays\/gpl-compatible.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 March 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Make+Your+Open+Source+Software+GPL-Compatible.+Or+Else&rft.atitle=dwheeler.com&rft.aulast=Wheeler%2C+David+A.&rft.au=Wheeler%2C+David+A.&rft.date=16+February+2014&rft_id=http%3A%2F%2Fwww.dwheeler.com%2Fessays%2Fgpl-compatible.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheelerTheFree07-76\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WheelerTheFree07_76-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wheeler, David A. (27 September 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.dwheeler.com\/essays\/floss-license-slide.html\" target=\"_blank\">\"The Free-Libre \/ Open Source Software (FLOSS) License Slide\"<\/a>. <i>dwheeler.com<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dwheeler.com\/essays\/floss-license-slide.html\" target=\"_blank\">http:\/\/www.dwheeler.com\/essays\/floss-license-slide.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 28 May 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Free-Libre+%2F+Open+Source+Software+%28FLOSS%29+License+Slide&rft.atitle=dwheeler.com&rft.aulast=Wheeler%2C+David+A.&rft.au=Wheeler%2C+David+A.&rft.date=27+September+2007&rft_id=http%3A%2F%2Fwww.dwheeler.com%2Fessays%2Ffloss-license-slide.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DockerWhatIs-77\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DockerWhatIs_77-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.docker.com\/whatisdocker\" target=\"_blank\">\"What Is Docker?\"<\/a>. <i>Docker.com<\/i>. Docker, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.docker.com\/whatisdocker\" target=\"_blank\">https:\/\/www.docker.com\/whatisdocker<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=What+Is+Docker%3F&rft.atitle=Docker.com&rft.pub=Docker%2C+Inc&rft_id=https%3A%2F%2Fwww.docker.com%2Fwhatisdocker&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KnorrWhen15-78\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KnorrWhen15_78-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Knorr, Eric (23 March 2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.infoworld.com\/article\/2900333\/cloud-computing\/scenes-from-the-docker-revolution.html\" target=\"_blank\">\"When will we see Docker in production?\"<\/a>. <i>InfoWorld<\/i>. InfoWorld, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.infoworld.com\/article\/2900333\/cloud-computing\/scenes-from-the-docker-revolution.html\" target=\"_blank\">http:\/\/www.infoworld.com\/article\/2900333\/cloud-computing\/scenes-from-the-docker-revolution.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=When+will+we+see+Docker+in+production%3F&rft.atitle=InfoWorld&rft.aulast=Knorr%2C+Eric&rft.au=Knorr%2C+Eric&rft.date=23+March+2015&rft.pub=InfoWorld%2C+Inc&rft_id=http%3A%2F%2Fwww.infoworld.com%2Farticle%2F2900333%2Fcloud-computing%2Fscenes-from-the-docker-revolution.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EmadeenBika10-79\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EmadeenBika10_79-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Emadeen, Hamza (01 August 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.goomedic.com\/bika-free-open-source-lims-laboratory-information-management-system-for-windows-linux-and-mac-osx.html\" target=\"_blank\">\"Bika : Free , Open source LIMS \u2013 Laboratory Information Management System for Windows , Linux and Mac OSX\"<\/a>. <i>Goomedic.com<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.goomedic.com\/bika-free-open-source-lims-laboratory-information-management-system-for-windows-linux-and-mac-osx.html\" target=\"_blank\">http:\/\/www.goomedic.com\/bika-free-open-source-lims-laboratory-information-management-system-for-windows-linux-and-mac-osx.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bika+%3A+Free+%2C+Open+source+LIMS+%E2%80%93+Laboratory+Information+Management+System+for+Windows+%2C+Linux+and+Mac+OSX&rft.atitle=Goomedic.com&rft.aulast=Emadeen%2C+Hamza&rft.au=Emadeen%2C+Hamza&rft.date=01+August+2010&rft_id=http%3A%2F%2Fwww.goomedic.com%2Fbika-free-open-source-lims-laboratory-information-management-system-for-windows-linux-and-mac-osx.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LimardoOpen11-80\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LimardoOpen11_80-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Limardo, J. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1304895280&title=Open-LIMS%20Analysis%20Results\" target=\"_blank\">\"Open-LIMS Analysis Results\"<\/a>. <i>LIMSExpert.com<\/i>. ForwardPhase Technologies, LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1304895280&title=Open-LIMS%20Analysis%20Results\" target=\"_blank\">http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1304895280&title=Open-LIMS%20Analysis%20Results<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 February 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Open-LIMS+Analysis+Results&rft.atitle=LIMSExpert.com&rft.aulast=Limardo%2C+J.&rft.au=Limardo%2C+J.&rft.date=2011&rft.pub=ForwardPhase+Technologies%2C+LLC&rft_id=http%3A%2F%2Fwww.limsexpert.com%2Fcgi-bin%2Fbixchange%2Fbixchange.cgi%3Fpom%3Dlimsexpert3%26iid%3DreadMore%3Bgo%3D1304895280%26title%3DOpen-LIMS%2520Analysis%2520Results&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LimardoScoring11-81\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LimardoScoring11_81-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Limardo, J. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1305075855&title=Scoring%20Open-LIMS\" target=\"_blank\">\"Scoring Open-LIMS\"<\/a>. <i>LIMSExpert.com<\/i>. ForwardPhase Technologies, LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1305075855&title=Scoring%20Open-LIMS\" target=\"_blank\">http:\/\/www.limsexpert.com\/cgi-bin\/bixchange\/bixchange.cgi?pom=limsexpert3&iid=readMore;go=1305075855&title=Scoring%20Open-LIMS<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 07 February 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Scoring+Open-LIMS&rft.atitle=LIMSExpert.com&rft.aulast=Limardo%2C+J.&rft.au=Limardo%2C+J.&rft.date=2011&rft.pub=ForwardPhase+Technologies%2C+LLC&rft_id=http%3A%2F%2Fwww.limsexpert.com%2Fcgi-bin%2Fbixchange%2Fbixchange.cgi%3Fpom%3Dlimsexpert3%26iid%3DreadMore%3Bgo%3D1305075855%26title%3DScoring%2520Open-LIMS&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-82\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-82\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Glover, H.D. (2013). \"Academia Should Embrace Open Access Scholarly Publishing\". <i>Open Journal of Accounting<\/i> <b>2<\/b> (4): 95\u201396. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4236%2Fojacct.2013.24012\" target=\"_blank\">10.4236\/ojacct.2013.24012<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Academia+Should+Embrace+Open+Access+Scholarly+Publishing&rft.jtitle=Open+Journal+of+Accounting&rft.aulast=Glover%2C+H.D.&rft.au=Glover%2C+H.D.&rft.date=2013&rft.volume=2&rft.issue=4&rft.pages=95%E2%80%9396&rft_id=info:doi\/10.4236%2Fojacct.2013.24012&rfr_id=info:sid\/en.wikipedia.org:Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This article has not officially been published in a journal. However, this presentation is largely faithful to the original paper. The content has been edited for grammar, punctuation, and spelling. Additional error correction of a few reference URLs and types as well as cleaning up of the glossary also occurred. Redundancies and references to entities that don't offer open-source software were removed from the FLOSS examples in Table 2. DOIs and other identifiers have been added to the references to make them more useful. This article is being made available for the first time under the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/\" target=\"_blank\">Creative Commons Attribution-ShareAlike 4.0 International<\/a> license, the same license used on this wiki.\n<\/p><p><br \/>\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210240\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.660 seconds\nReal time usage: 1.711 seconds\nPreprocessor visited node count: 54279\/1000000\nPreprocessor generated node count: 45172\/1000000\nPost\u2010expand include size: 599944\/2097152 bytes\nTemplate argument size: 146265\/2097152 bytes\nHighest expansion depth: 15\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 1624.257 1 - -total\n 89.14% 1447.888 1 - Template:Reflist\n 78.54% 1275.674 82 - Template:Citation\/core\n 55.26% 897.580 58 - Template:Cite_web\n 16.62% 269.917 14 - Template:Cite_book\n 10.77% 174.971 10 - Template:Cite_journal\n 5.21% 84.649 133 - Template:Citation\/make_link\n 3.93% 63.901 1 - Template:Infobox_journal_article\n 3.77% 61.166 1 - Template:Infobox\n 2.53% 41.077 31 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7821-0!*!0!!en!5!* and timestamp 20181213210238 and revision id 23011\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version\">https:\/\/www.limswiki.org\/index.php\/Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","13b3b6c47f60a3b75f007d8cd017d140_images":["https:\/\/www.limswiki.org\/images\/3\/3c\/Fig1_Joyce_2015.png","https:\/\/www.limswiki.org\/images\/9\/9f\/Fig2_Joyce_2015.png","https:\/\/www.limswiki.org\/images\/8\/84\/Fig3_Joyce_2015.png"],"13b3b6c47f60a3b75f007d8cd017d140_timestamp":1544734958,"2d0bddc8ac63fbe42ecf34c128173d02_type":"article","2d0bddc8ac63fbe42ecf34c128173d02_title":"Benefits of the community for partners of open source vendors (Groganz 2011)","2d0bddc8ac63fbe42ecf34c128173d02_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Benefits_of_the_community_for_partners_of_open_source_vendors","2d0bddc8ac63fbe42ecf34c128173d02_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Benefits of the community for partners of open source vendors\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nBenefits of the community for partners of open source vendorsJournal\n \nOpen Source Business ResourceAuthor(s)\n \nGroganz, SandroAuthor affiliation(s)\n \nAge of Peers Ltd., CampaignChain, Inc.Primary contact\n \nWeb: http:\/\/sandro.groganz.com\/contact\/Year published\n \n2011Volume and issue\n \nAugust 2011Distribution license\n \nCreative Commons Attribution 3.0 UnportedWebsite\n \nhttp:\/\/timreview.ca\/article\/463\n\n\n\n\n \n This article originally appeared in an online magazine and is not a traditional journal article, thus the lack of citations. \n\n\n\"The most important single central fact about a free market is that no exchange takes place unless both parties benefit.\" - Milton Friedman\nContents\n\n1 Abstract \n2 Introduction \n3 Vendor-driven ecosystems \n4 Business strategies for partners \n\n4.1 Key strategic elements \n4.2 Common business strategies \n4.3 Comparison of partner and vendor business strategies \n4.4 Common business scenarios \n\n\n5 Community benefits for partners \n\n5.1 Efficient marketing \n5.2 Cost savings and innovation \n5.3 Sustainable growth \n\n\n6 Conclusion \n7 Notes \n\n\n\nAbstract \nOpen source vendors can benefit from business ecosystems that form around their products. Partners of such vendors can utilize this ecosystem for their own business benefit by understanding the structure of the ecosystem, the key actors and their relationships, and the main levers of profitability. This article provides information on all of these aspects and identifies common business scenarios for partners of open source vendors. Armed with this information, partners can select a strategy that allows them to participate in the ecosystem while also maximizing their gains and driving adoption of their product or solution in the marketplace.\n\nIntroduction \nEvery free\/libre open source software (F\/LOSS) vendor strives to create a business ecosystem around its software product. Doing this offers two primary advantages from a sales and marketing perspective: i) it increases the viability and longevity of the product in both commercial and communal spaces, and ii) it opens up new channels for communication and innovation.\nAccess to source code, coupled with open communication channels, exposes each actor within the ecosystem to higher competitive pressures, but it also creates greater possibilities for cooperation. This so-called coopetition (simultaneous cooperation and competition) is a common phenomenon in open source communities. In addition, individual actors in an open system can achieve higher visibility, which adds to their reputation and public profile within the ecosystem and can make it easier for them to market themselves or their services.\nFor partners of F\/LOSS vendors, therefore, there are some key questions they must ask in order to maximize their gains from the ecosystem and attain positive business benefits from their involvement and investment of time or money:\n\n What is the structure of the ecosystem?\n Who are the actors in the ecosystem and how do they relate to each other?\n How do I benefit from co-opetition and higher visibility in this ecosystem?\nThis article presents a practitioner's view of an aspect of business ecosystems that has been largely neglected by analysts and authors. It illustrates how a partner network can leverage the same business strategies within an open source ecosystem as the vendor who owns the code. Attention is focused on how long-term, win-win situations can be established that will enable not only cost savings for partners but also innovation and the opening of new markets. Insight into the structures and relationships of open source ecosystems is offered with the aim of giving partners of F\/LOSS vendors a strategic foundation for their interactions with the community.\nThe ideas discussed in this article are illustrated with reference to an existing F\/LOSS product, OXID eShop, and its vendor, OXID eSales. OXID eShop is an ecommerce system written in PHP. In November 2008, OXID eSales made OXID eShop available under the GNU General Public Licence (GPL). This opening of the source code was accompanied by the transformation and expansion of the OXID business ecosystem, which previously centered on a proprietary product. Today it comprises the users, partners, and developers of OXID eShop, together with the vendor, OXID eSales.\n\nVendor-driven ecosystems \nEvery F\/LOSS vendor strives to create a vendor-driven, open source ecosystem around his or her software product. Doing this is advantageous for the vendor for two reasons: i) it increases the long-term viability and attractiveness of the product in the marketplace, and ii) it allows the vendor to derive a business advantage from its superior product expertise, as owner of the software copyright, trademark, and other intellectual property.\nThe best known example of such an ecosystem is MySQL. The former CEO of MySQL, M\u00e5rten Mickos, referred to this dual-licensing model as \"the cathedral in the bazaar\" (Figure 1). This is, of course, an allusion to Eric S. Raymond's famous essay \"The Cathedral and the Bazaar\", in which Raymond discusses the benefits of an open and public software development process (the \"bazaar\") as opposed to one in which development occurs within a restricted group (the \"cathedral\").\n\n\n\nFigure 1: A Vendor-Driven Ecosystem Model\nThe OXID ecosystem conforms closely to this model. The vendor, OXID eSales, is at the centre of the ecosystem. Business partners are grouped close to the vendor; they have direct relationships with both OXID eSales and with their own customers, and they serve as a two-way link for information exchange between customers and the vendor, and for joint cooperation on customer projects.\nPartners are also part of the community and have direct contact with users of the OXID eShop. For example, software developers of partner companies are active in the OXID eShop forums and mailing lists, participating and communicating with users by solving common problems, providing advice on deployment, or creating new modules.\nIn a vendor-driven ecosystem, the manufacturer acts as a link between the community and the customer. On the one hand, it is part of the community with which it trades; on the other hand, it serves customer wishes through its partner network (Figure 2).\n\n\n\nFigure 2: Relationships in a Vendor-Driven Ecosystem\nThe main difference between \"customer space\" and \"community space\" in this ecosystem is compensation for time spent. Customers pay money to partners to obtain services or additional software products. Within the community, recognition is the key currency. Or, to put it another way, in customer space, one invests money to save time whereas, in community space, one invests time to save money.\nThis ecosystem also supports one or more \"lone wolves\", which are recognized experts that independently implement customer projects or work with partners as subcontractors.\n\nBusiness strategies for partners \nA wide range of business strategies are available to partners of F\/LOSS vendors within a vendor-driven ecosystem under both proprietary and open source licensing models. Partners can employ any element of an open source strategy that is available to the manufacturer. The following sections present these elements, and various possible combinations thereof, within a F\/LOSS vendor-driven ecosystem. However, only the most important variants are presented; an exhaustive analysis is beyond the scope of this article.\n\nKey strategic elements \nThe key strategic elements of a business strategy can be illustrated using a model developed by the analyst firm The 451 Group, as shown in Figure 3. This model defines the key elements of an open source business strategy and serves as a useful tool to consider the strategically valuable levers in an open source ecosystem.\n\n\n\nFigure 3: Elements of an Open Source Business Strategy.\r\nImage used with permission, courtesy of Matthew Aslett and the 451 Group.\nThe following sections describe the important elements of this model for partners of F\/LOSS vendors and illustrate these elements with reference to the OXID ecosystem.\n\n1. Copyright Control: Who Owns What? The question of who owns the copyright to the source code is of central importance, because it determines to a great extent the business model of the vendor and, by extension, the strategies available to partners. For example, a vendor may follow a dual-licensing business model, a subscription model, or a service model.\nTo illustrate, consider that in the case of OXID eShop, it is OXID eSales that owns the copyright (in Figure 3, Copyright control: vendor). So, it is able to offer its product within the framework of a dual licensing model (End-user licensing: dual-licensing). There is an OXID eShop Community Edition (CE) available under an open source GPL license; an OXID eShop Professional Edition (PE) and an Enterprise Edition (EE) are also available under a proprietary license.\n2. Revenue Triggers: Who Earns What? A vendor can earn revenue from multiple sources for the same product. For example, a vendor can sell a commercial license to the product (e.g., Magnolia), a subscription (e.g., RedHat), or a support or training package (e.g., OTRS). Vendors can exploit their copyright control and central position in the business ecosystem to incentivize customers to \"upgrade\" from an open source license to a commercial one, or they can use their in-depth knowledge of the product to sell additional services such as support, training, or integration.\nBy putting the CE under the GPL, OXID eSales created an incentive to acquire the PE or EE as soon as the product is to be used together with other proprietary software (Revenue triggers: closed source license). The GPL is a license that, simply put, requires that software based on the OXID eShop, or that is used together with it, must also be open source (Software license: strong copyleft).\nIn addition, OXID eSales offers support and other services, partly only for the PE and EE (e.g., technical support), partly also for the CE (e.g., training), which establishes another source of income (Revenue triggers: support services and software services).\n3. Development Model: Who Does What? It is also important to consider whether development is driven by the vendor or the community, and if it is conducted in private, in public, or as a private\/public collaboration. On one hand, a completely vendor-driven model that is closed to community input risks alienating users and partnersbecause product managers may well prioritize paying customer feature requests over those of community members. A completely community-driven model, on the other hand, may lack the coordination and close supervision needed to satisfy commercial and service-level commitments.\nOXID eSales has chosen a public development model: the code, the bug tracker, feature tracker, forums, mailing lists, etc., are open to the public (Development model: public). However, the development process of OXID eShop is strongly driven by the manufacturer, OXID eSales (Development model: vendor).\nCommon business strategies \nIn a vendor-driven ecosystem, partners will typically offer services such as custom software development or product training. This is certainly the case with our sample ecosystem, where a large number of official OXID partners offer services as system integrators or web agencies.\nPartners usually own the copyright to customer specific adaptations or extensions of the product. A partner therefore has the entrepreneurial opportunity to offer its customers, for example, an extension as an independent module. Under what license this module can be offered depends on two factors:\n\nDoes the module make use of third-party, open source code?\nIf so, is it reciprocal (requiring that the module itself is also open source) or permissive (allowing the module to be either open source or proprietary)?\nDepending on how these questions are answered, a partner can offer such a module under a single or dual license. For example, the partner has the opportunity of building a dual-licensing model and distributing the module subject to a charge for the commercial license. Of course, the partner may also decide to offer the module under a single proprietary license (subject to a charge) or under a single open source license. If the module should also be offered partly or solely under an open source license, then the partner would benefit from the advantages of rapid dissemination and thus an increased demand for support and software services, such as support services, software services, and custom development.\nA partner module that is available as open source software could be developed publicly or purely in-house. The latter is usually the case for proprietary modules.\n\nComparison of partner and vendor business strategies \nComparing the elements of the open source business model of OXID eSales with those that are available for OXID partners, it is clear that the same elements are available for the partners as for OXID eSales, plus additional ones (Table 1). The strategic options of partners will only be limited, for example, by the license that includes third-party software that was built into a module.\n\n\n\n\n\n\n\nTable 1. Business Strategies in the OXID Ecosystem\n\n\n\n\n OXID eSales\n\n Partners\n\n\nCopyright control\n\nVendor\n\nVendor\n\n\nEnd user licensing\n\nDual licensing\n\nDual licensing\r\nSingle open source\r\nClosed\r\n... and more\n\n\nSoftware license\n\nReciprocal\n\nReciprocal\r\nPermissive\n\n\nRevenue triggers\n\nCommercial license\r\nSupport services\r\nSoftware services\n\nCommercial license\r\nSupport services\r\nSoftware services\r\nCustom development\r\n... and more\n\n\nDevelopment model\n\nPublic\r\nVendor\n\nPublic\r\nPrivate\r\nVendor\r\n... and more\n\n\n\nCommon business scenarios \nA partner will make the decision - about whether a module should be available partly or solely under an open source license - on the basis of its own entrepreneurial motivation and also after taking into account the competitive situation. For example:\n\n If no competing product already exists, it makes perfect sense for a first mover to provide a module under a proprietary license, subject to a fee. This approach will produce the highest margins.\n If a proprietary module already exists, it could be a sensible decision to provide the competing module for free and under an open source license. This approach will guarantee a high degree of distribution. A partner could then generate revenue from support and professional services, or it could also offer a paid version within the framework of a dual-license model.\nIn any case, partners in an open source ecosystem are under extreme pressure to be efficient and have to expect - even more than in a purely proprietary environment - that someone will offer the same module free of charge and that their revenue may be generated purely from services.\n\nCommunity benefits for partners \nAn open source ecosystem also offers other opportunities and benefits for partners. For example, it gives them the opportunity to establish a name for themselves through collaboration with the open source community in the development of modules, themes, libraries, widgets, and so on. This benefit accrues regardless of whether a partner has created an open source module or merely contributes to it. It also accrues to partners who generate their companies revenues exclusively through services.\nPartners of F\/LOSS vendors can thus choose from a wide range of elements for an open source business model within the vendor ecosystem. Moreover, they can also profit from the same advantages in marketing, software development costs, service and product innovation, and business growth, which the vendor has also gained for itself.\n\nEfficient marketing \nAs an open system with open communication channels, an open source community provides the opportunity to generate high visibility with little effort. So-called word-of-mouth marketing can be wonderfully implemented here, especially with social media tools, such as Twitter and Weblogs. Thus, for example, a new module may quickly become known within the community with little effort on the part of the vendor or partner.\nThis environment is ideal for lead generation, although usually this does not occur through direct contact with customers. A code snippet that facilitates previously missing functionality could, for example, come to the attention of the vendor or another partner. If this code can be used in a prospective customer project, but still needs additional extensions, who better than the author of the code for making the extensions?\nTo illustrate the efficiency benefits of co-marketing in the OXID eShop ecosystem, consider some examples.\n\n Joint press releases for a new case study benefit not only OXID eSales and the partners involved, but the entire OXID ecosystem. The community as a whole, which also includes the partner network, benefits from the overall higher visibility of OXID eShop, which in turn means a potential increase in demand for the product.\n Through its two freely accessible software distribution channels, OXID eXchange and its collaborative development channel OXIDprojects, OXID eSales gives partners the opportunity to offer their proprietary or open source extensions in a cost-effective manner.\n Officially-certified OXID partners have the advantage of enjoying higher confidence from potential customers than businesses that cannot show preferred business relationships with OXID eSales.\nCost savings and innovation \nCompared to the proprietary model, the open source development model is generally considered to be more efficient, including in terms of costs. Partners can take advantage of this efficiency within a vendor-driven ecosystem in a number of ways:\n\n1. Cost savings due to better error detection and correction. First, partners can use the existing software development infrastructure to develop their own open source extensions. The development community can see the source code, report errors, propose new features, or discuss ideas and issues in forums or mailing lists. For example, for the OXID community, OXIDprojects provides a collaborative, open development environment that allows partners and users to participate in the development of OXID eShop.\nThis mechanism allows for medium to long-term cost savings because errors are discovered by others before they lead to unnecessary additional costs or even recourse claims in a customer project. If it so happens that a community member provides the bug-fix or patch, it saves time in addition to cost.\n2. Innovation due to easier information exchange. The open ecosystem also leads to a high degree of innovation because of open information exchange. If community members try an open source extension and find it valuable, then they may generate ideas to expand it or use it in other ways.\nPartners acting within the open source community will benefit from continuous innovation, which also extends to in-house expertise and will increase the efficiency and quality of sales pitches and customer projects.\n3. Lower project risk due to easy availability of community expertise. The community also serves partners as a pool of free or permanent employees. Community members can be used at relatively short notice in customer projects on a consulting or freelance basis and, after successfully completing several joint customer projects, may even be considered for a permanent position. For a partner, this reduces the risk of hiring employees who are not well qualified, which also saves costs.\nSustainable growth \nA properly designed vendor-driven ecosystem offers partners the ability to grow in a sustainable manner. This is mainly due to the transparency of public communications, which produces a higher level of confidence. Within such an ecosystem, small companies that are not official partners and that only use the open source edition of the product for customer projects also have the chance to participate in the ecosystem. These companies are well received within the ecosystem because it allows them to grow in a sustainable manner.\nPartners can also easily build their own ecosystems within the vendor ecosystem and benefit from it. Consider Figure 4, which illustrates how a partner can sustain and grow its business within the scope of a broader vendor-driven ecosystem.\n\n\n\nFigure 4: Partner Ecosystems\nAs an example of the opportunities for growth, the OXID ecosystem includes firms that are not official partners, but that continue to use and deploy the CE product in client projects. The ecosystem sustains them and gives them the tools and support to transition from using only the CE product to eventually becoming an official OXID partner selling the PE and EE products along with vendor support. Apart from this, the OXID eShop ecosystem benefits from each user of the CE, because each user increases the overall product quality through feedback, and this in turn, is beneficial for the PE and the EE products.\n\nConclusion \nThis article has offered some insight into strategies that partners can use within a F\/LOSS vendor\u2019s ecosystem and the benefits that will accrue to those partners. The most important point to remember is this: an open source community is not a state but a process. When you act as a partner to an F\/LOSS vendor and build your own ecosystem within the vendor\u2019s ecosystem, you will be exposed to a diverse community and many new opportunities. Evolving your business to meet and exploit these opportunities is an exciting, ongoing process and one that, handled correctly, creates a foundation for long-term growth and sustainability.\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Benefits_of_the_community_for_partners_of_open_source_vendors\">https:\/\/www.limswiki.org\/index.php\/Journal:Benefits_of_the_community_for_partners_of_open_source_vendors<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on open source\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 19 January 2016, at 16:11.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,102 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","2d0bddc8ac63fbe42ecf34c128173d02_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Benefits_of_the_community_for_partners_of_open_source_vendors skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Benefits of the community for partners of open source vendors<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n<blockquote><i>\"The most important single central fact about a free market is that no exchange takes place unless both parties benefit.\"<\/i> - Milton Friedman<\/blockquote>\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Open source vendors can benefit from business ecosystems that form around their products. Partners of such vendors can utilize this ecosystem for their own business benefit by understanding the structure of the ecosystem, the key actors and their relationships, and the main levers of profitability. This article provides <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> on all of these aspects and identifies common business scenarios for partners of open source vendors. Armed with this information, partners can select a strategy that allows them to participate in the ecosystem while also maximizing their gains and driving adoption of their product or solution in the marketplace.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Every <a href=\"https:\/\/www.limswiki.org\/index.php\/Free_and_open-source_software#FLOSS\" title=\"Free and open-source software\" target=\"_blank\" class=\"wiki-link\" data-key=\"7ebfaa4419be758c507232ec468fbdfc\">free\/libre open source software<\/a> (F\/LOSS) vendor strives to create a business ecosystem around its software product. Doing this offers two primary advantages from a sales and marketing perspective: i) it increases the viability and longevity of the product in both commercial and communal spaces, and ii) it opens up new channels for communication and innovation.\n<\/p><p>Access to source code, coupled with open communication channels, exposes each actor within the ecosystem to higher competitive pressures, but it also creates greater possibilities for cooperation. This so-called coopetition (simultaneous cooperation and competition) is a common phenomenon in open source communities. In addition, individual actors in an open system can achieve higher visibility, which adds to their reputation and public profile within the ecosystem and can make it easier for them to market themselves or their services.\n<\/p><p>For partners of F\/LOSS vendors, therefore, there are some key questions they must ask in order to maximize their gains from the ecosystem and attain positive business benefits from their involvement and investment of time or money:\n<\/p>\n<ol><li> What is the structure of the ecosystem?<\/li>\n<li> Who are the actors in the ecosystem and how do they relate to each other?<\/li>\n<li> How do I benefit from co-opetition and higher visibility in this ecosystem?<\/li><\/ol>\n<p>This article presents a practitioner's view of an aspect of business ecosystems that has been largely neglected by analysts and authors. It illustrates how a partner network can leverage the same business strategies within an open source ecosystem as the vendor who owns the code. Attention is focused on how long-term, win-win situations can be established that will enable not only cost savings for partners but also innovation and the opening of new markets. Insight into the structures and relationships of open source ecosystems is offered with the aim of giving partners of F\/LOSS vendors a strategic foundation for their interactions with the community.\n<\/p><p>The ideas discussed in this article are illustrated with reference to an existing F\/LOSS product, <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.oxid-esales.com\/\" target=\"_blank\">OXID eShop<\/a>, and its vendor, OXID eSales. OXID eShop is an ecommerce system written in <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/php.net\/\" target=\"_blank\">PHP<\/a>. In November 2008, OXID eSales made OXID eShop available under the GNU General Public Licence (<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.gnu.org\/licenses\/gpl.html\" target=\"_blank\">GPL<\/a>). This opening of the source code was accompanied by the transformation and expansion of the OXID business ecosystem, which previously centered on a proprietary product. Today it comprises the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.oxid-esales.com\/en\/references\" target=\"_blank\">users<\/a>, <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.oxid-esales.com\/en\/partners\/partnerworld\/partner-model\" target=\"_blank\">partners<\/a>, and <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.oxidforge.org\/\" target=\"_blank\">developers<\/a> of OXID eShop, together with the vendor, OXID eSales.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Vendor-driven_ecosystems\">Vendor-driven ecosystems<\/span><\/h2>\n<p>Every F\/LOSS vendor strives to create a vendor-driven, open source ecosystem around his or her software product. Doing this is advantageous for the vendor for two reasons: i) it increases the long-term viability and attractiveness of the product in the marketplace, and ii) it allows the vendor to derive a business advantage from its superior product expertise, as owner of the software copyright, trademark, and other intellectual property.\n<\/p><p>The best known example of such an ecosystem is <a href=\"https:\/\/www.limswiki.org\/index.php\/MySQL\" title=\"MySQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"35005451bfcd508bce47c58e72260128\">MySQL<\/a>. The former CEO of MySQL, M\u00e5rten Mickos, referred to this dual-licensing model as \"the cathedral in the bazaar\" (Figure 1). This is, of course, an allusion to Eric S. Raymond's famous essay \"<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.catb.org\/esr\/writings\/cathedral-bazaar\/\" target=\"_blank\">The Cathedral and the Bazaar<\/a>\", in which Raymond discusses the benefits of an open and public software development process (the \"bazaar\") as opposed to one in which development occurs within a restricted group (the \"cathedral\").\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_GroganzOpenSourceBR2011_Aug.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2c7af7a505da6157279c36c7e9133e4a\"><img alt=\"Fig1 GroganzOpenSourceBR2011 Aug.png\" src=\"https:\/\/www.limswiki.org\/images\/0\/08\/Fig1_GroganzOpenSourceBR2011_Aug.png\" width=\"350\" height=\"360\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1: A Vendor-Driven Ecosystem Model<\/b><\/blockquote>\n<p>The OXID ecosystem conforms closely to this model. The vendor, OXID eSales, is at the centre of the ecosystem. Business partners are grouped close to the vendor; they have direct relationships with both OXID eSales and with their own customers, and they serve as a two-way link for information exchange between customers and the vendor, and for joint cooperation on customer projects.\n<\/p><p>Partners are also part of the community and have direct contact with users of the OXID eShop. For example, software developers of partner companies are active in the OXID eShop forums and mailing lists, participating and communicating with users by solving common problems, providing advice on deployment, or creating new modules.\n<\/p><p>In a vendor-driven ecosystem, the manufacturer acts as a link between the community and the customer. On the one hand, it is part of the community with which it trades; on the other hand, it serves customer wishes through its partner network (Figure 2).\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_GroganzOpenSourceBR2011_Aug.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"ff3b7c235aab04754c2e3568bd54085d\"><img alt=\"Fig2 GroganzOpenSourceBR2011 Aug.png\" src=\"https:\/\/www.limswiki.org\/images\/d\/da\/Fig2_GroganzOpenSourceBR2011_Aug.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2: Relationships in a Vendor-Driven Ecosystem<\/b><\/blockquote>\n<p>The main difference between \"customer space\" and \"community space\" in this ecosystem is compensation for time spent. Customers pay money to partners to obtain services or additional software products. Within the community, recognition is the key currency. Or, to put it another way, in customer space, one invests money to save time whereas, in community space, one invests time to save money.\n<\/p><p>This ecosystem also supports one or more \"lone wolves\", which are recognized experts that independently implement customer projects or work with partners as subcontractors.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Business_strategies_for_partners\">Business strategies for partners<\/span><\/h2>\n<p>A wide range of business strategies are available to partners of F\/LOSS vendors within a vendor-driven ecosystem under both proprietary and open source licensing models. Partners can employ any element of an open source strategy that is available to the manufacturer. The following sections present these elements, and various possible combinations thereof, within a F\/LOSS vendor-driven ecosystem. However, only the most important variants are presented; an exhaustive analysis is beyond the scope of this article.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Key_strategic_elements\">Key strategic elements<\/span><\/h3>\n<p>The key strategic elements of a business strategy can be illustrated using a model developed by the analyst firm <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.451group.com\/\" target=\"_blank\">The 451 Group<\/a>, as shown in Figure 3. This model defines the key elements of an open source business strategy and serves as a useful tool to consider the strategically valuable levers in an open source ecosystem.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_GroganzOpenSourceBR2011_Aug.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b4b5127e4693fd23b8432b1712548390\"><img alt=\"Fig3 GroganzOpenSourceBR2011 Aug.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/ba\/Fig3_GroganzOpenSourceBR2011_Aug.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3: Elements of an Open Source Business Strategy<\/b>.<br \/>Image used with permission, courtesy of <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/blogs.the451group.com\/opensource\/2011\/01\/06\/updated-open-source-business-strategy-framework\/\" target=\"_blank\">Matthew Aslett and the 451 Group<\/a>.<\/blockquote>\n<p>The following sections describe the important elements of this model for partners of F\/LOSS vendors and illustrate these elements with reference to the OXID ecosystem.\n<\/p>\n<dl><dd>1. <b>Copyright Control: Who Owns What?<\/b> The question of who owns the copyright to the source code is of central importance, because it determines to a great extent the business model of the vendor and, by extension, the strategies available to partners. For example, a vendor may follow a dual-licensing business model, a subscription model, or a service model.<\/dd><\/dl>\n<dl><dd>To illustrate, consider that in the case of OXID eShop, it is OXID eSales that owns the copyright (in Figure 3, Copyright control: vendor). So, it is able to offer its product within the framework of a dual licensing model (End-user licensing: dual-licensing). There is an OXID eShop Community Edition (CE) available under an open source GPL license; an OXID eShop Professional Edition (PE) and an Enterprise Edition (EE) are also available under a proprietary license.<\/dd><\/dl>\n<dl><dd>2. <b>Revenue Triggers: Who Earns What?<\/b> A vendor can earn revenue from multiple sources for the same product. For example, a vendor can sell a commercial license to the product (e.g., <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.magnolia-cms.com\/\" target=\"_blank\">Magnolia<\/a>), a subscription (e.g., <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.redhat.com\/\" target=\"_blank\">RedHat<\/a>), or a support or training package (e.g., <a href=\"https:\/\/www.limswiki.org\/index.php\/OTRS\" title=\"OTRS\" target=\"_blank\" class=\"wiki-link\" data-key=\"2991eabead1ef579b31f06171457ff16\">OTRS<\/a>). Vendors can exploit their copyright control and central position in the business ecosystem to incentivize customers to \"upgrade\" from an open source license to a commercial one, or they can use their in-depth knowledge of the product to sell additional services such as support, training, or integration.<\/dd><\/dl>\n<dl><dd>By putting the CE under the GPL, OXID eSales created an incentive to acquire the PE or EE as soon as the product is to be used together with other proprietary software (Revenue triggers: closed source license). The GPL is a license that, simply put, requires that software based on the OXID eShop, or that is used together with it, must also be open source (Software license: strong copyleft).<\/dd><\/dl>\n<dl><dd>In addition, OXID eSales offers support and other services, partly only for the PE and EE (e.g., technical support), partly also for the CE (e.g., training), which establishes another source of income (Revenue triggers: support services and software services).<\/dd><\/dl>\n<dl><dd>3. <b>Development Model: Who Does What?<\/b> It is also important to consider whether development is driven by the vendor or the community, and if it is conducted in private, in public, or as a private\/public collaboration. On one hand, a completely vendor-driven model that is closed to community input risks alienating users and partnersbecause product managers may well prioritize paying customer feature requests over those of community members. A completely community-driven model, on the other hand, may lack the coordination and close supervision needed to satisfy commercial and service-level commitments.<\/dd><\/dl>\n<dl><dd>OXID eSales has chosen a public development model: the code, the bug tracker, feature tracker, forums, mailing lists, etc., are open to the public (Development model: public). However, the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/wiki.oxidforge.org\/Development_Process\" target=\"_blank\">development process of OXID eShop<\/a> is strongly driven by the manufacturer, OXID eSales (Development model: vendor).<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"Common_business_strategies\">Common business strategies<\/span><\/h3>\n<p>In a vendor-driven ecosystem, partners will typically offer services such as custom software development or product training. This is certainly the case with our sample ecosystem, where a large number of official OXID partners offer services as system integrators or web agencies.\n<\/p><p>Partners usually own the copyright to customer specific adaptations or extensions of the product. A partner therefore has the entrepreneurial opportunity to offer its customers, for example, an extension as an independent module. Under what license this module can be offered depends on two factors:\n<\/p>\n<ol><li>Does the module make use of third-party, open source code?<\/li>\n<li>If so, is it reciprocal (requiring that the module itself is also open source) or permissive (allowing the module to be either open source or proprietary)?<\/li><\/ol>\n<p>Depending on how these questions are answered, a partner can offer such a module under a single or dual license. For example, the partner has the opportunity of building a dual-licensing model and distributing the module subject to a charge for the commercial license. Of course, the partner may also decide to offer the module under a single proprietary license (subject to a charge) or under a single open source license. If the module should also be offered partly or solely under an open source license, then the partner would benefit from the advantages of rapid dissemination and thus an increased demand for support and software services, such as support services, software services, and custom development.\n<\/p><p>A partner module that is available as open source software could be developed publicly or purely in-house. The latter is usually the case for proprietary modules.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Comparison_of_partner_and_vendor_business_strategies\">Comparison of partner and vendor business strategies<\/span><\/h3>\n<p>Comparing the elements of the open source business model of OXID eSales with those that are available for OXID partners, it is clear that the same elements are available for the partners as for OXID eSales, plus additional ones (Table 1). The strategic options of partners will only be limited, for example, by the license that includes third-party software that was built into a module.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 1. Business Strategies in the OXID Ecosystem<\/b>\n<\/td><\/tr>\n<tr>\n<th>\n<\/th>\n<th> OXID eSales\n<\/th>\n<th> Partners\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-top:10px;\"><b>Copyright control<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-top:10px;\">Vendor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-top:10px;\">Vendor\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>End user licensing<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Dual licensing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-top:10px;\">Dual licensing<br \/>Single open source<br \/>Closed<br \/>... and more\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Software license<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reciprocal\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-top:10px;\">Reciprocal<br \/>Permissive\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Revenue triggers<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Commercial license<br \/>Support services<br \/>Software services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-top:10px;\">Commercial license<br \/>Support services<br \/>Software services<br \/>Custom development<br \/>... and more\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>Development model<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Public<br \/>Vendor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; padding-top:10px;\">Public<br \/>Private<br \/>Vendor<br \/>... and more\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Common_business_scenarios\">Common business scenarios<\/span><\/h3>\n<p>A partner will make the decision - about whether a module should be available partly or solely under an open source license - on the basis of its own entrepreneurial motivation and also after taking into account the competitive situation. For example:\n<\/p>\n<ol><li> If no competing product already exists, it makes perfect sense for a first mover to provide a module under a proprietary license, subject to a fee. This approach will produce the highest margins.<\/li>\n<li> If a proprietary module already exists, it could be a sensible decision to provide the competing module for free and under an open source license. This approach will guarantee a high degree of distribution. A partner could then generate revenue from support and professional services, or it could also offer a paid version within the framework of a dual-license model.<\/li><\/ol>\n<p>In any case, partners in an open source ecosystem are under extreme pressure to be efficient and have to expect - even more than in a purely proprietary environment - that someone will offer the same module free of charge and that their revenue may be generated purely from services.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Community_benefits_for_partners\">Community benefits for partners<\/span><\/h2>\n<p>An open source ecosystem also offers other opportunities and benefits for partners. For example, it gives them the opportunity to establish a name for themselves through collaboration with the open source community in the development of modules, themes, libraries, widgets, and so on. This benefit accrues regardless of whether a partner has created an open source module or merely contributes to it. It also accrues to partners who generate their companies revenues exclusively through services.\n<\/p><p>Partners of F\/LOSS vendors can thus choose from a wide range of elements for an open source business model within the vendor ecosystem. Moreover, they can also profit from the same advantages in marketing, software development costs, service and product innovation, and business growth, which the vendor has also gained for itself.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Efficient_marketing\">Efficient marketing<\/span><\/h3>\n<p>As an open system with open communication channels, an open source community provides the opportunity to generate high visibility with little effort. So-called word-of-mouth marketing can be wonderfully implemented here, especially with social media tools, such as Twitter and Weblogs. Thus, for example, a new module may quickly become known within the community with little effort on the part of the vendor or partner.\n<\/p><p>This environment is ideal for lead generation, although usually this does not occur through direct contact with customers. A code snippet that facilitates previously missing functionality could, for example, come to the attention of the vendor or another partner. If this code can be used in a prospective customer project, but still needs additional extensions, who better than the author of the code for making the extensions?\n<\/p><p>To illustrate the efficiency benefits of co-marketing in the OXID eShop ecosystem, consider some examples.\n<\/p>\n<ol><li> Joint press releases for a new case study benefit not only OXID eSales and the partners involved, but the entire OXID ecosystem. The community as a whole, which also includes the partner network, benefits from the overall higher visibility of OXID eShop, which in turn means a potential increase in demand for the product.<\/li>\n<li> Through its two freely accessible software distribution channels, <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.oxid-esales.com\/en\/exchange\" target=\"_blank\">OXID eXchange<\/a> and its collaborative development channel <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/projects.oxidforge.org\/\" target=\"_blank\">OXIDprojects<\/a>, OXID eSales gives partners the opportunity to offer their proprietary or open source extensions in a cost-effective manner.<\/li>\n<li> Officially-certified OXID partners have the advantage of enjoying higher confidence from potential customers than businesses that cannot show preferred business relationships with OXID eSales.<\/li><\/ol>\n<h3><span class=\"mw-headline\" id=\"Cost_savings_and_innovation\">Cost savings and innovation<\/span><\/h3>\n<p>Compared to the proprietary model, the open source development model is generally considered to be more efficient, including in terms of costs. Partners can take advantage of this efficiency within a vendor-driven ecosystem in a number of ways:\n<\/p>\n<dl><dd>1. <b>Cost savings due to better error detection and correction.<\/b> First, partners can use the existing software development infrastructure to develop their own open source extensions. The development community can see the source code, report errors, propose new features, or discuss ideas and issues in forums or mailing lists. For example, for the OXID community, OXIDprojects provides a collaborative, open development environment that allows partners and users to participate in the development of OXID eShop.<\/dd><\/dl>\n<dl><dd>This mechanism allows for medium to long-term cost savings because errors are discovered by others before they lead to unnecessary additional costs or even recourse claims in a customer project. If it so happens that a community member provides the bug-fix or patch, it saves time in addition to cost.<\/dd><\/dl>\n<dl><dd>2. <b>Innovation due to easier information exchange.<\/b> The open ecosystem also leads to a high degree of innovation because of open information exchange. If community members try an open source extension and find it valuable, then they may generate ideas to expand it or use it in other ways.<\/dd><\/dl>\n<dl><dd>Partners acting within the open source community will benefit from continuous innovation, which also extends to in-house expertise and will increase the efficiency and quality of sales pitches and customer projects.<\/dd><\/dl>\n<dl><dd>3. <b>Lower project risk due to easy availability of community expertise.<\/b> The community also serves partners as a pool of free or permanent employees. Community members can be used at relatively short notice in customer projects on a consulting or freelance basis and, after successfully completing several joint customer projects, may even be considered for a permanent position. For a partner, this reduces the risk of hiring employees who are not well qualified, which also saves costs.<\/dd><\/dl>\n<h3><span class=\"mw-headline\" id=\"Sustainable_growth\">Sustainable growth<\/span><\/h3>\n<p>A properly designed vendor-driven ecosystem offers partners the ability to grow in a sustainable manner. This is mainly due to the transparency of public communications, which produces a higher level of confidence. Within such an ecosystem, small companies that are not official partners and that only use the open source edition of the product for customer projects also have the chance to participate in the ecosystem. These companies are well received within the ecosystem because it allows them to grow in a sustainable manner.\n<\/p><p>Partners can also easily build their own ecosystems within the vendor ecosystem and benefit from it. Consider Figure 4, which illustrates how a partner can sustain and grow its business within the scope of a broader vendor-driven ecosystem.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_GroganzOpenSourceBR2011_Aug.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"133cbd802adc4222f20a4e415eeb2eb4\"><img alt=\"Fig4 GroganzOpenSourceBR2011 Aug.png\" src=\"https:\/\/www.limswiki.org\/images\/4\/43\/Fig4_GroganzOpenSourceBR2011_Aug.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4: Partner Ecosystems<\/b><\/blockquote>\n<p>As an example of the opportunities for growth, the OXID ecosystem includes firms that are not official partners, but that continue to use and deploy the CE product in client projects. The ecosystem sustains them and gives them the tools and support to transition from using only the CE product to eventually becoming an official OXID partner selling the PE and EE products along with vendor support. Apart from this, the OXID eShop ecosystem benefits from each user of the CE, because each user increases the overall product quality through feedback, and this in turn, is beneficial for the PE and the EE products.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>This article has offered some insight into strategies that partners can use within a F\/LOSS vendor\u2019s ecosystem and the benefits that will accrue to those partners. The most important point to remember is this: an open source community is not a state but a process. When you act as a partner to an F\/LOSS vendor and build your own ecosystem within the vendor\u2019s ecosystem, you will be exposed to a diverse community and many new opportunities. Evolving your business to meet and exploit these opportunities is an exciting, ongoing process and one that, handled correctly, creates a foundation for long-term growth and sustainability.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210238\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.132 seconds\nReal time usage: 0.922 seconds\nPreprocessor visited node count: 2126\/1000000\nPreprocessor generated node count: 13247\/1000000\nPost\u2010expand include size: 9047\/2097152 bytes\nTemplate argument size: 2448\/2097152 bytes\nHighest expansion depth: 8\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 74.269 1 - -total\n 86.03% 63.890 1 - Template:Infobox_journal_article\n 82.83% 61.516 1 - Template:Infobox\n 47.71% 35.432 80 - Template:Infobox\/row\n 10.79% 8.014 1 - Template:Ombox\n 6.47% 4.807 1 - Template:Ombox\/core\n 2.21% 1.644 4 - Template:Clear\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7902-0!*!0!!en!5!* and timestamp 20181213210237 and revision id 23675\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Benefits_of_the_community_for_partners_of_open_source_vendors\">https:\/\/www.limswiki.org\/index.php\/Journal:Benefits_of_the_community_for_partners_of_open_source_vendors<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","2d0bddc8ac63fbe42ecf34c128173d02_images":["https:\/\/www.limswiki.org\/images\/0\/08\/Fig1_GroganzOpenSourceBR2011_Aug.png","https:\/\/www.limswiki.org\/images\/d\/da\/Fig2_GroganzOpenSourceBR2011_Aug.png","https:\/\/www.limswiki.org\/images\/b\/ba\/Fig3_GroganzOpenSourceBR2011_Aug.jpg","https:\/\/www.limswiki.org\/images\/4\/43\/Fig4_GroganzOpenSourceBR2011_Aug.png"],"2d0bddc8ac63fbe42ecf34c128173d02_timestamp":1544734957,"6865b9c249ccc54417e80c99d92bb99c_type":"article","6865b9c249ccc54417e80c99d92bb99c_title":"Why health services research needs geoinformatics: Rationale and case example (Onega et al. 2014)","6865b9c249ccc54417e80c99d92bb99c_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example","6865b9c249ccc54417e80c99d92bb99c_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Why health services research needs geoinformatics: Rationale and case example\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nWhy health services research needs geoinformatics: Rationale and case exampleJournal\n \nJournal of Health & Medical InformaticsAuthor(s)\n \nOnega, Tracy; Alford-Teaster, Jennifer; Andrews, Steven; Ganoe, Craig; Perez, Mike; King, David; Shi, XunAuthor affiliation(s)\n \nGeisel School of Medicine at Dartmouth; Exaptive, Inc.; Dartmouth CollegePrimary contact\n \nEmail: jennifer.a.alford-teaster@dartmouth.edu; Phone: 603-653-6088Year published\n \n2014Volume and issue\n \n5 (6)Page(s)\n \n176DOI\n \n10.4172\/2157-7420.1000176ISSN\n \n2157-7420Distribution license\n \nCreative Commons Attribution License (journal fails to specify which version)Website\n \nhttp:\/\/www.omicsonline.org\/open-access\/Download\n \nhttp:\/\/www.omicsonline.org\/open-access\/ (PDF)\n\nContents\n\n1 Introduction \n2 Shared history of geography and health services research: Successes and limitations \n3 A health services research problem: Diffusion of medical technology \n4 Measuring near-real time diffusion of breast imaging technology \n5 Conclusion \n6 Acknowledgements \n7 References \n8 Notes \n\n\n\nIntroduction \nDelivery of health care in the United States has become increasingly complex over the past 50 years, as health care markets have evolved, technology has diffused, population demographics have shifted, and cultural expectations of health and health care have been transformed. Identifying and understanding important patterns of health care services, accessibility, utilization, and outcomes can best be accomplished by combining data from all of these dimensions in near-real time. The Big Data paradigm provides a new framework to bring together very large volumes of data from a variety of sources and formats, with computing capacity to derive new information, hypotheses, and inferences.[1][2] The complementary fields of genomics and bioinformatics have already made great advances only made possible by Big Data approaches. Similar gains can be made by pairing health services research with geoinformatics \u2013- defined as \u201cthe science and technology dealing with the structure and character of spatial information, its capture, its classification and qualification, its storage, processing, portrayal and dissemination, including the infrastructure necessary to secure optimal use of this information\u201d.[3] Integrating geospatial technologies with health services research brings informatics approaches, data sciences, and spatial theories of health and healthcare together to explore relationships among geography, health, and delivery of care in novel ways made possible through geoinformatics. Syngergy between the two disciplines will enhance our ability to discover how health care is delivered most effectively for the greatest health benefits across populations.\n\nShared history of geography and health services research: Successes and limitations \nHealth services research and geography have intersected under the rubrics of medical geography, epidemiology of health care, small area analysis, and public health since ancient Roman times, but more formally since the 1930s. Then, James Glover \u2014 an English physician \u2014 noticed that tonsillectomies were occurring at highly variable rates across school districts, which could not be explained by geographic, socio-demographic, or clinical factors, and the most likely explanation suggested being differences in how physicians practice.[4] Similar work, begun in the 1970s, became the impetus for the Dartmouth Atlas of Health Care[5], which used health care utilization data to develop geographic units representing health care markets. These health care-based spatial units can be directly compared to identify patterns associated with both effective and ineffective care. Other notable examples of health services research linked to a geospatial framework can be found in the public health arena, with planning of population-based vaccination programs and designation of federally-qualified health centers. Despite these tremendous contributions, limitations exist that geoinformatics and health service research are now poised to overcome as information technology and the digital era expands data availability, accessibility, usability, and timely knowledge generation.\nUnderstanding spatiotemporal distributions of health services is a fundamental aspect of health services research upon which studies of utilization, outcomes, comparative effectiveness, resource allocation, and others are based. Four key limitations can be found in typical approaches to measuring health services distributions: 1. retrospective methods; 2. limited geographic extents; 3. ascertainment challenges; and 4. structured data only; which will be addressed below (Table 1). By combining geoinformatics with health services research, an important domain of questions within the field of medical informatics can be addressed, such as is illustrated with a case example.\n\n\n\n\n\n\n\n Key Current Limitations in Geographic Based Health Services Research\n\n Example Geoinformatics Approaches to Address Current Limitations\n\n\nRetrospective methods available data is usually retrospective with lag in timeliness, so pattern described are rarely current.\n\nUsing web content mining and mobile technology feeds, near\u2013 real time locational data can be obtained.\n\n\nLimited geographic extents Typically a tradeoff exists between rich\/granular data available for small geographic extents, or coarse\/broad data available for large geographic extents.\n\nUse of internet, and mobile- technology based locational data mining, allows for very broad geographic capture, without spatial scale limitations.\n\n\nAscertainment Challenges Locational\/spatial data is often not available at a point location (address) for health services and\/or patients. Locational data may be only at an area level (e.g. ZIP code), or not known with certainty or completeness at all.\n\nUse of internet, and mobile- technology based locational data mining, is based on either point locations (latitude-longitude) from IP address, or address mining that can be automatically geocoded to point location.\n\n\nStructured data only typically only spatial data in structured form \u2014 such as databases or files \u2014 is available. Manual abstraction of spatial data is possible, but limits the scale of examination.\n\nContent Mixing (text and images) allows the use of unstructured data from the internet and mobile technology feeds to obtain locational information that is not captured explicitly in an existing database.\n\n\nAsynchronous evaluation of technology with outcomes because evaluation of new technologies is typically limited by the factors above, evaluation of technology occurs after it is already being used in actual practice, and often in small or non representative areas and or populations. This asynchronicity creates the potential for detrimental outcomes to occur prior to establishing outcomes to be unavailable to populations who may benefit if known that a technology should be available.\n\nBy addressing the limitations noted above, evaluation of technology is more likely to occur in near real time, allowing for outcomes to be determined in a more timely manner, rather than with a temporal lag during which patients and populations may be impacted by negative outcomes not previously understood, or missing out on positive outcomes if the technology is reveled to not be located where a population may benefit.\n\n\n\nTable 1: Summary of major limitations in geographical based health services research and how geoinformatics approaches can be used to address them\nA health services research problem: Diffusion of medical technology \nTechnological innovation is a hallmark of the U.S. health care system, and relies on the backbone of translational research to evaluate effectiveness after efficacy has been established. The full potential of a new technology is determined as clinical improvements and impacts on population health are assessed. Yet typically diffusion and the research to establish effectiveness occur asynchronously. This limits the potential for timely assessment of broad clinical impact and leads to two concerns: 1. overuse of technologies with unproven or minimal benefits in the general population and concomitant unwarranted costs; 2. underuse of technologies that have beneficial effects and improve outcomes for particular patient populations. As medical technology diffuses into generalizable practice, data and research needs arise to address these concerns from a host of stakeholders perspectives, including patients, clinicians, researchers, health care facilities, commercial vendors, payers, health care systems, communities, and regulatory bodies. A prerequisite for assessing new technologies in community practice is knowledge of the locations or extent of diffusion and the populations reached. Data availability and timeliness are critical barriers to establishing this knowledge, contributing to spotty information that is retrospective, often with notable lags. For example, The Dartmouth Atlas of Healthcare[5] and other work[6][7][8] has described geographic variation of medical technologies at a national level, but relied on Medicare data, which are typically available for research with a lag of 2-3 years. Further, Medicare, as well as private insurers, are dependent on the new technology being approved for reimbursement since they rely on billing data (claims), and coverage of the technology may take years following FDA approval and commercial dissemination, thus not able to be ascertained until unique billing codes are implemented. Some data resources may be timely for monitoring diffusion \u2014 such as registries like the Breast Cancer Surveillance Consortium (BCSC) and the HMO Research Network (HMORN)[9][10], and other clinical\/provider networks with health information exchanges and\/or robust electronic health records. However, these data resources are limited in geographic extent, are often not population-based, and may not capture use of new technologies until they are uniquely coded (e.g. CPT \u2013 Common Procedural Terminology), or are reimbursable. Further, most data sources require existing structured data, although natural language processing (NLP) is increasingly applied to unstructured medical record information, which can be powerful, but also time-consuming, complicated and variable across settings.[11][12][13] Achieving national extent, fully ascertained, and timely data capture to characterize geographic and sociodemographic diffusion of new technologies is a critical need, particularly as the U.S. seeks to improve health and healthcare and limit health care costs to effective use within populations.\n\nMeasuring near-real time diffusion of breast imaging technology \nTo more fully understand dissemination of new technologies we need to be able to capture the occurrences of the technology in large geographic areas, and in a dynamic way that reflects the dynamic process that dissemination is. Figure 1 presents a schematic approach to do this, using a breast imaging technology diffusion cased example (Figure 1). We can address the existing limitations in measuring, monitoring, and characterizing health care diffusion \u2014 with breast imaging as a timely example, given new technologies, such as digital breast tomosynthesis (DBT)[14][15][16], and legislation related to breast density notification [17]. With a geospatial semantic web[17][18][19][20][21], which combines web mining techniques with geographic information systems and census data, one can ascertain geographic uptake of DBT nationally, estimate potential access overall and by population subgroups, and identify correlates of dissemination patterns.\n\n\n\nFigure 1. Schematic flow for a geoinformatics application in health services<br \/research to dynamically monitor technology diffusion \nWeb content mining is used in this project to identify instances of DBT based on taxonomy of terms. Using associated web pages from these instances, street address information is captured for the DBT instance. These addresses and related attributes (facility name, date of data capture, etc.) are brought into a GIS for geocoding, spatial joins with other layers (e.g. road network), travel time analysis, and service area creation. Population demographics and other census-derived data are attributed to DBT locations and service areas. This constitutes the database containing geographic extent of DBT diffusion and population characteristics served by DBT facilities, which is refreshed at regular intervals following external validation. This application characterizes heterogeneous diffusion both geographically and socio-demographically, which also serves as a proof of concept for higher-dimension data which incorporates spatial layers, as well as other technologies.\n\nConclusion \nThe application of geoinformatics to health services research has high potential for advancing currently used research methods to monitor and evaluate new technologies as they are translated from experimental settings into communities and populations. A successful application of this approach will yield a validated tool to dynamically integrate geospatial data, population data, and web content for automated discovery and monitoring of technology diffusion. For example, a user interface will provide static functions, such as maps of service locations, derived service areas, density of services available, and populations that are in service catchment areas. Interactive and near-real time functions could produce for user-defined areas, time, and populations: video\/time trends of service locations, derive-time defined service areas, time trends of populations coincident with services, and projected population coverage for actual or potential service locations. Such tools can be readily scalable, applicable to other new technologies, and foundational for further capabilities, such as using geostatistical methods for predictive modeling, visualization, and other \u201cBig Data\u201d analytics.\n\nAcknowledgements \nResearch reported in this (publication\/press release) was supported by The Dartmouth Clinical and Translational Science Institute, under award number UL1TR001086 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH). The content is solely the responsibility of the author(s) and does not necessarily represent the official views of the NIH.\n\nReferences \n\n\n\u2191 Philip Chen, C.L.; Zhang, C.Y. (2014). \"Data-intensive applications, challenges, techniques and technologies: A survey on Big Data\". Information Sciences 275: 314-347. doi:10.1016\/j.ins.2014.01.015.   \n\n\u2191 Kambatla, K.; Kolliasb, G.; Kumar, V.; Grama, A. (2014). \"Trends in big data analytics\". Journal of Parallel and Distributed Computing 74 (7): 2561-2573. doi:10.1016\/j.jpdc.2014.01.003.   \n\n\u2191 Raju, P.L.N. (2004). \"Fundamentals of Geographic Information Systems\" (PDF). Satellite Remote Sensing and GIS Applications in Agricultural Meteorology. World Meteorological Organisation. http:\/\/www.preventionweb.net\/files\/1682_9970.pdf .   \n\n\u2191 Glover, J.A. (1938). \"The incidence of tonsillectomy in school children\". Proceedings of the Royal Society of Medicine 31 (10): 95-113. PMC PMC2076749. PMID 18245048. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2076749 .   \n\n\u2191 5.0 5.1 \"The Dartmouth Atlas of Health Care\". The Dartmouth Institute. http:\/\/www.dartmouthatlas.org\/publications\/articles.aspx .   \n\n\u2191 Onega, T.; Toseston, T.D.; Wang, Q.; Hillner, B.E.; Song, Y.; Siegel, B.A.; Tosteson, A.N. (2012). \"Geographic and sociodemographic variation of PET use in Medicare beneficiaries with cancer\". Journal of the American College of Radiology 9 (9): 635-642. doi:10.1016\/j.jacr.2012.05.005. PMC PMC3830950. PMID 22954545. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3830950 .   \n\n\u2191 Onega, T.; Hubbard, R.; Hill, D.; Lee, C.I.; Haas, J.S.; Carlos, H.A.; Alford-Teaster, J.; Bogart, A.; DeMartini, W.B.; Kerlikowske, K.; Virnig, B.A.; Buist, D.S.; Henderson, L.; Tosteson, A.N. (2014). \"Geographic Access to Breast Imaging for U.S. Women\". Journal of the American College of Radiology 11 (9): 874-882. doi:10.1016\/j.jacr.2014.03.022. PMC PMC4156905. PMID 24889479. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4156905 .   \n\n\u2191 Onega, T.; Duell, E.J.; Shi, X.; Wang, D.; Demidenko, E.; Goodman, D. (2008). \"Geographic access to cancer care in the U.S.\". Cancer 112 (4): 909-918. doi:10.1002\/cncr.23229. PMID 18189295.   \n\n\u2191 \"Breast Cancer Surveillance Consortium\". National Cancer Institute. http:\/\/breastscreening.cancer.gov\/about\/ .   \n\n\u2191 \"HMO Research Network\". Kaiser Permanente Division of Research. http:\/\/www.hcsrn.org\/en\/About\/ .   \n\n\u2191 Xu, R.; Garten, Y.; Supekar, K.S.; Das, A.K.; Altman, R.B.; Garber, A.M. (2007). \"Extracting subject demographic information from abstracts of randomized clinical trial reports\". MEDINFO 2007 129: 550-554. PMID 17911777.   \n\n\u2191 Xu, R.; Superkar, K.; Huang, Y.; Das, A.K.; Garber, A.M. (2006). \"Combining text classification and Hidden Markov Modeling techniques for structuring randomized clinical trial abstracts\". AMIA Annual Symposium Proceedings 2006: 824-828. PMC PMC1839538. PMID 17238456. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839538 .   \n\n\u2191 Embley, D.W. (2004). \"Toward semantic understanding: An approach based on information extraction ontologies\". Proceedings of the Fifteenth Australasian Database Conference 27: 3-12.   \n\n\u2191 Dobbins, J.T. (2009). \"Tomosynthesis imaging: at a translational crossroads\". Medical Physics 36 (6): 1956-1967. PMC PMC2832060. PMID 19610284. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2832060 .   \n\n\u2191 Kopans, D.B. (2014). \"Digital breast tomosynthesis from concept to clinical care\". American Journal of Roentgenology 202 (2): 299-308. doi:10.2214\/AJR.13.11520. PMID 24450669.   \n\n\u2191 Lee, C.I.; Lehman, C.D. (2013). \"Digital breast tomosynthesis and the challenges of implementing an emerging breast cancer screening technology into clinical practice\". Journal of the American College of Radiology 10 (12): 913-917. doi:10.1016\/j.jacr.2013.09.010. PMID 24295940.   \n\n\u2191 Roman, D.; Klien, E.; Scharl, A. (Ed.); Tochtermann, K. (Ed.) (2007). \"SWING \u2013 A Semantic Framework for Geospatial Services\". The Geospatial Web: How Geobrowsers, Social Software and the Web 2.0 are Shaping the Network Society. Springer. ISBN 1846288266.   \n\n\u2191 Yue, P.; Di, L.; Yang, W.; Yu, G.; Zhao, P. (2007). \"Semantics-based automatic composition of geospatial Web service chains\". Computers & Geoscience 33 (5): 649\u2013665. doi:10.1016\/j.cageo.2006.09.003.   \n\n\u2191 Brodaric, B.; Fox, P.; McGuinness, D.L. (2009). \"Geoscience knowledge representation in cyberinfrastructure\". Computers & Geoscience 35 (4): 697-699. doi:10.1016\/j.cageo.2009.01.001.   \n\n\u2191 Egenhofer, M.J. (2002). \"Toward the Semantic Geospatial Web\". Proceedings of the 10th ACM International Symposium on Advances in Geographic Information Systems 2002: 1\u20134. doi:10.1145\/585147.585148.   \n\n\u2191 Yang, C.; Raskin, R.; Goodchild, M.; Gahegan, M. (2010). \"Geospatial Cyberinfrastructure: Past, present and future\". Computers, Environment and Urban Systems 34 (2010): 264\u2013277. doi:10.1016\/j.compenvurbsys.2010.04.001.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\">https:\/\/www.limswiki.org\/index.php\/Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on geoinformaticsLIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 21:05.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,350 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","6865b9c249ccc54417e80c99d92bb99c_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Why_health_services_research_needs_geoinformatics_Rationale_and_case_example skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Why health services research needs geoinformatics: Rationale and case example<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Delivery of health care in the United States has become increasingly complex over the past 50 years, as health care markets have evolved, technology has diffused, population demographics have shifted, and cultural expectations of health and health care have been transformed. Identifying and understanding important patterns of health care services, accessibility, utilization, and outcomes can best be accomplished by combining data from all of these dimensions in near-real time. The Big Data paradigm provides a new framework to bring together very large volumes of data from a variety of sources and formats, with computing capacity to derive new <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>, hypotheses, and inferences.<sup id=\"rdp-ebb-cite_ref-PhilipData14_1-0\" class=\"reference\"><a href=\"#cite_note-PhilipData14-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KambatlaTrends14_2-0\" class=\"reference\"><a href=\"#cite_note-KambatlaTrends14-2\" rel=\"external_link\">[2]<\/a><\/sup> The complementary fields of genomics and <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> have already made great advances only made possible by Big Data approaches. Similar gains can be made by pairing health services research with <a href=\"https:\/\/www.limswiki.org\/index.php\/Geoinformatics\" title=\"Geoinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"2dc37de467d4af308f4b02d8e2ba12d1\">geoinformatics<\/a> \u2013- defined as \u201cthe science and technology dealing with the structure and character of spatial information, its capture, its classification and qualification, its storage, processing, portrayal and dissemination, including the infrastructure necessary to secure optimal use of this information\u201d.<sup id=\"rdp-ebb-cite_ref-RajuFund04_3-0\" class=\"reference\"><a href=\"#cite_note-RajuFund04-3\" rel=\"external_link\">[3]<\/a><\/sup> Integrating geospatial technologies with health services research brings <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">informatics<\/a> approaches, data sciences, and spatial theories of health and healthcare together to explore relationships among geography, health, and delivery of care in novel ways made possible through geoinformatics. Syngergy between the two disciplines will enhance our ability to discover how health care is delivered most effectively for the greatest health benefits across populations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Shared_history_of_geography_and_health_services_research:_Successes_and_limitations\">Shared history of geography and health services research: Successes and limitations<\/span><\/h2>\n<p>Health services research and geography have intersected under the rubrics of medical geography, epidemiology of health care, small area analysis, and public health since ancient Roman times, but more formally since the 1930s. Then, James Glover \u2014 an English physician \u2014 noticed that tonsillectomies were occurring at highly variable rates across school districts, which could not be explained by geographic, socio-demographic, or clinical factors, and the most likely explanation suggested being differences in how physicians practice.<sup id=\"rdp-ebb-cite_ref-GloverThe38_4-0\" class=\"reference\"><a href=\"#cite_note-GloverThe38-4\" rel=\"external_link\">[4]<\/a><\/sup> Similar work, begun in the 1970s, became the impetus for the Dartmouth Atlas of Health Care<sup id=\"rdp-ebb-cite_ref-DartmouthAtlas14_5-0\" class=\"reference\"><a href=\"#cite_note-DartmouthAtlas14-5\" rel=\"external_link\">[5]<\/a><\/sup>, which used health care utilization data to develop geographic units representing health care markets. These health care-based spatial units can be directly compared to identify patterns associated with both effective and ineffective care. Other notable examples of health services research linked to a geospatial framework can be found in the public health arena, with planning of population-based vaccination programs and designation of <a href=\"https:\/\/www.limswiki.org\/index.php\/Federally_qualified_health_center\" title=\"Federally qualified health center\" target=\"_blank\" class=\"wiki-link\" data-key=\"72b1961c7abe7b1850ad53a7ca0694d6\">federally-qualified health centers<\/a>. Despite these tremendous contributions, limitations exist that geoinformatics and health service research are now poised to overcome as information technology and the digital era expands data availability, accessibility, usability, and timely knowledge generation.\n<\/p><p>Understanding spatiotemporal distributions of health services is a fundamental aspect of health services research upon which studies of utilization, outcomes, comparative effectiveness, resource allocation, and others are based. Four key limitations can be found in typical approaches to measuring health services distributions: 1. retrospective methods; 2. limited geographic extents; 3. ascertainment challenges; and 4. structured data only; which will be addressed below (Table 1). By combining geoinformatics with health services research, an important domain of questions within the field of <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_informatics\" title=\"Medical informatics\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"f89ecb3b26617b8c6e09bc5e050cfd5d\">medical informatics<\/a> can be addressed, such as is illustrated with a case example.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Key Current Limitations in Geographic Based Health Services Research\n<\/th>\n<th> Example Geoinformatics Approaches to Address Current Limitations\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Retrospective methods available data is usually retrospective with lag in timeliness, so pattern described are rarely current.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Using web content mining and mobile technology feeds, near\u2013 real time locational data can be obtained.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Limited geographic extents Typically a tradeoff exists between rich\/granular data available for small geographic extents, or coarse\/broad data available for large geographic extents.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Use of internet, and mobile- technology based locational data mining, allows for very broad geographic capture, without spatial scale limitations.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ascertainment Challenges Locational\/spatial data is often not available at a point location (address) for health services and\/or patients. Locational data may be only at an area level (e.g. ZIP code), or not known with certainty or completeness at all.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Use of internet, and mobile- technology based locational data mining, is based on either point locations (latitude-longitude) from IP address, or address mining that can be automatically geocoded to point location.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Structured data only typically only spatial data in structured form \u2014 such as databases or files \u2014 is available. Manual abstraction of spatial data is possible, but limits the scale of examination.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Content Mixing (text and images) allows the use of unstructured data from the internet and mobile technology feeds to obtain locational information that is not captured explicitly in an existing database.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Asynchronous evaluation of technology with outcomes because evaluation of new technologies is typically limited by the factors above, evaluation of technology occurs after it is already being used in actual practice, and often in <i>small or non representative areas and or populations<\/i>. This asynchronicity creates the potential for detrimental outcomes to occur prior to establishing outcomes to be unavailable to populations who may benefit if known that a technology should be available.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">By addressing the limitations noted above, evaluation of technology is more likely to occur in near real time, allowing for outcomes to be determined in a more timely manner, rather than with a temporal lag during which patients and populations may be impacted by negative outcomes not previously understood, or missing out on positive outcomes if the technology is reveled to not be located where a population may benefit.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<dl><dd><b>Table 1<\/b>: Summary of major limitations in geographical based health services research and how geoinformatics approaches can be used to address them<\/dd><\/dl>\n<h2><span class=\"mw-headline\" id=\"A_health_services_research_problem:_Diffusion_of_medical_technology\">A health services research problem: Diffusion of medical technology<\/span><\/h2>\n<p>Technological innovation is a hallmark of the U.S. health care system, and relies on the backbone of translational research to evaluate effectiveness after efficacy has been established. The full potential of a new technology is determined as clinical improvements and impacts on population health are assessed. Yet typically diffusion and the research to establish effectiveness occur asynchronously. This limits the potential for timely assessment of broad clinical impact and leads to two concerns: 1. overuse of technologies with unproven or minimal benefits in the general population and concomitant unwarranted costs; 2. underuse of technologies that have beneficial effects and improve outcomes for particular patient populations. As medical technology diffuses into generalizable practice, data and research needs arise to address these concerns from a host of stakeholders perspectives, including patients, clinicians, researchers, health care facilities, commercial vendors, payers, health care systems, communities, and regulatory bodies. A prerequisite for assessing new technologies in community practice is knowledge of the locations or extent of diffusion and the populations reached. Data availability and timeliness are critical barriers to establishing this knowledge, contributing to spotty information that is retrospective, often with notable lags. For example, The Dartmouth Atlas of Healthcare<sup id=\"rdp-ebb-cite_ref-DartmouthAtlas14_5-1\" class=\"reference\"><a href=\"#cite_note-DartmouthAtlas14-5\" rel=\"external_link\">[5]<\/a><\/sup> and other work<sup id=\"rdp-ebb-cite_ref-OnegaGeo12_6-0\" class=\"reference\"><a href=\"#cite_note-OnegaGeo12-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-7\" class=\"reference\"><a href=\"#cite_note-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OnegaGeo08_8-0\" class=\"reference\"><a href=\"#cite_note-OnegaGeo08-8\" rel=\"external_link\">[8]<\/a><\/sup> has described geographic variation of medical technologies at a national level, but relied on Medicare data, which are typically available for research with a lag of 2-3 years. Further, <a href=\"https:\/\/www.limswiki.org\/index.php\/Centers_for_Medicare_and_Medicaid_Services\" title=\"Centers for Medicare and Medicaid Services\" target=\"_blank\" class=\"wiki-link\" data-key=\"654b4449e4816e190325b420c264df1a\">Medicare<\/a>, as well as private insurers, are dependent on the new technology being approved for reimbursement since they rely on billing data (claims), and coverage of the technology may take years following FDA approval and commercial dissemination, thus not able to be ascertained until unique billing codes are implemented. Some data resources may be timely for monitoring diffusion \u2014 such as registries like the Breast Cancer Surveillance Consortium (BCSC) and the HMO Research Network (HMORN)<sup id=\"rdp-ebb-cite_ref-BCSC_9-0\" class=\"reference\"><a href=\"#cite_note-BCSC-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HMORN_10-0\" class=\"reference\"><a href=\"#cite_note-HMORN-10\" rel=\"external_link\">[10]<\/a><\/sup>, and other clinical\/provider networks with health information exchanges and\/or robust <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a>. However, these data resources are limited in geographic extent, are often not population-based, and may not capture use of new technologies until they are uniquely coded (e.g. CPT \u2013 Common Procedural Terminology), or are reimbursable. Further, most data sources require existing structured data, although natural language processing (NLP) is increasingly applied to unstructured medical record information, which can be powerful, but also time-consuming, complicated and variable across settings.<sup id=\"rdp-ebb-cite_ref-XuExt07_11-0\" class=\"reference\"><a href=\"#cite_note-XuExt07-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-XuComb06_12-0\" class=\"reference\"><a href=\"#cite_note-XuComb06-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-EmbleyToward04_13-0\" class=\"reference\"><a href=\"#cite_note-EmbleyToward04-13\" rel=\"external_link\">[13]<\/a><\/sup> Achieving national extent, fully ascertained, and timely data capture to characterize geographic and sociodemographic diffusion of new technologies is a critical need, particularly as the U.S. seeks to improve health and healthcare and limit health care costs to effective use within populations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Measuring_near-real_time_diffusion_of_breast_imaging_technology\">Measuring near-real time diffusion of breast imaging technology<\/span><\/h2>\n<p>To more fully understand dissemination of new technologies we need to be able to capture the occurrences of the technology in large geographic areas, and in a dynamic way that reflects the dynamic process that dissemination is. Figure 1 presents a schematic approach to do this, using a breast imaging technology diffusion cased example (Figure 1). We can address the existing limitations in measuring, monitoring, and characterizing health care diffusion \u2014 with breast imaging as a timely example, given new technologies, such as digital breast tomosynthesis (DBT)<sup id=\"rdp-ebb-cite_ref-DobbinsTomo09_14-0\" class=\"reference\"><a href=\"#cite_note-DobbinsTomo09-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KopansDigi14_15-0\" class=\"reference\"><a href=\"#cite_note-KopansDigi14-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeeDigi13_16-0\" class=\"reference\"><a href=\"#cite_note-LeeDigi13-16\" rel=\"external_link\">[16]<\/a><\/sup>, and legislation related to breast density notification [17]. With a geospatial semantic web<sup id=\"rdp-ebb-cite_ref-RomanSWING07_17-0\" class=\"reference\"><a href=\"#cite_note-RomanSWING07-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-YueSem07_18-0\" class=\"reference\"><a href=\"#cite_note-YueSem07-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BrodaricGeo09_19-0\" class=\"reference\"><a href=\"#cite_note-BrodaricGeo09-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-EgenhoferToward09_20-0\" class=\"reference\"><a href=\"#cite_note-EgenhoferToward09-20\" rel=\"external_link\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-YangGeo10_21-0\" class=\"reference\"><a href=\"#cite_note-YangGeo10-21\" rel=\"external_link\">[21]<\/a><\/sup>, which combines web mining techniques with geographic information systems and census data, one can ascertain geographic uptake of DBT nationally, estimate potential access overall and by population subgroups, and identify correlates of dissemination patterns.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Onega_JournalHMInformatics2014_5-6.gif\" class=\"image wiki-link\" target=\"_blank\" data-key=\"f14a527ba0bc2d910e42519fff917bb1\"><img alt=\"Fig1 Onega JournalHMInformatics2014 5-6.gif\" src=\"https:\/\/www.limswiki.org\/images\/1\/1b\/Fig1_Onega_JournalHMInformatics2014_5-6.gif\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Schematic flow for a geoinformatics application in health services<br \/research to dynamically monitor technology diffusion <\/blockquote>\n<p>Web content mining is used in this project to identify instances of DBT based on taxonomy of terms. Using associated web pages from these instances, street address information is captured for the DBT instance. These addresses and related attributes (facility name, date of data capture, etc.) are brought into a <a href=\"https:\/\/www.limswiki.org\/index.php\/Geographic_information_system\" title=\"Geographic information system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8981ab93f8ebf0730c3b38949b39ad99\">GIS<\/a> for geocoding, spatial joins with other layers (e.g. road network), travel time analysis, and service area creation. Population demographics and other census-derived data are attributed to DBT locations and service areas. This constitutes the database containing geographic extent of DBT diffusion and population characteristics served by DBT facilities, which is refreshed at regular intervals following external validation. This application characterizes heterogeneous diffusion both geographically and socio-demographically, which also serves as a proof of concept for higher-dimension data which incorporates spatial layers, as well as other technologies.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>The application of geoinformatics to health services research has high potential for advancing currently used research methods to monitor and evaluate new technologies as they are translated from experimental settings into communities and populations. A successful application of this approach will yield a validated tool to dynamically integrate geospatial data, population data, and web content for automated discovery and monitoring of technology diffusion. For example, a user interface will provide static functions, such as maps of service locations, derived service areas, density of services available, and populations that are in service catchment areas. Interactive and near-real time functions could produce for user-defined areas, time, and populations: video\/time trends of service locations, derive-time defined service areas, time trends of populations coincident with services, and projected population coverage for actual or potential service locations. Such tools can be readily scalable, applicable to other new technologies, and foundational for further capabilities, such as using geostatistical methods for predictive modeling, visualization, and other \u201cBig Data\u201d analytics.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>Research reported in this (publication\/press release) was supported by The Dartmouth Clinical and Translational Science Institute, under award number UL1TR001086 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH). The content is solely the responsibility of the author(s) and does not necessarily represent the official views of the NIH.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-PhilipData14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PhilipData14_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Philip Chen, C.L.; Zhang, C.Y. (2014). \"Data-intensive applications, challenges, techniques and technologies: A survey on Big Data\". <i>Information Sciences<\/i> <b>275<\/b>: 314-347. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ins.2014.01.015\" target=\"_blank\">10.1016\/j.ins.2014.01.015<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data-intensive+applications%2C+challenges%2C+techniques+and+technologies%3A+A+survey+on+Big+Data&rft.jtitle=Information+Sciences&rft.aulast=Philip+Chen%2C+C.L.%3B+Zhang%2C+C.Y.&rft.au=Philip+Chen%2C+C.L.%3B+Zhang%2C+C.Y.&rft.date=2014&rft.volume=275&rft.pages=314-347&rft_id=info:doi\/10.1016%2Fj.ins.2014.01.015&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KambatlaTrends14-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KambatlaTrends14_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kambatla, K.; Kolliasb, G.; Kumar, V.; Grama, A. (2014). \"Trends in big data analytics\". <i>Journal of Parallel and Distributed Computing<\/i> <b>74<\/b> (7): 2561-2573. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jpdc.2014.01.003\" target=\"_blank\">10.1016\/j.jpdc.2014.01.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Trends+in+big+data+analytics&rft.jtitle=Journal+of+Parallel+and+Distributed+Computing&rft.aulast=Kambatla%2C+K.%3B+Kolliasb%2C+G.%3B+Kumar%2C+V.%3B+Grama%2C+A.&rft.au=Kambatla%2C+K.%3B+Kolliasb%2C+G.%3B+Kumar%2C+V.%3B+Grama%2C+A.&rft.date=2014&rft.volume=74&rft.issue=7&rft.pages=2561-2573&rft_id=info:doi\/10.1016%2Fj.jpdc.2014.01.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RajuFund04-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RajuFund04_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Raju, P.L.N. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.preventionweb.net\/files\/1682_9970.pdf\" target=\"_blank\">\"Fundamentals of Geographic Information Systems\"<\/a> (PDF). <i>Satellite Remote Sensing and GIS Applications in Agricultural Meteorology<\/i>. World Meteorological Organisation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.preventionweb.net\/files\/1682_9970.pdf\" target=\"_blank\">http:\/\/www.preventionweb.net\/files\/1682_9970.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Fundamentals+of+Geographic+Information+Systems&rft.atitle=Satellite+Remote+Sensing+and+GIS+Applications+in+Agricultural+Meteorology&rft.aulast=Raju%2C+P.L.N.&rft.au=Raju%2C+P.L.N.&rft.date=2004&rft.pub=World+Meteorological+Organisation&rft_id=http%3A%2F%2Fwww.preventionweb.net%2Ffiles%2F1682_9970.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GloverThe38-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GloverThe38_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Glover, J.A. (1938). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2076749\" target=\"_blank\">\"The incidence of tonsillectomy in school children\"<\/a>. <i>Proceedings of the Royal Society of Medicine<\/i> <b>31<\/b> (10): 95-113. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2076749\/\" target=\"_blank\">PMC2076749<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18245048\" target=\"_blank\">18245048<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2076749\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2076749<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+incidence+of+tonsillectomy+in+school+children&rft.jtitle=Proceedings+of+the+Royal+Society+of+Medicine&rft.aulast=Glover%2C+J.A.&rft.au=Glover%2C+J.A.&rft.date=1938&rft.volume=31&rft.issue=10&rft.pages=95-113&rft_id=info:pmc\/PMC2076749&rft_id=info:pmid\/18245048&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2076749&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DartmouthAtlas14-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DartmouthAtlas14_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-DartmouthAtlas14_5-1\" rel=\"external_link\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.dartmouthatlas.org\/publications\/articles.aspx\" target=\"_blank\">\"The Dartmouth Atlas of Health Care\"<\/a>. The Dartmouth Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dartmouthatlas.org\/publications\/articles.aspx\" target=\"_blank\">http:\/\/www.dartmouthatlas.org\/publications\/articles.aspx<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Dartmouth+Atlas+of+Health+Care&rft.atitle=&rft.pub=The+Dartmouth+Institute&rft_id=http%3A%2F%2Fwww.dartmouthatlas.org%2Fpublications%2Farticles.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OnegaGeo12-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OnegaGeo12_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Onega, T.; Toseston, T.D.; Wang, Q.; Hillner, B.E.; Song, Y.; Siegel, B.A.; Tosteson, A.N. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3830950\" target=\"_blank\">\"Geographic and sociodemographic variation of PET use in Medicare beneficiaries with cancer\"<\/a>. <i>Journal of the American College of Radiology<\/i> <b>9<\/b> (9): 635-642. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jacr.2012.05.005\" target=\"_blank\">10.1016\/j.jacr.2012.05.005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3830950\/\" target=\"_blank\">PMC3830950<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22954545\" target=\"_blank\">22954545<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3830950\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3830950<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Geographic+and+sociodemographic+variation+of+PET+use+in+Medicare+beneficiaries+with+cancer&rft.jtitle=Journal+of+the+American+College+of+Radiology&rft.aulast=Onega%2C+T.%3B+Toseston%2C+T.D.%3B+Wang%2C+Q.%3B+Hillner%2C+B.E.%3B+Song%2C+Y.%3B+Siegel%2C+B.A.%3B+Tosteson%2C+A.N.&rft.au=Onega%2C+T.%3B+Toseston%2C+T.D.%3B+Wang%2C+Q.%3B+Hillner%2C+B.E.%3B+Song%2C+Y.%3B+Siegel%2C+B.A.%3B+Tosteson%2C+A.N.&rft.date=2012&rft.volume=9&rft.issue=9&rft.pages=635-642&rft_id=info:doi\/10.1016%2Fj.jacr.2012.05.005&rft_id=info:pmc\/PMC3830950&rft_id=info:pmid\/22954545&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3830950&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-7\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Onega, T.; Hubbard, R.; Hill, D.; Lee, C.I.; Haas, J.S.; Carlos, H.A.; Alford-Teaster, J.; Bogart, A.; DeMartini, W.B.; Kerlikowske, K.; Virnig, B.A.; Buist, D.S.; Henderson, L.; Tosteson, A.N. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4156905\" target=\"_blank\">\"Geographic Access to Breast Imaging for U.S. Women\"<\/a>. <i>Journal of the American College of Radiology<\/i> <b>11<\/b> (9): 874-882. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jacr.2014.03.022\" target=\"_blank\">10.1016\/j.jacr.2014.03.022<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4156905\/\" target=\"_blank\">PMC4156905<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24889479\" target=\"_blank\">24889479<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4156905\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4156905<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Geographic+Access+to+Breast+Imaging+for+U.S.+Women&rft.jtitle=Journal+of+the+American+College+of+Radiology&rft.aulast=Onega%2C+T.%3B+Hubbard%2C+R.%3B+Hill%2C+D.%3B+Lee%2C+C.I.%3B+Haas%2C+J.S.%3B+Carlos%2C+H.A.%3B+Alford-Teaster%2C+J.%3B+Bogart%2C+A.%3B+DeMartini%2C+W.B.%3B+Kerlikowske%2C+K.%3B+Virnig%2C+B.A.%3B+Buist%2C+D.S.%3B+Henderson%2C+L.%3B+Tosteson%2C+A.N.&rft.au=Onega%2C+T.%3B+Hubbard%2C+R.%3B+Hill%2C+D.%3B+Lee%2C+C.I.%3B+Haas%2C+J.S.%3B+Carlos%2C+H.A.%3B+Alford-Teaster%2C+J.%3B+Bogart%2C+A.%3B+DeMartini%2C+W.B.%3B+Kerlikowske%2C+K.%3B+Virnig%2C+B.A.%3B+Buist%2C+D.S.%3B+Henderson%2C+L.%3B+Tosteson%2C+A.N.&rft.date=2014&rft.volume=11&rft.issue=9&rft.pages=874-882&rft_id=info:doi\/10.1016%2Fj.jacr.2014.03.022&rft_id=info:pmc\/PMC4156905&rft_id=info:pmid\/24889479&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4156905&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OnegaGeo08-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OnegaGeo08_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Onega, T.; Duell, E.J.; Shi, X.; Wang, D.; Demidenko, E.; Goodman, D. (2008). \"Geographic access to cancer care in the U.S.\". <i>Cancer<\/i> <b>112<\/b> (4): 909-918. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fcncr.23229\" target=\"_blank\">10.1002\/cncr.23229<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18189295\" target=\"_blank\">18189295<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Geographic+access+to+cancer+care+in+the+U.S.&rft.jtitle=Cancer&rft.aulast=Onega%2C+T.%3B+Duell%2C+E.J.%3B+Shi%2C+X.%3B+Wang%2C+D.%3B+Demidenko%2C+E.%3B+Goodman%2C+D.&rft.au=Onega%2C+T.%3B+Duell%2C+E.J.%3B+Shi%2C+X.%3B+Wang%2C+D.%3B+Demidenko%2C+E.%3B+Goodman%2C+D.&rft.date=2008&rft.volume=112&rft.issue=4&rft.pages=909-918&rft_id=info:doi\/10.1002%2Fcncr.23229&rft_id=info:pmid\/18189295&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BCSC-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BCSC_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/breastscreening.cancer.gov\/about\/\" target=\"_blank\">\"Breast Cancer Surveillance Consortium\"<\/a>. National Cancer Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/breastscreening.cancer.gov\/about\/\" target=\"_blank\">http:\/\/breastscreening.cancer.gov\/about\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Breast+Cancer+Surveillance+Consortium&rft.atitle=&rft.pub=National+Cancer+Institute&rft_id=http%3A%2F%2Fbreastscreening.cancer.gov%2Fabout%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HMORN-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HMORN_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.hcsrn.org\/en\/About\/\" target=\"_blank\">\"HMO Research Network\"<\/a>. Kaiser Permanente Division of Research<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.hcsrn.org\/en\/About\/\" target=\"_blank\">http:\/\/www.hcsrn.org\/en\/About\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=HMO+Research+Network&rft.atitle=&rft.pub=Kaiser+Permanente+Division+of+Research&rft_id=http%3A%2F%2Fwww.hcsrn.org%2Fen%2FAbout%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XuExt07-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-XuExt07_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xu, R.; Garten, Y.; Supekar, K.S.; Das, A.K.; Altman, R.B.; Garber, A.M. (2007). \"Extracting subject demographic information from abstracts of randomized clinical trial reports\". <i>MEDINFO 2007<\/i> <b>129<\/b>: 550-554. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17911777\" target=\"_blank\">17911777<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Extracting+subject+demographic+information+from+abstracts+of+randomized+clinical+trial+reports&rft.jtitle=MEDINFO+2007&rft.aulast=Xu%2C+R.%3B+Garten%2C+Y.%3B+Supekar%2C+K.S.%3B+Das%2C+A.K.%3B+Altman%2C+R.B.%3B+Garber%2C+A.M.&rft.au=Xu%2C+R.%3B+Garten%2C+Y.%3B+Supekar%2C+K.S.%3B+Das%2C+A.K.%3B+Altman%2C+R.B.%3B+Garber%2C+A.M.&rft.date=2007&rft.volume=129&rft.pages=550-554&rft_id=info:pmid\/17911777&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XuComb06-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-XuComb06_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Xu, R.; Superkar, K.; Huang, Y.; Das, A.K.; Garber, A.M. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839538\" target=\"_blank\">\"Combining text classification and Hidden Markov Modeling techniques for structuring randomized clinical trial abstracts\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>2006<\/b>: 824-828. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1839538\/\" target=\"_blank\">PMC1839538<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17238456\" target=\"_blank\">17238456<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839538\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839538<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Combining+text+classification+and+Hidden+Markov+Modeling+techniques+for+structuring+randomized+clinical+trial+abstracts&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Xu%2C+R.%3B+Superkar%2C+K.%3B+Huang%2C+Y.%3B+Das%2C+A.K.%3B+Garber%2C+A.M.&rft.au=Xu%2C+R.%3B+Superkar%2C+K.%3B+Huang%2C+Y.%3B+Das%2C+A.K.%3B+Garber%2C+A.M.&rft.date=2006&rft.volume=2006&rft.pages=824-828&rft_id=info:pmc\/PMC1839538&rft_id=info:pmid\/17238456&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1839538&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EmbleyToward04-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EmbleyToward04_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Embley, D.W. (2004). \"Toward semantic understanding: An approach based on information extraction ontologies\". <i>Proceedings of the Fifteenth Australasian Database Conference<\/i> <b>27<\/b>: 3-12.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+semantic+understanding%3A+An+approach+based+on+information+extraction+ontologies&rft.jtitle=Proceedings+of+the+Fifteenth+Australasian+Database+Conference&rft.aulast=Embley%2C+D.W.&rft.au=Embley%2C+D.W.&rft.date=2004&rft.volume=27&rft.pages=3-12&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DobbinsTomo09-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DobbinsTomo09_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dobbins, J.T. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2832060\" target=\"_blank\">\"Tomosynthesis imaging: at a translational crossroads\"<\/a>. <i>Medical Physics<\/i> <b>36<\/b> (6): 1956-1967. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2832060\/\" target=\"_blank\">PMC2832060<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19610284\" target=\"_blank\">19610284<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2832060\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2832060<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Tomosynthesis+imaging%3A+at+a+translational+crossroads&rft.jtitle=Medical+Physics&rft.aulast=Dobbins%2C+J.T.&rft.au=Dobbins%2C+J.T.&rft.date=2009&rft.volume=36&rft.issue=6&rft.pages=1956-1967&rft_id=info:pmc\/PMC2832060&rft_id=info:pmid\/19610284&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2832060&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KopansDigi14-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KopansDigi14_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kopans, D.B. (2014). \"Digital breast tomosynthesis from concept to clinical care\". <i>American Journal of Roentgenology<\/i> <b>202<\/b> (2): 299-308. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2214%2FAJR.13.11520\" target=\"_blank\">10.2214\/AJR.13.11520<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24450669\" target=\"_blank\">24450669<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Digital+breast+tomosynthesis+from+concept+to+clinical+care&rft.jtitle=American+Journal+of+Roentgenology&rft.aulast=Kopans%2C+D.B.&rft.au=Kopans%2C+D.B.&rft.date=2014&rft.volume=202&rft.issue=2&rft.pages=299-308&rft_id=info:doi\/10.2214%2FAJR.13.11520&rft_id=info:pmid\/24450669&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeeDigi13-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeeDigi13_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lee, C.I.; Lehman, C.D. (2013). \"Digital breast tomosynthesis and the challenges of implementing an emerging breast cancer screening technology into clinical practice\". <i>Journal of the American College of Radiology<\/i> <b>10<\/b> (12): 913-917. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jacr.2013.09.010\" target=\"_blank\">10.1016\/j.jacr.2013.09.010<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24295940\" target=\"_blank\">24295940<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Digital+breast+tomosynthesis+and+the+challenges+of+implementing+an+emerging+breast+cancer+screening+technology+into+clinical+practice&rft.jtitle=Journal+of+the+American+College+of+Radiology&rft.aulast=Lee%2C+C.I.%3B+Lehman%2C+C.D.&rft.au=Lee%2C+C.I.%3B+Lehman%2C+C.D.&rft.date=2013&rft.volume=10&rft.issue=12&rft.pages=913-917&rft_id=info:doi\/10.1016%2Fj.jacr.2013.09.010&rft_id=info:pmid\/24295940&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RomanSWING07-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RomanSWING07_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Roman, D.; Klien, E.; Scharl, A. (Ed.); Tochtermann, K. (Ed.) (2007). \"SWING \u2013 A Semantic Framework for Geospatial Services\". <i>The Geospatial Web: How Geobrowsers, Social Software and the Web 2.0 are Shaping the Network Society<\/i>. Springer. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 1846288266.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SWING+%E2%80%93+A+Semantic+Framework+for+Geospatial+Services&rft.atitle=The+Geospatial+Web%3A+How+Geobrowsers%2C+Social+Software+and+the+Web+2.0+are+Shaping+the+Network+Society&rft.aulast=Roman%2C+D.%3B+Klien%2C+E.%3B+Scharl%2C+A.+%28Ed.%29%3B+Tochtermann%2C+K.+%28Ed.%29&rft.au=Roman%2C+D.%3B+Klien%2C+E.%3B+Scharl%2C+A.+%28Ed.%29%3B+Tochtermann%2C+K.+%28Ed.%29&rft.date=2007&rft.pub=Springer&rft.isbn=1846288266&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YueSem07-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YueSem07_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yue, P.; Di, L.; Yang, W.; Yu, G.; Zhao, P. (2007). \"Semantics-based automatic composition of geospatial Web service chains\". <i>Computers & Geoscience<\/i> <b>33<\/b> (5): 649\u2013665. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cageo.2006.09.003\" target=\"_blank\">10.1016\/j.cageo.2006.09.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Semantics-based+automatic+composition+of+geospatial+Web+service+chains&rft.jtitle=Computers+%26+Geoscience&rft.aulast=Yue%2C+P.%3B+Di%2C+L.%3B+Yang%2C+W.%3B+Yu%2C+G.%3B+Zhao%2C+P.&rft.au=Yue%2C+P.%3B+Di%2C+L.%3B+Yang%2C+W.%3B+Yu%2C+G.%3B+Zhao%2C+P.&rft.date=2007&rft.volume=33&rft.issue=5&rft.pages=649%E2%80%93665&rft_id=info:doi\/10.1016%2Fj.cageo.2006.09.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrodaricGeo09-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrodaricGeo09_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brodaric, B.; Fox, P.; McGuinness, D.L. (2009). \"Geoscience knowledge representation in cyberinfrastructure\". <i>Computers & Geoscience<\/i> <b>35<\/b> (4): 697-699. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.cageo.2009.01.001\" target=\"_blank\">10.1016\/j.cageo.2009.01.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Geoscience+knowledge+representation+in+cyberinfrastructure&rft.jtitle=Computers+%26+Geoscience&rft.aulast=Brodaric%2C+B.%3B+Fox%2C+P.%3B+McGuinness%2C+D.L.&rft.au=Brodaric%2C+B.%3B+Fox%2C+P.%3B+McGuinness%2C+D.L.&rft.date=2009&rft.volume=35&rft.issue=4&rft.pages=697-699&rft_id=info:doi\/10.1016%2Fj.cageo.2009.01.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EgenhoferToward09-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EgenhoferToward09_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Egenhofer, M.J. (2002). \"Toward the Semantic Geospatial Web\". <i>Proceedings of the 10th ACM International Symposium on Advances in Geographic Information Systems<\/i> <b>2002<\/b>: 1\u20134. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F585147.585148\" target=\"_blank\">10.1145\/585147.585148<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+the+Semantic+Geospatial+Web&rft.jtitle=Proceedings+of+the+10th+ACM+International+Symposium+on+Advances+in+Geographic+Information+Systems&rft.aulast=Egenhofer%2C+M.J.&rft.au=Egenhofer%2C+M.J.&rft.date=2002&rft.volume=2002&rft.pages=1%E2%80%934&rft_id=info:doi\/10.1145%2F585147.585148&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YangGeo10-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YangGeo10_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yang, C.; Raskin, R.; Goodchild, M.; Gahegan, M. (2010). \"Geospatial Cyberinfrastructure: Past, present and future\". <i>Computers, Environment and Urban Systems<\/i> <b>34<\/b> (2010): 264\u2013277. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.compenvurbsys.2010.04.001\" target=\"_blank\">10.1016\/j.compenvurbsys.2010.04.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Geospatial+Cyberinfrastructure%3A+Past%2C+present+and+future&rft.jtitle=Computers%2C+Environment+and+Urban+Systems&rft.aulast=Yang%2C+C.%3B+Raskin%2C+R.%3B+Goodchild%2C+M.%3B+Gahegan%2C+M.&rft.au=Yang%2C+C.%3B+Raskin%2C+R.%3B+Goodchild%2C+M.%3B+Gahegan%2C+M.&rft.date=2010&rft.volume=34&rft.issue=2010&rft.pages=264%E2%80%93277&rft_id=info:doi\/10.1016%2Fj.compenvurbsys.2010.04.001&rfr_id=info:sid\/en.wikipedia.org:Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210237\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.567 seconds\nReal time usage: 0.593 seconds\nPreprocessor visited node count: 16803\/1000000\nPreprocessor generated node count: 34792\/1000000\nPost\u2010expand include size: 125454\/2097152 bytes\nTemplate argument size: 42653\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 570.492 1 - -total\n 84.48% 481.975 1 - Template:Reflist\n 73.51% 419.355 21 - Template:Citation\/core\n 62.54% 356.768 16 - Template:Cite_journal\n 12.70% 72.466 4 - Template:Cite_web\n 11.40% 65.018 1 - Template:Infobox_journal_article\n 10.98% 62.628 1 - Template:Infobox\n 7.70% 43.909 26 - Template:Citation\/identifier\n 6.67% 38.067 80 - Template:Infobox\/row\n 4.11% 23.419 23 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7649-0!*!0!!en!5!* and timestamp 20181213210236 and revision id 23428\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example\">https:\/\/www.limswiki.org\/index.php\/Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","6865b9c249ccc54417e80c99d92bb99c_images":["https:\/\/www.limswiki.org\/images\/1\/1b\/Fig1_Onega_JournalHMInformatics2014_5-6.gif"],"6865b9c249ccc54417e80c99d92bb99c_timestamp":1544734956,"26a517162c5d1f67f3c6bbb8c4cd1a80_type":"article","26a517162c5d1f67f3c6bbb8c4cd1a80_title":"Use of handheld computers in clinical practice: A systematic review (Mickan et al. 2014)","26a517162c5d1f67f3c6bbb8c4cd1a80_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review","26a517162c5d1f67f3c6bbb8c4cd1a80_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Use of handheld computers in clinical practice: A systematic review\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUse of handheld computers in clinical practice: A systematic reviewJournal\n \nBMC Medical Informatics & Decision MakingAuthor(s)\n \nMickan, Sharon; Atherton, Helen; Roberts, Nia Wyn; Heneghan, Carl; Tilson, Julie KAuthor affiliation(s)\n \nNuffield Department of Primary Care Health Sciences, University of Oxford;\r\nDivision of Biokinesiology and Physical Therapy, University of Southern CaliforniaPrimary contact\n \nEmail: Sharon.mickan@phc.ox.ac.ukYear published\n \n2014Volume and issue\n \n14Page(s)\n \n56DOI\n \n10.1186\/1472-6947-14-56ISSN\n \n1472-6947Distribution license\n \nCreative Commons Attribution 2.0Website\n \nhttp:\/\/www.biomedcentral.com\/1472-6947\/14\/56Download\n \nhttp:\/\/www.biomedcentral.com\/content\/pdf\/1472-6947-14-56.pdf (PDF)\n\nContents\n\n1 Abstract \n\n1.1 Background \n1.2 Methods \n1.3 Results \n1.4 Conclusion \n\n\n2 Background \n3 Methods \n\n3.1 Search strategy \n3.2 Inclusion and exclusion criteria \n3.3 Study selection \n3.4 Data extraction \n3.5 Assessment of quality \n3.6 Data synthesis \n\n\n4 Results \n\n4.1 Information for clinical knowledge \n4.2 Adherence to guidelines \n4.3 Diagnostic decision making \n\n\n5 Discussion \n\n5.1 Key findings \n5.2 Strengths and limitations \n5.3 Implication for practice \n5.4 Further research \n\n\n6 Conclusion \n7 Abbreviations \n8 Competing interests \n9 Authors\u2019 contributions \n10 Acknowledgements \n11 References \n12 Pre-publication history \n13 Notes \n\n\n\nAbstract \nBackground \nMany healthcare professionals use smartphones and tablets to inform patient care. Contemporary research suggests that handheld computers may support aspects of clinical diagnosis and management. This systematic review was designed to synthesise high quality evidence to answer the question; Does healthcare professionals\u2019 use of handheld computers improve their access to information and support clinical decision making at the point of care?\n\nMethods \nA detailed search was conducted using Cochrane, MEDLINE, EMBASE, PsycINFO, Science and Social Science Citation Indices since 2001. Interventions promoting healthcare professionals seeking information or making clinical decisions using handheld computers were included. Classroom learning and the use of laptop computers were excluded. Two authors independently selected studies, assessed quality using the Cochrane Risk of Bias tool and extracted data. High levels of data heterogeneity negated statistical synthesis. Instead, evidence for effectiveness was summarised narratively, according to each study\u2019s aim for assessing the impact of handheld computer use.\n\nResults \nWe included seven randomised trials investigating medical or nursing staffs\u2019 use of Personal Digital Assistants. Effectiveness was demonstrated across three distinct functions that emerged from the data: accessing information for clinical knowledge, adherence to guidelines and diagnostic decision making. When healthcare professionals used handheld computers to access clinical information, their knowledge improved significantly more than peers who used paper resources. When clinical guideline recommendations were presented on handheld computers, clinicians made significantly safer prescribing decisions and adhered more closely to recommendations than peers using paper resources. Finally, healthcare professionals made significantly more appropriate diagnostic decisions using clinical decision making tools on handheld computers compared to colleagues who did not have access to these tools. For these clinical decisions, the numbers need to test\/screen were all less than 11.\n\nConclusion \nHealthcare professionals\u2019 use of handheld computers may improve their information seeking, adherence to guidelines and clinical decision making. Handheld computers can provide real time access to and analysis of clinical information. The integration of clinical decision support systems within handheld computers offers clinicians the highest level of synthesised evidence at the point of care. Future research is needed to replicate these early results and to identify beneficial clinical outcomes.\nKeywords: Handheld computers; Smartphone; Information-seeking behaviour; Evidence-based practice; Knowledge translation; Clinical decision support systems; Clinical guidelines; Diagnostic decision making\n\nBackground \nIncreasing numbers of healthcare professionals use handheld computers that offer instant access to vast amounts of information via the internet and healthcare applications (apps).[1] Over the last 10 years there has been a rapid and accelerating rate of innovation in handheld computers, from personal digital assistants (PDAs) towards more powerful, versatile and internet connected devices. As the rate of adoption of handheld computers has increased, individual patterns of usage have moved from that of communication and personal diary management towards information seeking and decision support.[2] Today\u2019s clinicians can use handheld computers to search the internet for evidence and guidance on drugs and clinical conditions, use clinical decision support systems (CDSS) and access highly detailed patient information from clinical and laboratory investigations.\nAt the same time, there has been a change in the acceptance of using handheld computers in healthcare settings. Now, most students and many professionals are enthusiastic about using smartphones and tablet computers, and they take them wherever they go.[3] Along with this increasing adoption of handheld computers, there has been a massive growth in the volume of synthesized research information, healthcare oriented apps, databases and CDSSs.\nThis has also sparked an increased production of feasibility research, which has yet to recommend strategies for engagement, efficacy or effectiveness of mobile health initiatives.[4] While both early and current systematic reviews offer tentative and sceptical conclusions, there is equipoise in the literature. A systematic review of the use of PDAs in clinical decision making reported an increase in data collection quality and concluded that the use of decision support software improved the appropriateness of diagnostic and treatment decisions.[2] In a broader and contemporary systematic review of mHealth technologies, modest benefits were reported for improved clinical diagnosis and management support, and mixed outcomes were reported for efficient and accurate documentation.[3] Further, there was no clear benefit for educational interventions and some evidence of reduced quality of clinical assessment, when using mobile technology based photos.\nWhen healthcare professionals communicate with patients, there is high quality evidence to support the use of mobile phones to transmit short message service (SMS) reminders to improve attendance at health care appointments.[5][6] Further, text messaging interventions were shown to increase adherence to antiretroviral therapy in low-income settings and increased smoking cessation in high income settings.[7]\nAn early review of computerised, rather than mobile, CDSSs for prescribing, described effectiveness in initiating and monitoring therapy, but provided little evidence on their impact in specific clinical settings.[8] A later review reported improved processes of care in 60% of included studies but improved patient outcomes in only 20% of studies.[9] It is not clear whether incorporating these computerised systems into mobile devices would produce similar results.\nA literature and commercial review of mobile CDSSs reported medical professionals using a growing number of apps across a wide range of fields.[10] A systematic review of smartphone healthcare apps identified seven functional categories in which apps have been developed for use by healthcare professionals: diagnosis, drug reference, medical calculators, literature search, clinical communication, access to hospital information systems, and medical training.[1] A scoping review from a further five systematic reviews concluded that there is evidence for effective use of handheld computers by healthcare professionals across four key functions: providing easy and timely access to information, enabling accurate and complete documentation, providing instant access to evidence-based decision support and patient management systems, and promoting efficient work practices.[11]\nMost published studies to date describe the design, development and implementation of handheld computers using observational study designs.[4] In order to determine the benefits of integrating handheld computer use in healthcare practice, it is important to summarise and quantify results from the highest quality randomised controlled trials (RCTs) of effectiveness studies. Based on the functions identified in the earlier scoping review[11], it is timely to better understand whether healthcare professionals\u2019 use of handheld computers facilitates information seeking and improved clinical decision making. The purpose of this review is to answer the research question \u201cDoes healthcare professionals\u2019 use of handheld computers improve their access to information and support clinical decision making at the point of care?\u201d.\n\nMethods \nThe protocol for this systematic review was registered with PROSPERO (CRD42011001632), updated and adhered to. http:\/\/www.crd.york.ac.uk\/PROSPERO\/display_record.asp?ID=CRD42011001632#.U7-vibFnDhA\n\nSearch strategy \nWe searched the following databases from 2001 to 19th August 2013: Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, EMBASE, PsycINFO, Science Citation Index and Social Science Citation Index. The MEDLINE search strategy can be found in the Additional file 1. Reference lists of included studies were hand searched.\n\n\n\n\n\n\n\n Additional file 1. Medline search strategy.\n\n\nFormat: DOC Size: 39KB Download file\r\n\nThis file can be viewed with: Microsoft Word Viewer\n\n\n\n\nInclusion and exclusion criteria \nWe included studies whose participants were healthcare professionals using handheld devices in clinical settings. Interventions of interest were those investigating the use of handheld computers to promote healthcare professionals\u2019 information seeking (outside of formal education courses), or to support informed clinical decision making. Our comparator was usual clinical practice. We excluded the use of laptops.\nStudy designs included were RCTs. The review was restricted to the English language. We searched from 2001 onwards to account for the changing nature of technology. We excluded studies that were presented as abstracts only, and where author contact confirmed the study had not been published in full.\n\nStudy selection \nTwo authors (SM and JT) screened titles and abstracts. Full text articles were obtained for those selected and screened for inclusion (SM and HA). Where necessary, authors of studies were contacted for clarification of inclusion status.\n\nData extraction \nA data extraction form was designed and piloted by two authors (SM and HA) to record study design, country, device used, aim, participants, setting, intervention, comparator, primary and secondary outcome data (as reported by the systematic review authors). The same authors independently extracted data. Disagreements were resolved by discussion.\n\nAssessment of quality \nAssessment of risk of bias was conducted at the study level using the Cochrane Risk of Bias tool.[12] Assessment was conducted independently by two authors (SM and HA) with disagreements resolved by discussion. Information on risk of bias status was used to aid interpretation of the included studies.\n\nData synthesis \nHigh levels of data heterogeneity and mixed data quality meant that statistical synthesis was not possible. We adopted a narrative approach to summarise the evidence for effectiveness according to the purpose for using the handheld computer.\n\nResults \nThe combined search strategies identified 5,888 titles. After duplicates were removed, 3,612 titles were screened for eligibility. Thirty-eight full text articles were read, of which 31 did not meet the inclusion and exclusion criteria and therefore seven studies were retained for data extraction (see Figure 1).\n\n\n\nFigure 1. Flow diagram\nCharacteristics of the seven included studies are summarised in Table 1. All were RCTs, mostly designed as pilot studies with comparatively small numbers of participants (range 12-76 participants). Although we intended to include studies investigating smartphones and tablets, to represent the most current forms of handheld computers, all included studies investigated the use of PDAs. Three studies were conducted in USA, two in Canada and one each in France and Australia. In five studies, the intervention group used a PDA while the control group used paper-based resources. In two studies, both groups used a PDA, but the intervention group had access to a specific clinical decision support system (CDSS) or information tool that the control group did not. Healthcare participants were either medical (residents, fellows, and family, general and emergency physicians) or nursing professionals. Where students were included, they were using a PDA in a clinical environment.\n\r\n\n\nTable 1: Characteristics of included studies\n\n\n\n\n\n\n Study, country\n\n Participants, setting\n\n Intervention\n\n Comparator\n\n Primary outcome\n\n Secondary outcome\n\n\nBerner 2006 USA[13]\n\n59 Internal medicine residents, University outpatient clinic\n\nPDA with rule for gastrointestinal risk assessment when prescribing NSAIDS\n\nPDA without rule for gastrointestinal risk assessment when prescribing NSAIDS\n\nDifference in unsafe NSAID prescriptions\n\nIdentification of key risk factors for standardised patient case\n\n\nBochicchio 2006 USA[14]\n\n12 1st year critical care fellows, University hospital\n\nPDA with John Hopkins Antibiotic Guide\n\nNo PDA, instructed to use written reference guides\n\nDifference in mean score for knowledge test\n\nAntibiotic decision accuracy\n\n\nFarrell 2008 Australia[15]\n\n76 nursing students, Medical-surgical wards\n\nPDA with pharmacological information and training session\n\nNo training or PDA\n\nDifference in mean score for pharmacology test\n\nN\/A\n\n\nPrice 2005 Canada[16]\n\n8 General practitioners, General practice (79 patients)\n\nPDA with reminder for 5 preventive measures\n\nSoftware provided after the study\n\nAdherence to five guidelines\n\nN\/A\n\n\nRoy 2009 France[17]\n\n24 Emergency physicians, 10 emergency departments (1645 patients)\n\nPDA with CDSS for pulmonary embolism\n\nPDA used for data collection only; Paper based guideline material\n\nAppropriate diagnostic strategy for pulmonary embolism\n\nAdherence to recommended diagnostic testing Number of tests per patient\n\n\nGreiver 2005 Canada[18]\n\n18 Family physicians, Family practice (65 patients)\n\nPDA with angina diagnosis software\n\nConventional care\n\nAppropriate referral for cardiac stress testing at presentation, and nuclear cardiology after cardiac stress testing\n\nReferral to cardiologists\n\n\nLee 2009 USA[19]\n\n29 registered nurses, Hospital and ambulatory care (1874 patients)\n\nPDA with CDSS for obesity diagnosis\n\nPDA without CDSS for obesity diagnosis\n\nAppropriate obesity related diagnosis\n\nMissed obesity related diagnosis\n\n\n\nThe risk of bias assessment for included studies is shown in Figure 2 and summarised in Figure 3. The studies were largely at low or unclear risk of bias. Only one study[13] was at low risk of bias for all domains. The highest risk of bias occurred for blinding of participants, which is not always practical when investigating the use of an obvious handheld computer. Participants could not be blinded in 5 studies where the intervention group used a PDA and the control group did not. Blinding was achieved, however, in two studies where both intervention and control groups were using a PDA. Four studies ensured that the outcome assessment was blinded.\n\n\n\nFigure 2. Risk of bias in included studies\n\r\n\n\n\n\nFigure 3. Risk of bias across individual domains of risk of bias\nHeterogeneity of outcomes (Table 1) negated any quantitative synthesis or meta-analysis in this review. Instead, narrative summary was used to describe evidence of effectiveness of three distinct functions of handheld computers that emerged from the data: accessing information for clinical knowledge, adherence to safety and quality guidelines, and diagnostic decision making.\n\nInformation for clinical knowledge \nThere is evidence from two pilot studies, one showing statistical significance[14], that when doctors and nurses used a handheld computer to access information in clinical environments, their clinical knowledge improved more than their peers who used traditional paper resources.[14][15]\nMedical Fellows allocated to use a PDA in hospital intensive care units increased their mean knowledge of infectious disease management more than those who used paper resources at both 3 months (p\u2009<\u20090.05) and 6 months (p\u2009<\u20090.01).[14] PDA use also increased fellows\u2019 antibiotic selection accuracy, a secondary aim of this project. Of the 125 antibiotic selections evaluated, fellows\u2019 antibiotic selection accuracy improved from 66% (33\/50) in the first 3 months, to 87% (65\/75) in the second 3 month period. In the second study, pharmacological knowledge of 76 nursing students was tested before and after a 3 week clinical placement on rather than in medical and surgical wards. All demonstrated increased pharmacological knowledge, with those who used a PDA increasing their mean score twice as much as those who used only paper resources. However, this was not significant (p\u2009=\u20090.17).[15]\n\nAdherence to guidelines \nTwo feasibility RCTs examined the impact of healthcare professionals\u2019 use of PDAs on adherence to safety and quality guidelines in clinical practice.[13][16] One identified a statistically significant result[13] and both found that the PDA promoted adherence to guidelines.\nIn one study, 59 internal medicine residents received a PDA.[13] The intervention group\u2019s PDA included software supporting the use of a prediction rule for assessing gastro-intestinal risk when prescribing non-steroidal anti-inflammatory drugs (NSAIDs). Thirteen standardised patients (all with moderate risks for an adverse gastro-intestinal event) were included in regular clinics and their medical records and prescriptions were independently reviewed by clinicians blinded to the residents\u2019 group assignment. Residents using a PDA with the prediction rule made fewer unsafe NSAID prescriptions (23% vs 45%, p\u2009<\u20090.05) and the mean proportion of cases per physician with unsafe prescriptions was significantly lower (0 vs 50%, p\u2009<\u20090.001). There was a non-significant trend towards identifying patients\u2019 risk factors (58% vs 45%, p\u2009>\u20090.05) and it was noted that no unsafe prescriptions were recommended when risk factors were correctly identified.\nAmong eight general practitioners screening 79 patients, those randomly allocated to use a PDA (n\u2009=\u20094) with clinical guideline decision support demonstrated greater adherence to four out of five preventive healthcare guidelines, compared to colleagues who did not have access to a PDA. Although the small sample prevented statistical comparison, there was a pattern of greater improvement in rates of adherence among PDA users for including the prophylactic use of aspirin in patients at risk of coronary artery disease (from 33% before to 81% after PDA access), colorectal screening (38-65%) and screening for cervical cancer (88-100%) and cholesterol (64-94%). In comparison, there was a slight reduction in the high rate of screening for hypertension (97-94%).[16]\n\nDiagnostic decision making \nThere is evidence from three pilot RCTs[17][18][19] that having a clinical decision support system (CDSS) on a handheld computer can improve clinical and diagnostic decision making (Table 2). When used on a handheld computer in a clinical setting, the CDSS can prompt clinicians to collect and analyse patient data to inform diagnostic testing choices.\n\r\n\n\nTable 2: Improved diagnostic decision making\n\n\n\n\n\n\n Diagnostic decision assessed\n\n Percentage of episodes with PDA (n\/n)\n\n Percentage of episodes without PDA (n\/n)\n\n Percentage absolute difference (95% CI)\n\n Number needed to test \/ screen\n\n P Value\n\n\nAppropriate diagnostic strategy for pulmonary embolism[17]\n\n55% 378\/694\n\n26% 245\/951\n\n29%\n\n4\n\n0.023\n\n\nAdherence to recommended diagnostic testing[17]\n\n41% 287\/694\n\n17% 162\/951\n\n24% (20-29)\n\n5\n\n0.030\n\n\nAppropriate referral for cardiac stress testing at presentation[18]\n\n49% 18\/37\n\n29% 8\/28\n\n20%\n\n5\n\n0.284\n\n\nAppropriate referral for nuclear cardiology after cardiac stress testing[18]\n\n63% 17\/27\n\n45% 5\/11\n\n18% (14-50)\n\n6\n\n0.4\n\n\nAppropriate obesity-related diagnosis[19]\n\n11% 91\/807\n\n1% 10\/997\n\n10% (8-13)\n\n10\n\n<0.05\n\n\nMissed obesity-related diagnosis[19]\n\n25% 51\/208\n\n67% 440\/662\n\n42% (35-48)\n\n3\n\n<0.05\n\n\n\nFor 24 emergency physicians, the use of a handheld CDSS led to greater improvement in diagnostic decision making for pulmonary emboli, compared to the use of paper guidelines.[17] While the use of a guideline improved appropriate diagnostic testing during the pre-intervention observational period, there was a greater clinical and statistically significant increase in appropriate diagnostic decision making by the physicians with a handheld CDSS during the intervention (p\u2009=\u20090.023). Physicians who used a CDSS on their PDA assessed a significantly greater proportion of patients\u2019 pre-test probabilities and provided more appropriate diagnostic testing, especially for patients for whom pulmonary embolism was ruled out. When physicians in the handheld CDSS group recorded pre-test probabilities, they demonstrated greater adherence to recommended diagnostic testing (p\u2009=\u20090.03), and performed fewer mean tests per patient (p\u2009<\u20090.001). Therefore, in the assessment of pulmonary embolism in emergency departments, a handheld computer with a CDSS could provide a more appropriate diagnostic strategy for one out every four patients reviewed.\nThere was a non-significant trend towards more appropriate referral patterns to cardiac stress and nuclear cardiology testing, for patients with intermediate cardiac risk, among family physicians randomly assigned to use a PDA loaded with an interactive guideline for diagnosing angina, compared to customary care.[18] Further, those with the PDA and interactive guideline did not refer a higher percentage of patients on to cardiologists. It seems that use of the interactive guideline on a PDA could benefit one in every five patients referred for cardiac stress testing and one in six patients requiring additional nuclear cardiology testing.\nWhen 13 advanced practice nurses were randomly allocated to use a CDSS that calculated an obesity-related diagnosis on their PDA, they identified significantly more obesity-related diagnoses (p\u2009<\u20090.05) and missed less obesity-related diagnoses (p\u2009<\u20090.05) than the control group who had a PDA but not the CDSS.[19] Further, one in ten patients with obesity could be more appropriately identified with obesity-related diagnoses and one in three patients may be prevented from having a missed diagnosis by providing nurses with a PDA and CDSS.\n\nDiscussion \nKey findings \nThis systematic review provides a summary of current research evidence regarding the use of handheld computers to support clinical decision-making, from seven randomised controlled trials. We identified evidence suggesting that clinicians can effectively use handheld computers to access information to enhance their clinical knowledge, adhere to guidelines and make accurate and appropriate diagnostic decisions. The potential impact of clinicians using handheld computers to facilitate clinical decision making is high. Across six different diagnostic decisions, the numbers need to test\/screen were all less than 11. If these results can be reproduced in larger studies, the practice benefits could be substantial.\nIt appears that a wide range of healthcare clinicians are engaging with multiple uses of handheld computers, in a complementary role to existing informational tools. Handheld computers can make synthesised information more accessible through the provision of detailed recommendations and across a range of conditions and settings. While one of the included studies demonstrated the benefit of providing GPs with patient-specific reminders about preventative guidelines at the point of care[16], a cluster randomised trial demonstrated that simpler strategies, such as text message reminders for outpatient paediatric malaria management, was associated with improved adherence to national guidelines in Kenya.[20]\nQuick access to accurate information is important in situations where the delivery of care becomes more complex.[14] It is likely that mobile CDSSs will enhance benefits already identified for computerised CDSSs, particularly for improving workflow efficiencies.[21] For example, when handheld computers are used with CDSSs, individual patient information can be integrated with synthesised research evidence to facilitate decision making at the point of care.[22]\nCDSSs are designed using either rule-based systems that represent knowledge in IF\u2026THEN rules, or machine learning models where mathematical functions estimate risks given patient observations.[10] Both systems were referred to in the included studies. Clinicians were prompted to ask patients for specific contextual information most likely to impact their clinical diagnosis and treatment. This facilitated critical thinking around screening and management of patients.[19] Further, decision making was improved when excluding a diagnosis through more consistent documentation of pre-test probabilities and more appropriate use of tests for diagnostic investigations.[17] This review supports the mounting evidence from observational studies, that mobile CDSSs improve adherence to guidelines and policy, facilitate patient monitoring, provide valuable predictive tools, distinguish different levels of patient impairment and model medical problems for individualised care. Further, if they are integrated with electronic medical records, then individual patient data can be automatically included.[10]\nThe evidence for using mobile CDSSs to directly improve patient outcomes remains sparse.[3] While the included studies do not provide statistically significant support for improved prescribing behaviour[13], there is convincing evidence from several large non-randomised studies. An observational study conducted within a rural US community randomised trial, demonstrated that a PDA-based CDSS steadily improved outpatient antibiotic prescribing rates within usual consultations.[23] Similarly, in a before\/after prospective cohort trial in an Australian university-affiliated hospital\u2019s intensive care unit (ICU), the use of a CDSS was associated with a reduction in antibiotic usage and patterns of use more consistent with clinical guidelines.[24] This study also demonstrated a decrease in mean patient length of stay in the ICU, which can be interpreted as a surrogate for patient outcomes and overall costs.\n\nStrengths and limitations \nWhile this review focussed exclusively on RCTs, the conclusions are similar to the current and broader review of this topic.[3] Of the seven included studies, only four reported convincing, statistically significant evidence. This may represent the early proliferation of small and lower quality feasibility projects, associated with the growth of handheld computer use in healthcare.[20] This pattern is consistent with a recent broader review of mHealth technologies, in which none of the 42 included controlled trials were of high quality.[3] The heterogeneity of study designs and purposes makes the synthesis of this literature difficult. It will be important, in future studies, to be specific about the components of each intervention, so that the mechanisms of action and the impact of each component can be explored. However, the most promising results in both reviews were reported in the use of handheld computers for clinical management, appropriate testing and diagnosis.\nAs we did not search for studies via clinical trials registers, in other forms of grey literature, or published in other languages, there is a possibility that we may have missed unpublished studies. It is highly likely that participants in included studies were early adopters, who were more enthusiastic about the use of novel technology. Furthermore, there is reason to be concerned about publication bias given the sparse reporting of negative findings.\nAlthough we did look to include smartphones and tablets in this review, we only found suitable trials of PDAs. It is also interesting to note that despite the proliferation of research into mobile technologies in health care, there were no RCTs published in the last 3 years sufficiently rigorous for inclusion in this analysis. However, it is expected that the functional benefits of smartphones and small tablets are likely to be consistent. As technology becomes more sophisticated and the range of software and apps increase, the way in which healthcare professionals use these resources is likely to build on the functionalities examined in the studies reported here.\n\nImplication for practice \nWith widespread adoption of handheld computers by healthcare professionals, there is potential for improved access to information and improved clinical decision making at the point of care. Handheld computers provide a tool for synthesising, organising and accessing a wide range of research evidence for use with individual patient data. While this review has identified effective use of handheld computers across a broad range of clinical situations, there is a need to demonstrate direct improvements in patient outcomes.\nFurther, there is a need to understand the conditions in which handheld computers work best.[25] It may be useful to conceptualise them as complex interventions, informed by a theory of behaviour change and supporting existing practices.[4] Understanding facilitators and barriers for their continued adoption is important. Potential facilitators could include high levels of access and motivation to use handheld technology and minimal training costs, using online and tailored training programmes. It will be necessary to highlight additional benefits and challenges as different user populations are studied, particularly in low and middle income countries. Further, partnerships between government, private investors and researchers will be important in developing mobile computing technology and its implementation strategies.[4]\n\nFurther research \nThe pace of technological change is moving faster than the time it takes to design, implement and report on rigorous research. However, there is evidence of a rise in the number of registered clinical trials of mHealth interventions in the USA.[26] Robust and novel research designs are required to rapidly evaluate the effectiveness of healthcare professionals using handheld computers to improve their access to information and their clinical decision making at the point of care.\nIt will be important to document how handheld computers can be integrated into normal work practices, and to demonstrate improved clinical outcomes such as prescribing rates and lengths of stay. It will also be important to carefully \u2018blind\u2019 participants to particular functions or apps if all clinicians are using handheld computers within their daily work routines.\nAreas of high impact decision making such as emergency departments and intensive care units should be targeted for early RCTs. Following on, it will be important to broaden investigations across healthcare professionals in different clinical and geographic contexts, and to critically evaluate implementation plans and cost- benefit comparisons.\nWith the burgeoning development of apps, it will be important to monitor their accuracy and reliability.[10] It will also be important to monitor rates of handheld computer use in usual clinical care, and to supplement this with qualitative investigations of patient and provider attitudes and expectations. Further evaluation is warranted to investigate nurses\u2019 perceptions that using PDAs in front of patients seemed rude and inconvenient.[15]\n\nConclusion \nThis review provides evidence that healthcare professionals\u2019 use of handheld computers can improve their clinical decision making through improved information seeking and adherence to clinical guidelines. Handheld computers show promise for real time access to and analysis of clinical information, across many medical and health specialities. Handheld computers can host a variety of CDSS tools, which enable individual patient information to be integrated with synthesised research evidence, and facilitate decision making at the point of care. For diagnostic decisions, the numbers need to test\/screen were all less than 11. However, to quantify costs and benefits for patients and healthcare systems, replication of these early results in more robust studies is urgently needed.\n\nAbbreviations \nPDA: Personal digital assistant\nSMS: Short message service\nRCT: Randomized controlled trial\nCDSS: Clinical decision support system\nNSAID: Non-steroidal anti-inflammatory drugs\nICU: Intensive Care Unit\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors\u2019 contributions \nJT and SM conceived and designed the study; NR designed and implemented the search strategy; JT and SM screened titles and abstracts; HA and SM read full texts, assessed quality and extracted data from included studies, and prepared the manuscript. All authors contributed to editing the manuscript, read and approved the final manuscript.\n\nAcknowledgements \nHelen Atherton is funded by a National Institute for Health Research School for Primary Care Fellowship.\n\nReferences \n\n\n\u2191 1.0 1.1 Mosa, A.S.; Yoo, I.; Sheets, L. (2012). \"A systematic review of healthcare applications for smartphones\". BMC Medical Informatics & Decision Making 12: 67. doi:10.1186\/1472-6947-12-67. PMID 22781312.   \n\n\u2191 2.0 2.1 Divall, P.; Camosso-Stefinovic, J.; Baker, R. (2013). \"The use of personal digital assistants in clinical decision making by health care professionals: a systematic review\". Health Informatics Journal 19 (1): 16-28. doi:10.1177\/1460458212446761. PMID 23486823.   \n\n\u2191 3.0 3.1 3.2 3.3 3.4 Free, C.; Phillips, G.; Watson, L.; Galli, L.; Felix, L.; Edwards, P.; Patel, V.; Haines, A. (2013). \"The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis\". PLoS Medicine 10 (1): e1001363. doi:10.1371\/journal.pmed.1001363. PMC PMC3566926. PMID 23458994. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3566926 .   \n\n\u2191 4.0 4.1 4.2 4.3 Tomlinson, M.; Rotheram-Borus, M.J.; Swartz, L.; Tsai, A.C. (2013). \"Scaling up mHealth: where is the evidence?\". PLoS Medicine 10 (2): e1001382. doi:10.1371\/journal.pmed.1001382. PMC PMC3570540. PMID 23424286. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3570540 .   \n\n\u2191 Car, J.; Gurol-Urganci, I.; de Jongh, T.; Vodopivec-Jamsek, V.; Atun, R. (2012). \"Mobile phone messaging reminders for attendance at healthcare appointments\". Cochrane Database of Systematic Reviews 7: Cd007458. doi:10.1002\/14651858.CD007458. PMID 22786507.   \n\n\u2191 Guy, R.; Hocking, J.; Wand, H.; Stott, S.; Ali, H.; Kaldor, J. (2012). \"How effective are short message service reminders at increasing clinic attendance? a meta-analysis and systematic review\". Health Services Research 47 (2): 614-632. doi:10.1111\/j.1475-6773.2011.01342.x. PMC PMC3419880. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3419880 .   \n\n\u2191 Free, C.; Phillips, G.; Galli, L.; Watson, L.; Felix, L.; Edwards, P.; Patel, V.; Haines, A. (2013). \"The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review\". PloS Medicine 10 (1): e1001362. doi:10.1371\/journal.pmed.1001362. PMC PMC3548655. PMID 23349621. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3548655 .   \n\n\u2191 Pearson, S-A.; Moxey, A.; Robertson, J.; Hains, I.; Williamson, M.; Reeve, J.; Newby, D. (2009). \"Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990-2007)\". BMC Health Services Research 9 (1): 154. doi:10.1186\/1472-6963-9-154. PMC PMC2744674. PMID 19715591. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2744674 .   \n\n\u2191 Nieuwlaat, R.; Connolly, S.; Mackay, J.; Weise-Kelly, L.; Navarro, T.; Wilczynski, N.; Brian Haynes, R.; Team tCSR (2011). \"Computerized clinical decision support systems for therapeutic drug monitoring and dosing: a decision-maker-researcher partnership systematic review\". Implementation Science 6 (1): 90. doi:10.1186\/1748-5908-6-90. PMC PMC3170236. PMID 21824384. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3170236 .   \n\n\u2191 10.0 10.1 10.2 10.3 Mart\u00ednez-P\u00e9rez, B.; de la Torre-D\u00edez, I.; L\u00f3pez-Coronado, M.; Sainz-de-Abajo, B.; Robles, M.; Garc\u00eda-G\u00f3mez, J.M. (2014). \"Mobile clinical decision support systems and applications: a literature and commercial review\". Journal of Medical Systems 38 (1): 4. doi:10.1007\/s10916-013-0004-y. PMID 24399281.   \n\n\u2191 11.0 11.1 Mickan, S.; Tilson, K.J.; Atherton, H.; Roberts, W.N.; Heneghan, C. (2013). \"Evidence of effectiveness of health care professionals using handheld computers: a scoping review of systematic reviews\". Journal of Medical Internet Research 15 (10): e212. doi:10.2196\/jmir.2530. PMC PMC3841346. PMID 24165786. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3841346 .   \n\n\u2191 Higgins, J.P.; Altman, D.G.; Tzsche, P.C.; Ni, P.; Moher, D.; Oxman, A.D.; Savovic, J.; Schulz, K.F.; Weeks, L.; Sterne, J.A.C. (2011). \"The Cochrane Collaboration's tool for assessing risk of bias in randomised trials\". BMJ 343: d5928. doi:10.1136\/bmj.d5928. PMC PMC3196245. PMID 22008217. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3196245 .   \n\n\u2191 13.0 13.1 13.2 13.3 13.4 13.5 Berner, E.S.; Houston, T.K.; Ray, M.N.; Allison, J.J.; Heudebert, G.R.; Chatham, W.W.; Kennedy, J.I.; Glandon, G.L.; Norton, P.A.; Crawford, M.A.; Maisiak, R.S. (2006). \"Improving ambulatory prescribing safety with a handheld decision support system: a randomized controlled trial\". Journal of the American Medical Informatics Association 13 (2): 171-179. doi:10.1197\/jamia.M1961. PMC PMC1447547. PMID 16357350. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447547 .   \n\n\u2191 14.0 14.1 14.2 14.3 14.4 Bochicchio, G.V.; Smit, P.A.; Moore, R.; Bochicchio, K.; Auwaerter, P.; Johnson, S.B.; Scalea, T.; Bartlett, J.G. (2006). \"Pilot study of a web-based antibiotic decision management guide\". Journal of the American College of Surgeons 202 (3): 459-467. doi:10.1016\/j.jamcollsurg.2005.11.010. PMID 16500251.   \n\n\u2191 15.0 15.1 15.2 15.3 Farrell, M.J.; Rose, L. (2008). \"Use of mobile handheld computers in clinical nursing education\". Journal of Nursing Education 47 (1): 13-19. PMID 18232610.   \n\n\u2191 16.0 16.1 16.2 16.3 Price, M. (2005). \"Can hand-held computers improve adherence to guidelines? A (Palm) Pilot study of family doctors in British Columbia\". Canadian Family Physician 51 (11): 1506-1507. PMC PMC1479487. PMID 16926943. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1479487 .   \n\n\u2191 17.0 17.1 17.2 17.3 17.4 17.5 Roy, P-M.; Durieux, P.; Gillaizeau, F.; Legall, C.; Armand-Perroux, A.; Martino, L.; Hachelaf, M.; Dubart, A-E.; Schmidt, J.; Cristiano, M.; Jean-Marie, C.; Arnaud, P.; Guy, M. (2009). \"A computerized handheld decision-support system to improve pulmonary embolism diagnosis: a randomized trial\". Annals of Internal Medicine 151 (10): 677-686. doi:10.7326\/0003-4819-151-10-200911170-00003. PMID 19920268.   \n\n\u2191 18.0 18.1 18.2 18.3 18.4 Greiver, M.; Drummond, N.; White, D.; Weshler, J.; Moineddin, R.; Network NTPCR (2005). \"Angina on the Palm: randomized controlled pilot trial of Palm PDA software for referrals for cardiac testing\". Canadian Family Physician 51 (3): 382-383. PMC PMC1472967. PMID 16926933. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967 .   \n\n\u2191 19.0 19.1 19.2 19.3 19.4 19.5 Lee, N.J.; Chen, E.S.; Currie, L.M.; Donovan, M.; Hall, E.K.; Jia, H.; John, R.M.; Bakken, S. (2009). \"The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters\". Advances in Nursing Science 32 (3): 211-221. doi:10.1097\/ANS.0b013e3181b0d6bf. PMC PMC1472967. PMID 19707090. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967 .   \n\n\u2191 20.0 20.1 Zurovac, D.; Sudoi, R.K.; Akhwale, W.S.; Ndiritu, M.; Hamer, D.H.; Rowe, A.K.; Snow, R.W. (2011). \"The effect of mobile phone text-message reminders on Kenyan health workers\u2019 adherence to malaria treatment guidelines: a cluster randomised trial\". The Lancet 378 (9793): 795-803. doi:10.1016\/S0140-6736(11)60783-6. PMC PMC3163847. PMID 21820166. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163847 .   \n\n\u2191 Bright, T.J.; Wong, A.; Dhurjati, R.; Bristow, E.; Bastian, L.; Coeytaux, R.R.; Samsa, G.; Hasselblad, V.; Williams, J.W.; Musty, M.D.; Wing, L.; Kendrick, A.S.; Sanders, G.D.; Lobach, D. (2012). \"Effect of clinical decision-support systems: a systematic review\". Annals of Internal Medicine 157 (1): 29-43. doi:10.7326\/0003-4819-157-1-201207030-00450. PMID 22751758.   \n\n\u2191 DiCenso, A.; Bayley, L.; Haynes, R.B. (2009). \"Accessing pre-appraised evidence: fine-tuning the 5S model into a 6S model\". Evidence Based Nursing 12 (4): 99-101. doi:10.1136\/ebn.12.4.99-b. PMID 19779069.   \n\n\u2191 Rubin, M.A.; Bateman, K.; Donnelly, S.; Stoddard, G.J.; Stevenson, K.; Gardner, R.M.; Samore, M.H. (2006). \"Use of a personal digital assistant for managing antibiotic prescribing for outpatient respiratory tract infections in rural communities\". Journal of the American Medical Informatics Association 13 (6): 627-634. doi:10.1197\/jamia.M2029. PMC PMC1656956. PMID 16929045. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1656956 .   \n\n\u2191 Sintchenko, V.; Iredell, J.R.; Gilbert, G.L.; Coiera, E. (2005). \"Handheld computer-based decision support reduces patient length of stay and antibiotic prescribing in critical care\". Journal of the American Medical Informatics Association 12 (4): 398-402. doi:10.1197\/jamia.M1798. PMC PMC1174884. PMID 15802478. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1174884 .   \n\n\u2191 PLOS Medicine Editors (2013). \"A reality checkpoint for mobile health: three challenges to overcome\". PloS Medicine 10 (2): e1001395. doi:10.1371\/journal.pmed.1001395. PMC PMC3582561. PMID 23468597. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3582561 .   \n\n\u2191 Labrique, A.; Vasudevan, L.; Chang, L.W.; Mehl, G. (2013). \"H_pe for mHealth: more \"y\" or \"o\" on the horizon?\". International Journal of Medical Informatics 82 (5): 467-469. doi:10.1016\/j.ijmedinf.2012.11.016. PMC PMC3849805. PMID 23279850. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849805 .   \n\n\nPre-publication history \nThe pre-publication history for this paper can be accessed here: http:\/\/www.biomedcentral.com\/1472-6947\/14\/56\/prepub\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. The list of studies in Table 1 was also ordered differently (by reference number rather than alphabetical order) in order to maintain the same reference numbers used in the paper. The table has been made sortable to partially compensate.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\">https:\/\/www.limswiki.org\/index.php\/Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 21:05.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 4,407 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","26a517162c5d1f67f3c6bbb8c4cd1a80_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Use_of_handheld_computers_in_clinical_practice_A_systematic_review skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Use of handheld computers in clinical practice: A systematic review<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h3>\n<p>Many healthcare professionals use smartphones and tablets to inform patient care. Contemporary research suggests that handheld computers may support aspects of clinical diagnosis and management. This systematic review was designed to synthesise high quality evidence to answer the question; Does healthcare professionals\u2019 use of handheld computers improve their access to <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> and support clinical decision making at the point of care?\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h3>\n<p>A detailed search was conducted using Cochrane, MEDLINE, EMBASE, PsycINFO, Science and Social Science Citation Indices since 2001. Interventions promoting healthcare professionals seeking information or making clinical decisions using handheld computers were included. Classroom learning and the use of laptop computers were excluded. Two authors independently selected studies, assessed quality using the Cochrane Risk of Bias tool and extracted data. High levels of data heterogeneity negated statistical synthesis. Instead, evidence for effectiveness was summarised narratively, according to each study\u2019s aim for assessing the impact of handheld computer use.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h3>\n<p>We included seven randomised trials investigating medical or nursing staffs\u2019 use of Personal Digital Assistants. Effectiveness was demonstrated across three distinct functions that emerged from the data: accessing information for clinical knowledge, adherence to guidelines and diagnostic decision making. When healthcare professionals used handheld computers to access clinical information, their knowledge improved significantly more than peers who used paper resources. When clinical guideline recommendations were presented on handheld computers, clinicians made significantly safer prescribing decisions and adhered more closely to recommendations than peers using paper resources. Finally, healthcare professionals made significantly more appropriate diagnostic decisions using clinical decision making tools on handheld computers compared to colleagues who did not have access to these tools. For these clinical decisions, the numbers need to test\/screen were all less than 11.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h3>\n<p>Healthcare professionals\u2019 use of handheld computers may improve their information seeking, adherence to guidelines and clinical decision making. Handheld computers can provide real time access to and analysis of clinical information. The integration of <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" target=\"_blank\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support systems<\/a> within handheld computers offers clinicians the highest level of synthesised evidence at the point of care. Future research is needed to replicate these early results and to identify beneficial clinical outcomes.\n<\/p><p><b>Keywords<\/b>: Handheld computers; Smartphone; Information-seeking behaviour; Evidence-based practice; Knowledge translation; Clinical decision support systems; Clinical guidelines; Diagnostic decision making\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background_2\">Background<\/span><\/h2>\n<p>Increasing numbers of healthcare professionals use handheld computers that offer instant access to vast amounts of information via the internet and healthcare applications (apps).<sup id=\"rdp-ebb-cite_ref-MolaASys12_1-0\" class=\"reference\"><a href=\"#cite_note-MolaASys12-1\" rel=\"external_link\">[1]<\/a><\/sup> Over the last 10 years there has been a rapid and accelerating rate of innovation in handheld computers, from personal digital assistants (PDAs) towards more powerful, versatile and internet connected devices. As the rate of adoption of handheld computers has increased, individual patterns of usage have moved from that of communication and personal diary management towards information seeking and decision support.<sup id=\"rdp-ebb-cite_ref-DivallThe13_2-0\" class=\"reference\"><a href=\"#cite_note-DivallThe13-2\" rel=\"external_link\">[2]<\/a><\/sup> Today\u2019s clinicians can use handheld computers to search the internet for evidence and guidance on drugs and clinical conditions, use clinical decision support systems (CDSS) and access highly detailed patient information from clinical and <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> investigations.\n<\/p><p>At the same time, there has been a change in the acceptance of using handheld computers in healthcare settings. Now, most students and many professionals are enthusiastic about using smartphones and tablet computers, and they take them wherever they go.<sup id=\"rdp-ebb-cite_ref-FreeThe13_3-0\" class=\"reference\"><a href=\"#cite_note-FreeThe13-3\" rel=\"external_link\">[3]<\/a><\/sup> Along with this increasing adoption of handheld computers, there has been a massive growth in the volume of synthesized research information, healthcare oriented apps, databases and CDSSs.\n<\/p><p>This has also sparked an increased production of feasibility research, which has yet to recommend strategies for engagement, efficacy or effectiveness of mobile health initiatives.<sup id=\"rdp-ebb-cite_ref-TomlinsonScaling13_4-0\" class=\"reference\"><a href=\"#cite_note-TomlinsonScaling13-4\" rel=\"external_link\">[4]<\/a><\/sup> While both early and current systematic reviews offer tentative and sceptical conclusions, there is equipoise in the literature. A systematic review of the use of PDAs in clinical decision making reported an increase in data collection quality and concluded that the use of decision support software improved the appropriateness of diagnostic and treatment decisions.<sup id=\"rdp-ebb-cite_ref-DivallThe13_2-1\" class=\"reference\"><a href=\"#cite_note-DivallThe13-2\" rel=\"external_link\">[2]<\/a><\/sup> In a broader and contemporary systematic review of mHealth technologies, modest benefits were reported for improved clinical diagnosis and management support, and mixed outcomes were reported for efficient and accurate documentation.<sup id=\"rdp-ebb-cite_ref-FreeThe13_3-1\" class=\"reference\"><a href=\"#cite_note-FreeThe13-3\" rel=\"external_link\">[3]<\/a><\/sup> Further, there was no clear benefit for educational interventions and some evidence of reduced quality of clinical assessment, when using mobile technology based photos.\n<\/p><p>When healthcare professionals communicate with patients, there is high quality evidence to support the use of mobile phones to transmit short message service (SMS) reminders to improve attendance at health care appointments.<sup id=\"rdp-ebb-cite_ref-CarMobile12_5-0\" class=\"reference\"><a href=\"#cite_note-CarMobile12-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GuyHow12_6-0\" class=\"reference\"><a href=\"#cite_note-GuyHow12-6\" rel=\"external_link\">[6]<\/a><\/sup> Further, text messaging interventions were shown to increase adherence to antiretroviral therapy in low-income settings and increased smoking cessation in high income settings.<sup id=\"rdp-ebb-cite_ref-FreeTheEff13_7-0\" class=\"reference\"><a href=\"#cite_note-FreeTheEff13-7\" rel=\"external_link\">[7]<\/a><\/sup>\n<\/p><p>An early review of computerised, rather than mobile, CDSSs for prescribing, described effectiveness in initiating and monitoring therapy, but provided little evidence on their impact in specific clinical settings.<sup id=\"rdp-ebb-cite_ref-PearsonDo09_8-0\" class=\"reference\"><a href=\"#cite_note-PearsonDo09-8\" rel=\"external_link\">[8]<\/a><\/sup> A later review reported improved processes of care in 60% of included studies but improved patient outcomes in only 20% of studies.<sup id=\"rdp-ebb-cite_ref-NieuwlaatComp11_9-0\" class=\"reference\"><a href=\"#cite_note-NieuwlaatComp11-9\" rel=\"external_link\">[9]<\/a><\/sup> It is not clear whether incorporating these computerised systems into mobile devices would produce similar results.\n<\/p><p>A literature and commercial review of mobile CDSSs reported medical professionals using a growing number of apps across a wide range of fields.<sup id=\"rdp-ebb-cite_ref-MartinezMobile14_10-0\" class=\"reference\"><a href=\"#cite_note-MartinezMobile14-10\" rel=\"external_link\">[10]<\/a><\/sup> A systematic review of smartphone healthcare apps identified seven functional categories in which apps have been developed for use by healthcare professionals: diagnosis, drug reference, medical calculators, literature search, clinical communication, access to <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital_information_system\" title=\"Hospital information system\" target=\"_blank\" class=\"wiki-link\" data-key=\"d8385de7b1f39a39d793f8ce349b448d\">hospital information systems<\/a>, and medical training.<sup id=\"rdp-ebb-cite_ref-MolaASys12_1-1\" class=\"reference\"><a href=\"#cite_note-MolaASys12-1\" rel=\"external_link\">[1]<\/a><\/sup> A scoping review from a further five systematic reviews concluded that there is evidence for effective use of handheld computers by healthcare professionals across four key functions: providing easy and timely access to information, enabling accurate and complete documentation, providing instant access to evidence-based decision support and patient management systems, and promoting efficient work practices.<sup id=\"rdp-ebb-cite_ref-MickanEvi13_11-0\" class=\"reference\"><a href=\"#cite_note-MickanEvi13-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/p><p>Most published studies to date describe the design, development and implementation of handheld computers using observational study designs.<sup id=\"rdp-ebb-cite_ref-TomlinsonScaling13_4-1\" class=\"reference\"><a href=\"#cite_note-TomlinsonScaling13-4\" rel=\"external_link\">[4]<\/a><\/sup> In order to determine the benefits of integrating handheld computer use in healthcare practice, it is important to summarise and quantify results from the highest quality randomised controlled trials (RCTs) of effectiveness studies. Based on the functions identified in the earlier scoping review<sup id=\"rdp-ebb-cite_ref-MickanEvi13_11-1\" class=\"reference\"><a href=\"#cite_note-MickanEvi13-11\" rel=\"external_link\">[11]<\/a><\/sup>, it is timely to better understand whether healthcare professionals\u2019 use of handheld computers facilitates information seeking and improved clinical decision making. The purpose of this review is to answer the research question \u201cDoes healthcare professionals\u2019 use of handheld computers improve their access to information and support clinical decision making at the point of care?\u201d.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods_2\">Methods<\/span><\/h2>\n<p>The protocol for this systematic review was registered with PROSPERO (CRD42011001632), updated and adhered to. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.crd.york.ac.uk\/PROSPERO\/display_record.asp?ID=CRD42011001632#.U7-vibFnDhA\" target=\"_blank\">http:\/\/www.crd.york.ac.uk\/PROSPERO\/display_record.asp?ID=CRD42011001632#.U7-vibFnDhA<\/a>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Search_strategy\">Search strategy<\/span><\/h3>\n<p>We searched the following databases from 2001 to 19th August 2013: Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, EMBASE, PsycINFO, Science Citation Index and Social Science Citation Index. The MEDLINE search strategy can be found in the Additional file 1. Reference lists of included studies were hand searched.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 1. Medline search strategy.\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Format: DOC Size: 39KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1472-6947-14-56-s1.doc\" target=\"_blank\">Download file<\/a><br \/>\n<p>This file can be viewed with: <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.microsoft.com\/downloads\/details.aspx?displaylang=en&FamilyID=3657ce88-7cfa-457a-9aec-f4f827f20cac\" target=\"_blank\">Microsoft Word Viewer<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Inclusion_and_exclusion_criteria\">Inclusion and exclusion criteria<\/span><\/h3>\n<p>We included studies whose participants were healthcare professionals using handheld devices in clinical settings. Interventions of interest were those investigating the use of handheld computers to promote healthcare professionals\u2019 information seeking (outside of formal education courses), or to support informed clinical decision making. Our comparator was usual clinical practice. We excluded the use of laptops.\n<\/p><p>Study designs included were RCTs. The review was restricted to the English language. We searched from 2001 onwards to account for the changing nature of technology. We excluded studies that were presented as abstracts only, and where author contact confirmed the study had not been published in full.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Study_selection\">Study selection<\/span><\/h3>\n<p>Two authors (SM and JT) screened titles and abstracts. Full text articles were obtained for those selected and screened for inclusion (SM and HA). Where necessary, authors of studies were contacted for clarification of inclusion status.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_extraction\">Data extraction<\/span><\/h3>\n<p>A data extraction form was designed and piloted by two authors (SM and HA) to record study design, country, device used, aim, participants, setting, intervention, comparator, primary and secondary outcome data (as reported by the systematic review authors). The same authors independently extracted data. Disagreements were resolved by discussion.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Assessment_of_quality\">Assessment of quality<\/span><\/h3>\n<p>Assessment of risk of bias was conducted at the study level using the Cochrane Risk of Bias tool.<sup id=\"rdp-ebb-cite_ref-HigginsThe11_12-0\" class=\"reference\"><a href=\"#cite_note-HigginsThe11-12\" rel=\"external_link\">[12]<\/a><\/sup> Assessment was conducted independently by two authors (SM and HA) with disagreements resolved by discussion. Information on risk of bias status was used to aid interpretation of the included studies.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_synthesis\">Data synthesis<\/span><\/h3>\n<p>High levels of data heterogeneity and mixed data quality meant that statistical synthesis was not possible. We adopted a narrative approach to summarise the evidence for effectiveness according to the purpose for using the handheld computer.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_2\">Results<\/span><\/h2>\n<p>The combined search strategies identified 5,888 titles. After duplicates were removed, 3,612 titles were screened for eligibility. Thirty-eight full text articles were read, of which 31 did not meet the inclusion and exclusion criteria and therefore seven studies were retained for data extraction (see Figure 1).\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Mickan_BMCMedInfoDecMak2014_14.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"3b8556c2421565364c8394249074cc61\"><img alt=\"Fig1 Mickan BMCMedInfoDecMak2014 14.jpg\" src=\"https:\/\/www.limswiki.org\/images\/2\/2e\/Fig1_Mickan_BMCMedInfoDecMak2014_14.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Flow diagram<\/blockquote>\n<p>Characteristics of the seven included studies are summarised in Table 1. All were RCTs, mostly designed as pilot studies with comparatively small numbers of participants (range 12-76 participants). Although we intended to include studies investigating smartphones and tablets, to represent the most current forms of handheld computers, all included studies investigated the use of PDAs. Three studies were conducted in USA, two in Canada and one each in France and Australia. In five studies, the intervention group used a PDA while the control group used paper-based resources. In two studies, both groups used a PDA, but the intervention group had access to a specific clinical decision support system (CDSS) or information tool that the control group did not. Healthcare participants were either medical (residents, fellows, and family, general and emergency physicians) or nursing professionals. Where students were included, they were using a PDA in a clinical environment.\n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 1<\/b>: Characteristics of included studies<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable sortable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Study, country\n<\/th>\n<th> Participants, setting\n<\/th>\n<th> Intervention\n<\/th>\n<th> Comparator\n<\/th>\n<th> Primary outcome\n<\/th>\n<th> Secondary outcome\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Berner 2006 USA<sup id=\"rdp-ebb-cite_ref-BernerImprov06_13-0\" class=\"reference\"><a href=\"#cite_note-BernerImprov06-13\" rel=\"external_link\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">59 Internal medicine residents, University outpatient clinic\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA with rule for gastrointestinal risk assessment when prescribing NSAIDS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA without rule for gastrointestinal risk assessment when prescribing NSAIDS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Difference in unsafe NSAID prescriptions\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Identification of key risk factors for standardised patient case\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bochicchio 2006 USA<sup id=\"rdp-ebb-cite_ref-BochicchioPilot06_14-0\" class=\"reference\"><a href=\"#cite_note-BochicchioPilot06-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12 1st year critical care fellows, University hospital\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA with John Hopkins Antibiotic Guide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No PDA, instructed to use written reference guides\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Difference in mean score for knowledge test\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Antibiotic decision accuracy\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Farrell 2008 Australia<sup id=\"rdp-ebb-cite_ref-FarrellUse08_15-0\" class=\"reference\"><a href=\"#cite_note-FarrellUse08-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76 nursing students, Medical-surgical wards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA with pharmacological information and training session\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No training or PDA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Difference in mean score for pharmacology test\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\/A\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Price 2005 Canada<sup id=\"rdp-ebb-cite_ref-PriceCan05_16-0\" class=\"reference\"><a href=\"#cite_note-PriceCan05-16\" rel=\"external_link\">[16]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8 General practitioners, General practice (79 patients)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA with reminder for 5 preventive measures\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Software provided after the study\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Adherence to five guidelines\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\/A\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Roy 2009 France<sup id=\"rdp-ebb-cite_ref-RoyAComp09_17-0\" class=\"reference\"><a href=\"#cite_note-RoyAComp09-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24 Emergency physicians, 10 emergency departments (1645 patients)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA with CDSS for pulmonary embolism\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA used for data collection only; Paper based guideline material\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Appropriate diagnostic strategy for pulmonary embolism\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Adherence to recommended diagnostic testing Number of tests per patient\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Greiver 2005 Canada<sup id=\"rdp-ebb-cite_ref-GreiverAng05_18-0\" class=\"reference\"><a href=\"#cite_note-GreiverAng05-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18 Family physicians, Family practice (65 patients)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA with angina diagnosis software\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Conventional care\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Appropriate referral for cardiac stress testing at presentation, and nuclear cardiology after cardiac stress testing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Referral to cardiologists\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lee 2009 USA<sup id=\"rdp-ebb-cite_ref-LeeThe09_19-0\" class=\"reference\"><a href=\"#cite_note-LeeThe09-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29 registered nurses, Hospital and ambulatory care (1874 patients)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA with CDSS for obesity diagnosis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PDA without CDSS for obesity diagnosis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Appropriate obesity related diagnosis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Missed obesity related diagnosis\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The risk of bias assessment for included studies is shown in Figure 2 and summarised in Figure 3. The studies were largely at low or unclear risk of bias. Only one study<sup id=\"rdp-ebb-cite_ref-BernerImprov06_13-1\" class=\"reference\"><a href=\"#cite_note-BernerImprov06-13\" rel=\"external_link\">[13]<\/a><\/sup> was at low risk of bias for all domains. The highest risk of bias occurred for blinding of participants, which is not always practical when investigating the use of an obvious handheld computer. Participants could not be blinded in 5 studies where the intervention group used a PDA and the control group did not. Blinding was achieved, however, in two studies where both intervention and control groups were using a PDA. Four studies ensured that the outcome assessment was blinded.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Mickan_BMCMedInfoDecMak2014_14.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"6d366acf4530e506de498b782f04bb28\"><img alt=\"Fig2 Mickan BMCMedInfoDecMak2014 14.jpg\" src=\"https:\/\/www.limswiki.org\/images\/7\/71\/Fig2_Mickan_BMCMedInfoDecMak2014_14.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> Risk of bias in included studies<\/blockquote>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Mickan_BMCMedInfoDecMak2014_14.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"39d73b61fe706b1315c595f2328ff90f\"><img alt=\"Fig3 Mickan BMCMedInfoDecMak2014 14.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/bf\/Fig3_Mickan_BMCMedInfoDecMak2014_14.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3.<\/b> Risk of bias across individual domains of risk of bias<\/blockquote>\n<p>Heterogeneity of outcomes (Table 1) negated any quantitative synthesis or meta-analysis in this review. Instead, narrative summary was used to describe evidence of effectiveness of three distinct functions of handheld computers that emerged from the data: accessing information for clinical knowledge, adherence to safety and quality guidelines, and diagnostic decision making.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Information_for_clinical_knowledge\">Information for clinical knowledge<\/span><\/h3>\n<p>There is evidence from two pilot studies, one showing statistical significance<sup id=\"rdp-ebb-cite_ref-BochicchioPilot06_14-1\" class=\"reference\"><a href=\"#cite_note-BochicchioPilot06-14\" rel=\"external_link\">[14]<\/a><\/sup>, that when doctors and nurses used a handheld computer to access information in clinical environments, their clinical knowledge improved more than their peers who used traditional paper resources.<sup id=\"rdp-ebb-cite_ref-BochicchioPilot06_14-2\" class=\"reference\"><a href=\"#cite_note-BochicchioPilot06-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FarrellUse08_15-1\" class=\"reference\"><a href=\"#cite_note-FarrellUse08-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/p><p>Medical Fellows allocated to use a PDA in hospital intensive care units increased their mean knowledge of infectious disease management more than those who used paper resources at both 3 months (p\u2009<\u20090.05) and 6 months (p\u2009<\u20090.01).<sup id=\"rdp-ebb-cite_ref-BochicchioPilot06_14-3\" class=\"reference\"><a href=\"#cite_note-BochicchioPilot06-14\" rel=\"external_link\">[14]<\/a><\/sup> PDA use also increased fellows\u2019 antibiotic selection accuracy, a secondary aim of this project. Of the 125 antibiotic selections evaluated, fellows\u2019 antibiotic selection accuracy improved from 66% (33\/50) in the first 3 months, to 87% (65\/75) in the second 3 month period. In the second study, pharmacological knowledge of 76 nursing students was tested before and after a 3 week clinical placement on rather than in medical and surgical wards. All demonstrated increased pharmacological knowledge, with those who used a PDA increasing their mean score twice as much as those who used only paper resources. However, this was not significant (p\u2009=\u20090.17).<sup id=\"rdp-ebb-cite_ref-FarrellUse08_15-2\" class=\"reference\"><a href=\"#cite_note-FarrellUse08-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Adherence_to_guidelines\">Adherence to guidelines<\/span><\/h3>\n<p>Two feasibility RCTs examined the impact of healthcare professionals\u2019 use of PDAs on adherence to safety and quality guidelines in clinical practice.<sup id=\"rdp-ebb-cite_ref-BernerImprov06_13-2\" class=\"reference\"><a href=\"#cite_note-BernerImprov06-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PriceCan05_16-1\" class=\"reference\"><a href=\"#cite_note-PriceCan05-16\" rel=\"external_link\">[16]<\/a><\/sup> One identified a statistically significant result<sup id=\"rdp-ebb-cite_ref-BernerImprov06_13-3\" class=\"reference\"><a href=\"#cite_note-BernerImprov06-13\" rel=\"external_link\">[13]<\/a><\/sup> and both found that the PDA promoted adherence to guidelines.\n<\/p><p>In one study, 59 internal medicine residents received a PDA.<sup id=\"rdp-ebb-cite_ref-BernerImprov06_13-4\" class=\"reference\"><a href=\"#cite_note-BernerImprov06-13\" rel=\"external_link\">[13]<\/a><\/sup> The intervention group\u2019s PDA included software supporting the use of a prediction rule for assessing gastro-intestinal risk when prescribing non-steroidal anti-inflammatory drugs (NSAIDs). Thirteen standardised patients (all with moderate risks for an adverse gastro-intestinal event) were included in regular clinics and their medical records and prescriptions were independently reviewed by clinicians blinded to the residents\u2019 group assignment. Residents using a PDA with the prediction rule made fewer unsafe NSAID prescriptions (23% vs 45%, p\u2009<\u20090.05) and the mean proportion of cases per physician with unsafe prescriptions was significantly lower (0 vs 50%, p\u2009<\u20090.001). There was a non-significant trend towards identifying patients\u2019 risk factors (58% vs 45%, p\u2009>\u20090.05) and it was noted that no unsafe prescriptions were recommended when risk factors were correctly identified.\n<\/p><p>Among eight general practitioners screening 79 patients, those randomly allocated to use a PDA (n\u2009=\u20094) with clinical guideline decision support demonstrated greater adherence to four out of five preventive healthcare guidelines, compared to colleagues who did not have access to a PDA. Although the small sample prevented statistical comparison, there was a pattern of greater improvement in rates of adherence among PDA users for including the prophylactic use of aspirin in patients at risk of coronary artery disease (from 33% before to 81% after PDA access), colorectal screening (38-65%) and screening for cervical cancer (88-100%) and cholesterol (64-94%). In comparison, there was a slight reduction in the high rate of screening for hypertension (97-94%).<sup id=\"rdp-ebb-cite_ref-PriceCan05_16-2\" class=\"reference\"><a href=\"#cite_note-PriceCan05-16\" rel=\"external_link\">[16]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Diagnostic_decision_making\">Diagnostic decision making<\/span><\/h3>\n<p>There is evidence from three pilot RCTs<sup id=\"rdp-ebb-cite_ref-RoyAComp09_17-1\" class=\"reference\"><a href=\"#cite_note-RoyAComp09-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GreiverAng05_18-1\" class=\"reference\"><a href=\"#cite_note-GreiverAng05-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeeThe09_19-1\" class=\"reference\"><a href=\"#cite_note-LeeThe09-19\" rel=\"external_link\">[19]<\/a><\/sup> that having a clinical decision support system (CDSS) on a handheld computer can improve clinical and diagnostic decision making (Table 2). When used on a handheld computer in a clinical setting, the CDSS can prompt clinicians to collect and analyse patient data to inform diagnostic testing choices.\n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 2<\/b>: Improved diagnostic decision making<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable sortable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Diagnostic decision assessed\n<\/th>\n<th> Percentage of episodes with PDA (n\/n)\n<\/th>\n<th> Percentage of episodes without PDA (n\/n)\n<\/th>\n<th> Percentage absolute difference (95% CI)\n<\/th>\n<th> Number needed to test \/ screen\n<\/th>\n<th> P Value\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Appropriate diagnostic strategy for pulmonary embolism<sup id=\"rdp-ebb-cite_ref-RoyAComp09_17-2\" class=\"reference\"><a href=\"#cite_note-RoyAComp09-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">55% 378\/694\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26% 245\/951\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.023\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Adherence to recommended diagnostic testing<sup id=\"rdp-ebb-cite_ref-RoyAComp09_17-3\" class=\"reference\"><a href=\"#cite_note-RoyAComp09-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">41% 287\/694\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">17% 162\/951\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">24% (20-29)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.030\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Appropriate referral for cardiac stress testing at presentation<sup id=\"rdp-ebb-cite_ref-GreiverAng05_18-2\" class=\"reference\"><a href=\"#cite_note-GreiverAng05-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">49% 18\/37\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29% 8\/28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.284\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Appropriate referral for nuclear cardiology after cardiac stress testing<sup id=\"rdp-ebb-cite_ref-GreiverAng05_18-3\" class=\"reference\"><a href=\"#cite_note-GreiverAng05-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">63% 17\/27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">45% 5\/11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18% (14-50)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Appropriate obesity-related diagnosis<sup id=\"rdp-ebb-cite_ref-LeeThe09_19-2\" class=\"reference\"><a href=\"#cite_note-LeeThe09-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11% 91\/807\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1% 10\/997\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10% (8-13)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.05\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Missed obesity-related diagnosis<sup id=\"rdp-ebb-cite_ref-LeeThe09_19-3\" class=\"reference\"><a href=\"#cite_note-LeeThe09-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25% 51\/208\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">67% 440\/662\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">42% (35-48)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><0.05\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For 24 emergency physicians, the use of a handheld CDSS led to greater improvement in diagnostic decision making for pulmonary emboli, compared to the use of paper guidelines.<sup id=\"rdp-ebb-cite_ref-RoyAComp09_17-4\" class=\"reference\"><a href=\"#cite_note-RoyAComp09-17\" rel=\"external_link\">[17]<\/a><\/sup> While the use of a guideline improved appropriate diagnostic testing during the pre-intervention observational period, there was a greater clinical and statistically significant increase in appropriate diagnostic decision making by the physicians with a handheld CDSS during the intervention (p\u2009=\u20090.023). Physicians who used a CDSS on their PDA assessed a significantly greater proportion of patients\u2019 pre-test probabilities and provided more appropriate diagnostic testing, especially for patients for whom pulmonary embolism was ruled out. When physicians in the handheld CDSS group recorded pre-test probabilities, they demonstrated greater adherence to recommended diagnostic testing (p\u2009=\u20090.03), and performed fewer mean tests per patient (p\u2009<\u20090.001). Therefore, in the assessment of pulmonary embolism in emergency departments, a handheld computer with a CDSS could provide a more appropriate diagnostic strategy for one out every four patients reviewed.\n<\/p><p>There was a non-significant trend towards more appropriate referral patterns to cardiac stress and nuclear cardiology testing, for patients with intermediate cardiac risk, among family physicians randomly assigned to use a PDA loaded with an interactive guideline for diagnosing angina, compared to customary care.<sup id=\"rdp-ebb-cite_ref-GreiverAng05_18-4\" class=\"reference\"><a href=\"#cite_note-GreiverAng05-18\" rel=\"external_link\">[18]<\/a><\/sup> Further, those with the PDA and interactive guideline did not refer a higher percentage of patients on to cardiologists. It seems that use of the interactive guideline on a PDA could benefit one in every five patients referred for cardiac stress testing and one in six patients requiring additional nuclear cardiology testing.\n<\/p><p>When 13 advanced practice nurses were randomly allocated to use a CDSS that calculated an obesity-related diagnosis on their PDA, they identified significantly more obesity-related diagnoses (p\u2009<\u20090.05) and missed less obesity-related diagnoses (p\u2009<\u20090.05) than the control group who had a PDA but not the CDSS.<sup id=\"rdp-ebb-cite_ref-LeeThe09_19-4\" class=\"reference\"><a href=\"#cite_note-LeeThe09-19\" rel=\"external_link\">[19]<\/a><\/sup> Further, one in ten patients with obesity could be more appropriately identified with obesity-related diagnoses and one in three patients may be prevented from having a missed diagnosis by providing nurses with a PDA and CDSS.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Key_findings\">Key findings<\/span><\/h3>\n<p>This systematic review provides a summary of current research evidence regarding the use of handheld computers to support clinical decision-making, from seven randomised controlled trials. We identified evidence suggesting that clinicians can effectively use handheld computers to access information to enhance their clinical knowledge, adhere to guidelines and make accurate and appropriate diagnostic decisions. The potential impact of clinicians using handheld computers to facilitate clinical decision making is high. Across six different diagnostic decisions, the numbers need to test\/screen were all less than 11. If these results can be reproduced in larger studies, the practice benefits could be substantial.\n<\/p><p>It appears that a wide range of healthcare clinicians are engaging with multiple uses of handheld computers, in a complementary role to existing informational tools. Handheld computers can make synthesised information more accessible through the provision of detailed recommendations and across a range of conditions and settings. While one of the included studies demonstrated the benefit of providing GPs with patient-specific reminders about preventative guidelines at the point of care<sup id=\"rdp-ebb-cite_ref-PriceCan05_16-3\" class=\"reference\"><a href=\"#cite_note-PriceCan05-16\" rel=\"external_link\">[16]<\/a><\/sup>, a cluster randomised trial demonstrated that simpler strategies, such as text message reminders for outpatient paediatric malaria management, was associated with improved adherence to national guidelines in Kenya.<sup id=\"rdp-ebb-cite_ref-ZurovacThe11_20-0\" class=\"reference\"><a href=\"#cite_note-ZurovacThe11-20\" rel=\"external_link\">[20]<\/a><\/sup>\n<\/p><p>Quick access to accurate information is important in situations where the delivery of care becomes more complex.<sup id=\"rdp-ebb-cite_ref-BochicchioPilot06_14-4\" class=\"reference\"><a href=\"#cite_note-BochicchioPilot06-14\" rel=\"external_link\">[14]<\/a><\/sup> It is likely that mobile CDSSs will enhance benefits already identified for computerised CDSSs, particularly for improving workflow efficiencies.<sup id=\"rdp-ebb-cite_ref-BrightEffect12_21-0\" class=\"reference\"><a href=\"#cite_note-BrightEffect12-21\" rel=\"external_link\">[21]<\/a><\/sup> For example, when handheld computers are used with CDSSs, individual patient information can be integrated with synthesised research evidence to facilitate decision making at the point of care.<sup id=\"rdp-ebb-cite_ref-DiCensoAcc09_22-0\" class=\"reference\"><a href=\"#cite_note-DiCensoAcc09-22\" rel=\"external_link\">[22]<\/a><\/sup>\n<\/p><p>CDSSs are designed using either rule-based systems that represent knowledge in IF\u2026THEN rules, or machine learning models where mathematical functions estimate risks given patient observations.<sup id=\"rdp-ebb-cite_ref-MartinezMobile14_10-1\" class=\"reference\"><a href=\"#cite_note-MartinezMobile14-10\" rel=\"external_link\">[10]<\/a><\/sup> Both systems were referred to in the included studies. Clinicians were prompted to ask patients for specific contextual information most likely to impact their clinical diagnosis and treatment. This facilitated critical thinking around screening and management of patients.<sup id=\"rdp-ebb-cite_ref-LeeThe09_19-5\" class=\"reference\"><a href=\"#cite_note-LeeThe09-19\" rel=\"external_link\">[19]<\/a><\/sup> Further, decision making was improved when excluding a diagnosis through more consistent documentation of pre-test probabilities and more appropriate use of tests for diagnostic investigations.<sup id=\"rdp-ebb-cite_ref-RoyAComp09_17-5\" class=\"reference\"><a href=\"#cite_note-RoyAComp09-17\" rel=\"external_link\">[17]<\/a><\/sup> This review supports the mounting evidence from observational studies, that mobile CDSSs improve adherence to guidelines and policy, facilitate patient monitoring, provide valuable predictive tools, distinguish different levels of patient impairment and model medical problems for individualised care. Further, if they are integrated with electronic medical records, then individual patient data can be automatically included.<sup id=\"rdp-ebb-cite_ref-MartinezMobile14_10-2\" class=\"reference\"><a href=\"#cite_note-MartinezMobile14-10\" rel=\"external_link\">[10]<\/a><\/sup>\n<\/p><p>The evidence for using mobile CDSSs to directly improve patient outcomes remains sparse.<sup id=\"rdp-ebb-cite_ref-FreeThe13_3-2\" class=\"reference\"><a href=\"#cite_note-FreeThe13-3\" rel=\"external_link\">[3]<\/a><\/sup> While the included studies do not provide statistically significant support for improved prescribing behaviour<sup id=\"rdp-ebb-cite_ref-BernerImprov06_13-5\" class=\"reference\"><a href=\"#cite_note-BernerImprov06-13\" rel=\"external_link\">[13]<\/a><\/sup>, there is convincing evidence from several large non-randomised studies. An observational study conducted within a rural US community randomised trial, demonstrated that a PDA-based CDSS steadily improved outpatient antibiotic prescribing rates within usual consultations.<sup id=\"rdp-ebb-cite_ref-RubinUse06_23-0\" class=\"reference\"><a href=\"#cite_note-RubinUse06-23\" rel=\"external_link\">[23]<\/a><\/sup> Similarly, in a before\/after prospective cohort trial in an Australian university-affiliated hospital\u2019s intensive care unit (ICU), the use of a CDSS was associated with a reduction in antibiotic usage and patterns of use more consistent with clinical guidelines.<sup id=\"rdp-ebb-cite_ref-SintchenkoHand05_24-0\" class=\"reference\"><a href=\"#cite_note-SintchenkoHand05-24\" rel=\"external_link\">[24]<\/a><\/sup> This study also demonstrated a decrease in mean patient length of stay in the ICU, which can be interpreted as a surrogate for patient outcomes and overall costs.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Strengths_and_limitations\">Strengths and limitations<\/span><\/h3>\n<p>While this review focussed exclusively on RCTs, the conclusions are similar to the current and broader review of this topic.<sup id=\"rdp-ebb-cite_ref-FreeThe13_3-3\" class=\"reference\"><a href=\"#cite_note-FreeThe13-3\" rel=\"external_link\">[3]<\/a><\/sup> Of the seven included studies, only four reported convincing, statistically significant evidence. This may represent the early proliferation of small and lower quality feasibility projects, associated with the growth of handheld computer use in healthcare.<sup id=\"rdp-ebb-cite_ref-ZurovacThe11_20-1\" class=\"reference\"><a href=\"#cite_note-ZurovacThe11-20\" rel=\"external_link\">[20]<\/a><\/sup> This pattern is consistent with a recent broader review of mHealth technologies, in which none of the 42 included controlled trials were of high quality.<sup id=\"rdp-ebb-cite_ref-FreeThe13_3-4\" class=\"reference\"><a href=\"#cite_note-FreeThe13-3\" rel=\"external_link\">[3]<\/a><\/sup> The heterogeneity of study designs and purposes makes the synthesis of this literature difficult. It will be important, in future studies, to be specific about the components of each intervention, so that the mechanisms of action and the impact of each component can be explored. However, the most promising results in both reviews were reported in the use of handheld computers for clinical management, appropriate testing and diagnosis.\n<\/p><p>As we did not search for studies via clinical trials registers, in other forms of grey literature, or published in other languages, there is a possibility that we may have missed unpublished studies. It is highly likely that participants in included studies were early adopters, who were more enthusiastic about the use of novel technology. Furthermore, there is reason to be concerned about publication bias given the sparse reporting of negative findings.\n<\/p><p>Although we did look to include smartphones and tablets in this review, we only found suitable trials of PDAs. It is also interesting to note that despite the proliferation of research into mobile technologies in health care, there were no RCTs published in the last 3 years sufficiently rigorous for inclusion in this analysis. However, it is expected that the functional benefits of smartphones and small tablets are likely to be consistent. As technology becomes more sophisticated and the range of software and apps increase, the way in which healthcare professionals use these resources is likely to build on the functionalities examined in the studies reported here.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implication_for_practice\">Implication for practice<\/span><\/h3>\n<p>With widespread adoption of handheld computers by healthcare professionals, there is potential for improved access to information and improved clinical decision making at the point of care. Handheld computers provide a tool for synthesising, organising and accessing a wide range of research evidence for use with individual patient data. While this review has identified effective use of handheld computers across a broad range of clinical situations, there is a need to demonstrate direct improvements in patient outcomes.\n<\/p><p>Further, there is a need to understand the conditions in which handheld computers work best.<sup id=\"rdp-ebb-cite_ref-PMEdsAReal13_25-0\" class=\"reference\"><a href=\"#cite_note-PMEdsAReal13-25\" rel=\"external_link\">[25]<\/a><\/sup> It may be useful to conceptualise them as complex interventions, informed by a theory of behaviour change and supporting existing practices.<sup id=\"rdp-ebb-cite_ref-TomlinsonScaling13_4-2\" class=\"reference\"><a href=\"#cite_note-TomlinsonScaling13-4\" rel=\"external_link\">[4]<\/a><\/sup> Understanding facilitators and barriers for their continued adoption is important. Potential facilitators could include high levels of access and motivation to use handheld technology and minimal training costs, using online and tailored training programmes. It will be necessary to highlight additional benefits and challenges as different user populations are studied, particularly in low and middle income countries. Further, partnerships between government, private investors and researchers will be important in developing mobile computing technology and its implementation strategies.<sup id=\"rdp-ebb-cite_ref-TomlinsonScaling13_4-3\" class=\"reference\"><a href=\"#cite_note-TomlinsonScaling13-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Further_research\">Further research<\/span><\/h3>\n<p>The pace of technological change is moving faster than the time it takes to design, implement and report on rigorous research. However, there is evidence of a rise in the number of registered clinical trials of mHealth interventions in the USA.<sup id=\"rdp-ebb-cite_ref-LabriqueHpe13_26-0\" class=\"reference\"><a href=\"#cite_note-LabriqueHpe13-26\" rel=\"external_link\">[26]<\/a><\/sup> Robust and novel research designs are required to rapidly evaluate the effectiveness of healthcare professionals using handheld computers to improve their access to information and their clinical decision making at the point of care.\n<\/p><p>It will be important to document how handheld computers can be integrated into normal work practices, and to demonstrate improved clinical outcomes such as prescribing rates and lengths of stay. It will also be important to carefully \u2018blind\u2019 participants to particular functions or apps if all clinicians are using handheld computers within their daily work routines.\n<\/p><p>Areas of high impact decision making such as emergency departments and intensive care units should be targeted for early RCTs. Following on, it will be important to broaden investigations across healthcare professionals in different clinical and geographic contexts, and to critically evaluate implementation plans and cost- benefit comparisons.\n<\/p><p>With the burgeoning development of apps, it will be important to monitor their accuracy and reliability.<sup id=\"rdp-ebb-cite_ref-MartinezMobile14_10-3\" class=\"reference\"><a href=\"#cite_note-MartinezMobile14-10\" rel=\"external_link\">[10]<\/a><\/sup> It will also be important to monitor rates of handheld computer use in usual clinical care, and to supplement this with qualitative investigations of patient and provider attitudes and expectations. Further evaluation is warranted to investigate nurses\u2019 perceptions that using PDAs in front of patients seemed rude and inconvenient.<sup id=\"rdp-ebb-cite_ref-FarrellUse08_15-3\" class=\"reference\"><a href=\"#cite_note-FarrellUse08-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion_2\">Conclusion<\/span><\/h2>\n<p>This review provides evidence that healthcare professionals\u2019 use of handheld computers can improve their clinical decision making through improved information seeking and adherence to clinical guidelines. Handheld computers show promise for real time access to and analysis of clinical information, across many medical and health specialities. Handheld computers can host a variety of CDSS tools, which enable individual patient information to be integrated with synthesised research evidence, and facilitate decision making at the point of care. For diagnostic decisions, the numbers need to test\/screen were all less than 11. However, to quantify costs and benefits for patients and healthcare systems, replication of these early results in more robust studies is urgently needed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p>PDA: Personal digital assistant\nSMS: Short message service\nRCT: Randomized controlled trial\nCDSS: Clinical decision support system\nNSAID: Non-steroidal anti-inflammatory drugs\nICU: Intensive Care Unit\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h2>\n<p>JT and SM conceived and designed the study; NR designed and implemented the search strategy; JT and SM screened titles and abstracts; HA and SM read full texts, assessed quality and extracted data from included studies, and prepared the manuscript. All authors contributed to editing the manuscript, read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>Helen Atherton is funded by a National Institute for Health Research School for Primary Care Fellowship.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-MolaASys12-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MolaASys12_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-MolaASys12_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mosa, A.S.; Yoo, I.; Sheets, L. (2012). \"A systematic review of healthcare applications for smartphones\". <i>BMC Medical Informatics & Decision Making<\/i> <b>12<\/b>: 67. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-12-67\" target=\"_blank\">10.1186\/1472-6947-12-67<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22781312\" target=\"_blank\">22781312<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+systematic+review+of+healthcare+applications+for+smartphones&rft.jtitle=BMC+Medical+Informatics+%26+Decision+Making&rft.aulast=Mosa%2C+A.S.%3B+Yoo%2C+I.%3B+Sheets%2C+L.&rft.au=Mosa%2C+A.S.%3B+Yoo%2C+I.%3B+Sheets%2C+L.&rft.date=2012&rft.volume=12&rft.pages=67&rft_id=info:doi\/10.1186%2F1472-6947-12-67&rft_id=info:pmid\/22781312&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DivallThe13-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DivallThe13_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-DivallThe13_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Divall, P.; Camosso-Stefinovic, J.; Baker, R. (2013). \"The use of personal digital assistants in clinical decision making by health care professionals: a systematic review\". <i>Health Informatics Journal<\/i> <b>19<\/b> (1): 16-28. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1460458212446761\" target=\"_blank\">10.1177\/1460458212446761<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23486823\" target=\"_blank\">23486823<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+use+of+personal+digital+assistants+in+clinical+decision+making+by+health+care+professionals%3A+a+systematic+review&rft.jtitle=Health+Informatics+Journal&rft.aulast=Divall%2C+P.%3B+Camosso-Stefinovic%2C+J.%3B+Baker%2C+R.&rft.au=Divall%2C+P.%3B+Camosso-Stefinovic%2C+J.%3B+Baker%2C+R.&rft.date=2013&rft.volume=19&rft.issue=1&rft.pages=16-28&rft_id=info:doi\/10.1177%2F1460458212446761&rft_id=info:pmid\/23486823&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FreeThe13-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FreeThe13_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-FreeThe13_3-1\" rel=\"external_link\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-FreeThe13_3-2\" rel=\"external_link\">3.2<\/a><\/sup> <sup><a href=\"#cite_ref-FreeThe13_3-3\" rel=\"external_link\">3.3<\/a><\/sup> <sup><a href=\"#cite_ref-FreeThe13_3-4\" rel=\"external_link\">3.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Free, C.; Phillips, G.; Watson, L.; Galli, L.; Felix, L.; Edwards, P.; Patel, V.; Haines, A. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3566926\" target=\"_blank\">\"The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis\"<\/a>. <i>PLoS Medicine<\/i> <b>10<\/b> (1): e1001363. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1001363\" target=\"_blank\">10.1371\/journal.pmed.1001363<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3566926\/\" target=\"_blank\">PMC3566926<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23458994\" target=\"_blank\">23458994<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3566926\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3566926<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+effectiveness+of+mobile-health+technologies+to+improve+health+care+service+delivery+processes%3A+a+systematic+review+and+meta-analysis&rft.jtitle=PLoS+Medicine&rft.aulast=Free%2C+C.%3B+Phillips%2C+G.%3B+Watson%2C+L.%3B+Galli%2C+L.%3B+Felix%2C+L.%3B+Edwards%2C+P.%3B+Patel%2C+V.%3B+Haines%2C+A.&rft.au=Free%2C+C.%3B+Phillips%2C+G.%3B+Watson%2C+L.%3B+Galli%2C+L.%3B+Felix%2C+L.%3B+Edwards%2C+P.%3B+Patel%2C+V.%3B+Haines%2C+A.&rft.date=2013&rft.volume=10&rft.issue=1&rft.pages=e1001363&rft_id=info:doi\/10.1371%2Fjournal.pmed.1001363&rft_id=info:pmc\/PMC3566926&rft_id=info:pmid\/23458994&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3566926&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TomlinsonScaling13-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TomlinsonScaling13_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-TomlinsonScaling13_4-1\" rel=\"external_link\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-TomlinsonScaling13_4-2\" rel=\"external_link\">4.2<\/a><\/sup> <sup><a href=\"#cite_ref-TomlinsonScaling13_4-3\" rel=\"external_link\">4.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tomlinson, M.; Rotheram-Borus, M.J.; Swartz, L.; Tsai, A.C. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3570540\" target=\"_blank\">\"Scaling up mHealth: where is the evidence?\"<\/a>. <i>PLoS Medicine<\/i> <b>10<\/b> (2): e1001382. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1001382\" target=\"_blank\">10.1371\/journal.pmed.1001382<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3570540\/\" target=\"_blank\">PMC3570540<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23424286\" target=\"_blank\">23424286<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3570540\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3570540<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Scaling+up+mHealth%3A+where+is+the+evidence%3F&rft.jtitle=PLoS+Medicine&rft.aulast=Tomlinson%2C+M.%3B+Rotheram-Borus%2C+M.J.%3B+Swartz%2C+L.%3B+Tsai%2C+A.C.&rft.au=Tomlinson%2C+M.%3B+Rotheram-Borus%2C+M.J.%3B+Swartz%2C+L.%3B+Tsai%2C+A.C.&rft.date=2013&rft.volume=10&rft.issue=2&rft.pages=e1001382&rft_id=info:doi\/10.1371%2Fjournal.pmed.1001382&rft_id=info:pmc\/PMC3570540&rft_id=info:pmid\/23424286&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3570540&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarMobile12-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CarMobile12_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Car, J.; Gurol-Urganci, I.; de Jongh, T.; Vodopivec-Jamsek, V.; Atun, R. (2012). \"Mobile phone messaging reminders for attendance at healthcare appointments\". <i>Cochrane Database of Systematic Reviews<\/i> <b>7<\/b>: Cd007458. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2F14651858.CD007458\" target=\"_blank\">10.1002\/14651858.CD007458<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22786507\" target=\"_blank\">22786507<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Mobile+phone+messaging+reminders+for+attendance+at+healthcare+appointments&rft.jtitle=Cochrane+Database+of+Systematic+Reviews&rft.aulast=Car%2C+J.%3B+Gurol-Urganci%2C+I.%3B+de+Jongh%2C+T.%3B+Vodopivec-Jamsek%2C+V.%3B+Atun%2C+R.&rft.au=Car%2C+J.%3B+Gurol-Urganci%2C+I.%3B+de+Jongh%2C+T.%3B+Vodopivec-Jamsek%2C+V.%3B+Atun%2C+R.&rft.date=2012&rft.volume=7&rft.pages=Cd007458&rft_id=info:doi\/10.1002%2F14651858.CD007458&rft_id=info:pmid\/22786507&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuyHow12-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuyHow12_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guy, R.; Hocking, J.; Wand, H.; Stott, S.; Ali, H.; Kaldor, J. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3419880\" target=\"_blank\">\"How effective are short message service reminders at increasing clinic attendance? a meta-analysis and systematic review\"<\/a>. <i>Health Services Research<\/i> <b>47<\/b> (2): 614-632. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1475-6773.2011.01342.x\" target=\"_blank\">10.1111\/j.1475-6773.2011.01342.x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3419880\/\" target=\"_blank\">PMC3419880<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3419880\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3419880<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=How+effective+are+short+message+service+reminders+at+increasing+clinic+attendance%3F+a+meta-analysis+and+systematic+review&rft.jtitle=Health+Services+Research&rft.aulast=Guy%2C+R.%3B+Hocking%2C+J.%3B+Wand%2C+H.%3B+Stott%2C+S.%3B+Ali%2C+H.%3B+Kaldor%2C+J.&rft.au=Guy%2C+R.%3B+Hocking%2C+J.%3B+Wand%2C+H.%3B+Stott%2C+S.%3B+Ali%2C+H.%3B+Kaldor%2C+J.&rft.date=2012&rft.volume=47&rft.issue=2&rft.pages=614-632&rft_id=info:doi\/10.1111%2Fj.1475-6773.2011.01342.x&rft_id=info:pmc\/PMC3419880&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3419880&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FreeTheEff13-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FreeTheEff13_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Free, C.; Phillips, G.; Galli, L.; Watson, L.; Felix, L.; Edwards, P.; Patel, V.; Haines, A. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3548655\" target=\"_blank\">\"The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review\"<\/a>. <i>PloS Medicine<\/i> <b>10<\/b> (1): e1001362. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1001362\" target=\"_blank\">10.1371\/journal.pmed.1001362<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3548655\/\" target=\"_blank\">PMC3548655<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23349621\" target=\"_blank\">23349621<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3548655\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3548655<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+effectiveness+of+mobile-health+technology-based+health+behaviour+change+or+disease+management+interventions+for+health+care+consumers%3A+a+systematic+review&rft.jtitle=PloS+Medicine&rft.aulast=Free%2C+C.%3B+Phillips%2C+G.%3B+Galli%2C+L.%3B+Watson%2C+L.%3B+Felix%2C+L.%3B+Edwards%2C+P.%3B+Patel%2C+V.%3B+Haines%2C+A.&rft.au=Free%2C+C.%3B+Phillips%2C+G.%3B+Galli%2C+L.%3B+Watson%2C+L.%3B+Felix%2C+L.%3B+Edwards%2C+P.%3B+Patel%2C+V.%3B+Haines%2C+A.&rft.date=2013&rft.volume=10&rft.issue=1&rft.pages=e1001362&rft_id=info:doi\/10.1371%2Fjournal.pmed.1001362&rft_id=info:pmc\/PMC3548655&rft_id=info:pmid\/23349621&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3548655&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PearsonDo09-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PearsonDo09_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pearson, S-A.; Moxey, A.; Robertson, J.; Hains, I.; Williamson, M.; Reeve, J.; Newby, D. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2744674\" target=\"_blank\">\"Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990-2007)\"<\/a>. <i>BMC Health Services Research<\/i> <b>9<\/b> (1): 154. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6963-9-154\" target=\"_blank\">10.1186\/1472-6963-9-154<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2744674\/\" target=\"_blank\">PMC2744674<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19715591\" target=\"_blank\">19715591<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2744674\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2744674<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Do+computerised+clinical+decision+support+systems+for+prescribing+change+practice%3F+A+systematic+review+of+the+literature+%281990-2007%29&rft.jtitle=BMC+Health+Services+Research&rft.aulast=Pearson%2C+S-A.%3B+Moxey%2C+A.%3B+Robertson%2C+J.%3B+Hains%2C+I.%3B+Williamson%2C+M.%3B+Reeve%2C+J.%3B+Newby%2C+D.&rft.au=Pearson%2C+S-A.%3B+Moxey%2C+A.%3B+Robertson%2C+J.%3B+Hains%2C+I.%3B+Williamson%2C+M.%3B+Reeve%2C+J.%3B+Newby%2C+D.&rft.date=2009&rft.volume=9&rft.issue=1&rft.pages=154&rft_id=info:doi\/10.1186%2F1472-6963-9-154&rft_id=info:pmc\/PMC2744674&rft_id=info:pmid\/19715591&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2744674&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NieuwlaatComp11-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NieuwlaatComp11_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nieuwlaat, R.; Connolly, S.; Mackay, J.; Weise-Kelly, L.; Navarro, T.; Wilczynski, N.; Brian Haynes, R.; Team tCSR (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3170236\" target=\"_blank\">\"Computerized clinical decision support systems for therapeutic drug monitoring and dosing: a decision-maker-researcher partnership systematic review\"<\/a>. <i>Implementation Science<\/i> <b>6<\/b> (1): 90. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1748-5908-6-90\" target=\"_blank\">10.1186\/1748-5908-6-90<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3170236\/\" target=\"_blank\">PMC3170236<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21824384\" target=\"_blank\">21824384<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3170236\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3170236<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Computerized+clinical+decision+support+systems+for+therapeutic+drug+monitoring+and+dosing%3A+a+decision-maker-researcher+partnership+systematic+review&rft.jtitle=Implementation+Science&rft.aulast=Nieuwlaat%2C+R.%3B+Connolly%2C+S.%3B+Mackay%2C+J.%3B+Weise-Kelly%2C+L.%3B+Navarro%2C+T.%3B+Wilczynski%2C+N.%3B+Brian+Haynes%2C+R.%3B+Team+tCSR&rft.au=Nieuwlaat%2C+R.%3B+Connolly%2C+S.%3B+Mackay%2C+J.%3B+Weise-Kelly%2C+L.%3B+Navarro%2C+T.%3B+Wilczynski%2C+N.%3B+Brian+Haynes%2C+R.%3B+Team+tCSR&rft.date=2011&rft.volume=6&rft.issue=1&rft.pages=90&rft_id=info:doi\/10.1186%2F1748-5908-6-90&rft_id=info:pmc\/PMC3170236&rft_id=info:pmid\/21824384&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3170236&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MartinezMobile14-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MartinezMobile14_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-MartinezMobile14_10-1\" rel=\"external_link\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-MartinezMobile14_10-2\" rel=\"external_link\">10.2<\/a><\/sup> <sup><a href=\"#cite_ref-MartinezMobile14_10-3\" rel=\"external_link\">10.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mart\u00ednez-P\u00e9rez, B.; de la Torre-D\u00edez, I.; L\u00f3pez-Coronado, M.; Sainz-de-Abajo, B.; Robles, M.; Garc\u00eda-G\u00f3mez, J.M. (2014). \"Mobile clinical decision support systems and applications: a literature and commercial review\". <i>Journal of Medical Systems<\/i> <b>38<\/b> (1): 4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10916-013-0004-y\" target=\"_blank\">10.1007\/s10916-013-0004-y<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24399281\" target=\"_blank\">24399281<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Mobile+clinical+decision+support+systems+and+applications%3A+a+literature+and+commercial+review&rft.jtitle=Journal+of+Medical+Systems&rft.aulast=Mart%C3%ADnez-P%C3%A9rez%2C+B.%3B+de+la+Torre-D%C3%ADez%2C+I.%3B+L%C3%B3pez-Coronado%2C+M.%3B+Sainz-de-Abajo%2C+B.%3B+Robles%2C+M.%3B+Garc%C3%ADa-G%C3%B3mez%2C+J.M.&rft.au=Mart%C3%ADnez-P%C3%A9rez%2C+B.%3B+de+la+Torre-D%C3%ADez%2C+I.%3B+L%C3%B3pez-Coronado%2C+M.%3B+Sainz-de-Abajo%2C+B.%3B+Robles%2C+M.%3B+Garc%C3%ADa-G%C3%B3mez%2C+J.M.&rft.date=2014&rft.volume=38&rft.issue=1&rft.pages=4&rft_id=info:doi\/10.1007%2Fs10916-013-0004-y&rft_id=info:pmid\/24399281&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MickanEvi13-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MickanEvi13_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-MickanEvi13_11-1\" rel=\"external_link\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mickan, S.; Tilson, K.J.; Atherton, H.; Roberts, W.N.; Heneghan, C. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3841346\" target=\"_blank\">\"Evidence of effectiveness of health care professionals using handheld computers: a scoping review of systematic reviews\"<\/a>. <i>Journal of Medical Internet Research<\/i> <b>15<\/b> (10): e212. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fjmir.2530\" target=\"_blank\">10.2196\/jmir.2530<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3841346\/\" target=\"_blank\">PMC3841346<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24165786\" target=\"_blank\">24165786<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3841346\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3841346<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evidence+of+effectiveness+of+health+care+professionals+using+handheld+computers%3A+a+scoping+review+of+systematic+reviews&rft.jtitle=Journal+of+Medical+Internet+Research&rft.aulast=Mickan%2C+S.%3B+Tilson%2C+K.J.%3B+Atherton%2C+H.%3B+Roberts%2C+W.N.%3B+Heneghan%2C+C.&rft.au=Mickan%2C+S.%3B+Tilson%2C+K.J.%3B+Atherton%2C+H.%3B+Roberts%2C+W.N.%3B+Heneghan%2C+C.&rft.date=2013&rft.volume=15&rft.issue=10&rft.pages=e212&rft_id=info:doi\/10.2196%2Fjmir.2530&rft_id=info:pmc\/PMC3841346&rft_id=info:pmid\/24165786&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3841346&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HigginsThe11-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HigginsThe11_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Higgins, J.P.; Altman, D.G.; Tzsche, P.C.; Ni, P.; Moher, D.; Oxman, A.D.; Savovic, J.; Schulz, K.F.; Weeks, L.; Sterne, J.A.C. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3196245\" target=\"_blank\">\"The Cochrane Collaboration's tool for assessing risk of bias in randomised trials\"<\/a>. <i>BMJ<\/i> <b>343<\/b>: d5928. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.d5928\" target=\"_blank\">10.1136\/bmj.d5928<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3196245\/\" target=\"_blank\">PMC3196245<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22008217\" target=\"_blank\">22008217<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3196245\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3196245<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Cochrane+Collaboration%27s+tool+for+assessing+risk+of+bias+in+randomised+trials&rft.jtitle=BMJ&rft.aulast=Higgins%2C+J.P.%3B+Altman%2C+D.G.%3B+Tzsche%2C+P.C.%3B+Ni%2C+P.%3B+Moher%2C+D.%3B+Oxman%2C+A.D.%3B+Savovic%2C+J.%3B+Schulz%2C+K.F.%3B+Weeks%2C+L.%3B+Sterne%2C+J.A.C.&rft.au=Higgins%2C+J.P.%3B+Altman%2C+D.G.%3B+Tzsche%2C+P.C.%3B+Ni%2C+P.%3B+Moher%2C+D.%3B+Oxman%2C+A.D.%3B+Savovic%2C+J.%3B+Schulz%2C+K.F.%3B+Weeks%2C+L.%3B+Sterne%2C+J.A.C.&rft.date=2011&rft.volume=343&rft.pages=d5928&rft_id=info:doi\/10.1136%2Fbmj.d5928&rft_id=info:pmc\/PMC3196245&rft_id=info:pmid\/22008217&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3196245&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BernerImprov06-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BernerImprov06_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-BernerImprov06_13-1\" rel=\"external_link\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-BernerImprov06_13-2\" rel=\"external_link\">13.2<\/a><\/sup> <sup><a href=\"#cite_ref-BernerImprov06_13-3\" rel=\"external_link\">13.3<\/a><\/sup> <sup><a href=\"#cite_ref-BernerImprov06_13-4\" rel=\"external_link\">13.4<\/a><\/sup> <sup><a href=\"#cite_ref-BernerImprov06_13-5\" rel=\"external_link\">13.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berner, E.S.; Houston, T.K.; Ray, M.N.; Allison, J.J.; Heudebert, G.R.; Chatham, W.W.; Kennedy, J.I.; Glandon, G.L.; Norton, P.A.; Crawford, M.A.; Maisiak, R.S. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447547\" target=\"_blank\">\"Improving ambulatory prescribing safety with a handheld decision support system: a randomized controlled trial\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>13<\/b> (2): 171-179. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M1961\" target=\"_blank\">10.1197\/jamia.M1961<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1447547\/\" target=\"_blank\">PMC1447547<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16357350\" target=\"_blank\">16357350<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447547\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447547<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improving+ambulatory+prescribing+safety+with+a+handheld+decision+support+system%3A+a+randomized+controlled+trial&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Berner%2C+E.S.%3B+Houston%2C+T.K.%3B+Ray%2C+M.N.%3B+Allison%2C+J.J.%3B+Heudebert%2C+G.R.%3B+Chatham%2C+W.W.%3B+Kennedy%2C+J.I.%3B+Glandon%2C+G.L.%3B+Norton%2C+P.A.%3B+Crawford%2C+M.A.%3B+Maisiak%2C+R.S.&rft.au=Berner%2C+E.S.%3B+Houston%2C+T.K.%3B+Ray%2C+M.N.%3B+Allison%2C+J.J.%3B+Heudebert%2C+G.R.%3B+Chatham%2C+W.W.%3B+Kennedy%2C+J.I.%3B+Glandon%2C+G.L.%3B+Norton%2C+P.A.%3B+Crawford%2C+M.A.%3B+Maisiak%2C+R.S.&rft.date=2006&rft.volume=13&rft.issue=2&rft.pages=171-179&rft_id=info:doi\/10.1197%2Fjamia.M1961&rft_id=info:pmc\/PMC1447547&rft_id=info:pmid\/16357350&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1447547&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BochicchioPilot06-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BochicchioPilot06_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-BochicchioPilot06_14-1\" rel=\"external_link\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-BochicchioPilot06_14-2\" rel=\"external_link\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-BochicchioPilot06_14-3\" rel=\"external_link\">14.3<\/a><\/sup> <sup><a href=\"#cite_ref-BochicchioPilot06_14-4\" rel=\"external_link\">14.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bochicchio, G.V.; Smit, P.A.; Moore, R.; Bochicchio, K.; Auwaerter, P.; Johnson, S.B.; Scalea, T.; Bartlett, J.G. (2006). \"Pilot study of a web-based antibiotic decision management guide\". <i>Journal of the American College of Surgeons<\/i> <b>202<\/b> (3): 459-467. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jamcollsurg.2005.11.010\" target=\"_blank\">10.1016\/j.jamcollsurg.2005.11.010<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16500251\" target=\"_blank\">16500251<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pilot+study+of+a+web-based+antibiotic+decision+management+guide&rft.jtitle=Journal+of+the+American+College+of+Surgeons&rft.aulast=Bochicchio%2C+G.V.%3B+Smit%2C+P.A.%3B+Moore%2C+R.%3B+Bochicchio%2C+K.%3B+Auwaerter%2C+P.%3B+Johnson%2C+S.B.%3B+Scalea%2C+T.%3B+Bartlett%2C+J.G.&rft.au=Bochicchio%2C+G.V.%3B+Smit%2C+P.A.%3B+Moore%2C+R.%3B+Bochicchio%2C+K.%3B+Auwaerter%2C+P.%3B+Johnson%2C+S.B.%3B+Scalea%2C+T.%3B+Bartlett%2C+J.G.&rft.date=2006&rft.volume=202&rft.issue=3&rft.pages=459-467&rft_id=info:doi\/10.1016%2Fj.jamcollsurg.2005.11.010&rft_id=info:pmid\/16500251&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FarrellUse08-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FarrellUse08_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-FarrellUse08_15-1\" rel=\"external_link\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-FarrellUse08_15-2\" rel=\"external_link\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-FarrellUse08_15-3\" rel=\"external_link\">15.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Farrell, M.J.; Rose, L. (2008). \"Use of mobile handheld computers in clinical nursing education\". <i>Journal of Nursing Education<\/i> <b>47<\/b> (1): 13-19. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18232610\" target=\"_blank\">18232610<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+mobile+handheld+computers+in+clinical+nursing+education&rft.jtitle=Journal+of+Nursing+Education&rft.aulast=Farrell%2C+M.J.%3B+Rose%2C+L.&rft.au=Farrell%2C+M.J.%3B+Rose%2C+L.&rft.date=2008&rft.volume=47&rft.issue=1&rft.pages=13-19&rft_id=info:pmid\/18232610&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PriceCan05-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PriceCan05_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-PriceCan05_16-1\" rel=\"external_link\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-PriceCan05_16-2\" rel=\"external_link\">16.2<\/a><\/sup> <sup><a href=\"#cite_ref-PriceCan05_16-3\" rel=\"external_link\">16.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Price, M. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1479487\" target=\"_blank\">\"Can hand-held computers improve adherence to guidelines? A (Palm) Pilot study of family doctors in British Columbia\"<\/a>. <i>Canadian Family Physician<\/i> <b>51<\/b> (11): 1506-1507. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1479487\/\" target=\"_blank\">PMC1479487<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16926943\" target=\"_blank\">16926943<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1479487\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1479487<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Can+hand-held+computers+improve+adherence+to+guidelines%3F+A+%28Palm%29+Pilot+study+of+family+doctors+in+British+Columbia&rft.jtitle=Canadian+Family+Physician&rft.aulast=Price%2C+M.&rft.au=Price%2C+M.&rft.date=2005&rft.volume=51&rft.issue=11&rft.pages=1506-1507&rft_id=info:pmc\/PMC1479487&rft_id=info:pmid\/16926943&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1479487&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoyAComp09-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RoyAComp09_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-RoyAComp09_17-1\" rel=\"external_link\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-RoyAComp09_17-2\" rel=\"external_link\">17.2<\/a><\/sup> <sup><a href=\"#cite_ref-RoyAComp09_17-3\" rel=\"external_link\">17.3<\/a><\/sup> <sup><a href=\"#cite_ref-RoyAComp09_17-4\" rel=\"external_link\">17.4<\/a><\/sup> <sup><a href=\"#cite_ref-RoyAComp09_17-5\" rel=\"external_link\">17.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Roy, P-M.; Durieux, P.; Gillaizeau, F.; Legall, C.; Armand-Perroux, A.; Martino, L.; Hachelaf, M.; Dubart, A-E.; Schmidt, J.; Cristiano, M.; Jean-Marie, C.; Arnaud, P.; Guy, M. (2009). \"A computerized handheld decision-support system to improve pulmonary embolism diagnosis: a randomized trial\". <i>Annals of Internal Medicine<\/i> <b>151<\/b> (10): 677-686. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7326%2F0003-4819-151-10-200911170-00003\" target=\"_blank\">10.7326\/0003-4819-151-10-200911170-00003<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19920268\" target=\"_blank\">19920268<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+computerized+handheld+decision-support+system+to+improve+pulmonary+embolism+diagnosis%3A+a+randomized+trial&rft.jtitle=Annals+of+Internal+Medicine&rft.aulast=Roy%2C+P-M.%3B+Durieux%2C+P.%3B+Gillaizeau%2C+F.%3B+Legall%2C+C.%3B+Armand-Perroux%2C+A.%3B+Martino%2C+L.%3B+Hachelaf%2C+M.%3B+Dubart%2C+A-E.%3B+Schmidt%2C+J.%3B+Cristiano%2C+M.%3B+Jean-Marie%2C+C.%3B+Arnaud%2C+P.%3B+Guy%2C+M.&rft.au=Roy%2C+P-M.%3B+Durieux%2C+P.%3B+Gillaizeau%2C+F.%3B+Legall%2C+C.%3B+Armand-Perroux%2C+A.%3B+Martino%2C+L.%3B+Hachelaf%2C+M.%3B+Dubart%2C+A-E.%3B+Schmidt%2C+J.%3B+Cristiano%2C+M.%3B+Jean-Marie%2C+C.%3B+Arnaud%2C+P.%3B+Guy%2C+M.&rft.date=2009&rft.volume=151&rft.issue=10&rft.pages=677-686&rft_id=info:doi\/10.7326%2F0003-4819-151-10-200911170-00003&rft_id=info:pmid\/19920268&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GreiverAng05-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GreiverAng05_18-0\" rel=\"external_link\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-GreiverAng05_18-1\" rel=\"external_link\">18.1<\/a><\/sup> <sup><a href=\"#cite_ref-GreiverAng05_18-2\" rel=\"external_link\">18.2<\/a><\/sup> <sup><a href=\"#cite_ref-GreiverAng05_18-3\" rel=\"external_link\">18.3<\/a><\/sup> <sup><a href=\"#cite_ref-GreiverAng05_18-4\" rel=\"external_link\">18.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Greiver, M.; Drummond, N.; White, D.; Weshler, J.; Moineddin, R.; Network NTPCR (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967\" target=\"_blank\">\"Angina on the Palm: randomized controlled pilot trial of Palm PDA software for referrals for cardiac testing\"<\/a>. <i>Canadian Family Physician<\/i> <b>51<\/b> (3): 382-383. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1472967\/\" target=\"_blank\">PMC1472967<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16926933\" target=\"_blank\">16926933<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Angina+on+the+Palm%3A+randomized+controlled+pilot+trial+of+Palm+PDA+software+for+referrals+for+cardiac+testing&rft.jtitle=Canadian+Family+Physician&rft.aulast=Greiver%2C+M.%3B+Drummond%2C+N.%3B+White%2C+D.%3B+Weshler%2C+J.%3B+Moineddin%2C+R.%3B+Network+NTPCR&rft.au=Greiver%2C+M.%3B+Drummond%2C+N.%3B+White%2C+D.%3B+Weshler%2C+J.%3B+Moineddin%2C+R.%3B+Network+NTPCR&rft.date=2005&rft.volume=51&rft.issue=3&rft.pages=382-383&rft_id=info:pmc\/PMC1472967&rft_id=info:pmid\/16926933&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1472967&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeeThe09-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LeeThe09_19-0\" rel=\"external_link\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-LeeThe09_19-1\" rel=\"external_link\">19.1<\/a><\/sup> <sup><a href=\"#cite_ref-LeeThe09_19-2\" rel=\"external_link\">19.2<\/a><\/sup> <sup><a href=\"#cite_ref-LeeThe09_19-3\" rel=\"external_link\">19.3<\/a><\/sup> <sup><a href=\"#cite_ref-LeeThe09_19-4\" rel=\"external_link\">19.4<\/a><\/sup> <sup><a href=\"#cite_ref-LeeThe09_19-5\" rel=\"external_link\">19.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lee, N.J.; Chen, E.S.; Currie, L.M.; Donovan, M.; Hall, E.K.; Jia, H.; John, R.M.; Bakken, S. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967\" target=\"_blank\">\"The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters\"<\/a>. <i>Advances in Nursing Science<\/i> <b>32<\/b> (3): 211-221. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FANS.0b013e3181b0d6bf\" target=\"_blank\">10.1097\/ANS.0b013e3181b0d6bf<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1472967\/\" target=\"_blank\">PMC1472967<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19707090\" target=\"_blank\">19707090<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1472967<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+effect+of+a+mobile+clinical+decision+support+system+on+the+diagnosis+of+obesity+and+overweight+in+acute+and+primary+care+encounters&rft.jtitle=Advances+in+Nursing+Science&rft.aulast=Lee%2C+N.J.%3B+Chen%2C+E.S.%3B+Currie%2C+L.M.%3B+Donovan%2C+M.%3B+Hall%2C+E.K.%3B+Jia%2C+H.%3B+John%2C+R.M.%3B+Bakken%2C+S.&rft.au=Lee%2C+N.J.%3B+Chen%2C+E.S.%3B+Currie%2C+L.M.%3B+Donovan%2C+M.%3B+Hall%2C+E.K.%3B+Jia%2C+H.%3B+John%2C+R.M.%3B+Bakken%2C+S.&rft.date=2009&rft.volume=32&rft.issue=3&rft.pages=211-221&rft_id=info:doi\/10.1097%2FANS.0b013e3181b0d6bf&rft_id=info:pmc\/PMC1472967&rft_id=info:pmid\/19707090&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1472967&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZurovacThe11-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ZurovacThe11_20-0\" rel=\"external_link\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-ZurovacThe11_20-1\" rel=\"external_link\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zurovac, D.; Sudoi, R.K.; Akhwale, W.S.; Ndiritu, M.; Hamer, D.H.; Rowe, A.K.; Snow, R.W. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163847\" target=\"_blank\">\"The effect of mobile phone text-message reminders on Kenyan health workers\u2019 adherence to malaria treatment guidelines: a cluster randomised trial\"<\/a>. <i>The Lancet<\/i> <b>378<\/b> (9793): 795-803. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0140-6736%2811%2960783-6\" target=\"_blank\">10.1016\/S0140-6736(11)60783-6<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3163847\/\" target=\"_blank\">PMC3163847<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21820166\" target=\"_blank\">21820166<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163847\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163847<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+effect+of+mobile+phone+text-message+reminders+on+Kenyan+health+workers%E2%80%99+adherence+to+malaria+treatment+guidelines%3A+a+cluster+randomised+trial&rft.jtitle=The+Lancet&rft.aulast=Zurovac%2C+D.%3B+Sudoi%2C+R.K.%3B+Akhwale%2C+W.S.%3B+Ndiritu%2C+M.%3B+Hamer%2C+D.H.%3B+Rowe%2C+A.K.%3B+Snow%2C+R.W.&rft.au=Zurovac%2C+D.%3B+Sudoi%2C+R.K.%3B+Akhwale%2C+W.S.%3B+Ndiritu%2C+M.%3B+Hamer%2C+D.H.%3B+Rowe%2C+A.K.%3B+Snow%2C+R.W.&rft.date=2011&rft.volume=378&rft.issue=9793&rft.pages=795-803&rft_id=info:doi\/10.1016%2FS0140-6736%2811%2960783-6&rft_id=info:pmc\/PMC3163847&rft_id=info:pmid\/21820166&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3163847&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrightEffect12-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrightEffect12_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bright, T.J.; Wong, A.; Dhurjati, R.; Bristow, E.; Bastian, L.; Coeytaux, R.R.; Samsa, G.; Hasselblad, V.; Williams, J.W.; Musty, M.D.; Wing, L.; Kendrick, A.S.; Sanders, G.D.; Lobach, D. (2012). \"Effect of clinical decision-support systems: a systematic review\". <i>Annals of Internal Medicine<\/i> <b>157<\/b> (1): 29-43. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7326%2F0003-4819-157-1-201207030-00450\" target=\"_blank\">10.7326\/0003-4819-157-1-201207030-00450<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22751758\" target=\"_blank\">22751758<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Effect+of+clinical+decision-support+systems%3A+a+systematic+review&rft.jtitle=Annals+of+Internal+Medicine&rft.aulast=Bright%2C+T.J.%3B+Wong%2C+A.%3B+Dhurjati%2C+R.%3B+Bristow%2C+E.%3B+Bastian%2C+L.%3B+Coeytaux%2C+R.R.%3B+Samsa%2C+G.%3B+Hasselblad%2C+V.%3B+Williams%2C+J.W.%3B+Musty%2C+M.D.%3B+Wing%2C+L.%3B+Kendrick%2C+A.S.%3B+Sanders%2C+G.D.%3B+Lobach%2C+D.&rft.au=Bright%2C+T.J.%3B+Wong%2C+A.%3B+Dhurjati%2C+R.%3B+Bristow%2C+E.%3B+Bastian%2C+L.%3B+Coeytaux%2C+R.R.%3B+Samsa%2C+G.%3B+Hasselblad%2C+V.%3B+Williams%2C+J.W.%3B+Musty%2C+M.D.%3B+Wing%2C+L.%3B+Kendrick%2C+A.S.%3B+Sanders%2C+G.D.%3B+Lobach%2C+D.&rft.date=2012&rft.volume=157&rft.issue=1&rft.pages=29-43&rft_id=info:doi\/10.7326%2F0003-4819-157-1-201207030-00450&rft_id=info:pmid\/22751758&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DiCensoAcc09-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DiCensoAcc09_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DiCenso, A.; Bayley, L.; Haynes, R.B. (2009). \"Accessing pre-appraised evidence: fine-tuning the 5S model into a 6S model\". <i>Evidence Based Nursing<\/i> <b>12<\/b> (4): 99-101. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Febn.12.4.99-b\" target=\"_blank\">10.1136\/ebn.12.4.99-b<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19779069\" target=\"_blank\">19779069<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Accessing+pre-appraised+evidence%3A+fine-tuning+the+5S+model+into+a+6S+model&rft.jtitle=Evidence+Based+Nursing&rft.aulast=DiCenso%2C+A.%3B+Bayley%2C+L.%3B+Haynes%2C+R.B.&rft.au=DiCenso%2C+A.%3B+Bayley%2C+L.%3B+Haynes%2C+R.B.&rft.date=2009&rft.volume=12&rft.issue=4&rft.pages=99-101&rft_id=info:doi\/10.1136%2Febn.12.4.99-b&rft_id=info:pmid\/19779069&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RubinUse06-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RubinUse06_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rubin, M.A.; Bateman, K.; Donnelly, S.; Stoddard, G.J.; Stevenson, K.; Gardner, R.M.; Samore, M.H. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1656956\" target=\"_blank\">\"Use of a personal digital assistant for managing antibiotic prescribing for outpatient respiratory tract infections in rural communities\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>13<\/b> (6): 627-634. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2029\" target=\"_blank\">10.1197\/jamia.M2029<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1656956\/\" target=\"_blank\">PMC1656956<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16929045\" target=\"_blank\">16929045<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1656956\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1656956<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+a+personal+digital+assistant+for+managing+antibiotic+prescribing+for+outpatient+respiratory+tract+infections+in+rural+communities&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Rubin%2C+M.A.%3B+Bateman%2C+K.%3B+Donnelly%2C+S.%3B+Stoddard%2C+G.J.%3B+Stevenson%2C+K.%3B+Gardner%2C+R.M.%3B+Samore%2C+M.H.&rft.au=Rubin%2C+M.A.%3B+Bateman%2C+K.%3B+Donnelly%2C+S.%3B+Stoddard%2C+G.J.%3B+Stevenson%2C+K.%3B+Gardner%2C+R.M.%3B+Samore%2C+M.H.&rft.date=2006&rft.volume=13&rft.issue=6&rft.pages=627-634&rft_id=info:doi\/10.1197%2Fjamia.M2029&rft_id=info:pmc\/PMC1656956&rft_id=info:pmid\/16929045&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1656956&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SintchenkoHand05-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SintchenkoHand05_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sintchenko, V.; Iredell, J.R.; Gilbert, G.L.; Coiera, E. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1174884\" target=\"_blank\">\"Handheld computer-based decision support reduces patient length of stay and antibiotic prescribing in critical care\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>12<\/b> (4): 398-402. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M1798\" target=\"_blank\">10.1197\/jamia.M1798<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1174884\/\" target=\"_blank\">PMC1174884<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15802478\" target=\"_blank\">15802478<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1174884\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1174884<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Handheld+computer-based+decision+support+reduces+patient+length+of+stay+and+antibiotic+prescribing+in+critical+care&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Sintchenko%2C+V.%3B+Iredell%2C+J.R.%3B+Gilbert%2C+G.L.%3B+Coiera%2C+E.&rft.au=Sintchenko%2C+V.%3B+Iredell%2C+J.R.%3B+Gilbert%2C+G.L.%3B+Coiera%2C+E.&rft.date=2005&rft.volume=12&rft.issue=4&rft.pages=398-402&rft_id=info:doi\/10.1197%2Fjamia.M1798&rft_id=info:pmc\/PMC1174884&rft_id=info:pmid\/15802478&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1174884&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PMEdsAReal13-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PMEdsAReal13_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">PLOS Medicine Editors (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3582561\" target=\"_blank\">\"A reality checkpoint for mobile health: three challenges to overcome\"<\/a>. <i>PloS Medicine<\/i> <b>10<\/b> (2): e1001395. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1001395\" target=\"_blank\">10.1371\/journal.pmed.1001395<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3582561\/\" target=\"_blank\">PMC3582561<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23468597\" target=\"_blank\">23468597<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3582561\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3582561<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+reality+checkpoint+for+mobile+health%3A+three+challenges+to+overcome&rft.jtitle=PloS+Medicine&rft.aulast=PLOS+Medicine+Editors&rft.au=PLOS+Medicine+Editors&rft.date=2013&rft.volume=10&rft.issue=2&rft.pages=e1001395&rft_id=info:doi\/10.1371%2Fjournal.pmed.1001395&rft_id=info:pmc\/PMC3582561&rft_id=info:pmid\/23468597&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3582561&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LabriqueHpe13-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LabriqueHpe13_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Labrique, A.; Vasudevan, L.; Chang, L.W.; Mehl, G. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849805\" target=\"_blank\">\"H_pe for mHealth: more \"y\" or \"o\" on the horizon?\"<\/a>. <i>International Journal of Medical Informatics<\/i> <b>82<\/b> (5): 467-469. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2012.11.016\" target=\"_blank\">10.1016\/j.ijmedinf.2012.11.016<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3849805\/\" target=\"_blank\">PMC3849805<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23279850\" target=\"_blank\">23279850<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849805\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849805<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=H_pe+for+mHealth%3A+more+%22y%22+or+%22o%22+on+the+horizon%3F&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Labrique%2C+A.%3B+Vasudevan%2C+L.%3B+Chang%2C+L.W.%3B+Mehl%2C+G.&rft.au=Labrique%2C+A.%3B+Vasudevan%2C+L.%3B+Chang%2C+L.W.%3B+Mehl%2C+G.&rft.date=2013&rft.volume=82&rft.issue=5&rft.pages=467-469&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2012.11.016&rft_id=info:pmc\/PMC3849805&rft_id=info:pmid\/23279850&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3849805&rfr_id=info:sid\/en.wikipedia.org:Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Pre-publication_history\">Pre-publication history<\/span><\/h2>\n<p>The pre-publication history for this paper can be accessed here: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.biomedcentral.com\/1472-6947\/14\/56\/prepub\" target=\"_blank\">http:\/\/www.biomedcentral.com\/1472-6947\/14\/56\/prepub<\/a>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. The list of studies in Table 1 was also ordered differently (by reference number rather than alphabetical order) in order to maintain the same reference numbers used in the paper. The table has been made sortable to partially compensate.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210236\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.711 seconds\nReal time usage: 0.738 seconds\nPreprocessor visited node count: 23353\/1000000\nPreprocessor generated node count: 33698\/1000000\nPost\u2010expand include size: 230689\/2097152 bytes\nTemplate argument size: 80436\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 697.187 1 - -total\n 82.77% 577.064 1 - Template:Reflist\n 76.27% 531.773 26 - Template:Cite_journal\n 73.07% 509.429 26 - Template:Citation\/core\n 12.11% 84.429 65 - Template:Citation\/identifier\n 8.84% 61.598 1 - Template:Infobox_journal_article\n 8.46% 58.969 1 - Template:Infobox\n 5.02% 34.993 80 - Template:Infobox\/row\n 4.38% 30.515 147 - Template:Hide_in_print\n 4.31% 30.080 26 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7643-0!*!0!!en!5!* and timestamp 20181213210235 and revision id 23427\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review\">https:\/\/www.limswiki.org\/index.php\/Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","26a517162c5d1f67f3c6bbb8c4cd1a80_images":["https:\/\/www.limswiki.org\/images\/2\/2e\/Fig1_Mickan_BMCMedInfoDecMak2014_14.jpg","https:\/\/www.limswiki.org\/images\/7\/71\/Fig2_Mickan_BMCMedInfoDecMak2014_14.jpg","https:\/\/www.limswiki.org\/images\/b\/bf\/Fig3_Mickan_BMCMedInfoDecMak2014_14.jpg"],"26a517162c5d1f67f3c6bbb8c4cd1a80_timestamp":1544734955,"4d5313302570d2f6c92b0f91883ad392_type":"article","4d5313302570d2f6c92b0f91883ad392_title":"Unravelling the tangled taxonomies of health informatics (Barrett et al. 2014)","4d5313302570d2f6c92b0f91883ad392_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Unravelling_the_tangled_taxonomies_of_health_informatics","4d5313302570d2f6c92b0f91883ad392_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Unravelling the tangled taxonomies of health informatics\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUnravelling the tangled taxonomies of health informaticsJournal\n \nInformatics in Primary CareAuthor(s)\n \nBarrett, David; Liaw, S. T.; de Lusignan, SimonAuthor affiliation(s)\n \nDepartment of Nursing, University of Hull; UNSW Medicine; Department of Health Care Management & Policy, University of SurreyPrimary contact\n \nEmail: d.i.barrett@hull.ac.ukYear published\n \n2014Volume and issue\n \n21 (3)Page(s)\n \n152\u2013155DOI\n \n10.14236\/jhi.v21i3.78ISSN\n \n2058-4563Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/hijournal.bcs.org\/index.php\/jhi\/article\/view\/78\/109Download\n \nhttp:\/\/hijournal.bcs.org\/index.php\/jhi\/article\/download\/78\/110 (PDF)\n\nContents\n\n1 Abstract \n2 The problem of the tangled taxonomies \n3 Definitions of health informatics \n\n3.1 Existing taxonomies \n\n\n4 Dimensions of health informatics \n\n4.1 Multidisciplinary \n4.2 Interdisciplinary \n4.3 Patient focus \n4.4 Level of expertise and sophistication \n4.5 Technology application \n4.6 Data granularity \n4.7 Recognition, academic and learned societies \n\n\n5 Summary - Untangling the taxonomies \n6 References \n7 Notes \n\n\n\nAbstract \nEven though informatics is a term used commonly in healthcare, it can be a confusing and disengaging one. Many definitions exist in the literature, and attempts have been made to develop a clear taxonomy. Despite this, informatics is still a term that lacks clarity in both its scope and the classification of sub-terms that it encompasses.\nThis paper reviews the importance of an agreed taxonomy and explores the challenges of establishing exactly what is meant by health informatics (HI). It reviews what a taxonomy should do, summarises previous attempts at categorising and organising HI and suggests the elements to consider when seeking to develop a system of classification.\nThe paper does not provide all the answers, but it does clarify the questions. By plotting a path towards a taxonomy of HI, it will be possible to enhance understanding and optimise the benefits of embracing technology in clinical practice.\nKeywords: clinical informatics, health informatics, taxonomy\n\nThe problem of the tangled taxonomies \nInformatics: a word that conjures up a host of definitions, applications and systems. Within healthcare, \u2018informatics\u2019 is used as a descriptor in a way that can be confusing and in some cases disengaging.[1] This confusion stems partly from the meaning of the word itself (Box 1), and partly from the plethora of sub-terms, sub-definitions and applications that can be connected to it. The focus of many of these sub-terms is on the technologies used in the delivery of care, providing a conceptual overlap between health information management and health (clinical) informatics. So, what is informatics?\nWhere do different concepts fit and interrelate? And does it really matter if we do not know?\n\n\n\n\n\n\n\n Box 1. Zuboff's definition of informatics.[1]\n\n\nShoshana Zuboff is accredited with having coined the term \u2018to informate\u2019 in the book \u2018In the Age of Smart Machine\u2019. Informating was the process of turning activities, events and objects into information. Not only do machine processes replace human ones, but also as a byproduct they produce new information streams.\n\n\n\nTerms such as digital health, eHealth, mHealth and technology-enabled care are used interchangeably and without any clear boundaries or criteria. It can be argued that this is unimportant, and that because specific applications (such as electronic patient records, electronic prescribing and clinical decision support software) can be described with some clarity, the need for clear categorisation is redundant. We would refute this. In any area of healthcare, a clear taxonomy \u2014 essentially, a system of classification \u2014 is necessary to underpin the commissioning and provision of services, and for documentation of care, workforce development and evidence-based generation. Without clarity, we struggle to describe to others what health informatics (HI) means for them and what the benefits are to patients, practitioners and organisations.\n\nDefinitions of health informatics \nIt is hard to develop a taxonomy without first defining the area you are looking to classify. Fortunately, overarching definitions of HI vary little across organisations and countries. Most are centred on the principle that HI relates to information and communication technologies applied to healthcare to achieve desired outcomes.\nFor example, the UK Department of Health definition of HI is\n\nThe knowledge, skills and tools that enable information to be collected, managed, used and shared to support the delivery of healthcare and to promote health and wellbeing.[2]\nThe Australian Health Informatics Education Council has a much more scientific, discipline-based definition, describing HI as\n\n...the application of information science and computer science to healthcare[3]\nA recent, comprehensive, yet succinct definition of clinical informatics encompasses much of the scope of HI:\n\nClinical informatics is not simply \u201ccomputers in medicine\u201d but rather is a body of knowledge, methods and theories that focus on the effective use of information and knowledge to improve the quality, safety and costeffectiveness of patient care as well as the health of both individuals and populations.[4]\nThese definitions align on the principle and purpose of informatics, but \u2013 as is the case with most definitions \u2013 do not provide any detail or clarity on the boundaries or component elements of HI.\nA discussion of the scope of HI is therefore needed, possibly even debating whether specific applications fall under the informatics umbrella. For example, do any or all of the following applications form part of HI?\n\n ePrescribing\n Remote blood pressure monitoring\n The provision of peer support via social media\nExisting taxonomies \nAttempts have previously been made to create a taxonomy for HI and associated areas:\n\n Dixon, McGowan and colleagues progressively developed first a glossary of terms aimed at novices to health information[5] and then a taxonomy for health information technology[6], finally looking to enhance this by adapting their taxonomy according to users\u2019 preferred search terms.[7] However, their approach was based on the scope of library classifications such as Medical Subject Headings. Others have seen the development of similar vocabularies as a key piece of the infrastructure to enable the definition of HI as a discipline.[8]\n Boonstra and Broekhuis proposed a taxonomy focussed on the barriers to the adoption of HI applications (specifically, computerised medical records). They identified eight key elements: (1) financial, (2) technical, (3) time, (4) psychological, (5) social, (6) legal, (7) organisational and (8) change process limitations.[9] Such a taxonomy might be applied more widely to HI and beyond.\n Taxonomies have also been described for the HI platforms, HealthGrids, which may enable linkage of multiple informatics systems.[10] These highlight how systems used in health lag behind those routinely used in business.\n Stagger and Thompson suggested that there might be (1) technology-, (2) role- and (3) concept-orientated definitions of HI.11 In the Staggers and Thompson taxonomy, terms such as telehealth, eHealth and mHealth are simply technology-orientated definitions, focussed purely on the devices or media that serve as facilitators of care. Role-orientated definitions might relate to the need to use informatics within a specific clinical discipline \u2013 for example, primary care informatics[11] \u2013 or may be linked to an individual\u2019s role. For example, the term \u2018health informatician\u2019 may be used to describe someone with HI skills who may be specially trained or have relevant experience. (A recent JAMA paper described the establishment of clinical informatics as a subspecialty.[4]) Individuals may have a specific professional role (e.g. nursing or pathology informatics), or a generic, organisational role, such as chief clinical information officer.[12] Finally, concept-orientated definitions of informatics attempt to define what HI is, some deliberately opting for conceptually defining informatics as a science that should be research and evidence based.[13]\nDefining the broad scope of HI is only the first step on the road to a clear taxonomy. Assuming that HI encompasses more specific concepts such as eHealth, telemonitoring and telemedicine, it is necessary to explore how these (and others) are categorised and how they interrelate.[14]\nTo move this debate forward, we need to explore the relevant concepts that help define HI, providing some clarity and allowing its development as a science. At the crudest level, these concepts can be viewed as a checklist to be considered in any future attempts at developing a taxonomy of HI.\n\nDimensions of health informatics \nMultidisciplinary \nHI is a multidisciplinary science, which implies an intuitive relationship to the multidisciplinary health care team in promoting information-enhanced integrated care.[15] Herein lies another problem of tangled taxonomies as we see it today. Though both HI and health care are inherently multidisciplinary in approach, both can involve very different groups of professionals with their own skill sets, approaches to practice and views on terminology. This exacerbates the complexity of classification and implementation.\n\nInterdisciplinary \nAlong with technical and conceptual definitions and associated issues, there are inter-professional and inter-disciplinary questions to be addressed in any taxonomy. In an ideal world, HI could well be the unifying mechanism for the health professions, and conceptual research in this area could lead to more coordinated and accessible care. To date, interdisciplinary initiatives extend little beyond educational establishments.[16][17]\n\nPatient focus \nBeing patient centred has long been part of health care. Informatics has the potential to empower patients to manage their health, with or without the input of the clinical professions.[14][18] However, a taxonomy needs to acknowledge and clarify the role of different elements of informatics in terms of the role of \u2014 and impact on \u2014 patients.\n\nLevel of expertise and sophistication \nWithin HI, we must also be able to describe the level of expertise and sophistication of people working within the discipline. To date, most attempts to do this have been at the regional or national level.[4][19]\n\nTechnology application \nHI is dependent on the implementation and adoption of a growing range of technological solutions. From data input devices (such as digital pens) to user interfaces on a multitude of platforms, HI comes in many shapes and sizes. Device-orientated terms (such as mHealth) exist in the literature and may figure within any taxonomy.\n\nData granularity \nSome taxonomies of HI have looked at the granularity of the data that are processed. Some of the motivation for this was to avoid the separation of HI from bioinformatics.[20] Regardless, the scope of HI can be described using a taxonomy related to the degree of granularity as the primary subject of interest (Figure 1).\n\nRecognition, academic and learned societies \nCourses, appointments and societies (national or specialist) recognised by international groups and journals all provide markers of what defines HI and its subspecialties. Regulation may be required to ensure that its processes are safe for patients[21], and existing mechanisms to organise, verify, accredit and recognise interventions may need acknowledging in any taxonomy.\n\n\n\nFigure 1. A taxonomy of HI based on the granularity of the primary focus of the HI subspecialty\nSummary - Untangling the taxonomies \nHI is evolving as a multidisciplinary science and should be defined as such. Conceptual research and development is required to optimize and guide taxonomical evolution over the coming years.\nThrough journals such as Informatics in Primary Care, some consensus needs to be reached regarding the scope, definitions and categories of applications. Clarity will aid clinicians, researchers, commissioners, managers and educators to understand HI, build the evidence base, implement services and share knowledge. The development of an agreed taxonomy is not necessarily an end in itself, but is a means to an end: greater clarity provides greater understanding and underpins future research that informs clinicians on how best to use technology to enhance the delivery of health care.\n\nReferences \n\n\u2191 1.0 1.1 Hersh, W. (2009). \"A stimulus to define informatics and health information technology\". BMC Medical Informatics and Decision Making 9: 24. doi:10.1186\/1472-6947-9-24. PMID 19445665.   \n\n\u2191 Department of Health (October 2002). \"Making Information Count: A Human Resources Strategy for Health Informatics Professionals\". The Health and Social Care Information Centre. http:\/\/systems.hscic.gov.uk\/icd\/whatis .   \n\n\u2191 Australian Health Informatics Education Council (November 2011). \"Health Informatics: Scope, Careers and Competencies - Version 1.9\" (PDF). http:\/\/www.ahiec.org.au\/docs\/AHIEC_HI_Scope_Careers_and_Competencies_V1-9.pdf .   \n\n\u2191 4.0 4.1 4.2 Detmer, D.E.; Shortliffe, E. (2014). \"Clinical informatics: prospects for a new medical subspecialty\". JAMA 311 (20): 2067\u20138. doi:10.1001\/jama.2014.3514.   \n\n\u2191 Cravens, G.D.; Dixon, B.E.; Zafar, A.; and McGowan, J.J. (2008). \"A health information technology glossary for novices\". AMIA Annual Symposium Proceedings: 917. PMID 18998981.   \n\n\u2191 Dixon, B.E.; Zafar, A.; McGowan, J.J. (2007). \"Development of a taxonomy for health information technology\". Studies in Health Technology and Informatics 129 (Pt 1): 616\u201320. PMID 17911790.   \n\n\u2191 Dixon, B.E.; McGowan, J.J. (2010). \"Enhancing a taxonomy for health information technology: an exploratory study of user input towards folksonomy\". Studies in Health Technology and Informatics 160 (Pt 2): 1055\u20139. PMID 20841845.   \n\n\u2191 Sperzel, W.D.; Broverman, C.A.; Kapusnik-Uner, J.E.; Schlesinger, J.M. (1998). \"The need for a concept-based medication vocabulary as an enabling infrastructure in health informatics\". Annual Symposium Proceedings: 865\u20139. PMID 9929342.   \n\n\u2191 Boonstra, A.; Broekhuis, M. (2010). \"Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions\". BMC Health Services Research 10: 231. doi:10.1186\/1472-6963-10-231.   \n\n\u2191 Naseer, A.; Stergioulas, L.K. (2010). \"HealthGrids in health informatics: a taxonomy\". In Khoumbati, K.; Dwivedi, Y.; Srivastava, A.; Lal, B. Handbook of Research on Advances in Health Informatics and Electronic Healthcare Applications: Global Adoption and Impact of Information Communication Technologies. Medical Information Science Reference. pp. 124\u201343. doi:10.4018\/978-1-60566-030-1.ch008.   \n\n\u2191 Staggers, N.; Thompson, C.R. (2002). \"The evolution of definitions for nursing informatics: a critical analysis and revised definition\". Journal of the American Medical Informatics Association 9: 255\u201361. doi:10.1197\/jamia.M0946.   \n\n\u2191 Whatling, J. (September 2011). \"Chief Clinical Information Officer campaign (CCIO) launch\". British Computer Society (BCS). http:\/\/www.bcs.org\/content\/conWebDoc\/41603 .   \n\n\u2191 de Lusignan, S. (2003). \"What is primary care informatics?\". Journal of the American Medical Informatics Association 10 (4): 304\u20139. doi:10.1197\/jamia.M1187. PMID 12668690.   \n\n\u2191 14.0 14.1 Wyatt, J.C.; Sullivan, F. (2005). \"eHealth and the future: promise or peril?\". British Medical Journal 331 (7529): 1391\u20133. doi:10.1136\/bmj.331.7529.1391. PMID 16339252.   \n\n\u2191 Geissbuhler, A.; Kimura, M.; Kulikowski, C.A.; Murray, P.J.; Ohno-Machado, L.; Park, H.A. et al. (2011). \"Confluence of disciplines in health informatics: an international perspective\". Methods of Information in Medicine 50 (6): 545\u201355. doi:10.3414\/ME11-06-0005. PMID 22146917.   \n\n\u2191 Demiris, G. (2007). \"Interdisciplinary innovations in biomedical and health informatics graduate education\". Methods of Information in Medicine 46 (1): 63\u20136. PMID 17224983.   \n\n\u2191 Kushniruk, A.; Lau, F.; Borycki, E.; Pratti, D. (2006). \"The School of Health Information Science at the University of Victoria: towards an intergrative model for health informatics education and research\". Yearbook of Medical Informatics: 159\u201365. PMID 17051310.   \n\n\u2191 Alamantariotou, K.; Zisi, D. (2010). \"Consumer health informatics and interactive visual learning tools for health\". International Journal of Electronic Healthcare 5 (4): 414\u201324. doi:10.1504\/IJEH.2010.036211.   \n\n\u2191 McCullagh, P.; McAllister, G.; Hanna, P.; Finlay, D.; Comac, P. (2011). \"Professional development of health informatics in Northern Ireland\". Studies in Health Technology and Informatics 169: 218\u201322. PMID 21893745.   \n\n\u2191 Maojo, V.; Kulikowski, C. (2006). \"Medical informatics and bioinformatics: integration or evolution through scientific crises?\". Methods of Information in Medicine 45 (5): 474\u201382. PMID 17019500.   \n\n\u2191 Rigby, M.; Forsstr\u00f6m, J.; Roberts, R.; Wyatt, J. (2001). \"Verifying quality and safety in health informatics services\". British Medical Journal 323 (7312): 552\u20136. doi:10.1136\/bmj.323.7312.552. PMID 11546703.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\">https:\/\/www.limswiki.org\/index.php\/Journal:Unravelling_the_tangled_taxonomies_of_health_informatics<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 21:03.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,468 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","4d5313302570d2f6c92b0f91883ad392_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Unravelling_the_tangled_taxonomies_of_health_informatics skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Unravelling the tangled taxonomies of health informatics<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Even though <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">informatics<\/a> is a term used commonly in healthcare, it can be a confusing and disengaging one. Many definitions exist in the literature, and attempts have been made to develop a clear taxonomy. Despite this, informatics is still a term that lacks clarity in both its scope and the classification of sub-terms that it encompasses.\n<\/p><p>This paper reviews the importance of an agreed taxonomy and explores the challenges of establishing exactly what is meant by <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health informatics<\/a> (HI). It reviews what a taxonomy should do, summarises previous attempts at categorising and organising HI and suggests the elements to consider when seeking to develop a system of classification.\n<\/p><p>The paper does not provide all the answers, but it does clarify the questions. By plotting a path towards a taxonomy of HI, it will be possible to enhance understanding and optimise the benefits of embracing technology in clinical practice.\n<\/p><p><b>Keywords:<\/b> clinical informatics, health informatics, taxonomy\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_problem_of_the_tangled_taxonomies\">The problem of the tangled taxonomies<\/span><\/h2>\n<p>Informatics: a word that conjures up a host of definitions, applications and systems. Within healthcare, \u2018informatics\u2019 is used as a descriptor in a way that can be confusing and in some cases disengaging.<sup id=\"rdp-ebb-cite_ref-HershStim09_1-0\" class=\"reference\"><a href=\"#cite_note-HershStim09-1\" rel=\"external_link\">[1]<\/a><\/sup> This confusion stems partly from the meaning of the word itself (Box 1), and partly from the plethora of sub-terms, sub-definitions and applications that can be connected to it. The focus of many of these sub-terms is on the technologies used in the delivery of care, providing a conceptual overlap between health information management and health (clinical) informatics. So, what is informatics?\n<\/p><p>Where do different concepts fit and interrelate? And does it really matter if we do not know?\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Box 1. Zuboff's definition of informatics.<sup id=\"rdp-ebb-cite_ref-HershStim09_1-1\" class=\"reference\"><a href=\"#cite_note-HershStim09-1\" rel=\"external_link\">[1]<\/a><\/sup>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Shoshana Zuboff is accredited with having coined the term \u2018to informate\u2019 in the book \u2018In the Age of Smart Machine\u2019. Informating was the process of turning activities, events and objects into information. Not only do machine processes replace human ones, but also as a byproduct they produce new information streams.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Terms such as digital health, eHealth, mHealth and technology-enabled care are used interchangeably and without any clear boundaries or criteria. It can be argued that this is unimportant, and that because specific applications (such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic patient records<\/a>, electronic prescribing and <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" target=\"_blank\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support software<\/a>) can be described with some clarity, the need for clear categorisation is redundant. We would refute this. In any area of healthcare, a clear taxonomy \u2014 essentially, a system of classification \u2014 is necessary to underpin the commissioning and provision of services, and for documentation of care, workforce development and evidence-based generation. Without clarity, we struggle to describe to others what health informatics (HI) means for them and what the benefits are to patients, practitioners and organisations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Definitions_of_health_informatics\">Definitions of health informatics<\/span><\/h2>\n<p>It is hard to develop a taxonomy without first defining the area you are looking to classify. Fortunately, overarching definitions of HI vary little across organisations and countries. Most are centred on the principle that HI relates to information and communication technologies applied to healthcare to achieve desired outcomes.\n<\/p><p>For example, the UK Department of Health definition of HI is\n<\/p>\n<blockquote><i>The knowledge, skills and tools that enable information to be collected, managed, used and shared to support the delivery of healthcare and to promote health and wellbeing.<\/i><sup id=\"rdp-ebb-cite_ref-DoHInfo02_2-0\" class=\"reference\"><a href=\"#cite_note-DoHInfo02-2\" rel=\"external_link\">[2]<\/a><\/sup><\/blockquote>\n<p>The Australian Health Informatics Education Council has a much more scientific, discipline-based definition, describing HI as\n<\/p>\n<blockquote><i>...the application of information science and computer science to healthcare<\/i><sup id=\"rdp-ebb-cite_ref-AHIEC11_3-0\" class=\"reference\"><a href=\"#cite_note-AHIEC11-3\" rel=\"external_link\">[3]<\/a><\/sup><\/blockquote>\n<p>A recent, comprehensive, yet succinct definition of clinical informatics encompasses much of the scope of HI:\n<\/p>\n<blockquote><i>Clinical informatics is not simply \u201ccomputers in medicine\u201d but rather is a body of knowledge, methods and theories that focus on the effective use of information and knowledge to improve the quality, safety and costeffectiveness of patient care as well as the health of both individuals and populations.<\/i><sup id=\"rdp-ebb-cite_ref-DetmerCI14_4-0\" class=\"reference\"><a href=\"#cite_note-DetmerCI14-4\" rel=\"external_link\">[4]<\/a><\/sup><\/blockquote>\n<p>These definitions align on the principle and purpose of informatics, but \u2013 as is the case with most definitions \u2013 do not provide any detail or clarity on the boundaries or component elements of HI.\n<\/p><p>A discussion of the scope of HI is therefore needed, possibly even debating whether specific applications fall under the informatics umbrella. For example, do any or all of the following applications form part of HI?\n<\/p>\n<ul><li> ePrescribing<\/li>\n<li> Remote blood pressure monitoring<\/li>\n<li> The provision of peer support via social media<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Existing_taxonomies\">Existing taxonomies<\/span><\/h3>\n<p>Attempts have previously been made to create a taxonomy for HI and associated areas:\n<\/p>\n<ul><li> Dixon, McGowan and colleagues progressively developed first a glossary of terms aimed at novices to health information<sup id=\"rdp-ebb-cite_ref-CravensHI08_5-0\" class=\"reference\"><a href=\"#cite_note-CravensHI08-5\" rel=\"external_link\">[5]<\/a><\/sup> and then a taxonomy for health information technology<sup id=\"rdp-ebb-cite_ref-DixonTax07_6-0\" class=\"reference\"><a href=\"#cite_note-DixonTax07-6\" rel=\"external_link\">[6]<\/a><\/sup>, finally looking to enhance this by adapting their taxonomy according to users\u2019 preferred search terms.<sup id=\"rdp-ebb-cite_ref-DixonTax10_7-0\" class=\"reference\"><a href=\"#cite_note-DixonTax10-7\" rel=\"external_link\">[7]<\/a><\/sup> However, their approach was based on the scope of library classifications such as Medical Subject Headings. Others have seen the development of similar vocabularies as a key piece of the infrastructure to enable the definition of HI as a discipline.<sup id=\"rdp-ebb-cite_ref-SperzelVocab98_8-0\" class=\"reference\"><a href=\"#cite_note-SperzelVocab98-8\" rel=\"external_link\">[8]<\/a><\/sup><\/li><\/ul>\n<ul><li> Boonstra and Broekhuis proposed a taxonomy focussed on the barriers to the adoption of HI applications (specifically, <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">computerised medical records<\/a>). They identified eight key elements: (1) financial, (2) technical, (3) time, (4) psychological, (5) social, (6) legal, (7) organisational and (8) change process limitations.<sup id=\"rdp-ebb-cite_ref-BoonstraBar10_9-0\" class=\"reference\"><a href=\"#cite_note-BoonstraBar10-9\" rel=\"external_link\">[9]<\/a><\/sup> Such a taxonomy might be applied more widely to HI and beyond.<\/li><\/ul>\n<ul><li> Taxonomies have also been described for the HI platforms, HealthGrids, which may enable linkage of multiple informatics systems.<sup id=\"rdp-ebb-cite_ref-NaseerHand10_10-0\" class=\"reference\"><a href=\"#cite_note-NaseerHand10-10\" rel=\"external_link\">[10]<\/a><\/sup> These highlight how systems used in health lag behind those routinely used in business.<\/li><\/ul>\n<ul><li> Stagger and Thompson suggested that there might be (1) technology-, (2) role- and (3) concept-orientated definitions of HI.11 In the Staggers and Thompson taxonomy, terms such as telehealth, eHealth and mHealth are simply technology-orientated definitions, focussed purely on the devices or media that serve as facilitators of care. Role-orientated definitions might relate to the need to use informatics within a specific clinical discipline \u2013 for example, primary care informatics<sup id=\"rdp-ebb-cite_ref-StaggersEvo02_11-0\" class=\"reference\"><a href=\"#cite_note-StaggersEvo02-11\" rel=\"external_link\">[11]<\/a><\/sup> \u2013 or may be linked to an individual\u2019s role. For example, the term \u2018health informatician\u2019 may be used to describe someone with HI skills who may be specially trained or have relevant experience. (A recent JAMA paper described the establishment of clinical informatics as a subspecialty.<sup id=\"rdp-ebb-cite_ref-DetmerCI14_4-1\" class=\"reference\"><a href=\"#cite_note-DetmerCI14-4\" rel=\"external_link\">[4]<\/a><\/sup>) Individuals may have a specific professional role (e.g. nursing or pathology informatics), or a generic, organisational role, such as chief clinical information officer.<sup id=\"rdp-ebb-cite_ref-WhatlingChief11_12-0\" class=\"reference\"><a href=\"#cite_note-WhatlingChief11-12\" rel=\"external_link\">[12]<\/a><\/sup> Finally, concept-orientated definitions of informatics attempt to define what HI is, some deliberately opting for conceptually defining informatics as a science that should be research and evidence based.<sup id=\"rdp-ebb-cite_ref-LusignanWhat03_13-0\" class=\"reference\"><a href=\"#cite_note-LusignanWhat03-13\" rel=\"external_link\">[13]<\/a><\/sup><\/li><\/ul>\n<p>Defining the broad scope of HI is only the first step on the road to a clear taxonomy. Assuming that HI encompasses more specific concepts such as eHealth, telemonitoring and telemedicine, it is necessary to explore how these (and others) are categorised and how they interrelate.<sup id=\"rdp-ebb-cite_ref-WyattEhe05_14-0\" class=\"reference\"><a href=\"#cite_note-WyattEhe05-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p><p>To move this debate forward, we need to explore the relevant concepts that help define HI, providing some clarity and allowing its development as a science. At the crudest level, these concepts can be viewed as a checklist to be considered in any future attempts at developing a taxonomy of HI.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Dimensions_of_health_informatics\">Dimensions of health informatics<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Multidisciplinary\">Multidisciplinary<\/span><\/h3>\n<p>HI is a multidisciplinary science, which implies an intuitive relationship to the multidisciplinary health care team in promoting information-enhanced integrated care.<sup id=\"rdp-ebb-cite_ref-GeissbuhlerCon11_15-0\" class=\"reference\"><a href=\"#cite_note-GeissbuhlerCon11-15\" rel=\"external_link\">[15]<\/a><\/sup> Herein lies another problem of tangled taxonomies as we see it today. Though both HI and health care are inherently multidisciplinary in approach, both can involve very different groups of professionals with their own skill sets, approaches to practice and views on terminology. This exacerbates the complexity of classification and implementation.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Interdisciplinary\">Interdisciplinary<\/span><\/h3>\n<p>Along with technical and conceptual definitions and associated issues, there are inter-professional and inter-disciplinary questions to be addressed in any taxonomy. In an ideal world, HI could well be the unifying mechanism for the health professions, and conceptual research in this area could lead to more coordinated and accessible care. To date, interdisciplinary initiatives extend little beyond educational establishments.<sup id=\"rdp-ebb-cite_ref-DemirisInter07_16-0\" class=\"reference\"><a href=\"#cite_note-DemirisInter07-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KushnirukSch06_17-0\" class=\"reference\"><a href=\"#cite_note-KushnirukSch06-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Patient_focus\">Patient focus<\/span><\/h3>\n<p>Being patient centred has long been part of health care. Informatics has the potential to empower patients to manage their health, with or without the input of the clinical professions.<sup id=\"rdp-ebb-cite_ref-WyattEhe05_14-1\" class=\"reference\"><a href=\"#cite_note-WyattEhe05-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlamantariotouCon10_18-0\" class=\"reference\"><a href=\"#cite_note-AlamantariotouCon10-18\" rel=\"external_link\">[18]<\/a><\/sup> However, a taxonomy needs to acknowledge and clarify the role of different elements of informatics in terms of the role of \u2014 and impact on \u2014 patients.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Level_of_expertise_and_sophistication\">Level of expertise and sophistication<\/span><\/h3>\n<p>Within HI, we must also be able to describe the level of expertise and sophistication of people working within the discipline. To date, most attempts to do this have been at the regional or national level.<sup id=\"rdp-ebb-cite_ref-DetmerCI14_4-2\" class=\"reference\"><a href=\"#cite_note-DetmerCI14-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McCullaghPro11_19-0\" class=\"reference\"><a href=\"#cite_note-McCullaghPro11-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Technology_application\">Technology application<\/span><\/h3>\n<p>HI is dependent on the implementation and adoption of a growing range of technological solutions. From data input devices (such as digital pens) to user interfaces on a multitude of platforms, HI comes in many shapes and sizes. Device-orientated terms (such as mHealth) exist in the literature and may figure within any taxonomy.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_granularity\">Data granularity<\/span><\/h3>\n<p>Some taxonomies of HI have looked at the granularity of the data that are processed. Some of the motivation for this was to avoid the separation of HI from <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a>.<sup id=\"rdp-ebb-cite_ref-MaojoMI06_20-0\" class=\"reference\"><a href=\"#cite_note-MaojoMI06-20\" rel=\"external_link\">[20]<\/a><\/sup> Regardless, the scope of HI can be described using a taxonomy related to the degree of granularity as the primary subject of interest (Figure 1).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Recognition.2C_academic_and_learned_societies\">Recognition, academic and learned societies<\/span><\/h3>\n<p>Courses, appointments and societies (national or specialist) recognised by international groups and journals all provide markers of what defines HI and its subspecialties. Regulation may be required to ensure that its processes are safe for patients<sup id=\"rdp-ebb-cite_ref-RigbyVeri01_21-0\" class=\"reference\"><a href=\"#cite_note-RigbyVeri01-21\" rel=\"external_link\">[21]<\/a><\/sup>, and existing mechanisms to organise, verify, accredit and recognise interventions may need acknowledging in any taxonomy.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Barrett_InformaticsPC2014_21-3.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"46b5663903b898d67434332bdcf293cb\"><img alt=\"Fig1 Barrett InformaticsPC2014 21-3.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f7\/Fig1_Barrett_InformaticsPC2014_21-3.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> A taxonomy of HI based on the granularity of the primary focus of the HI subspecialty<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Summary_-_Untangling_the_taxonomies\">Summary - Untangling the taxonomies<\/span><\/h2>\n<p>HI is evolving as a multidisciplinary science and should be defined as such. Conceptual research and development is required to optimize and guide taxonomical evolution over the coming years.\n<\/p><p>Through journals such as <i>Informatics in Primary Care<\/i>, some consensus needs to be reached regarding the scope, definitions and categories of applications. Clarity will aid clinicians, researchers, commissioners, managers and educators to understand HI, build the evidence base, implement services and share knowledge. The development of an agreed taxonomy is not necessarily an end in itself, but is a means to an end: greater clarity provides greater understanding and underpins future research that informs clinicians on how best to use technology to enhance the delivery of health care.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<ol class=\"references\">\n<li id=\"cite_note-HershStim09-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HershStim09_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-HershStim09_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hersh, W. (2009). \"A stimulus to define informatics and health information technology\". <i>BMC Medical Informatics and Decision Making<\/i> <b>9<\/b>: 24. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-9-24\" target=\"_blank\">10.1186\/1472-6947-9-24<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19445665\" target=\"_blank\">19445665<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+stimulus+to+define+informatics+and+health+information+technology&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Hersh%2C+W.&rft.au=Hersh%2C+W.&rft.date=2009&rft.volume=9&rft.pages=24&rft_id=info:doi\/10.1186%2F1472-6947-9-24&rft_id=info:pmid\/19445665&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DoHInfo02-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DoHInfo02_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Department of Health (October 2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/systems.hscic.gov.uk\/icd\/whatis\" target=\"_blank\">\"Making Information Count: A Human Resources Strategy for Health Informatics Professionals\"<\/a>. <i>The Health and Social Care Information Centre<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/systems.hscic.gov.uk\/icd\/whatis\" target=\"_blank\">http:\/\/systems.hscic.gov.uk\/icd\/whatis<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Making+Information+Count%3A+A+Human+Resources+Strategy+for+Health+Informatics+Professionals&rft.atitle=The+Health+and+Social+Care+Information+Centre&rft.aulast=Department+of+Health&rft.au=Department+of+Health&rft.date=October+2002&rft_id=http%3A%2F%2Fsystems.hscic.gov.uk%2Ficd%2Fwhatis&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AHIEC11-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AHIEC11_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Australian Health Informatics Education Council (November 2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ahiec.org.au\/docs\/AHIEC_HI_Scope_Careers_and_Competencies_V1-9.pdf\" target=\"_blank\">\"Health Informatics: Scope, Careers and Competencies - Version 1.9\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ahiec.org.au\/docs\/AHIEC_HI_Scope_Careers_and_Competencies_V1-9.pdf\" target=\"_blank\">http:\/\/www.ahiec.org.au\/docs\/AHIEC_HI_Scope_Careers_and_Competencies_V1-9.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Health+Informatics%3A+Scope%2C+Careers+and+Competencies+-+Version+1.9&rft.atitle=&rft.aulast=Australian+Health+Informatics+Education+Council&rft.au=Australian+Health+Informatics+Education+Council&rft.date=November+2011&rft_id=http%3A%2F%2Fwww.ahiec.org.au%2Fdocs%2FAHIEC_HI_Scope_Careers_and_Competencies_V1-9.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DetmerCI14-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DetmerCI14_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerCI14_4-1\" rel=\"external_link\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerCI14_4-2\" rel=\"external_link\">4.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Detmer, D.E.; Shortliffe, E. (2014). \"Clinical informatics: prospects for a new medical subspecialty\". <i>JAMA<\/i> <b>311<\/b> (20): 2067\u20138. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2014.3514\" target=\"_blank\">10.1001\/jama.2014.3514<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Clinical+informatics%3A+prospects+for+a+new+medical+subspecialty&rft.jtitle=JAMA&rft.aulast=Detmer%2C+D.E.%3B+Shortliffe%2C+E.&rft.au=Detmer%2C+D.E.%3B+Shortliffe%2C+E.&rft.date=2014&rft.volume=311&rft.issue=20&rft.pages=2067%E2%80%938&rft_id=info:doi\/10.1001%2Fjama.2014.3514&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CravensHI08-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CravensHI08_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cravens, G.D.; Dixon, B.E.; Zafar, A.; and McGowan, J.J. (2008). \"A health information technology glossary for novices\". <i>AMIA Annual Symposium Proceedings<\/i>: 917. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18998981\" target=\"_blank\">18998981<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+health+information+technology+glossary+for+novices&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Cravens%2C+G.D.%3B+Dixon%2C+B.E.%3B+Zafar%2C+A.%3B+and+McGowan%2C+J.J.&rft.au=Cravens%2C+G.D.%3B+Dixon%2C+B.E.%3B+Zafar%2C+A.%3B+and+McGowan%2C+J.J.&rft.date=2008&rft.pages=917&rft_id=info:pmid\/18998981&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DixonTax07-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DixonTax07_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dixon, B.E.; Zafar, A.; McGowan, J.J. (2007). \"Development of a taxonomy for health information technology\". <i>Studies in Health Technology and Informatics<\/i> <b>129<\/b> (Pt 1): 616\u201320. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17911790\" target=\"_blank\">17911790<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Development+of+a+taxonomy+for+health+information+technology&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Dixon%2C+B.E.%3B+Zafar%2C+A.%3B+McGowan%2C+J.J.&rft.au=Dixon%2C+B.E.%3B+Zafar%2C+A.%3B+McGowan%2C+J.J.&rft.date=2007&rft.volume=129&rft.issue=Pt+1&rft.pages=616%E2%80%9320&rft_id=info:pmid\/17911790&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DixonTax10-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DixonTax10_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dixon, B.E.; McGowan, J.J. (2010). \"Enhancing a taxonomy for health information technology: an exploratory study of user input towards folksonomy\". <i>Studies in Health Technology and Informatics<\/i> <b>160<\/b> (Pt 2): 1055\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20841845\" target=\"_blank\">20841845<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enhancing+a+taxonomy+for+health+information+technology%3A+an+exploratory+study+of+user+input+towards+folksonomy&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Dixon%2C+B.E.%3B+McGowan%2C+J.J.&rft.au=Dixon%2C+B.E.%3B+McGowan%2C+J.J.&rft.date=2010&rft.volume=160&rft.issue=Pt+2&rft.pages=1055%E2%80%939&rft_id=info:pmid\/20841845&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SperzelVocab98-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SperzelVocab98_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sperzel, W.D.; Broverman, C.A.; Kapusnik-Uner, J.E.; Schlesinger, J.M. (1998). \"The need for a concept-based medication vocabulary as an enabling infrastructure in health informatics\". <i>Annual Symposium Proceedings<\/i>: 865\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/9929342\" target=\"_blank\">9929342<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+need+for+a+concept-based+medication+vocabulary+as+an+enabling+infrastructure+in+health+informatics&rft.jtitle=Annual+Symposium+Proceedings&rft.aulast=Sperzel%2C+W.D.%3B+Broverman%2C+C.A.%3B+Kapusnik-Uner%2C+J.E.%3B+Schlesinger%2C+J.M.&rft.au=Sperzel%2C+W.D.%3B+Broverman%2C+C.A.%3B+Kapusnik-Uner%2C+J.E.%3B+Schlesinger%2C+J.M.&rft.date=1998&rft.pages=865%E2%80%939&rft_id=info:pmid\/9929342&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BoonstraBar10-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BoonstraBar10_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Boonstra, A.; Broekhuis, M. (2010). \"Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions\". <i>BMC Health Services Research<\/i> <b>10<\/b>: 231. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6963-10-231\" target=\"_blank\">10.1186\/1472-6963-10-231<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Barriers+to+the+acceptance+of+electronic+medical+records+by+physicians+from+systematic+review+to+taxonomy+and+interventions&rft.jtitle=BMC+Health+Services+Research&rft.aulast=Boonstra%2C+A.%3B+Broekhuis%2C+M.&rft.au=Boonstra%2C+A.%3B+Broekhuis%2C+M.&rft.date=2010&rft.volume=10&rft.pages=231&rft_id=info:doi\/10.1186%2F1472-6963-10-231&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NaseerHand10-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NaseerHand10_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Naseer, A.; Stergioulas, L.K. (2010). \"HealthGrids in health informatics: a taxonomy\". In Khoumbati, K.; Dwivedi, Y.; Srivastava, A.; Lal, B. <i>Handbook of Research on Advances in Health Informatics and Electronic Healthcare Applications: Global Adoption and Impact of Information Communication Technologies<\/i>. Medical Information Science Reference. pp. 124\u201343. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4018%2F978-1-60566-030-1.ch008\" target=\"_blank\">10.4018\/978-1-60566-030-1.ch008<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=HealthGrids+in+health+informatics%3A+a+taxonomy&rft.atitle=Handbook+of+Research+on+Advances+in+Health+Informatics+and+Electronic+Healthcare+Applications%3A+Global+Adoption+and+Impact+of+Information+Communication+Technologies&rft.aulast=Naseer%2C+A.%3B+Stergioulas%2C+L.K.&rft.au=Naseer%2C+A.%3B+Stergioulas%2C+L.K.&rft.date=2010&rft.pages=pp.%26nbsp%3B124%E2%80%9343&rft.pub=Medical+Information+Science+Reference&rft_id=info:doi\/10.4018%2F978-1-60566-030-1.ch008&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StaggersEvo02-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StaggersEvo02_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Staggers, N.; Thompson, C.R. (2002). \"The evolution of definitions for nursing informatics: a critical analysis and revised definition\". <i>Journal of the American Medical Informatics Association<\/i> <b>9<\/b>: 255\u201361. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M0946\" target=\"_blank\">10.1197\/jamia.M0946<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+evolution+of+definitions+for+nursing+informatics%3A+a+critical+analysis+and+revised+definition&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Staggers%2C+N.%3B+Thompson%2C+C.R.&rft.au=Staggers%2C+N.%3B+Thompson%2C+C.R.&rft.date=2002&rft.volume=9&rft.pages=255%E2%80%9361&rft_id=info:doi\/10.1197%2Fjamia.M0946&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WhatlingChief11-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WhatlingChief11_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Whatling, J. (September 2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.bcs.org\/content\/conWebDoc\/41603\" target=\"_blank\">\"Chief Clinical Information Officer campaign (CCIO) launch\"<\/a>. British Computer Society (BCS)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.bcs.org\/content\/conWebDoc\/41603\" target=\"_blank\">http:\/\/www.bcs.org\/content\/conWebDoc\/41603<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chief+Clinical+Information+Officer+campaign+%28CCIO%29+launch&rft.atitle=&rft.aulast=Whatling%2C+J.&rft.au=Whatling%2C+J.&rft.date=September+2011&rft.pub=British+Computer+Society+%28BCS%29&rft_id=http%3A%2F%2Fwww.bcs.org%2Fcontent%2FconWebDoc%2F41603&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LusignanWhat03-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LusignanWhat03_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">de Lusignan, S. (2003). \"What is primary care informatics?\". <i>Journal of the American Medical Informatics Association<\/i> <b>10<\/b> (4): 304\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M1187\" target=\"_blank\">10.1197\/jamia.M1187<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12668690\" target=\"_blank\">12668690<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+is+primary+care+informatics%3F&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=de+Lusignan%2C+S.&rft.au=de+Lusignan%2C+S.&rft.date=2003&rft.volume=10&rft.issue=4&rft.pages=304%E2%80%939&rft_id=info:doi\/10.1197%2Fjamia.M1187&rft_id=info:pmid\/12668690&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WyattEhe05-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WyattEhe05_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-WyattEhe05_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wyatt, J.C.; Sullivan, F. (2005). \"eHealth and the future: promise or peril?\". <i>British Medical Journal<\/i> <b>331<\/b> (7529): 1391\u20133. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.331.7529.1391\" target=\"_blank\">10.1136\/bmj.331.7529.1391<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16339252\" target=\"_blank\">16339252<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=eHealth+and+the+future%3A+promise+or+peril%3F&rft.jtitle=British+Medical+Journal&rft.aulast=Wyatt%2C+J.C.%3B+Sullivan%2C+F.&rft.au=Wyatt%2C+J.C.%3B+Sullivan%2C+F.&rft.date=2005&rft.volume=331&rft.issue=7529&rft.pages=1391%E2%80%933&rft_id=info:doi\/10.1136%2Fbmj.331.7529.1391&rft_id=info:pmid\/16339252&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeissbuhlerCon11-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeissbuhlerCon11_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Geissbuhler, A.; Kimura, M.; Kulikowski, C.A.; Murray, P.J.; Ohno-Machado, L.; Park, H.A. et al. (2011). \"Confluence of disciplines in health informatics: an international perspective\". <i>Methods of Information in Medicine<\/i> <b>50<\/b> (6): 545\u201355. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3414%2FME11-06-0005\" target=\"_blank\">10.3414\/ME11-06-0005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22146917\" target=\"_blank\">22146917<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Confluence+of+disciplines+in+health+informatics%3A+an+international+perspective&rft.jtitle=Methods+of+Information+in+Medicine&rft.aulast=Geissbuhler%2C+A.%3B+Kimura%2C+M.%3B+Kulikowski%2C+C.A.%3B+Murray%2C+P.J.%3B+Ohno-Machado%2C+L.%3B+Park%2C+H.A.+et+al.&rft.au=Geissbuhler%2C+A.%3B+Kimura%2C+M.%3B+Kulikowski%2C+C.A.%3B+Murray%2C+P.J.%3B+Ohno-Machado%2C+L.%3B+Park%2C+H.A.+et+al.&rft.date=2011&rft.volume=50&rft.issue=6&rft.pages=545%E2%80%9355&rft_id=info:doi\/10.3414%2FME11-06-0005&rft_id=info:pmid\/22146917&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DemirisInter07-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DemirisInter07_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Demiris, G. (2007). \"Interdisciplinary innovations in biomedical and health informatics graduate education\". <i>Methods of Information in Medicine<\/i> <b>46<\/b> (1): 63\u20136. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17224983\" target=\"_blank\">17224983<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interdisciplinary+innovations+in+biomedical+and+health+informatics+graduate+education&rft.jtitle=Methods+of+Information+in+Medicine&rft.aulast=Demiris%2C+G.&rft.au=Demiris%2C+G.&rft.date=2007&rft.volume=46&rft.issue=1&rft.pages=63%E2%80%936&rft_id=info:pmid\/17224983&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KushnirukSch06-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KushnirukSch06_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kushniruk, A.; Lau, F.; Borycki, E.; Pratti, D. (2006). \"The School of Health Information Science at the University of Victoria: towards an intergrative model for health informatics education and research\". <i>Yearbook of Medical Informatics<\/i>: 159\u201365. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17051310\" target=\"_blank\">17051310<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+School+of+Health+Information+Science+at+the+University+of+Victoria%3A+towards+an+intergrative+model+for+health+informatics+education+and+research&rft.jtitle=Yearbook+of+Medical+Informatics&rft.aulast=Kushniruk%2C+A.%3B+Lau%2C+F.%3B+Borycki%2C+E.%3B+Pratti%2C+D.&rft.au=Kushniruk%2C+A.%3B+Lau%2C+F.%3B+Borycki%2C+E.%3B+Pratti%2C+D.&rft.date=2006&rft.pages=159%E2%80%9365&rft_id=info:pmid\/17051310&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlamantariotouCon10-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlamantariotouCon10_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Alamantariotou, K.; Zisi, D. (2010). \"Consumer health informatics and interactive visual learning tools for health\". <i>International Journal of Electronic Healthcare<\/i> <b>5<\/b> (4): 414\u201324. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1504%2FIJEH.2010.036211\" target=\"_blank\">10.1504\/IJEH.2010.036211<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Consumer+health+informatics+and+interactive+visual+learning+tools+for+health&rft.jtitle=International+Journal+of+Electronic+Healthcare&rft.aulast=Alamantariotou%2C+K.%3B+Zisi%2C+D.&rft.au=Alamantariotou%2C+K.%3B+Zisi%2C+D.&rft.date=2010&rft.volume=5&rft.issue=4&rft.pages=414%E2%80%9324&rft_id=info:doi\/10.1504%2FIJEH.2010.036211&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McCullaghPro11-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McCullaghPro11_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McCullagh, P.; McAllister, G.; Hanna, P.; Finlay, D.; Comac, P. (2011). \"Professional development of health informatics in Northern Ireland\". <i>Studies in Health Technology and Informatics<\/i> <b>169<\/b>: 218\u201322. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21893745\" target=\"_blank\">21893745<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Professional+development+of+health+informatics+in+Northern+Ireland&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=McCullagh%2C+P.%3B+McAllister%2C+G.%3B+Hanna%2C+P.%3B+Finlay%2C+D.%3B+Comac%2C+P.&rft.au=McCullagh%2C+P.%3B+McAllister%2C+G.%3B+Hanna%2C+P.%3B+Finlay%2C+D.%3B+Comac%2C+P.&rft.date=2011&rft.volume=169&rft.pages=218%E2%80%9322&rft_id=info:pmid\/21893745&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaojoMI06-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaojoMI06_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maojo, V.; Kulikowski, C. (2006). \"Medical informatics and bioinformatics: integration or evolution through scientific crises?\". <i>Methods of Information in Medicine<\/i> <b>45<\/b> (5): 474\u201382. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17019500\" target=\"_blank\">17019500<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+informatics+and+bioinformatics%3A+integration+or+evolution+through+scientific+crises%3F&rft.jtitle=Methods+of+Information+in+Medicine&rft.aulast=Maojo%2C+V.%3B+Kulikowski%2C+C.&rft.au=Maojo%2C+V.%3B+Kulikowski%2C+C.&rft.date=2006&rft.volume=45&rft.issue=5&rft.pages=474%E2%80%9382&rft_id=info:pmid\/17019500&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RigbyVeri01-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RigbyVeri01_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rigby, M.; Forsstr\u00f6m, J.; Roberts, R.; Wyatt, J. (2001). \"Verifying quality and safety in health informatics services\". <i>British Medical Journal<\/i> <b>323<\/b> (7312): 552\u20136. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.323.7312.552\" target=\"_blank\">10.1136\/bmj.323.7312.552<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11546703\" target=\"_blank\">11546703<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Verifying+quality+and+safety+in+health+informatics+services&rft.jtitle=British+Medical+Journal&rft.aulast=Rigby%2C+M.%3B+Forsstr%C3%B6m%2C+J.%3B+Roberts%2C+R.%3B+Wyatt%2C+J.&rft.au=Rigby%2C+M.%3B+Forsstr%C3%B6m%2C+J.%3B+Roberts%2C+R.%3B+Wyatt%2C+J.&rft.date=2001&rft.volume=323&rft.issue=7312&rft.pages=552%E2%80%936&rft_id=info:doi\/10.1136%2Fbmj.323.7312.552&rft_id=info:pmid\/11546703&rfr_id=info:sid\/en.wikipedia.org:Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210235\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.508 seconds\nReal time usage: 0.535 seconds\nPreprocessor visited node count: 16721\/1000000\nPreprocessor generated node count: 34729\/1000000\nPost\u2010expand include size: 110789\/2097152 bytes\nTemplate argument size: 38154\/2097152 bytes\nHighest expansion depth: 14\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 494.414 1 - -total\n 73.29% 362.379 21 - Template:Citation\/core\n 64.26% 317.719 17 - Template:Cite_journal\n 13.01% 64.316 1 - Template:Infobox_journal_article\n 12.51% 61.839 1 - Template:Infobox\n 9.76% 48.277 3 - Template:Cite_web\n 7.39% 36.545 80 - Template:Infobox\/row\n 7.05% 34.848 23 - Template:Citation\/identifier\n 4.90% 24.217 1 - Template:Cite_book\n 3.64% 17.997 23 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7629-0!*!0!!en!5!* and timestamp 20181213210234 and revision id 23424\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Unravelling_the_tangled_taxonomies_of_health_informatics\">https:\/\/www.limswiki.org\/index.php\/Journal:Unravelling_the_tangled_taxonomies_of_health_informatics<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","4d5313302570d2f6c92b0f91883ad392_images":["https:\/\/www.limswiki.org\/images\/f\/f7\/Fig1_Barrett_InformaticsPC2014_21-3.jpg"],"4d5313302570d2f6c92b0f91883ad392_timestamp":1544734954,"302bc46eb57fbea71e5d7745898c7261_type":"article","302bc46eb57fbea71e5d7745898c7261_title":"Undertaking sociotechnical evaluations of health information technologies (Cresswell et al. 2014)","302bc46eb57fbea71e5d7745898c7261_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies","302bc46eb57fbea71e5d7745898c7261_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Undertaking sociotechnical evaluations of health information technologies\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUndertaking sociotechnical evaluations of health information technologiesJournal\n \nInformatics in Primary CareAuthor(s)\n \nCresswell, Kathrin M.; Sheikh, AzizAuthor affiliation(s)\n \nSchool of Health in Social Science, University of Edinburgh; Centre for Population Health Sciences, University of EdinburghPrimary contact\n \nEmail: kathrin.beyer@ed.ac.ukYear published\n \n2014Volume and issue\n \n21 (2)Page(s)\n \n78\u201383DOI\n \n10.14236\/jhi.v21i2.54ISSN\n \n2058-4563Distribution license\n \nCreative Commons Attribution 2.5 GenericWebsite\n \nhttp:\/\/hijournal.bcs.org\/index.php\/jhi\/article\/view\/54\/80Download\n \nhttp:\/\/hijournal.bcs.org\/index.php\/jhi\/article\/download\/54\/79 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 What are sociotechnical evaluations and when should this approach be used? \n4 What study designs lend themselves to sociotechnical evaluations? \n5 What data to collect and how to analyse these \n6 Potential pitfalls and how to avoid these \n\n6.1 Practical challenges associated with sociotechnical evaluations \n6.2 Conceptual challenges associated with sociotechnical evaluations \n\n\n7 Conclusions \n8 Contributors and sources \n9 Acknowledgements \n10 Funding \n11 Competing interest declaration \n12 References \n13 Notes \n\n\n\nAbstract \nThere is an increasing international recognition that the evaluation of health information technologies should involve assessments of both the technology and the social\/organisational contexts into which it is deployed. There is, however, a lack of agreement on definitions, published guidance on how such \u2018sociotechnical evaluations\u2019 should be undertaken, and how they distinguish themselves from other approaches. We explain what sociotechnical evaluations are, consider the contexts in which these are most usefully undertaken, explain what they entail, reflect on the potential pitfalls associated with such research, and suggest possible ways to avoid these.\nKeywords: Evaluation, health information technology, sociotechnical\n\nIntroduction \nInternationally, there is a growing political drive to implement ever more complex information technologies into healthcare settings, in the hope that these will help improve the quality, safety, and efficiency of healthcare.[1] There is in parallel a growing appreciation that such interventions need to be formally evaluated, as the benefits of technologies should not simply be assumed. Complex systems such as electronic health records, and electronic prescribing and telemonitoring technologies are often very costly to procure and maintain, and so, even if effectiveness in relation to the quality of care is established, cost effectiveness needs to be examined. There is now also a growing body of work indicating that technologies may inadvertently introduce new risks, largely arising from difficulties of systems to integrate with existing work processes.[2]\nThe study of technological innovation into healthcare settings should therefore \u2014 particularly if the technology is likely to be disruptive \u2014 offer an opportunity to understand and evaluate the changing inter-relationships between technology and human\/organisational (or socio-) factors. Whilst there is a growing theoretical and empirical evidence base on this subject, there is as yet little practical guidance explaining how such sociotechnical evaluations should be undertaken.[3][4][5][6][7][8][9]\nAlthough there is an increasing appreciation of the complex processes involved in using and implementing technology in social contexts to improve the safety and quality of healthcare[3][4][5][6][7][8][9], current sociotechnical approaches somewhat fail to distinguish themselves from other methodologies such as usability testing and context-sensitive methods of investigation. This may be due to the lack of agreed existing definitions of what constitutes sociotechnical approaches to evaluation and the range of disciplinary backgrounds involved.\nDrawing on our experience of conducting a number of recent studies of complex health information technologies[10][11][12][13][14][15][16], we aim to provide a practical guide to undertaking sociotechnical evaluations: we consider the contexts in which such approaches are most usefully employed, explain what sociotechnical evaluations involve, and reflect on the potential pitfalls associated with such work and how these might be avoided.\n\nWhat are sociotechnical evaluations and when should this approach be used? \nSociotechnical perspectives assume that \u2018organisational and human (socio) factors and information technology system factors (technical) are inter-related parts of one system, each shaping the other.[12] In line with this, sociotechnical evaluations involve researching the way technical and social dimensions change and shape each other over time.[17][18][19]\nWe summarise some typical components of a sociotechnical evaluation in Box 1. This approach is potentially most appropriate when there is a complex, iterative relationship between the technology and social processes in the environment into which it is introduced. Conversely, it is less useful when there is likely to be a simple, linear cause-and-effect relationship between the technology and social processes. The dimensions explored in an evaluation may encompass investigating how technologies change social processes (e.g. the way care is delivered by, for example, introducing electronic health records), and how technologies themselves can change over time as a result of user\/organisational requirements (e.g. ongoing customisation to improve usability) (Figure 1).[5] Such adaptation is important as if the use of the technology results in perceived adverse consequences for the delivery of care, the technology is likely to frustrate busy clinical staff and may be abandoned altogether.[19]\nA further defining component of sociotechnical evaluations is the attempt to study processes associated with the introduction of a new technology in social\/organisational settings, as these mediators can offer important insights into potentially transferable lessons.[12][14][15][20] This focus on processes is important, because of the increasing number of technological functionalities and vast differences in implementation contexts. In contrast, evaluations that focus solely on investigating the impact of technology on outcomes often have limited generalisability beyond the immediate clinical setting in which the research was undertaken.\n\n\n\n\n\n\n\n Box 1. A summary of typical components of a sociotechnical evaluation\n\n\nAims\n1. Identify processes, benefits, and negative impacts of the new system across a variety of dimensions (see below).\n2. Extract overall potentially transferable lessons to a greater number of organisations.\n3. Liaise with decision makers to inform implementation.\n\r\nMethods\n\u2022 Longitudinal: tracking changes over time.\n\u2022 Ideally mixed methods: qualitative and quantitative work feeding into each other, focus on processes.\n\u2022 Can be case study based.\n\u2022 Drawing on existing theory.\n\r\nParticipants\nPurposeful sampling of individuals within care settings (managers, implementation team members and IT staff, doctors, nurses, allied health professionals, administrative staff, patients, and carers), and also stakeholders outside the immediate care setting (e.g. policy makers and system developers).\n\r\nIndividual dimensions that may be studied\n\u2022 Implementation strategies and experiences (e.g. technical, clinical, and organisational issues) \u2014 qualitative work, e.g. interviews and documentary analysis of strategic documents such as business cases; participants: organisational decision makers.\n\u2022 Attitudes, expectations, and experiences of individuals \u2014 qualitative work, e.g. interviews with organisational decision makers, policy makers, and users.\n\u2022 Organisational consequences such as changes in data quality, workflows, and organisational roles and responsibilities \u2014 qualitative work, e.g. interviews with organisational decision makers and users, and observations of technology use; some quantitative measurements may be possible.\n\u2022 Assessment of implementation costs \u2014 health economic work.\n\u2022 Assessing the impact of systems on errors, safety, and quality of care \u2014 considering key quantifiable benefits in relation to improving quality and\/or safety of care, with a focus on those outcomes that are most likely to be influenced by the technology in question.\n\u2022 Recommendations for implementation and evaluation of similar initiatives \u2014 may contain a qualitative element interviewing policy makers and system developers, and examining policy documents.\n\n\n\n\n\n\n\nFigure 1. Dimensions commonly explored in sociotechnical evaluations of health information technologies\nWhat study designs lend themselves to sociotechnical evaluations? \nThe focus on investigating and exploring processes lends itself best to a naturalistic approach, but sociotechnical evaluations may also incorporate aspects of more positivist designs, particularly in the context of undertaking quasi-experimental studies, when investigators do not have direct control over the technology that is to be implemented.[15]\nSociotechnical evaluations should ideally be undertaken using a prospective design, as this can help to map and understand the interplay between the technology and the social context, and thereby identify important insights into how the technology is received and used.[11][12][14][15][16][20][21]\nAny appropriate design with a longitudinal dimension that allows mediating processes to be understood in detail is therefore potentially suitable for a sociotechnical evaluation. Mixed-methods sociotechnical evaluations are becoming more popular and are likely to represent an important expansion area for this research approach.[11]\n\nWhat data to collect and how to analyse these \nA range of qualitative and quantitative data may be gathered during sociotechnical evaluations (see Box 1). Qualitative data can help to shed light on social processes and perceived technical features such as individual attitudes and expectations (interviews and focus groups), planned organisational strategies and policies (documents), and use of technology in context (observations). These data may be complemented by quantitative work investigating the measurable impacts of technology on social systems. For instance, collecting health economic and cost data can provide insights into investment and maintenance costs, benefits, and returns on investment. Quantitative data collection can also help to explore the impact of systems on the safety and quality of care, e.g. by measuring reductions in errors and increased efficiencies associated with the move from paper-based to electronic systems.\nA key distinctive feature in relation to analysis is the focus on exploring the dynamic relationship between technical and social factors over time. It is therefore important to obtain insights into technical characteristics and social processes before the introduction of the new technology into a new social context, changes to technical and social aspects once the technology is introduced, potential underlying relationships between technical and social dimensions, and the changes that occur over time as the technology becomes more embedded within the new social context. Learning across implementations can be promoted by identifying what mechanisms underlie observations and hypothesising if\/how these may be applicable to other contexts. This may be informed by a realistic evaluation perspective, assessing contexts (existing and desirable conditions for certain outcomes to be produced), mechanisms (potential causal pathways that may lead to an outcome), and outcomes (the observable effects produced).[22] The approach may help to overcome traditional boundaries associated with the separation of technical and social spheres, as all three aspects (contexts, mechanisms, and outcomes) are neither distinctively technical nor distinctively social \u2014 they emerge out of the interplay of both.\n\nPotential pitfalls and how to avoid these \nThere are, however, a number of challenges associated with sociotechnical evaluations of health information technologies (summarised in Table 1). These stem from a lack of existing agreement on various components of a sociotechnical system, possible study designs, and data analysis strategies. They range from practical issues surrounding the management of the work itself and coping with shifting implementation timelines, to more conceptual challenges surrounding the extraction of a \u2018bottom line\u2019, and the pragmatic use of theory.\n\n\n\n\n\n\n\nTable 1. Summary of potential challenges and how to avoid these\n\n\n Challenges\n\n How to avoid these\n\n\nLarge datasets and multi-disciplinary teams\n\nIntegrating data and perspectives through robust project management allocating lead researchers to aspects of the work\n\n\nShifting implementation landscapes and timelines\n\nAdapting the aims and methodologies originally envisaged, and realistic timeframes of evaluations over longer periods of time\n\n\nLinking local implementation processes to their structuring conditions\n\nResearching wider political and commercial processes in which local developments are taking place\n\n\nExtracting a \u2018bottom line\u2019\n\nJudging \u2018success\u2019 and \u2018failure\u2019 is not constructive; there is a complex story to be told but transferable lessons are possible\n\n\nDifferent research traditions vary significantly in the way technologies, processes, and stakeholders are conceptualised\n\nThe pragmatic use of theory, drawing on existing theoretical frameworks\n\n\n\nPractical challenges associated with sociotechnical evaluations \nImplementations of complex health information technologies such as electronic health records and electronic prescribing systems affect many aspects of organisational functioning and therefore tend to require complex evaluations. This is compounded with increasing organisational size and\/or if more than one organisation is studied. Specific expertise needed in the research team will vary depending on the research questions being studied and methods employed, but may include methodological (qualitative, statistical, epidemiological, and health economic), theoretical (organisational change, management, and human factors), managerial (applied team management), healthcare professional (doctors, nurses, and allied health professions), and technical (information technology specialists and system developers) expertise. Exploitation of individual strengths whilst ensuring coordinated efforts can in our experience be greatly facilitated by assigning lead researchers to individual aspects of the work investigating different impacts\/consequences (e.g. cost and changes to individual work practices), and organisations to be studied.\nA further practical challenge that evaluators are likely to face is the impact of shifting implementation landscapes and timelines, potentially resulting in original methodologies having to be adapted accordingly.[12] This is a common problem in evaluations of technical and\/or health policy interventions, where data collection activities depend on the planned introduction of technologies.[14] As a result of expanding timelines, originally anticipated quantitative measurement making before, during, and after assessments of technology introduction may not be possible, and evaluators may lack opportunities to investigate the systems once they are routinely used within organisations.[11][12][13][14] Longitudinal evaluative work over extended periods of time is therefore important.[16] This should be characterised by early and close collaboration between evaluators and decision makers, so that changes in strategic direction and potential consequences for evaluation activities can be planned for in advance.[15]\n\nConceptual challenges associated with sociotechnical evaluations \nThere are also a number of conceptual challenges associated with conducting sociotechnical evaluations of complex health information technologies. The first relates to the researching of context surrounding the technology. Here, it is important to explore the use of technology by individuals, but also the wider environment in which these processes are situated (e.g. professional, organisational, and political contexts), as this can impact significantly on the way technology is adopted and changed over time.[2][14][15] With this in mind, it is, for example, often insufficient to only explore healthcare professional perspectives on technology implementation and adoption; there is also a need to gain insights into managerial and organisational perspectives to get a more rounded understanding of implementation processes and reasons underlying strategic decisions. Similarly, broader political and commercial developments may play a role in shaping user experiences and the deployment\/design of technologies.[12][19]\nAnother common challenge facing sociotechnical evaluations is the extraction of a \u2018bottom line\u2019. Large complex processes tend to be most accurately described with the help of large complex stories, but there is also a need to ensure that messages emerging from evaluations are heard by decision makers, and this is often only possible with a limited number of straightforward key messages. In the face of this pressure, many evaluators have resorted to making bold statements surrounding the \u2018success\u2019 or \u2018failure\u2019 of evaluations of complex technologies, but in our experience, this is neither constructive in relation to future decision making nor accurate in representing reality. Evaluators need to acknowledge the existence of different notions of success as well as different temporal dimensions associated with perceived \u2018failures\u2019 (e.g. something that was initially observed as a \u2018failure\u2019 may on reflection turn out to be a \u2018success\u2019). A better way to conceptualise outputs of sociotechnical evaluations is through a summary of key lessons learned, outlining how and why these may be transferable to other settings.\nThe final conceptual challenge takes a more theoretical angle. It is now commonly recognised that theory can help to develop transferable lessons between settings, and there are many theories that draw on sociotechnical principles.[23] However, different existing research traditions vary significantly in the way technologies, processes, and stakeholders are conceptualised. This results in a lack of existing overall framework through which implementations can be examined, although some existing approaches are summarised in Box 2.[24] In addition, theoretical lenses are often hard to understand for non-academic audiences and lack pragmatism. This inhibits learning from experience and also widens the existing gap between academic research and frontline practice. With this in mind, it is key not to ignore the issue surrounding theory as is so often done in existing evaluations \u2014 particularly those carried out by healthcare organisations that lack the necessary expertise. More usable integrative theoretically informed evaluation frameworks are currently in development, but in the meantime, we advocate drawing on those that encompass not only social and technical dimensions, but also the wider contexts in which local developments are taking place (Box 2).\n\n\n\n\n\n\n\n Box 2. Theoretical frameworks that encompass wider contexts in which local developments are taking place\n\n\n\u2022 The Theory of the Diffusion of Innovations: This can help to explain why or why not certain innovations spread in and across organisations as well as how this spread may occur. Rogers E. Diffusion of Innovations. New York: Free Press, 1983.\n\u2022 Normalisation Process Theory: Can help to explore how innovations become embedded within clinical practice over time and what inhibits and\/or facilitates this process. May C. A rational model for assessing and evaluating complex interventions in healthcare. BMC Health Serv Res 2006;6:86.\n\u2022 Social Shaping of Technology: Can help to examine how information technology is related to and shaped by historical, cultural, and economic factors. Williams R and Edge D. The social shaping of technology. Res Pol 1996;25:865.\n\u2022 HOT-fit: Can help to explore the alignment between technology, human, and organisational factors. Yusof MM, Kuljis J, Papazafeiropoulou A and Stergioulas LK. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform 2008;77:386.\n\u2022 Cornford et al.\u2019s evaluation framework: Can help to explore interrelationships between system functions, human perspectives, and organisational context. Cornford T, Doukidis G and Forster D. Experience outcome with a structure, process and framework for evaluating an information system. Omega, Int J Manag Sci 1994;22:491\u2013504.\n\n\n\n\nConclusions \nSociotechnical evaluations are a powerful tool to research complex technological change, particularly if the aim is to investigate non-linear relationships between technology and social processes. However, there are also challenges associated with their conduct as they involve investigating complex processes over time and within complex changing environments. The existing literature has, due to a lack of existing agreement on defining components of such work, not always fully appreciated these.\nWe hope that our suggested definitions, experiences, and reflections will contribute to a more integrated approach to conducting sociotechnical evaluations of technological systems in healthcare settings. Given the very considerable policy interest and substantial financial resources being expended in implementing health information technologies in an attempt to achieve the triple aims of enhancing the safety\/quality of care, improving health outcomes, and maximising the efficiency of care, we believe that greater use of sociotechnical evaluations will help to understand key processes in the interrelationships between technology, people, and organisations. This understanding will help to derive important and potentially transferable lessons.\n\nContributors and sources \nAziz Sheikh conceived this work and is the guarantor. Aziz Sheikh was the principal investigator of the project discussed and is currently leading a National Institute for Health Research-funded national evaluation of electronic prescribing and medicines administration systems. He is supported by a Harkness Health Policy and Practice fellowship from The Commonwealth Fund. Kathrin M Cresswell was employed as a researcher and led on the write-up and drafting the initial version of the paper, with Aziz Sheikh commenting on various drafts.\n\nAcknowledgements \nThe authors are very grateful to all participants who kindly gave their time and to the extended project team of the work drawn on. The authors are also grateful for the helpful comments on an earlier version of this manuscript by Dr. Allison Worth.\n\nFunding \nThis work has drawn on research funded by the NHS Connecting for Health Evaluation Programme (NHS CFHEP 001, NHS CFHEP 005, NHS CFHEP 009, NHS CFHEP 010) and the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research scheme (RP-PG-1209-10099). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health. AS is supported by The Commonwealth Fund, a private independent foundation based in New York City. The views presented here are those of the author and not necessarily those of The Commonwealth Fund, its directors, officers, or staff.\n\nCompeting interest declaration \nThe authors have no competing interests.\n\nReferences \n\n\u2191 Coiera, E. (2009). \"Building a National Health IT System from the middle out\". Journal of the American Medical Informatics Association 16: 271\u20133. doi:10.1197\/jamia.M3183. PMC PMC2732241. PMID 19407078. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732241 .   \n\n\u2191 2.0 2.1 Greenhalgh, T.; Stones, R. (2010). \"Theorising big IT programmes in healthcare: strong structuration theory meets actor-network theory\". Social Science and Medicine 70: 1285\u201394. doi:10.1016\/j.socscimed.2009.12.034. PMID 20185218.   \n\n\u2191 3.0 3.1 Catwell, L.; Sheikh, A. (2009). \"Evaluating eHealth interven-tions: the need for continuous systemic evaluation\". PLoS Medicine 6: e1000126. doi:10.1371\/journal.pmed.1000126. PMC PMC2719100. PMID 19688038. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2719100 .   \n\n\u2191 4.0 4.1 Berg, M.; Aarts, J.; van der Lei, J. (2003). \"ICT in health care: socio-technical approaches\". Methods of Information in Medicine 42: 297\u2013301. PMID 14534625.   \n\n\u2191 5.0 5.1 5.2 Cresswell, K.; Worth, A.; Sheikh, A. (2010). \"Actor-network theory and its role in understanding the implementation of information technology developments in healthcare\". BMC Medical Informatics and Decision Making 10: 67. doi:10.1186\/1472-6947-10-67. PMC PMC2988706. PMID 21040575. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2988706 .   \n\n\u2191 6.0 6.1 \"Health IT and Patient Safety: Building Safer Systems for Better Care\" (PDF). Institute of Medicine. 2011. http:\/\/www.iom.edu\/~\/media\/Files\/Report%20Files\/2011\/Health-IT\/HIT%20and%20Patient%20Safety.pdf . Retrieved 16 July 2013 .   \n\n\u2191 7.0 7.1 Cherns, A. (1987). \"Principles of sociotechnical design revisited\". Human Relations 40: 153\u201361. doi:10.1177\/001872678704000303.   \n\n\u2191 8.0 8.1 Clegg, C.W. (2000). \"Sociotechnical principles for system design\". Applied Ergonomics 31: 463\u201377. doi:10.1016\/S0003-6870(00)00009-0.   \n\n\u2191 9.0 9.1 Harrison, M.I.; Koppel, R.; Bar-Lev, S. (2007). \"Unintended consequences of information technologies in health care\u2014an interactive socio-technical analysis\". Journal of the American Medical Informatics Association 14: 542\u20139. doi:10.1197\/jamia.M2384. PMC PMC1975796. PMID 17600093. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1975796 .   \n\n\u2191 Black, A.D.; Car, J.; Pagliari, C.; Anandan, C.; Cresswell, K.; Bokun, T.; et al. (2011). \"The impact of eHealth on the quality and safety of health care: a systematic overview\". PLoS Medicine 8: e1000387.   \n\n\u2191 11.0 11.1 11.2 11.3 Sheikh, A.; Cornford, T.; Barber, N.; Avery, A.; Takian, A.; Lichtner, V.; et al. (2011). \"Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from a prospective national evaluation in \u201cearly adopter\u201d hospitals\". British Medical Journal 343: d6054. doi:10.1136\/bmj.d6054. PMC PMC3195310. PMID 22006942. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3195310 .   \n\n\u2191 12.0 12.1 12.2 12.3 12.4 12.5 12.6 Robertson, A.; Cresswell, K.; Takian, A.; Petrakaki, D.; Crowe, S.; Cornford, T.; et al. (2010). \"Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation\". British Medical Journal 341: c4564.   \n\n\u2191 13.0 13.1 Takian, A.; Petrakaki, D.; Cornford, T.; Sheikh, A.; Barber, N. (2012). \"Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems\". BMC Health Services Research 12: 105\u201329. doi:10.1186\/1472-6963-12-105. PMC PMC3469374. PMID 22545646. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3469374 .   \n\n\u2191 14.0 14.1 14.2 14.3 14.4 14.5 Westbrook, J.; Braithwaite, J.; Georgiou, A.; Ampt, A.; Creswick, A.; Coiera, E.; et al. (2007). \"Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory\". Journal of the American Medical Informatics Association 14: 746\u201355. doi:10.1197\/jamia.M2462. PMC PMC2213479. PMID 17712083. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213479 .   \n\n\u2191 15.0 15.1 15.2 15.3 15.4 15.5 Greenhalgh, T.; Hinder, S.; Stramer, K.; Bratan, T.; Russell, J. (2010). \"Adoption, non-adoption, and abandonment of a personal electronic health record: case study of HealthSpace\". British Medical Journal 341: c5814.   \n\n\u2191 16.0 16.1 16.2 Barber, N.; Cornford, T.; Klecun, E. (2007). \"Qualitative evaluation of an electronic prescribing and administration system\". Quality and Safety in Health Care 16: 271\u20138. doi:10.1136\/qshc.2006.019505. PMC PMC2464937. PMID 17693675. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2464937 .   \n\n\u2191 Berg, M. (1999). \"Patient care information systems and health care work: a sociotechnical approach\". International Journal of Medical Informatics 55: 87\u2013101. doi:10.1016\/S1386-5056(99)00011-8.   \n\n\u2191 Cornford, T.; Doukidis, G.; Forster, D. (1994). \"Experience outcome with a structure, process and framework for evaluating an information system\". Omega, International Journal of Management Science 22: 491\u2013504. doi:10.1016\/0305-0483(94)90030-2.   \n\n\u2191 19.0 19.1 19.2 Cresswell, K.; Worth, A.; Sheikh, A. (2012). \"Comparative case study investigating sociotechnical processes of change in the con-text of a national electronic health record implementation\". Health Informatics Journal 18: 251\u201370. doi:10.1177\/1460458212445399. PMID 23257056.   \n\n\u2191 20.0 20.1 Crowe, C.; Cresswell, K.; Robertson, R.; Huby, G.; Avery, A.; Sheikh, A. (2011). \"The case study approach\". BMC Medical Research Methodology 11: 100. doi:10.1186\/1471-2288-11-100. PMC PMC3141799. PMID 21707982. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3141799 .   \n\n\u2191 Cresswell, K.M.; Bates, D.W.; Sheikh, A. (2013). \"Ten key considerations for the successful implementation and adoption of large-scale health information technology\". Journal of the American Medical Informatics Association 20: e9\u201313.   \n\n\u2191 Pawson, R.; Tilley, N. (1997). Realistic Evaluation. Sage Publications.   \n\n\u2191 Wacker, J.G. (1998). \"A definition of theory: research guidelines for different theory-building research methods in operations management\". Journal of Operations Management 16: 361\u201385. doi:10.1016\/S0272-6963(98)00019-9.   \n\n\u2191 Greenhalgh, T.; Potts, H.; Wong, G.; Bark, P.; Swinglehurst, D. (2009). \"Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method\". Milbank Quarterly 87: 729\u201388. doi:10.1111\/j.1468-0009.2009.00578.x. PMC PMC2888022. PMID 20021585. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2888022 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\">https:\/\/www.limswiki.org\/index.php\/Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on softwareHidden category: Pages using duplicate arguments in template calls\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 21:02.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,557 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","302bc46eb57fbea71e5d7745898c7261_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Undertaking_sociotechnical_evaluations_of_health_information_technologies skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Undertaking sociotechnical evaluations of health information technologies<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>There is an increasing international recognition that the evaluation of health information technologies should involve assessments of both the technology and the social\/organisational contexts into which it is deployed. There is, however, a lack of agreement on definitions, published guidance on how such \u2018sociotechnical evaluations\u2019 should be undertaken, and how they distinguish themselves from other approaches. We explain what sociotechnical evaluations are, consider the contexts in which these are most usefully undertaken, explain what they entail, reflect on the potential pitfalls associated with such research, and suggest possible ways to avoid these.\n<\/p><p><b>Keywords<\/b>: Evaluation, health information technology, sociotechnical\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Internationally, there is a growing political drive to implement ever more complex information technologies into healthcare settings, in the hope that these will help improve the quality, safety, and efficiency of healthcare.<sup id=\"rdp-ebb-cite_ref-CoieraBuild09_1-0\" class=\"reference\"><a href=\"#cite_note-CoieraBuild09-1\" rel=\"external_link\">[1]<\/a><\/sup> There is in parallel a growing appreciation that such interventions need to be formally evaluated, as the benefits of technologies should not simply be assumed. Complex systems such as , and electronic prescribing and telemonitoring technologies are often very costly to procure and maintain, and so, even if effectiveness in relation to the quality of care is established, cost effectiveness needs to be examined. There is now also a growing body of work indicating that technologies may inadvertently introduce new risks, largely arising from difficulties of systems to integrate with existing work processes.<sup id=\"rdp-ebb-cite_ref-GreenhalghTheo10_2-0\" class=\"reference\"><a href=\"#cite_note-GreenhalghTheo10-2\" rel=\"external_link\">[2]<\/a><\/sup>\n<\/p><p>The study of technological innovation into healthcare settings should therefore \u2014 particularly if the technology is likely to be disruptive \u2014 offer an opportunity to understand and evaluate the changing inter-relationships between technology and human\/organisational (or socio-) factors. Whilst there is a growing theoretical and empirical evidence base on this subject, there is as yet little practical guidance explaining how such sociotechnical evaluations should be undertaken.<sup id=\"rdp-ebb-cite_ref-CatwellEval09_3-0\" class=\"reference\"><a href=\"#cite_note-CatwellEval09-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BergICT03_4-0\" class=\"reference\"><a href=\"#cite_note-BergICT03-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CresswellActor10_5-0\" class=\"reference\"><a href=\"#cite_note-CresswellActor10-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-IoMHealth11_6-0\" class=\"reference\"><a href=\"#cite_note-IoMHealth11-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChernsPrinc87_7-0\" class=\"reference\"><a href=\"#cite_note-ChernsPrinc87-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Clegg_8-0\" class=\"reference\"><a href=\"#cite_note-Clegg-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HarrisonUn07_9-0\" class=\"reference\"><a href=\"#cite_note-HarrisonUn07-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/p><p>Although there is an increasing appreciation of the complex processes involved in using and implementing technology in social contexts to improve the safety and quality of healthcare<sup id=\"rdp-ebb-cite_ref-CatwellEval09_3-1\" class=\"reference\"><a href=\"#cite_note-CatwellEval09-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BergICT03_4-1\" class=\"reference\"><a href=\"#cite_note-BergICT03-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CresswellActor10_5-1\" class=\"reference\"><a href=\"#cite_note-CresswellActor10-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-IoMHealth11_6-1\" class=\"reference\"><a href=\"#cite_note-IoMHealth11-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChernsPrinc87_7-1\" class=\"reference\"><a href=\"#cite_note-ChernsPrinc87-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Clegg_8-1\" class=\"reference\"><a href=\"#cite_note-Clegg-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HarrisonUn07_9-1\" class=\"reference\"><a href=\"#cite_note-HarrisonUn07-9\" rel=\"external_link\">[9]<\/a><\/sup>, current sociotechnical approaches somewhat fail to distinguish themselves from other methodologies such as usability testing and context-sensitive methods of investigation. This may be due to the lack of agreed existing definitions of what constitutes sociotechnical approaches to evaluation and the range of disciplinary backgrounds involved.\n<\/p><p>Drawing on our experience of conducting a number of recent studies of complex health information technologies<sup id=\"rdp-ebb-cite_ref-BlackThe09_10-0\" class=\"reference\"><a href=\"#cite_note-BlackThe09-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SheikhImp11_11-0\" class=\"reference\"><a href=\"#cite_note-SheikhImp11-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RobertsonImp10_12-0\" class=\"reference\"><a href=\"#cite_note-RobertsonImp10-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TakianBuild12_13-0\" class=\"reference\"><a href=\"#cite_note-TakianBuild12-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WestbrookMulti07_14-0\" class=\"reference\"><a href=\"#cite_note-WestbrookMulti07-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GreenhalghAdop10_15-0\" class=\"reference\"><a href=\"#cite_note-GreenhalghAdop10-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Barberqual07_16-0\" class=\"reference\"><a href=\"#cite_note-Barberqual07-16\" rel=\"external_link\">[16]<\/a><\/sup>, we aim to provide a practical guide to undertaking sociotechnical evaluations: we consider the contexts in which such approaches are most usefully employed, explain what sociotechnical evaluations involve, and reflect on the potential pitfalls associated with such work and how these might be avoided.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"What_are_sociotechnical_evaluations_and_when_should_this_approach_be_used.3F\">What are sociotechnical evaluations and when should this approach be used?<\/span><\/h2>\n<p>Sociotechnical perspectives assume that \u2018organisational and human (socio) factors and information technology system factors (technical) are inter-related parts of one system, each shaping the other.<sup id=\"rdp-ebb-cite_ref-RobertsonImp10_12-1\" class=\"reference\"><a href=\"#cite_note-RobertsonImp10-12\" rel=\"external_link\">[12]<\/a><\/sup> In line with this, sociotechnical evaluations involve researching the way technical and social dimensions change and shape each other over time.<sup id=\"rdp-ebb-cite_ref-BergPatient99_17-0\" class=\"reference\"><a href=\"#cite_note-BergPatient99-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CornfordExp94_18-0\" class=\"reference\"><a href=\"#cite_note-CornfordExp94-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CresswellComp12_19-0\" class=\"reference\"><a href=\"#cite_note-CresswellComp12-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/p><p>We summarise some typical components of a sociotechnical evaluation in Box 1. This approach is potentially most appropriate when there is a complex, iterative relationship between the technology and social processes in the environment into which it is introduced. Conversely, it is less useful when there is likely to be a simple, linear cause-and-effect relationship between the technology and social processes. The dimensions explored in an evaluation may encompass investigating how technologies change social processes (e.g. the way care is delivered by, for example, introducing electronic health records), and how technologies themselves can change over time as a result of user\/organisational requirements (e.g. ongoing customisation to improve usability) (Figure 1).<sup id=\"rdp-ebb-cite_ref-CresswellActor10_5-2\" class=\"reference\"><a href=\"#cite_note-CresswellActor10-5\" rel=\"external_link\">[5]<\/a><\/sup> Such adaptation is important as if the use of the technology results in perceived adverse consequences for the delivery of care, the technology is likely to frustrate busy clinical staff and may be abandoned altogether.<sup id=\"rdp-ebb-cite_ref-CresswellComp12_19-1\" class=\"reference\"><a href=\"#cite_note-CresswellComp12-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/p><p>A further defining component of sociotechnical evaluations is the attempt to study processes associated with the introduction of a new technology in social\/organisational settings, as these mediators can offer important insights into potentially transferable lessons.<sup id=\"rdp-ebb-cite_ref-RobertsonImp10_12-2\" class=\"reference\"><a href=\"#cite_note-RobertsonImp10-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WestbrookMulti07_14-1\" class=\"reference\"><a href=\"#cite_note-WestbrookMulti07-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GreenhalghAdop10_15-1\" class=\"reference\"><a href=\"#cite_note-GreenhalghAdop10-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CroweCase11_20-0\" class=\"reference\"><a href=\"#cite_note-CroweCase11-20\" rel=\"external_link\">[20]<\/a><\/sup> This focus on processes is important, because of the increasing number of technological functionalities and vast differences in implementation contexts. In contrast, evaluations that focus solely on investigating the impact of technology on outcomes often have limited generalisability beyond the immediate clinical setting in which the research was undertaken.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Box 1. A summary of typical components of a sociotechnical evaluation\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><big><b>Aims<\/b><\/big>\n<p>1. Identify processes, benefits, and negative impacts of the new system across a variety of dimensions (see below).\n<\/p><p>2. Extract overall potentially transferable lessons to a greater number of organisations.\n<\/p><p>3. Liaise with decision makers to inform implementation.\n<\/p><p><br \/><big><b>Methods<\/b><\/big>\n<\/p><p>\u2022 Longitudinal: tracking changes over time.\n<\/p><p>\u2022 Ideally mixed methods: qualitative and quantitative work feeding into each other, focus on processes.\n<\/p><p>\u2022 Can be case study based.\n<\/p><p>\u2022 Drawing on existing theory.\n<\/p><p><br \/><big><b>Participants<\/b><\/big>\n<\/p><p>Purposeful sampling of individuals within care settings (managers, implementation team members and IT staff, doctors, nurses, allied health professionals, administrative staff, patients, and carers), and also stakeholders outside the immediate care setting (e.g. policy makers and system developers).\n<\/p><p><br \/><big><b>Individual dimensions that may be studied<\/b><\/big>\n<\/p><p>\u2022 <b>Implementation strategies and experiences<\/b> (e.g. technical, clinical, and organisational issues) \u2014 qualitative work, e.g. interviews and documentary analysis of strategic documents such as business cases; participants: organisational decision makers.\n<\/p><p>\u2022 <b>Attitudes, expectations, and experiences of individuals<\/b> \u2014 qualitative work, e.g. interviews with organisational decision makers, policy makers, and users.\n<\/p><p>\u2022 <b>Organisational consequences<\/b> such as changes in data quality, workflows, and organisational roles and responsibilities \u2014 qualitative work, e.g. interviews with organisational decision makers and users, and observations of technology use; some quantitative measurements may be possible.\n<\/p><p>\u2022 Assessment of <b>implementation costs<\/b> \u2014 health economic work.\n<\/p><p>\u2022 Assessing the <b>impact of systems on errors, safety, and quality of care<\/b> \u2014 considering key quantifiable benefits in relation to improving quality and\/or safety of care, with a focus on those outcomes that are most likely to be influenced by the technology in question.\n<\/p><p>\u2022 <b>Recommendations for implementation and evaluation<\/b> of similar initiatives \u2014 may contain a qualitative element interviewing policy makers and system developers, and examining policy documents.\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Cresswell_InformaticsPC2014_21-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"dfa411c5fd92c37b35ba67d0a5229a9b\"><img alt=\"Fig1 Cresswell InformaticsPC2014 21-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/e\/e6\/Fig1_Cresswell_InformaticsPC2014_21-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Dimensions commonly explored in sociotechnical evaluations of health information technologies<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"What_study_designs_lend_themselves_to_sociotechnical_evaluations.3F\">What study designs lend themselves to sociotechnical evaluations?<\/span><\/h2>\n<p>The focus on investigating and exploring processes lends itself best to a naturalistic approach, but sociotechnical evaluations may also incorporate aspects of more positivist designs, particularly in the context of undertaking quasi-experimental studies, when investigators do not have direct control over the technology that is to be implemented.<sup id=\"rdp-ebb-cite_ref-GreenhalghAdop10_15-2\" class=\"reference\"><a href=\"#cite_note-GreenhalghAdop10-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/p><p>Sociotechnical evaluations should ideally be undertaken using a prospective design, as this can help to map and understand the interplay between the technology and the social context, and thereby identify important insights into how the technology is received and used.<sup id=\"rdp-ebb-cite_ref-SheikhImp11_11-1\" class=\"reference\"><a href=\"#cite_note-SheikhImp11-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RobertsonImp10_12-3\" class=\"reference\"><a href=\"#cite_note-RobertsonImp10-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WestbrookMulti07_14-2\" class=\"reference\"><a href=\"#cite_note-WestbrookMulti07-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GreenhalghAdop10_15-3\" class=\"reference\"><a href=\"#cite_note-GreenhalghAdop10-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Barberqual07_16-1\" class=\"reference\"><a href=\"#cite_note-Barberqual07-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CroweCase11_20-1\" class=\"reference\"><a href=\"#cite_note-CroweCase11-20\" rel=\"external_link\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CresswellTen13_21-0\" class=\"reference\"><a href=\"#cite_note-CresswellTen13-21\" rel=\"external_link\">[21]<\/a><\/sup>\n<\/p><p>Any appropriate design with a longitudinal dimension that allows mediating processes to be understood in detail is therefore potentially suitable for a sociotechnical evaluation. Mixed-methods sociotechnical evaluations are becoming more popular and are likely to represent an important expansion area for this research approach.<sup id=\"rdp-ebb-cite_ref-SheikhImp11_11-2\" class=\"reference\"><a href=\"#cite_note-SheikhImp11-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"What_data_to_collect_and_how_to_analyse_these\">What data to collect and how to analyse these<\/span><\/h2>\n<p>A range of qualitative and quantitative data may be gathered during sociotechnical evaluations (see Box 1). Qualitative data can help to shed light on social processes and perceived technical features such as individual attitudes and expectations (interviews and focus groups), planned organisational strategies and policies (documents), and use of technology in context (observations). These data may be complemented by quantitative work investigating the measurable impacts of technology on social systems. For instance, collecting health economic and cost data can provide insights into investment and maintenance costs, benefits, and returns on investment. Quantitative data collection can also help to explore the impact of systems on the safety and quality of care, e.g. by measuring reductions in errors and increased efficiencies associated with the move from paper-based to electronic systems.\n<\/p><p>A key distinctive feature in relation to analysis is the focus on exploring the dynamic relationship between technical and social factors over time. It is therefore important to obtain insights into technical characteristics and social processes before the introduction of the new technology into a new social context, changes to technical and social aspects once the technology is introduced, potential underlying relationships between technical and social dimensions, and the changes that occur over time as the technology becomes more embedded within the new social context. Learning across implementations can be promoted by identifying what mechanisms underlie observations and hypothesising if\/how these may be applicable to other contexts. This may be informed by a realistic evaluation perspective, assessing contexts (existing and desirable conditions for certain outcomes to be produced), mechanisms (potential causal pathways that may lead to an outcome), and outcomes (the observable effects produced).<sup id=\"rdp-ebb-cite_ref-PawsonReal97_22-0\" class=\"reference\"><a href=\"#cite_note-PawsonReal97-22\" rel=\"external_link\">[22]<\/a><\/sup> The approach may help to overcome traditional boundaries associated with the separation of technical and social spheres, as all three aspects (contexts, mechanisms, and outcomes) are neither distinctively technical nor distinctively social \u2014 they emerge out of the interplay of both.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Potential_pitfalls_and_how_to_avoid_these\">Potential pitfalls and how to avoid these<\/span><\/h2>\n<p>There are, however, a number of challenges associated with sociotechnical evaluations of health information technologies (summarised in Table 1). These stem from a lack of existing agreement on various components of a sociotechnical system, possible study designs, and data analysis strategies. They range from practical issues surrounding the management of the work itself and coping with shifting implementation timelines, to more conceptual challenges surrounding the extraction of a \u2018bottom line\u2019, and the pragmatic use of theory.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1. Summary of potential challenges and how to avoid these<\/b>\n<\/td><\/tr>\n<tr>\n<th> Challenges\n<\/th>\n<th> How to avoid these\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Large datasets and multi-disciplinary teams\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Integrating data and perspectives through robust project management allocating lead researchers to aspects of the work\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Shifting implementation landscapes and timelines\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Adapting the aims and methodologies originally envisaged, and realistic timeframes of evaluations over longer periods of time\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Linking local implementation processes to their structuring conditions\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Researching wider political and commercial processes in which local developments are taking place\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Extracting a \u2018bottom line\u2019\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Judging \u2018success\u2019 and \u2018failure\u2019 is not constructive; there is a complex story to be told but transferable lessons are possible\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Different research traditions vary significantly in the way technologies, processes, and stakeholders are conceptualised\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The pragmatic use of theory, drawing on existing theoretical frameworks\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Practical_challenges_associated_with_sociotechnical_evaluations\">Practical challenges associated with sociotechnical evaluations<\/span><\/h3>\n<p>Implementations of complex health information technologies such as electronic health records and electronic prescribing systems affect many aspects of organisational functioning and therefore tend to require complex evaluations. This is compounded with increasing organisational size and\/or if more than one organisation is studied. Specific expertise needed in the research team will vary depending on the research questions being studied and methods employed, but may include methodological (qualitative, statistical, epidemiological, and health economic), theoretical (organisational change, management, and human factors), managerial (applied team management), healthcare professional (doctors, nurses, and allied health professions), and technical (information technology specialists and system developers) expertise. Exploitation of individual strengths whilst ensuring coordinated efforts can in our experience be greatly facilitated by assigning lead researchers to individual aspects of the work investigating different impacts\/consequences (e.g. cost and changes to individual work practices), and organisations to be studied.\n<\/p><p>A further practical challenge that evaluators are likely to face is the impact of shifting implementation landscapes and timelines, potentially resulting in original methodologies having to be adapted accordingly.<sup id=\"rdp-ebb-cite_ref-RobertsonImp10_12-4\" class=\"reference\"><a href=\"#cite_note-RobertsonImp10-12\" rel=\"external_link\">[12]<\/a><\/sup> This is a common problem in evaluations of technical and\/or health policy interventions, where data collection activities depend on the planned introduction of technologies.<sup id=\"rdp-ebb-cite_ref-WestbrookMulti07_14-3\" class=\"reference\"><a href=\"#cite_note-WestbrookMulti07-14\" rel=\"external_link\">[14]<\/a><\/sup> As a result of expanding timelines, originally anticipated quantitative measurement making before, during, and after assessments of technology introduction may not be possible, and evaluators may lack opportunities to investigate the systems once they are routinely used within organisations.<sup id=\"rdp-ebb-cite_ref-SheikhImp11_11-3\" class=\"reference\"><a href=\"#cite_note-SheikhImp11-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RobertsonImp10_12-5\" class=\"reference\"><a href=\"#cite_note-RobertsonImp10-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TakianBuild12_13-1\" class=\"reference\"><a href=\"#cite_note-TakianBuild12-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WestbrookMulti07_14-4\" class=\"reference\"><a href=\"#cite_note-WestbrookMulti07-14\" rel=\"external_link\">[14]<\/a><\/sup> Longitudinal evaluative work over extended periods of time is therefore important.<sup id=\"rdp-ebb-cite_ref-Barberqual07_16-2\" class=\"reference\"><a href=\"#cite_note-Barberqual07-16\" rel=\"external_link\">[16]<\/a><\/sup> This should be characterised by early and close collaboration between evaluators and decision makers, so that changes in strategic direction and potential consequences for evaluation activities can be planned for in advance.<sup id=\"rdp-ebb-cite_ref-GreenhalghAdop10_15-4\" class=\"reference\"><a href=\"#cite_note-GreenhalghAdop10-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conceptual_challenges_associated_with_sociotechnical_evaluations\">Conceptual challenges associated with sociotechnical evaluations<\/span><\/h3>\n<p>There are also a number of conceptual challenges associated with conducting sociotechnical evaluations of complex health information technologies. The first relates to the researching of context surrounding the technology. Here, it is important to explore the use of technology by individuals, but also the wider environment in which these processes are situated (e.g. professional, organisational, and political contexts), as this can impact significantly on the way technology is adopted and changed over time.<sup id=\"rdp-ebb-cite_ref-GreenhalghTheo10_2-1\" class=\"reference\"><a href=\"#cite_note-GreenhalghTheo10-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WestbrookMulti07_14-5\" class=\"reference\"><a href=\"#cite_note-WestbrookMulti07-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GreenhalghAdop10_15-5\" class=\"reference\"><a href=\"#cite_note-GreenhalghAdop10-15\" rel=\"external_link\">[15]<\/a><\/sup> With this in mind, it is, for example, often insufficient to only explore healthcare professional perspectives on technology implementation and adoption; there is also a need to gain insights into managerial and organisational perspectives to get a more rounded understanding of implementation processes and reasons underlying strategic decisions. Similarly, broader political and commercial developments may play a role in shaping user experiences and the deployment\/design of technologies.<sup id=\"rdp-ebb-cite_ref-RobertsonImp10_12-6\" class=\"reference\"><a href=\"#cite_note-RobertsonImp10-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CresswellComp12_19-2\" class=\"reference\"><a href=\"#cite_note-CresswellComp12-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/p><p>Another common challenge facing sociotechnical evaluations is the extraction of a \u2018bottom line\u2019. Large complex processes tend to be most accurately described with the help of large complex stories, but there is also a need to ensure that messages emerging from evaluations are heard by decision makers, and this is often only possible with a limited number of straightforward key messages. In the face of this pressure, many evaluators have resorted to making bold statements surrounding the \u2018success\u2019 or \u2018failure\u2019 of evaluations of complex technologies, but in our experience, this is neither constructive in relation to future decision making nor accurate in representing reality. Evaluators need to acknowledge the existence of different notions of success as well as different temporal dimensions associated with perceived \u2018failures\u2019 (e.g. something that was initially observed as a \u2018failure\u2019 may on reflection turn out to be a \u2018success\u2019). A better way to conceptualise outputs of sociotechnical evaluations is through a summary of key lessons learned, outlining how and why these may be transferable to other settings.\n<\/p><p>The final conceptual challenge takes a more theoretical angle. It is now commonly recognised that theory can help to develop transferable lessons between settings, and there are many theories that draw on sociotechnical principles.<sup id=\"rdp-ebb-cite_ref-WackerADef98_23-0\" class=\"reference\"><a href=\"#cite_note-WackerADef98-23\" rel=\"external_link\">[23]<\/a><\/sup> However, different existing research traditions vary significantly in the way technologies, processes, and stakeholders are conceptualised. This results in a lack of existing overall framework through which implementations can be examined, although some existing approaches are summarised in Box 2.<sup id=\"rdp-ebb-cite_ref-GreenhalghTens09_24-0\" class=\"reference\"><a href=\"#cite_note-GreenhalghTens09-24\" rel=\"external_link\">[24]<\/a><\/sup> In addition, theoretical lenses are often hard to understand for non-academic audiences and lack pragmatism. This inhibits learning from experience and also widens the existing gap between academic research and frontline practice. With this in mind, it is key not to ignore the issue surrounding theory as is so often done in existing evaluations \u2014 particularly those carried out by healthcare organisations that lack the necessary expertise. More usable integrative theoretically informed evaluation frameworks are currently in development, but in the meantime, we advocate drawing on those that encompass not only social and technical dimensions, but also the wider contexts in which local developments are taking place (Box 2).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Box 2. Theoretical frameworks that encompass wider contexts in which local developments are taking place\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022 The Theory of the Diffusion of Innovations: This can help to explain why or why not certain innovations spread in and across organisations as well as how this spread may occur. Rogers E. Diffusion of Innovations. New York: Free Press, 1983.\n<p>\u2022 Normalisation Process Theory: Can help to explore how innovations become embedded within clinical practice over time and what inhibits and\/or facilitates this process. May C. A rational model for assessing and evaluating complex interventions in healthcare. <i>BMC Health Serv Res<\/i> 2006;6:86.\n<\/p><p>\u2022 Social Shaping of Technology: Can help to examine how information technology is related to and shaped by historical, cultural, and economic factors. Williams R and Edge D. The social shaping of technology. <i>Res Pol<\/i> 1996;25:865.\n<\/p><p>\u2022 HOT-fit: Can help to explore the alignment between technology, human, and organisational factors. Yusof MM, Kuljis J, Papazafeiropoulou A and Stergioulas LK. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). <i>Int J Med Inform<\/i> 2008;77:386.\n<\/p><p>\u2022 Cornford et al.\u2019s evaluation framework: Can help to explore interrelationships between system functions, human perspectives, and organisational context. Cornford T, Doukidis G and Forster D. Experience outcome with a structure, process and framework for evaluating an information system. <i>Omega, Int J Manag Sci<\/i> 1994;22:491\u2013504.\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Sociotechnical evaluations are a powerful tool to research complex technological change, particularly if the aim is to investigate non-linear relationships between technology and social processes. However, there are also challenges associated with their conduct as they involve investigating complex processes over time and within complex changing environments. The existing literature has, due to a lack of existing agreement on defining components of such work, not always fully appreciated these.\n<\/p><p>We hope that our suggested definitions, experiences, and reflections will contribute to a more integrated approach to conducting sociotechnical evaluations of technological systems in healthcare settings. Given the very considerable policy interest and substantial financial resources being expended in implementing health information technologies in an attempt to achieve the triple aims of enhancing the safety\/quality of care, improving health outcomes, and maximising the efficiency of care, we believe that greater use of sociotechnical evaluations will help to understand key processes in the interrelationships between technology, people, and organisations. This understanding will help to derive important and potentially transferable lessons.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Contributors_and_sources\">Contributors and sources<\/span><\/h2>\n<p>Aziz Sheikh conceived this work and is the guarantor. Aziz Sheikh was the principal investigator of the project discussed and is currently leading a National Institute for Health Research-funded national evaluation of electronic prescribing and medicines administration systems. He is supported by a Harkness Health Policy and Practice fellowship from The Commonwealth Fund. Kathrin M Cresswell was employed as a researcher and led on the write-up and drafting the initial version of the paper, with Aziz Sheikh commenting on various drafts.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors are very grateful to all participants who kindly gave their time and to the extended project team of the work drawn on. The authors are also grateful for the helpful comments on an earlier version of this manuscript by Dr. Allison Worth.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h2>\n<p>This work has drawn on research funded by the NHS Connecting for Health Evaluation Programme (NHS CFHEP 001, NHS CFHEP 005, NHS CFHEP 009, NHS CFHEP 010) and the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research scheme (RP-PG-1209-10099). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health. AS is supported by The Commonwealth Fund, a private independent foundation based in New York City. The views presented here are those of the author and not necessarily those of The Commonwealth Fund, its directors, officers, or staff.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interest_declaration\">Competing interest declaration<\/span><\/h2>\n<p>The authors have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<ol class=\"references\">\n<li id=\"cite_note-CoieraBuild09-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CoieraBuild09_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coiera, E. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732241\" target=\"_blank\">\"Building a National Health IT System from the middle out\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>16<\/b>: 271\u20133. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M3183\" target=\"_blank\">10.1197\/jamia.M3183<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2732241\/\" target=\"_blank\">PMC2732241<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19407078\" target=\"_blank\">19407078<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732241\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2732241<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+a+National+Health+IT+System+from+the+middle+out&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Coiera%2C+E.&rft.au=Coiera%2C+E.&rft.date=2009&rft.volume=16&rft.pages=271%E2%80%933&rft_id=info:doi\/10.1197%2Fjamia.M3183&rft_id=info:pmc\/PMC2732241&rft_id=info:pmid\/19407078&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2732241&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GreenhalghTheo10-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GreenhalghTheo10_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-GreenhalghTheo10_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Greenhalgh, T.; Stones, R. (2010). \"Theorising big IT programmes in healthcare: strong structuration theory meets actor-network theory\". <i>Social Science and Medicine<\/i> <b>70<\/b>: 1285\u201394. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.socscimed.2009.12.034\" target=\"_blank\">10.1016\/j.socscimed.2009.12.034<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20185218\" target=\"_blank\">20185218<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Theorising+big+IT+programmes+in+healthcare%3A+strong+structuration+theory+meets+actor-network+theory&rft.jtitle=Social+Science+and+Medicine&rft.aulast=Greenhalgh%2C+T.%3B+Stones%2C+R.&rft.au=Greenhalgh%2C+T.%3B+Stones%2C+R.&rft.date=2010&rft.volume=70&rft.pages=1285%E2%80%9394&rft_id=info:doi\/10.1016%2Fj.socscimed.2009.12.034&rft_id=info:pmid\/20185218&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CatwellEval09-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CatwellEval09_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-CatwellEval09_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Catwell, L.; Sheikh, A. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2719100\" target=\"_blank\">\"Evaluating eHealth interven-tions: the need for continuous systemic evaluation\"<\/a>. <i>PLoS Medicine<\/i> <b>6<\/b>: e1000126. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1000126\" target=\"_blank\">10.1371\/journal.pmed.1000126<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2719100\/\" target=\"_blank\">PMC2719100<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19688038\" target=\"_blank\">19688038<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2719100\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2719100<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluating+eHealth+interven-tions%3A+the+need+for+continuous+systemic+evaluation&rft.jtitle=PLoS+Medicine&rft.aulast=Catwell%2C+L.%3B+Sheikh%2C+A.&rft.au=Catwell%2C+L.%3B+Sheikh%2C+A.&rft.date=2009&rft.volume=6&rft.pages=e1000126&rft_id=info:doi\/10.1371%2Fjournal.pmed.1000126&rft_id=info:pmc\/PMC2719100&rft_id=info:pmid\/19688038&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2719100&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BergICT03-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BergICT03_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-BergICT03_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berg, M.; Aarts, J.; van der Lei, J. (2003). \"ICT in health care: socio-technical approaches\". <i>Methods of Information in Medicine<\/i> <b>42<\/b>: 297\u2013301. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/14534625\" target=\"_blank\">14534625<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ICT+in+health+care%3A+socio-technical+approaches&rft.jtitle=Methods+of+Information+in+Medicine&rft.aulast=Berg%2C+M.%3B+Aarts%2C+J.%3B+van+der+Lei%2C+J.&rft.au=Berg%2C+M.%3B+Aarts%2C+J.%3B+van+der+Lei%2C+J.&rft.date=2003&rft.volume=42&rft.pages=297%E2%80%93301&rft_id=info:pmid\/14534625&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CresswellActor10-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CresswellActor10_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-CresswellActor10_5-1\" rel=\"external_link\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-CresswellActor10_5-2\" rel=\"external_link\">5.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cresswell, K.; Worth, A.; Sheikh, A. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2988706\" target=\"_blank\">\"Actor-network theory and its role in understanding the implementation of information technology developments in healthcare\"<\/a>. <i>BMC Medical Informatics and Decision Making<\/i> <b>10<\/b>: 67. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-10-67\" target=\"_blank\">10.1186\/1472-6947-10-67<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2988706\/\" target=\"_blank\">PMC2988706<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21040575\" target=\"_blank\">21040575<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2988706\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2988706<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Actor-network+theory+and+its+role+in+understanding+the+implementation+of+information+technology+developments+in+healthcare&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Cresswell%2C+K.%3B+Worth%2C+A.%3B+Sheikh%2C+A.&rft.au=Cresswell%2C+K.%3B+Worth%2C+A.%3B+Sheikh%2C+A.&rft.date=2010&rft.volume=10&rft.pages=67&rft_id=info:doi\/10.1186%2F1472-6947-10-67&rft_id=info:pmc\/PMC2988706&rft_id=info:pmid\/21040575&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2988706&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IoMHealth11-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-IoMHealth11_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-IoMHealth11_6-1\" rel=\"external_link\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.iom.edu\/~\/media\/Files\/Report%20Files\/2011\/Health-IT\/HIT%20and%20Patient%20Safety.pdf\" target=\"_blank\">\"Health IT and Patient Safety: Building Safer Systems for Better Care\"<\/a> (PDF). Institute of Medicine. 2011<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.iom.edu\/~\/media\/Files\/Report%20Files\/2011\/Health-IT\/HIT%20and%20Patient%20Safety.pdf\" target=\"_blank\">http:\/\/www.iom.edu\/~\/media\/Files\/Report%20Files\/2011\/Health-IT\/HIT%20and%20Patient%20Safety.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 16 July 2013<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Health+IT+and+Patient+Safety%3A+Building+Safer+Systems+for+Better+Care&rft.atitle=&rft.date=2011&rft.pub=Institute+of+Medicine&rft_id=http%3A%2F%2Fwww.iom.edu%2F%7E%2Fmedia%2FFiles%2FReport%2520Files%2F2011%2FHealth-IT%2FHIT%2520and%2520Patient%2520Safety.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChernsPrinc87-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ChernsPrinc87_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-ChernsPrinc87_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cherns, A. (1987). \"Principles of sociotechnical design revisited\". <i>Human Relations<\/i> <b>40<\/b>: 153\u201361. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F001872678704000303\" target=\"_blank\">10.1177\/001872678704000303<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Principles+of+sociotechnical+design+revisited&rft.jtitle=Human+Relations&rft.aulast=Cherns%2C+A.&rft.au=Cherns%2C+A.&rft.date=1987&rft.volume=40&rft.pages=153%E2%80%9361&rft_id=info:doi\/10.1177%2F001872678704000303&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Clegg-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Clegg_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-Clegg_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Clegg, C.W. (2000). \"Sociotechnical principles for system design\". <i>Applied Ergonomics<\/i> <b>31<\/b>: 463\u201377. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0003-6870%2800%2900009-0\" target=\"_blank\">10.1016\/S0003-6870(00)00009-0<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sociotechnical+principles+for+system+design&rft.jtitle=Applied+Ergonomics&rft.aulast=Clegg%2C+C.W.&rft.au=Clegg%2C+C.W.&rft.date=2000&rft.volume=31&rft.pages=463%E2%80%9377&rft_id=info:doi\/10.1016%2FS0003-6870%2800%2900009-0&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarrisonUn07-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HarrisonUn07_9-0\" rel=\"external_link\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-HarrisonUn07_9-1\" rel=\"external_link\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Harrison, M.I.; Koppel, R.; Bar-Lev, S. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1975796\" target=\"_blank\">\"Unintended consequences of information technologies in health care\u2014an interactive socio-technical analysis\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>14<\/b>: 542\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2384\" target=\"_blank\">10.1197\/jamia.M2384<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1975796\/\" target=\"_blank\">PMC1975796<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17600093\" target=\"_blank\">17600093<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1975796\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1975796<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Unintended+consequences+of+information+technologies+in+health+care%E2%80%94an+interactive+socio-technical+analysis&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Harrison%2C+M.I.%3B+Koppel%2C+R.%3B+Bar-Lev%2C+S.&rft.au=Harrison%2C+M.I.%3B+Koppel%2C+R.%3B+Bar-Lev%2C+S.&rft.date=2007&rft.volume=14&rft.pages=542%E2%80%939&rft_id=info:doi\/10.1197%2Fjamia.M2384&rft_id=info:pmc\/PMC1975796&rft_id=info:pmid\/17600093&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1975796&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlackThe09-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlackThe09_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Black, A.D.; Car, J.; Pagliari, C.; Anandan, C.; Cresswell, K.; Bokun, T.; et al. (2011). \"The impact of eHealth on the quality and safety of health care: a systematic overview\". <i>PLoS Medicine<\/i> <b>8<\/b>: e1000387.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+eHealth+on+the+quality+and+safety+of+health+care%3A+a+systematic+overview&rft.jtitle=PLoS+Medicine&rft.aulast=Black%2C+A.D.%3B+Car%2C+J.%3B+Pagliari%2C+C.%3B+Anandan%2C+C.%3B+Cresswell%2C+K.%3B+Bokun%2C+T.%3B+et+al.&rft.au=Black%2C+A.D.%3B+Car%2C+J.%3B+Pagliari%2C+C.%3B+Anandan%2C+C.%3B+Cresswell%2C+K.%3B+Bokun%2C+T.%3B+et+al.&rft.date=2011&rft.volume=8&rft.pages=e1000387&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SheikhImp11-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SheikhImp11_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-SheikhImp11_11-1\" rel=\"external_link\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-SheikhImp11_11-2\" rel=\"external_link\">11.2<\/a><\/sup> <sup><a href=\"#cite_ref-SheikhImp11_11-3\" rel=\"external_link\">11.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sheikh, A.; Cornford, T.; Barber, N.; Avery, A.; Takian, A.; Lichtner, V.; et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3195310\" target=\"_blank\">\"Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from a prospective national evaluation in \u201cearly adopter\u201d hospitals\"<\/a>. <i>British Medical Journal<\/i> <b>343<\/b>: d6054. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.d6054\" target=\"_blank\">10.1136\/bmj.d6054<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3195310\/\" target=\"_blank\">PMC3195310<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22006942\" target=\"_blank\">22006942<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3195310\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3195310<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementation+and+adoption+of+nationwide+electronic+health+records+in+secondary+care+in+England%3A+final+qualitative+results+from+a+prospective+national+evaluation+in+%E2%80%9Cearly+adopter%E2%80%9D+hospitals&rft.jtitle=British+Medical+Journal&rft.aulast=Sheikh%2C+A.%3B+Cornford%2C+T.%3B+Barber%2C+N.%3B+Avery%2C+A.%3B+Takian%2C+A.%3B+Lichtner%2C+V.%3B+et+al.&rft.au=Sheikh%2C+A.%3B+Cornford%2C+T.%3B+Barber%2C+N.%3B+Avery%2C+A.%3B+Takian%2C+A.%3B+Lichtner%2C+V.%3B+et+al.&rft.date=2011&rft.volume=343&rft.pages=d6054&rft_id=info:doi\/10.1136%2Fbmj.d6054&rft_id=info:pmc\/PMC3195310&rft_id=info:pmid\/22006942&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3195310&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RobertsonImp10-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RobertsonImp10_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-RobertsonImp10_12-1\" rel=\"external_link\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-RobertsonImp10_12-2\" rel=\"external_link\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-RobertsonImp10_12-3\" rel=\"external_link\">12.3<\/a><\/sup> <sup><a href=\"#cite_ref-RobertsonImp10_12-4\" rel=\"external_link\">12.4<\/a><\/sup> <sup><a href=\"#cite_ref-RobertsonImp10_12-5\" rel=\"external_link\">12.5<\/a><\/sup> <sup><a href=\"#cite_ref-RobertsonImp10_12-6\" rel=\"external_link\">12.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Robertson, A.; Cresswell, K.; Takian, A.; Petrakaki, D.; Crowe, S.; Cornford, T.; et al. (2010). \"Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation\". <i>British Medical Journal<\/i> <b>341<\/b>: c4564.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementation+and+adoption+of+nationwide+electronic+health+records+in+secondary+care+in+England%3A+qualitative+analysis+of+interim+results+from+a+prospective+national+evaluation&rft.jtitle=British+Medical+Journal&rft.aulast=Robertson%2C+A.%3B+Cresswell%2C+K.%3B+Takian%2C+A.%3B+Petrakaki%2C+D.%3B+Crowe%2C+S.%3B+Cornford%2C+T.%3B+et+al.&rft.au=Robertson%2C+A.%3B+Cresswell%2C+K.%3B+Takian%2C+A.%3B+Petrakaki%2C+D.%3B+Crowe%2C+S.%3B+Cornford%2C+T.%3B+et+al.&rft.date=2010&rft.volume=341&rft.pages=c4564&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TakianBuild12-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TakianBuild12_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-TakianBuild12_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Takian, A.; Petrakaki, D.; Cornford, T.; Sheikh, A.; Barber, N. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3469374\" target=\"_blank\">\"Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems\"<\/a>. <i>BMC Health Services Research<\/i> <b>12<\/b>: 105\u201329. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6963-12-105\" target=\"_blank\">10.1186\/1472-6963-12-105<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3469374\/\" target=\"_blank\">PMC3469374<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22545646\" target=\"_blank\">22545646<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3469374\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3469374<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Building+a+house+on+shifting+sand%3A+methodological+considerations+when+evaluating+the+implementation+and+adoption+of+national+electronic+health+record+systems&rft.jtitle=BMC+Health+Services+Research&rft.aulast=Takian%2C+A.%3B+Petrakaki%2C+D.%3B+Cornford%2C+T.%3B+Sheikh%2C+A.%3B+Barber%2C+N.&rft.au=Takian%2C+A.%3B+Petrakaki%2C+D.%3B+Cornford%2C+T.%3B+Sheikh%2C+A.%3B+Barber%2C+N.&rft.date=2012&rft.volume=12&rft.pages=105%E2%80%9329&rft_id=info:doi\/10.1186%2F1472-6963-12-105&rft_id=info:pmc\/PMC3469374&rft_id=info:pmid\/22545646&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3469374&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WestbrookMulti07-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WestbrookMulti07_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-WestbrookMulti07_14-1\" rel=\"external_link\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-WestbrookMulti07_14-2\" rel=\"external_link\">14.2<\/a><\/sup> <sup><a href=\"#cite_ref-WestbrookMulti07_14-3\" rel=\"external_link\">14.3<\/a><\/sup> <sup><a href=\"#cite_ref-WestbrookMulti07_14-4\" rel=\"external_link\">14.4<\/a><\/sup> <sup><a href=\"#cite_ref-WestbrookMulti07_14-5\" rel=\"external_link\">14.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Westbrook, J.; Braithwaite, J.; Georgiou, A.; Ampt, A.; Creswick, A.; Coiera, E.; et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213479\" target=\"_blank\">\"Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>14<\/b>: 746\u201355. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2462\" target=\"_blank\">10.1197\/jamia.M2462<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2213479\/\" target=\"_blank\">PMC2213479<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17712083\" target=\"_blank\">17712083<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213479\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2213479<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multimethod+evaluation+of+information+and+communication+technologies+in+health+in+the+context+of+wicked+problems+and+sociotechnical+theory&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Westbrook%2C+J.%3B+Braithwaite%2C+J.%3B+Georgiou%2C+A.%3B+Ampt%2C+A.%3B+Creswick%2C+A.%3B+Coiera%2C+E.%3B+et+al.&rft.au=Westbrook%2C+J.%3B+Braithwaite%2C+J.%3B+Georgiou%2C+A.%3B+Ampt%2C+A.%3B+Creswick%2C+A.%3B+Coiera%2C+E.%3B+et+al.&rft.date=2007&rft.volume=14&rft.pages=746%E2%80%9355&rft_id=info:doi\/10.1197%2Fjamia.M2462&rft_id=info:pmc\/PMC2213479&rft_id=info:pmid\/17712083&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2213479&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GreenhalghAdop10-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GreenhalghAdop10_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-GreenhalghAdop10_15-1\" rel=\"external_link\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-GreenhalghAdop10_15-2\" rel=\"external_link\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-GreenhalghAdop10_15-3\" rel=\"external_link\">15.3<\/a><\/sup> <sup><a href=\"#cite_ref-GreenhalghAdop10_15-4\" rel=\"external_link\">15.4<\/a><\/sup> <sup><a href=\"#cite_ref-GreenhalghAdop10_15-5\" rel=\"external_link\">15.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Greenhalgh, T.; Hinder, S.; Stramer, K.; Bratan, T.; Russell, J. (2010). \"Adoption, non-adoption, and abandonment of a personal electronic health record: case study of HealthSpace\". <i>British Medical Journal<\/i> <b>341<\/b>: c5814.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adoption%2C+non-adoption%2C+and+abandonment+of+a+personal+electronic+health+record%3A+case+study+of+HealthSpace&rft.jtitle=British+Medical+Journal&rft.aulast=Greenhalgh%2C+T.%3B+Hinder%2C+S.%3B+Stramer%2C+K.%3B+Bratan%2C+T.%3B+Russell%2C+J.&rft.au=Greenhalgh%2C+T.%3B+Hinder%2C+S.%3B+Stramer%2C+K.%3B+Bratan%2C+T.%3B+Russell%2C+J.&rft.date=2010&rft.volume=341&rft.pages=c5814&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Barberqual07-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Barberqual07_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-Barberqual07_16-1\" rel=\"external_link\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-Barberqual07_16-2\" rel=\"external_link\">16.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barber, N.; Cornford, T.; Klecun, E. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2464937\" target=\"_blank\">\"Qualitative evaluation of an electronic prescribing and administration system\"<\/a>. <i>Quality and Safety in Health Care<\/i> <b>16<\/b>: 271\u20138. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fqshc.2006.019505\" target=\"_blank\">10.1136\/qshc.2006.019505<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2464937\/\" target=\"_blank\">PMC2464937<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17693675\" target=\"_blank\">17693675<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2464937\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2464937<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Qualitative+evaluation+of+an+electronic+prescribing+and+administration+system&rft.jtitle=Quality+and+Safety+in+Health+Care&rft.aulast=Barber%2C+N.%3B+Cornford%2C+T.%3B+Klecun%2C+E.&rft.au=Barber%2C+N.%3B+Cornford%2C+T.%3B+Klecun%2C+E.&rft.date=2007&rft.volume=16&rft.pages=271%E2%80%938&rft_id=info:doi\/10.1136%2Fqshc.2006.019505&rft_id=info:pmc\/PMC2464937&rft_id=info:pmid\/17693675&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2464937&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BergPatient99-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BergPatient99_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berg, M. (1999). \"Patient care information systems and health care work: a sociotechnical approach\". <i>International Journal of Medical Informatics<\/i> <b>55<\/b>: 87\u2013101. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1386-5056%2899%2900011-8\" target=\"_blank\">10.1016\/S1386-5056(99)00011-8<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patient+care+information+systems+and+health+care+work%3A+a+sociotechnical+approach&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Berg%2C+M.&rft.au=Berg%2C+M.&rft.date=1999&rft.volume=55&rft.pages=87%E2%80%93101&rft_id=info:doi\/10.1016%2FS1386-5056%2899%2900011-8&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CornfordExp94-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CornfordExp94_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cornford, T.; Doukidis, G.; Forster, D. (1994). \"Experience outcome with a structure, process and framework for evaluating an information system\". <i>Omega, International Journal of Management Science<\/i> <b>22<\/b>: 491\u2013504. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2F0305-0483%2894%2990030-2\" target=\"_blank\">10.1016\/0305-0483(94)90030-2<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Experience+outcome+with+a+structure%2C+process+and+framework+for+evaluating+an+information+system&rft.jtitle=Omega%2C+International+Journal+of+Management+Science&rft.aulast=Cornford%2C+T.%3B+Doukidis%2C+G.%3B+Forster%2C+D.&rft.au=Cornford%2C+T.%3B+Doukidis%2C+G.%3B+Forster%2C+D.&rft.date=1994&rft.volume=22&rft.pages=491%E2%80%93504&rft_id=info:doi\/10.1016%2F0305-0483%2894%2990030-2&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CresswellComp12-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CresswellComp12_19-0\" rel=\"external_link\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-CresswellComp12_19-1\" rel=\"external_link\">19.1<\/a><\/sup> <sup><a href=\"#cite_ref-CresswellComp12_19-2\" rel=\"external_link\">19.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cresswell, K.; Worth, A.; Sheikh, A. (2012). \"Comparative case study investigating sociotechnical processes of change in the con-text of a national electronic health record implementation\". <i>Health Informatics Journal<\/i> <b>18<\/b>: 251\u201370. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1460458212445399\" target=\"_blank\">10.1177\/1460458212445399<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23257056\" target=\"_blank\">23257056<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparative+case+study+investigating+sociotechnical+processes+of+change+in+the+con-text+of+a+national+electronic+health+record+implementation&rft.jtitle=Health+Informatics+Journal&rft.aulast=Cresswell%2C+K.%3B+Worth%2C+A.%3B+Sheikh%2C+A.&rft.au=Cresswell%2C+K.%3B+Worth%2C+A.%3B+Sheikh%2C+A.&rft.date=2012&rft.volume=18&rft.pages=251%E2%80%9370&rft_id=info:doi\/10.1177%2F1460458212445399&rft_id=info:pmid\/23257056&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CroweCase11-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CroweCase11_20-0\" rel=\"external_link\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-CroweCase11_20-1\" rel=\"external_link\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Crowe, C.; Cresswell, K.; Robertson, R.; Huby, G.; Avery, A.; Sheikh, A. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3141799\" target=\"_blank\">\"The case study approach\"<\/a>. <i>BMC Medical Research Methodology<\/i> <b>11<\/b>: 100. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2288-11-100\" target=\"_blank\">10.1186\/1471-2288-11-100<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3141799\/\" target=\"_blank\">PMC3141799<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21707982\" target=\"_blank\">21707982<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3141799\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3141799<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+case+study+approach&rft.jtitle=BMC+Medical+Research+Methodology&rft.aulast=Crowe%2C+C.%3B+Cresswell%2C+K.%3B+Robertson%2C+R.%3B+Huby%2C+G.%3B+Avery%2C+A.%3B+Sheikh%2C+A.&rft.au=Crowe%2C+C.%3B+Cresswell%2C+K.%3B+Robertson%2C+R.%3B+Huby%2C+G.%3B+Avery%2C+A.%3B+Sheikh%2C+A.&rft.date=2011&rft.volume=11&rft.pages=100&rft_id=info:doi\/10.1186%2F1471-2288-11-100&rft_id=info:pmc\/PMC3141799&rft_id=info:pmid\/21707982&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3141799&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CresswellTen13-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CresswellTen13_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cresswell, K.M.; Bates, D.W.; Sheikh, A. (2013). \"Ten key considerations for the successful implementation and adoption of large-scale health information technology\". <i>Journal of the American Medical Informatics Association<\/i> <b>20<\/b>: e9\u201313.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ten+key+considerations+for+the+successful+implementation+and+adoption+of+large-scale+health+information+technology&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Cresswell%2C+K.M.%3B+Bates%2C+D.W.%3B+Sheikh%2C+A.&rft.au=Cresswell%2C+K.M.%3B+Bates%2C+D.W.%3B+Sheikh%2C+A.&rft.date=2013&rft.volume=20&rft.pages=e9%E2%80%9313&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PawsonReal97-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PawsonReal97_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Pawson, R.; Tilley, N. (1997). <i>Realistic Evaluation<\/i>. Sage Publications.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Realistic+Evaluation&rft.aulast=Pawson%2C+R.%3B+Tilley%2C+N.&rft.au=Pawson%2C+R.%3B+Tilley%2C+N.&rft.date=1997&rft.pub=Sage+Publications&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WackerADef98-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WackerADef98_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wacker, J.G. (1998). \"A definition of theory: research guidelines for different theory-building research methods in operations management\". <i>Journal of Operations Management<\/i> <b>16<\/b>: 361\u201385. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0272-6963%2898%2900019-9\" target=\"_blank\">10.1016\/S0272-6963(98)00019-9<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+definition+of+theory%3A+research+guidelines+for+different+theory-building+research+methods+in+operations+management&rft.jtitle=Journal+of+Operations+Management&rft.aulast=Wacker%2C+J.G.&rft.au=Wacker%2C+J.G.&rft.date=1998&rft.volume=16&rft.pages=361%E2%80%9385&rft_id=info:doi\/10.1016%2FS0272-6963%2898%2900019-9&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GreenhalghTens09-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GreenhalghTens09_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Greenhalgh, T.; Potts, H.; Wong, G.; Bark, P.; Swinglehurst, D. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2888022\" target=\"_blank\">\"Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method\"<\/a>. <i>Milbank Quarterly<\/i> <b>87<\/b>: 729\u201388. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1468-0009.2009.00578.x\" target=\"_blank\">10.1111\/j.1468-0009.2009.00578.x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2888022\/\" target=\"_blank\">PMC2888022<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20021585\" target=\"_blank\">20021585<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2888022\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2888022<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Tensions+and+paradoxes+in+electronic+patient+record+research%3A+a+systematic+literature+review+using+the+meta-narrative+method&rft.jtitle=Milbank+Quarterly&rft.aulast=Greenhalgh%2C+T.%3B+Potts%2C+H.%3B+Wong%2C+G.%3B+Bark%2C+P.%3B+Swinglehurst%2C+D.&rft.au=Greenhalgh%2C+T.%3B+Potts%2C+H.%3B+Wong%2C+G.%3B+Bark%2C+P.%3B+Swinglehurst%2C+D.&rft.date=2009&rft.volume=87&rft.pages=729%E2%80%9388&rft_id=info:doi\/10.1111%2Fj.1468-0009.2009.00578.x&rft_id=info:pmc\/PMC2888022&rft_id=info:pmid\/20021585&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2888022&rfr_id=info:sid\/en.wikipedia.org:Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210234\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.638 seconds\nReal time usage: 0.672 seconds\nPreprocessor visited node count: 20159\/1000000\nPreprocessor generated node count: 37027\/1000000\nPost\u2010expand include size: 165151\/2097152 bytes\nTemplate argument size: 56156\/2097152 bytes\nHighest expansion depth: 17\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 621.998 1 - -total\n 74.42% 462.887 24 - Template:Citation\/core\n 73.61% 457.882 22 - Template:Cite_journal\n 10.19% 63.399 1 - Template:Infobox_journal_article\n 9.64% 59.977 1 - Template:Infobox\n 9.36% 58.244 40 - Template:Citation\/identifier\n 5.70% 35.442 80 - Template:Infobox\/row\n 4.60% 28.618 24 - Template:Citation\/make_link\n 3.54% 22.032 90 - Template:Hide_in_print\n 3.21% 19.940 1 - Template:Cite_book\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7638-0!*!0!!en!5!* and timestamp 20181213210233 and revision id 23422\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies\">https:\/\/www.limswiki.org\/index.php\/Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","302bc46eb57fbea71e5d7745898c7261_images":["https:\/\/www.limswiki.org\/images\/e\/e6\/Fig1_Cresswell_InformaticsPC2014_21-2.jpg"],"302bc46eb57fbea71e5d7745898c7261_timestamp":1544734953,"5c6e2bf767feb103119262aff1c6473b_type":"article","5c6e2bf767feb103119262aff1c6473b_title":"The evolution, use, and effects of integrated personal health records: A narrative review (Zieth et al. 2014)","5c6e2bf767feb103119262aff1c6473b_url":"https:\/\/www.limswiki.org\/index.php\/Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review","5c6e2bf767feb103119262aff1c6473b_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:The evolution, use, and effects of integrated personal health records: A narrative review\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nThe evolution, use, and effects of integrated personal health records: A narrative reviewJournal\n \nelectronic Journal of Health InformaticsAuthor(s)\n \nZieth, Caroline R.; Chia, Lichun Rebecca; Roberts, Mark S.; Fischer, Gary S.; Clark, Sunday; Weimer, Melissa; Hess, RachelAuthor affiliation(s)\n \nCenter for Research on Health Care, University of Pittsburgh; University of Pittsburgh School of Medicine;\r\nWeil Cornell Medical College; University of Pittsburgh Graduate School of Public HealthPrimary contact\n \nCaroline Zieth - Email: ziethcr@upmc.eduYear published\n \n2014Volume and issue\n \n8 (2)Page(s)\n \ne17DOI\n \nNoneISSN\n \n1446-4381Distribution license\n \nCreative Commons Attribution-NonCommercial-ShareAlike 3.0Website\n \nhttp:\/\/www.ejhi.net\/ojs\/index.php\/ejhi\/article\/view\/247Download\n \nhttp:\/\/www.ejhi.net\/ojs\/index.php\/ejhi\/article\/download\/247\/169 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods \n4 Results \n\n4.1 Evolution and adoption of PHRs \n4.2 Attitudes toward adoption and use of PHRs \n4.3 Barriers toward adoption and use of PHRs \n4.4 Role of PHRs in self-management \n\n\n5 Discussion \n6 Conclusion \n7 Conflicts of interest \n8 References \n9 Notes \n\n\n\nAbstract \nObjective: To present a summarized literature review of the evolution, use, and effects of Personal Health Records (PHRs). \nMethods: Medline and PubMed were searched for \u2018personal health records\u2019. Seven hundred thirty-three references were initially screened resulting in 230 studies selected as relevant based on initial title and abstract review. After further review, a total of 52 articles provided relevant information and were included in this paper. These articles were reviewed by one author and grouped into the following categories: PHR evolution and adoption, patient user attitudes toward PHRs, patient reported barriers to use, and the role of PHRs in self-management. \nResults: Eleven papers described evolution and adoption, 17 papers described PHR user attitudes, 10 papers described barriers to use, and 11 papers described PHR use in self-management. Three papers were not grouped into a category but were used to inform the Discussion. PHRs have evolved from patient-maintained paper health records to provider-linked electronic health records. Patients report enthusiasm for the potential of modern PHRs, yet few patients actually use an electronic PHR. Low patient adoption of PHRs is associated with poor interface design and low health and computer literacy on the part of patient users. \nConclusion: PHR systems that account for patient\u2019s needs and skills can facilitate their adoption. Common barriers are avoidable when patients receive adequate guidance on useful features as well as technical support. When implemented effectively, PHRs can increase patient participation in health management, and improve patient-physician communication and health related decision making.\nKeywords: Personal Health Records; Electronic Health Records; Electronic Medical Records\n\nIntroduction \nInvolving patients in their health care using information technology (IT) such as interoperable personal health records (PHRs) may increase healthcare efficiency and improve quality while reducing medical errors.[1] The idea behind a PHR has existed for decades.[2][3] Initially, PHRs were in the form of paper medical records created and maintained by patients and used to augment healthcare provided in person.[4] These records, compiled and stored on paper by patients and families, allowed individuals to preserve their complete medical history.[3] As IT evolved, patients and families began to electronically store many health-related documents, including clinical notes from different health care providers, laboratory test results, and medication prescription records, essentially creating electronic versions of paper PHRs.[3][4]\nEarly patient-initiated PHRs have evolved into a wide variety of computer-based applications that allow patients to securely store health-related information such as laboratory test results; these can be maintained solely by patients or by both patients and clinicians.[3][5] Currently, employers, healthcare providers, and third party organizations have deployed a variety of electronic PHRs differing in architecture and function.[6][7] These PHRs range from original, stand-alone applications where patients enter the majority of their medical information to those integrated into the clinical health record.[4] In beginning to create standards for PHRs, the Markle Foundation\u2019s Connecting for Health Collaborative defined a PHR as \u201can electronic application through which individuals can access, manage and share their health information, and that of others for whom they are authorized, in a private, secure, and confidential environment\".[8] Currently, an estimated 70 million insured patients have access to some form of electronic PHR and those who do not would like access.[9][10]\nWhen integrated with provider-maintained electronic health records (EHRs), PHRs are electronically linked to clinical information in the EHRs, which are repositories of all electronically available patient medical information from multiple sources, and which are updated by health care providers.[11][12] With integrated PHRs, patients can view automatically populated medical information, such as laboratory test results.[4][6][11] They have advantages over stand-alone PHRs by allowing securely linked patient-provider communication outside traditional clinical encounters. Integrated PHRs provide a means to create a shared patient record through evolving features including patient-physician collaborative tools and interactive decision-making tools, personalized management tools for chronic conditions, integrated and linked health information resources, and patient-entered information.[3][4][11][13][14][15][16]\nTo inform healthcare decision-making, PHR development, and future research on clinical outcomes, this literature review summarizes the evolution, use, and effects of PHRs, with a focus on integrated PHRs. Four broad areas are address: (1) the characteristics of PHR use, including their evolution and adoption, (2) patient and provider attitudes toward PHRs, (3) barriers to PHR adoption and use, and (4) the effects of PHRs on patient management and outcomes.\n\nMethods \nMedline and PubMed were searched to identify English-language articles focused on PHRs and published between 1970 and 2011. Studies were included based on the content of titles and abstracts, removing those for which the abbreviation PHR did not stand for \u2018personal health record\u2019. Of the 733 retrieved references initially identified in the search, 230 studies were selected as relevant based on initial title and abstract. Two independent reviewers (LRC and RH) screened these and a single reviewer (CRZ) verified the studies contained information regarding PHR evolution, consumer attitudes, barriers to use, and\/or self-management through in-depth investigation of study content. Information from the final 52 papers, representing both qualitative and quantitative studies, was placed into four main categories: (1) the characteristics of PHR use, including their evolution and adoption, (2) patient and provider attitudes toward PHRs, (3) barriers to PHR adoption and use, and (4) the effects of PHRs on patient management and outcomes. Papers could contribute information to multiple categories (Tables 1-3).\n\nResults \nEvolution and adoption of PHRs \nEleven papers described the evolution of PHRs and the settings in which they are used. Early examples of PHRs were paper-based and patient-maintained.[3] Use of paper-based PHRs has continued even after computerized information systems had become available. For example, parents have routinely collected their children\u2019s basic medical information and tracked their child\u2019s development and immunizations using baby books or have carried wallet cards containing basic personal medical data (i.e., emergency contacts, blood types and allergy information).[3][17] Patients continue to keep some paper records despite the rise of electronic PHRs.[18] Currently almost half (42%) of Americans keep some form of a PHR, defined in this article as any single place where medical information is kept, and the majority (87%) are paper-based.[19]\nAs IT developed, patients began creating digital rather than paper records. Electronic PHRs evolved as patients started entering their health information into computer-based applications.[3][4] PHR functionality expanded to give patients the ability to view personal health information stored in their health care provider\u2019s records.[4] Web-based PHRs originated in the emergency room and included online emergency medical records.[3][5] As practice and hospital-based EHRs evolved, they merged with PHRs and have become a major source of the information contained within integrated PHRs. Patients now have access to integrated PHRs through their insurers or healthcare providers; however, patient adoption of PHRs has lagged behind this access. Preliminary estimates from the National Ambulatory Medical Care Survey (NAMCS) found that 51% of physicians reported providing patients with access to an integrated or partially integrated PHR (i.e. an electronic medical record (EMR) or EHR linked PHR).[20] In a 2010 Consumer and Health Information Technology survey, however, only 7% of Americans reported having used either a stand-alone or an integrated PHR; a 2008 Markle Foundation survey reported fewer than 3%.[13][21][22][23]\n\nAttitudes toward adoption and use of PHRs \nSeventeen papers described patient or provider attitudes toward electronic and paper-based PHRs (Table 1). Patients are eager to use PHRs for their potential to improve health care delivery and outcomes but these positive attitudes do not translate into use.[24][25][26][27][28][29] Patients have mixed reasons for using or not using a PHR some of which are tied to their motivation to improve health outcomes and their relationship with their physician. Understanding patient motivation is important, particularly when designing and adopting PHRs.[24] Patients seek the ability to control access to their health information and believe they should have access.[30][31] Patients view integrated PHRs favorably with one report finding that 60% of patients indicate they would use an integrated PHR to look up test results and record their medication, and another survey finding that 75% of patients would communicate with physicians electronically if given the option.[10][13] Patients\u2019 motivation to participate in their health care fosters their interest in viewing their PHRs and viewing PHRs influences patients\u2019 care-related decision-making.[24][27][32] Diabetic patient users of an integrated PHR reported receiving care more quickly, and connecting with their doctor more easily.[15] An integrated PHR with features such as secure patient-physician messaging, medication history updating, and online requests for medication renewals was highly valued by elderly and disabled patients, patients with chronic conditions, and middle-aged female patients.[33][34][35] The ability to contact health care providers through secure messaging in an integrated PHR provided a feeling of security for patients in the Netherlands.[26] Patients want to view their records in order to have detailed information about their health, and those using an integrated PHR reported feeling more in control of their chronic conditions and a sense of illness-ownership, which motivated them to contribute information to their EHR.[24][27][36] Patients\u2019 satisfaction with their physicians influenced their use of an integrated PHR. Patients expressing satisfaction with their patient-provider relationship were less likely to use an integrated PHR than patients expressing dissatisfaction.[25] Those expressing dissatisfaction viewed access to their PHR as a means of gaining knowledge or control over their health. \nFive studies reported on ease of use for patients accessing an integrated PHR.[26][28][33][34][35] More than 60% of patients with head or neck cancer in the Netherlands and the majority of middle-aged adult patients in the United Kingdom found an integrated PHR easy to navigate.[26][28] Female patients who used an integrated PHR rated various functions easier to use than males.[33] Several studies, however, reported that patients did not maintain health information in their PHRs despite ease of use. Elderly patients found value in using an integrated PHR for updating medications, health problems, and lab test information, yet failed to annotate certain health information such as immunizations and laboratory test results, which the authors attributed to difficulties with the user interface of the integrated PHR.[34]\n\r\n\n\nTable 1: Attitudes toward Adoption and Use of PHR\n\n\n\n\n\n\n Authors\/Year\n\n Population\n\n Design\n\n Results\n\n Implications According to Authors\n\n\nKrist et al. (2012)[13]\n\nn=4,500, patient users of an interactive PHR (IPHR) from 8 primary care practices\n\nEMR and survey data were analyzed to determine IPHR effectiveness\n\n\u2022Patients (25% ) who used the IPHR were up-to-date on all services; double the rate for non-users\n\nPractices need to inform patients about PHR\u2019s value and relevance to care\n\n\nWitry et al. (2010)[16]\n\nn=28, physicians from 4 family medicine practices\n\n4 focus groups to explore physician views of PHRs\n\n\u2022Physicians thought PHRs beneficial for certain patient populations (e.g., mobile populations)\r\n\u2022Physicians expressed doubt about the likelihood of patients updating their PHRs\n\nProviders are unfamiliar with electronic PHRs; physician use of electronic PHRs may increase with exposure\n\n\nTobacman et al. (1996) [17]\n\nn=100, patient users of a standalone PHR\n\nTelephone questionnaire to determine acceptance, usefulness, and impact of PHR\n\n\u2022Adult patients were interested in PHRs\n\nThere is an unmet need among patients for PHR access\n\n\nWinkelman et al. (2005)[24]\n\n=12, inflammatory bowel disease patient users of an EMR linked PHR in Canada\n\nQualitative study using in-depth interviews and focus groups\n\n\u2022PHR access promoted a sense of illness ownership, healthy practices, and participation in illness management\n\nDirect patient participation is crucial for developing and designing a PHR system\n\n\nZickmund et al. (2007)[25]\n\nn=39, diabetic patient users of an integrated PHR from 4 primary care practices\n\n10 focus groups conducted to determine impact of an electronic PHR on patient-physician relationship\n\n\u2022Patients appreciated having access to lab results\r\n\u2022Patients appreciated using the PHR to communicate directly with their PCP\n\nA good patient-provider relationship may diminish the perceived utility of a PHR\n\n\nBrink et al. (2005)[26]\n\nn=36, cancer patient users of an electronic PHR and (n=36) general practitioners in the Netherlands\n\nQuestionnaires completed on PHR use after 6 week period\n\n\u2022PHR system was highly valued by patients\r\n\u2022Patients (64%) reported increased knowledge of illness and treatment\n\nUsing electronic PHR in conjunction with clinical practice is recommended\n\n\nJones et al. (1999)[27]\n\nn=783, general medicine and dental practitioners, and patient users of an electronic, stand-alone medical-dental PHR in the United Kingdom\n\n3 part study: survey, randomized trial, and assessment of PHR records\n\nPatients felt positively about the PHR and recognized their role in maintaining information within\r\n\u2022Dentists and doctors expressed positive attitudes towards the PHR\n\nPatient\u2019s positive attitudes towards PHRs may increase after extended utilization\n\n\nPyper et al. (2004)[28]\n\nn=606, patient users of an integrated PHR in the United Kingdom\n\nPostal distribution of questionnaire\n\n\u2022Patients concerned with the security and privacy of the PHR\r\n\u2022Majority of patients wanted access to their records\n\nThe ideal PHR should be developed by a patient-physician partnership\n\n\nZayas-Caban et al. (2007)[29]\n\nn=7, patient users of a hypothetical PHR system\n\n2-part interviews conducted to determine how well patients understand PHRs\n\n\u2022Adult patients expressed positive attitude concerning access to electronic PHR\r\n\u2022Patients understood how they would use PHR to manage their health\n\nPatients\u2019 needs should be addressed in the design of and education about PHRs\n\n\nCox et al. (2008)[30]\n\nn=1202, stakeholders, healthcare consumers, and patient users of an integrated PHR\n\nTelephone, paper and web surveys plus focus groups assessed consumer\u2019s views\n\n\u2022Stakeholders trusted a PHR developed by a not-for-profit over one developed by government or private-sector organizations\r\n\u2022Consumers expressed privacy concerns\n\nA community outreach and communications program may inform the development of a PHR\n\n\nKeselman et al. (2007)[32]\n\nn=103, chronic disease patient users of both paper-based and electronic PHRs\n\nSurvey to determine patient needs and experiences with their health records for optimizing PHR design\n\n\u2022Patients felt a secure, online PHR provided easy access to EHR\r\n\u2022Patients felt medical records should be reviewed and discussed with health professionals to enhance patient comprehension\n\nPHRs should offer the ability to view radiology reports, physicians\u2019 notes, and diagnostic images\n\n\nHassol et al. (2004)[33]\n\nn=1,421, patient users of an integrated PHR and (n=10) primary care physicians\n\nOnline survey and focus groups with patients; one-on-one interviews with physicians\n\n\u2022Positive patient reports on ease of use and satisfaction with PHR\n\nPHRs improve efficiency and effectiveness of health care\n\n\nKim et al. (2007)[34]\n\nn=46, Elderly and disabled patient users of an electronic PHR\n\nPHR database log analyzed to determine patient usage patterns\n\n\u2022Amount of activity by participants suggests that users found the electronic PHR valuable\n\nInformation for behavior of users\/nonusers should be studied to optimize electronic PHRs for underserved populations\n\n\nHess et al. (2007)[35]\n\nn=39, patient users of an integrated PHR\n\nFocus groups, patient\u2019s reactions to an electronic PHR in diabetes self-management\n\n\u2022Trusted, centralized health information was useful to patients\r\n\u2022Participants most appreciated electronic communication feature and health-related management tools\r\n\u2022Frequent users reported a sense of empowerment\n\nPatients highly valued the secure, electronic messaging feature of the electronic PHR\n\n\nCimino et al. (2002)[36]\n\nn=13, patient and physician users of an integrated PHR\n\nPHR log file, online questionnaire, and telephone interview data analyzed\n\n\u2022Patients felt increased ownership of their health\r\n\u2022Patients were impressed with the PHR\r\n\u2022Physicians reported improved communication with patients\n\nPHR systems may improve health outcomes for select groups of patients\n\n\nWiljer et al. (2010)[38]\n\nn=250, cancer patient users of an integrated PHR in Canada\n\nLog files analyzed to assess the impact of electronic PHR access\n\n\u2022Patients preferred receiving PHR support by phone as opposed to email\n\nOptimizing PHR functionality may decrease human error\n\n\nPyper et al. (2004)[52]\n\nn=100, patient users of an integrated PHR in the United Kingdom\n\nInterviews conducted to evaluate first-time patient-users\u2019 experience with an electronic PHR\n\n\u2022Patients found the PHR useful and easy to use\n\nPHR access may improve patient care; requires additional resources to support future demand\n\n\n\nBarriers toward adoption and use of PHRs \nTen papers described patient or physician barriers to using electronic and paper-based PHRs (Table 2). A broad range of barriers to PHR adoption exists, many of which may be overcome by providing adequate technical support. Trends noted with paper-based PHRs, such as failure to document adult immunizations, laboratory test results, allergies and blood sugar, continued into electronic PHRs.[17] Difficult concepts, unfamiliar medical terms, and unknown abbreviations are commonly cited barriers.[32] Low computer literacy, low health literacy, and computer anxiety are additional patient-reported barriers in accessing electronic PHRs.[37] The time requirement for learning and, when information is not tightly linked between the PHR and EHR, entering personal health information into an electronic PHR system is problematic for patients as well as health care support staff.[25][37] Nearly all breast cancer patients (98%) in one Canadian study required technical support when accessing their electronic PHR.[38] Barriers to using an integrated PHR included lost or unknown user names and passwords, and patients\u2019 lack of awareness of useful features.[35] \nPatients are better able to access and maintain a PHR when given tailored education, technical assistance, self-management support, consumer-friendly PHR interface design, and access to trained staff.[4][29][32][37][38] Implementing these support mechanisms may require additional resources. Patients battling cancer found learning how to use an integrated PHR system was not difficult after receiving personal instruction.[26] Integrated PHRs could alleviate comprehension barriers by providing online terminology support such as using a text translator to clarify medical terms.[32][39] In addition, offering patients emotional, informational and\/or tangible support when accessing EHR\/EMR data may increase the perceived and actual utility of an integrated PHR.[24]\n\r\n\n\nTable 2: Barriers toward Adoption and Use of PHRs\n\n\n\n\n\n\n Authors\/Year\n\n Population\n\n Design\n\n Results\n\n Implications According to Authors\n\n\nTobacman et al. (1996) [17]\n\nn=100, patient users of a standalone PHR\n\nTelephone questionnaire to determine acceptance, usefulness, and impact of PHR\n\n\u2022Patients failed to record medical information\n\nPatient use of a PHR may increased over time\n\n\nZickmund et al. (2007)[25]\n\nn=39, diabetic patient users of an integrated PHR from 4 primary care practices\n\n10 focus groups conducted to determine impact of an electronic PHR on patient-physician relationship\n\n\u2022Patients expressed privacy concerns with email feature\r\n\u2022Patients who expressed satisfaction with their physician were less likely to want PHR access\n\nEducating patients about e-mail security and offering direct provider e-mail may encourage patient use\n\n\nBrink et al. (2005)[26]\n\nn=36, cancer patient users of an electronic PHR and (n=36) general practitioners in the Netherlands\n\nQuestionnaires completed on PHR use after 6 week period\n\n\u2022Patients (25%) encountered technical difficulties while using the PHR\r\n\u2022General practitioners questioned PHR utility; the majority did not use the PHR\n\nIntegrated PHRs may increase patient and provider use\n\n\nKeselman et al. (2007)[32]\n\nn=103, chronic disease patient users of both paper-based and electronic PHRs\n\nSurvey to determine patient information needs and experiences with their health records\n\n\u2022Lack of patient comprehension of medical terms\/abbreviations, medical record information, and lab test results\n\nPatients need carefully designed PHRs and comprehensive patient data\n\n\nHassol et al. (2004)[33]\n\nn=1,421, patient users of an integrated PHR and (n=10) primary care physicians\n\nOnline survey and focus groups with patients; one-on-one interviews with physicians\n\n\u2022Patients reported problems with information accuracy and completeness in their medical records\n\nPatient and provider needs are best met when EHR data is accurate and complete\n\n\nKim et al. (2007)[34]\n\nn=46, Elderly and disabled patient users of an electronic PHR\n\nPHR database log analyzed to determine patient usage patterns\n\n\u2022Patients did not fill certain fields in their PHRs with \u2019Immunizations\u2019 being the least updated information category\n\nAnalyzing system usage logs led to improvements of the studied PHR\n\n\nHess et al. (2007)[35]\n\nn=39, patient users of an integrated PHR\n\nFocus groups, patient\u2019s reactions to an electronic PHR in diabetes self-management\n\n\u2022Patients reported not knowing their username, password, and features included in access to PHR\r\n\u2022Patients reported inaccurate information and missing lab results\n\nPatients may abandon PHRs if their expectations are not met\n\n\nLober et al. (2006)[37]\n\nn=38, elderly, disabled patient users of an integrated PHR\n\nSurveys distributed to measure patient barriers with an electronic PHR system\n\n\u2022Patients experienced computer literacy issues and computer anxiety\r\n\u2022Cognitive impairments among participants affected their ability to maintain information in their PHRs\n\nPatients could create and maintain PHRs with assistance from registered nurses\n\n\nWiljer et al. (2010)[38]\n\nn=250, cancer patient users of an integrated PHR in Canada\n\nLog files analyzed to assess the impact of access to an electronic PHR\n\n\u2022Patients experienced technical difficulties and required technical support while using their PHRs\n\nPHR access requires technical and healthcare staff support\n\n\nPyper et al. (2004)[52]\n\nn=100, patient users of an integrated PHR in the United Kingdom\n\nInterviews conducted to evaluate first-time patient-users\u2019 experience with an electronic PHR\n\n\u2022Patients requested explanation of medical terms and tests\/results\r\n\u2022Patients (70%) found errors in their medical records\n\nPatients need time and assess to support staff when using their PHRs\n\n\n\nRole of PHRs in self-management \nEleven papers examined patient use of electronic and paper-based PHRs in self-management (Table 3). PHR functions helping patients to better manage their health care allow them to enter, record, and track their own health information. Interactive features within EHR-linked PHRs have the potential to increase patient participation in illness management, improve patient-physician communication, and increase a patient\u2019s sense of illness-ownership.[14][32][36] As it stands, there is currently a lack of concrete evidence that PHRs fulfilled the expectations set for them. While several studies have reported on improved clinical markers for patients with chronic conditions using integrated PHRs, few have demonstrated effects on clinical outcomes.[15][23] This may be due to the inherent complexity of conducting health IT research, the variability in characteristics and features of electronic PHRs from practice to practice, or the absence of agreement about the standard definition of a PHR.[34][40]\nPatients with chronic conditions may have the most to gain from using integrated PHRs. Improvements in diabetes-related clinical markers have been reported in studies using PHRs that incorporated systematic, active interactions between patients and providers, including improved glycemic control and blood pressure.[15][41][42] Adult patients with type 2 diabetes using an integrated PHR were more optimistic about their chronic disease management and their relationship with their primary care physician.[42] Although patients appreciate viewing their medical records, some have a greater need to record daily personal health information in a PHR.[43] Providing patients with type 2 diabetes with the option to upload measurements such as blood glucose levels can enable physicians to make between-visit medication adjustments.[15] Physicians and patients from primary care practices in mainly urban settings regarded the integrated PHR as an essential component of care and a means to establish active patient involvement.[44]\n\r\n\n\nTable 3: Role of PHRs in Self-Management\n\n\n\n\n\n\n Authors\/Year\n\n Population\n\n Design\n\n Results\n\n Implications According to Authors\n\n\nKrist et al. (2012)[13]\n\nn=4,500, patient users of an interactive PHR (IPHR) from 8 primary care practices\n\nEMR and survey data were analyzed to determine IPHR effectiveness\n\n\u2022Patients who used the IPHR increased uptake of preventative services\n\nPHRs with high functionality may improve patient outcome\n\n\nRalston et al. (2009)[15]\n\nn=83, type 2 diabetes patients using an integrated PHR\n\nCase study to evaluate the success of an initiative targeting patient access to care\n\n\u2022Patients received care faster and easier\r\n\u2022Connecting with physicians made easier\r\n\u2022Patient glycated hemoglobin levels decreased\n\nDiabetes patients benefit from care management features in PHRs\n\n\nTenforde et al. (2011)[23]\n\nn=21, published research articles on PHR adoption and attitudes, clinical outcomes, and existing PHRs\n\nLiterature review of MEDLINE articles on PHRs (published between 2000-2010)\n\n\u2022Affects PHR access on patient outcomes difficult to determine\r\n\u2022Study limitations included lack of patient blinding and generalizability\r\n\u2022Studies show inconsistent improvement in diabetes care with PHR access\n\nAlthough evidence supporting PHRs, clinical value is limited; PHRs may empower patients in self-management\n\n\nWinkelman et al. (2005)[24]\n\n=12, inflammatory bowel disease patient users of an EMR linked PHR in Canada\n\nQualitative study using in-depth interviews and focus groups\n\n\u2022Having the patient\u2019s voice guide EMR structure promotes a sense of illness ownership\r\n\u2022A PHR that supports self-care is most useful for patients\n\nA useful PHR is customizable, provides informational, emotional, and tangible support\n\n\nBrink et al. (2005)[26]\n\nn=36, cancer patient users of an electronic PHR and (n=36) general practitioners in the Netherlands\n\nQuestionnaires completed on PHR use after 6 week period\n\n\u2022PHR detected patient problems undiscovered during regular visits\r\n\u2022General practitioners (61%) expected health IT to play an important role in oncology care\n\nAn elderly, relatively computer illiterate patient group could use and appreciate the value of a PHR\n\n\nKeselman et al. (2007)[32]\n\nn=103, chronic disease patient users of both paper-based and electronic PHRs\n\nSurvey to determine patient needs and experiences with their health records for optimizing PHR design\n\n\u2022Desire for health self=management prompted patient interest in PHRs\r\n\u2022Viewing PHRs resulted in patient engagement in care-related decisions and actions\n\nOptimization of PHR functionality will enable patient participation in health care\n\n\nKim et al. (2007)[34]\n\nn=46, Elderly and disabled patient users of an electronic PHR\n\nPHR database log analyzed to determine patient usage patterns\n\n\u2022Patients (96%-100%) updated their medications and recorded past and present health problems\r\n\u2022Free-text field was used to record questions, note, and reminders\n\nInteractive features in integrated PHRs are important and useful for patients\n\n\nCimino et al. (2002)[36]\n\nn=13, patient and physician users of an integrated PHR\n\nPHR log file, online questionnaire, and telephone interview data analyzed\n\n\u2022Improved interactions with health care providers\r\n\u2022Patients reported taking a more active role in their healthcare\n\nAccess to PHRs enhanced patients\u2019 understanding of their conditions and improved communication with their physicians\n\n\nGrant et al. (2005)[41]\n\nn=244, diabetes patient users of an integrated, diabetes-specific PHR\n\nRandomized control trial among 11 primary care practices\n\n\u2022Patients who used the PHR were most likely to modify their medication regimens\n\nIntegrated PHRs may facilitate medication initiation and dosage adjustment, leading to improved clinical outcomes\n\n\nHolbrook et al. (2009)[42]\n\nn=511, type 2 diabetes patients using an integrated PHR in Canada\n\nQuestionnaires to evaluate PHR use\n\n\u2022Patient optimism increased in the intervention group compared to control\r\n\u2022Patients in the intervention group were satisfied or more satisfied about their diabetes care after using the PHR\n\nPHRs may contribute to improvements in blood pressure and glycated hemoglobin levels\n\n\nVentres et al. (2006)[44]\n\nUnknown, patients and physicians using an electronic PHR\n\nAn ethnographic study including participant observation, interviews, and taped encounters, to identify factors influencing physician and patient use\n\n\u2022PHR use enhanced patient-physician communication\r\n\u2022Patient concerns were influenced by their level of comfort with computers\r\n\u2022Physicians expressed frustration about the time needed to learn how to use the PHR\n\nThe effects of introducing PHRs into clinical practice may not be universally positive; consideration of factors influencing how PHRs are viewed and used in medical practice is advised\n\n\n\nDiscussion \nPatients embrace the idea of PHRs, but adoption has been slow as patients lack the knowledge and training required to fully engage with integrated PHRs, and remain unaware of useful features.[16][35][45] Patients\u2019 understanding of integrative PHRs\u2019 value and relevance to care may increase patient adoption; therefore, practices may need to offer a wide range of materials and processes to inform patients.[13] Overall, patients prefer integrated PHRs due to privacy concerns and their desire to be connected to their doctors.[9][18][46] Patients report an interest in communicating with their providers and are less interested in adding health-related data.[18][46] Specific features of integrated PHRs, such as the ability to look up test results and email physicians, appeal to patients, yet patients are deterred when confronted with unfamiliar medical terms and the need to memorize user names and passwords, or are lacking in health and computer literacy. PHRs will likely be more successful and effective when practices provide guidance to health care professionals in health record management and when patients receive both educational and technical support when accessing PHRs.[29][45] When used effectively, PHRs can engage patients in their health care, resulting in increased illness ownership and positive health outcomes. However, conclusive evidence of the clinical value of using PHRs is needed as studies have yet to demonstrate the effects of PHR adoption on clinical outcomes.[18]\nThe ideal PHR actively engages patients in sharing and exchanging health information with their clinicians by offering a full spectrum of high functioning health IT capabilities to improve patient care and outcomes beyond the traditional clinical encounter. However, a consensus on what information to include in an integrated PHR has yet to be reached.[18] Some integrated PHR systems offer such functionality, but do not provide patients with all possible capabilities.[47] Future integrated PHRs will likely offer secure patient-physician messaging, incorporate decision-support systems, evaluate patents\u2019 needs using evidence-based guidelines, and contain applications that offer behavioral feedback and the capabilities for individualized recommendations.[18][35][42][47][48] Furthermore, the adoption of national standards for integrated PHRs will be necessary to ensure the integrity and long-term sustainability of PHRs.[49] The Certification Commission for Health IT has recommended certification for specific PHR features, including privacy, security, interoperability, and functionality.[50]\nMultiple studies have examined patient attitudes towards PHRs and usage among special populations. Specific patient populations, such as those with chronic or multiple illnesses, may benefit directly from integrated PHRs, which could motivate continued use.[18] However, current evidence demonstrating the benefits of PHRs in chronic disease management is lacking. The existing evidence showing that PHRs have value is restricted to diabetes management.[15][35][41][42] Patients in developed countries with diabetes tend to utilize PHRs more than healthy patients.[15][35][41][42][46]\nStudies examining the effects of age, race, ethnicity, and sex on patient attitudes and usage have been inconclusive, demonstrating effects of these variables in some comparisons but not others.[9][18][34][37][46] Factors such diagnosis, age, gender, and country of origin may account for the reported variation. Patient motivations for using a PHR are tied to and highly influenced by the patient-physician relationship, physician attitudes toward PHRs[16][18][25][46], and the desire for illness ownership[16] and control over the dissemination of medical information.[25][28][30][31] Patients are likely to adopt, use, and value an integrated PHR that is user-friendly[26][28][34], provides a high level of privacy and security[25][30], and offers advanced features such as messaging, editing, and medication renewal capabilities.[33][34][35] Patient adoption and use will increase as health IT developers integrate patient feedback in the development process.\nThis review includes several limitations. First, the majority of studies included in this review were authored in the United States. However, 13 of the 52 papers selected were authored outside the United States, including five from Canada[18][24][38][40][42], four from the United Kingdom[14][27][28][51], and the remaining from Spain[11], Serbia[12], Germany[7], and the Netherlands.[26] Second, biases in the literature include small sample size[15][41], non-random, self-selected samples[25][30][32], and limited ethnic, racial, and socioeconomic diversity.[13][15][28][33][41][51] Only one paper included non-English speaking participants.[44] Finally, although we have discussed many of the important findings and themes in the literature, it was not possible to detail every factor affecting PHR evolution and adoption, patient user attitudes toward PHRs, patient reported barriers to use, and the role of PHRs in self-management.\n\nConclusion \nIntegrated PHRs have the potential to improve the patient-provider relationship, enable shared decision-making, and allow the healthcare system to move toward a more personalized healthcare delivery system. Integrated PHRs will have a broader impact on public health as they evolve higher levels of functionality and as physicians increase PHR adoption.[13] Since 2012, the federal government has given over $5 billion to providers and hospitals for the adoption and meaningful use of qualifying PHRs.[46] Beginning in 2014, providers and hospital are required to make PHRs available to 50% of patients and achieve an adoption rate of 10%.[46] Incentivizing providers and hospitals to adopt and become meaningful users of EHRs will provide more patients with the opportunity to use PHRs. However, the available evidence demonstrating that PHRs can support their intended use is insufficient as the existing literature on PHRs is limited and inconclusive.[9][18] Further patient- and physician-focused research on factors affecting PHR adoption and frequency of use is needed to improve PHR functionality, inform health IT development, and determine what motivates patients to not only adopt but to continue using PHRs.\n\nConflicts of interest \nNo potential conflict of interest to report.\n\nReferences \n\n\n\u2191 US Department of Health and Human Services (June 2008). \"The ONC-Coordinated Federal Health Information Technology Strategic Plan: 2008U2012\" (PDF). https:\/\/www.healthit.gov\/sites\/default\/files\/hit-strategic-plan-summary-508-2.pdf .   \n\n\u2191 US Department of Health and Human Services (February 2006). \"Personal Health Records and Personal Health Record Systems: A Report and Recommendation from the National Committee on Vital and Health Statistics\" (PDF). http:\/\/www.ncvhs.hhs.gov\/wp-content\/uploads\/2014\/05\/0602nhiirpt.pdf .   \n\n\u2191 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 Detmer, D.; Bloomrosen, M.; Raymond, B.; Tang, P. (2008). \"Integrated personal health records: transformative tools for consumer-centric care\". BioMed Central Medical Informatics and Decision Making 8: 45. doi:10.1186\/1472-6947-8-45. PMC PMC2596104. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2596104 .   \n\n\u2191 4.0 4.1 4.2 4.3 4.4 4.5 4.6 4.7 Tang, P.C.; Ash, J.S.; Bates, D.W.; Overhage, J.M.; Sands, D.Z. (2006). \"Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption\". Journal of the American Medical Informatics Association 13 (2): 121-126. doi:10.1197\/jamia.M2025. PMC PMC1447551. PMID 16357345. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551 .   \n\n\u2191 5.0 5.1 Gearon, C. (June 2007). \"Perspectives on the future of personal health records\" (PDF). California HealthCare Foundation. http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/P\/PDF%20PHRPerspectives.pdf .   \n\n\u2191 6.0 6.1 Kaelber, D.; Pan, E.C. (2008). \"The value of personal health records (PHR) systems\". American Medical Informatics Association Annual Symposium Proceedings 2008: 343\u2013347. PMC PMC2655982. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655982 .   \n\n\u2191 7.0 7.1 Sunyaev, A.; Kaletsch, A.; Krcmar, H. (January 2010). \"Comparative evaluation of Google Health API vs. Microsoft Healthvault API\" (PDF). Proceedings of the Third International Conference on Health Informatics 2010: 195\u2013201. Archived from the original on 11 May 2013. https:\/\/web.archive.org\/web\/20130511163734\/http:\/\/www.alexander-kaletsch.de\/downloads\/api.pdf .   \n\n\u2191 Markle Foundation (01 July 2003). \"Connecting for Health: A Public-Private Collaborative\" (PDF). p. 14. http:\/\/www.policyarchive.org\/handle\/10207\/bitstreams\/15473.pdf .   \n\n\u2191 9.0 9.1 9.2 9.3 Kaelber, D.C.; Jha, A.K.; Johnston, D.; Middleton, B.; Bates, D.W. (2008). \"A research agenda for personal health records (PHRs)\". Journal of the American Medical Informatics Association 15 (6): 729-736. doi:10.1197\/jamia.M2547. PMC PMC2585530. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2585530 .   \n\n\u2191 10.0 10.1 Harris Interactive (22 September 2006). \"Few patients use or have access to online services for communicating with their doctors, but most would like to\". The Wall Street Journal Online 5 (16): 1\u20137. http:\/\/www.prnewswire.com\/news-releases\/few-patients-use-or-have-access-to-online-services-for-communicating-with-their-doctors-but-most-would-like-to-57121547.html .   \n\n\u2191 11.0 11.1 11.2 11.3 Carri\u00f3n, I.; Fern\u00e1ndez-Alem\u00e1n, J.L.; Jayne, C.; Palmer-Brown, D.; Toval, A.; Carrillo-de-Gea, J.M. (January 2012). \"Evaluation and neuronal network-based classification of the PHRs privacy policies\". Proceedings of the 45th Hawaii International Conference on System Sciences 2012: 2840-2849. doi:10.1109\/HICSS.2012.257.   \n\n\u2191 12.0 12.1 Stakic, S.B.; Teodosijevic, N. (August 2010). \"Agreement based distribution of responsibilities in national electronic health record systems\". Proceedings of the IEEE International Conference on Social Computing \/ IEEE International Conference on Privacy, Security, Risk and Trust 2010: 607-614. doi:10.1109\/SocialCom.2010.95.   \n\n\u2191 13.0 13.1 13.2 13.3 13.4 13.5 13.6 13.7 Krist, A.H.; Woolf, S.H.; Rothemich, S.F.; Johnson, R.E.; Peele, J.E.; Cunningham, T.D.; Longo, D.R.; Bello, G.A.; Matzke, G.R. (2012). \"Interactive preventive health record to enhance delivery of recommended care: a randomized trial\". Annals of Family Medicine 10 (4): 312-9. doi:10.1370\/afm.1383. PMC PMC3392290. PMID 22778119. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3392290 .   \n\n\u2191 14.0 14.1 14.2 Pagliari, C.; Detmer, D.; Singleton, P. (2007). \"Potential of electronic personal health records\". British Medical Journal 335 (7615): 330-333. doi:10.1136\/bmj.39279.482963.AD. PMC PMC1949437. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1949437 .   \n\n\u2191 15.0 15.1 15.2 15.3 15.4 15.5 15.6 15.7 15.8 15.9 Ralston, J.D.; Hirsch, I.B.; Hoath, J.; Mullen, M.; Cheadle, A.; Goldberg, H.I. (2009). \"Web-based collaborative care for type 2 diabetes: a pilot randomized trial\". Diabetes Care 32 (2): 234-9. doi:10.2337\/dc08-1220. PMC PMC2628685. PMID 19017773. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2628685 .   \n\n\u2191 16.0 16.1 16.2 16.3 16.4 Witry, M.J.; Doucette, W.R.; Daly, J.M.; Levy, B.T.; Chrischilles, E.A. (2010). \"Family physician perceptions of personal health records\". Perspectives in Health Information Management 7 (Winter): 1d. PMC PMC2805556. PMID 20697465. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2805556 .   \n\n\u2191 17.0 17.1 17.2 17.3 Tobacman, J.; Nolan, P. (1996). \"Utilization of a personal health record in a general medicine clinic\". Journal of General Internal Medicine 11 (6): 370-372. PMID 8803745.   \n\n\u2191 18.00 18.01 18.02 18.03 18.04 18.05 18.06 18.07 18.08 18.09 18.10 Archer, N.; Fevrier-Thomas, U.; Lokker, C.; McKibbon, K.A.; Straus, S.E. (2011). \"Personal health records: a scoping review\". Journal of the American Medical Informatics Association 18 (4): 515-522. doi:10.1136\/amiajnl-2011-000105. PMC PMC3128401. PMID 21672914. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3128401 .   \n\n\u2191 Taylor, H. (2004). \"Two in five adults keep personal or family health records and almost everybody thinks this is a good idea: electronic health records likely to grow rapidly\". Health Care News 4 (13). http:\/\/www.prnewswire.com\/news-releases\/two-in-five-adults-keep-personal-or-family-health-records-and-almost-everybody-thinks-this-is-a-good-idea-71581677.html .   \n\n\u2191 Hsiao, C-J.; Hing, E.; Socey, T.C.; Cai, B. (December 2010). \"Electronic medical record\/electronic health record systems of office-based physicians: United States, 2009 and preliminary 2010 state estimates\" (PDF). National Center for Health Statistics. http:\/\/www.cdc.gov\/nchs\/data\/hestat\/emr_ehr_09\/emr_ehr_09.pdf .   \n\n\u2191 Undem, T. (April 2010). \"Consumers and health information technology: a national survey\" (PDF). California HealthCare Foundation. pp. 40. http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/C\/PDF%20ConsumersHealthInfoTechnologyNationalSurvey.pdf .   \n\n\u2191 Markle Foundation (June 2008). \"Americans overwhelmingly believe electronic personal health records could improve their health\" (PDF). pp. 7. http:\/\/www.markle.org\/sites\/default\/files\/ResearchBrief-200806.pdf .   \n\n\u2191 23.0 23.1 23.2 Tenforde, M.; Jain, A.; Hickner, J. (2011). \"The value of personal health records for chronic disease management: what do we know?\". Family Medicine 43 (5): 351-354. PMID 21557106.   \n\n\u2191 24.0 24.1 24.2 24.3 24.4 24.5 24.6 24.7 Winkelman, W.; Leonard, K.; Rossos, P. (2005). \"Patient-perceived usefulness of online electronic medical records: employing grounded theory in the development of information and communication technologies for use by patients living with chronic illness\". Journal of the American Medical Informatics Association 12 (3): 306-314. PMC PMC1090462. PMID 15684128. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090462 .   \n\n\u2191 25.0 25.1 25.2 25.3 25.4 25.5 25.6 25.7 25.8 Zickmund, S.; Hess, R.; Bryce, C.; McTigue, K.; Olshansky, E.; Fitzgerald, K.; Fischer, G. (2007). \"Interest in the use of computerized patient portals: role of the provider-patient relationship\". Journal of General Internal Medicine 23 (S1): 20-26. doi:10.1007\/s11606-007-0273-6. PMC PMC2338160. PMID 18095039. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2338160 .   \n\n\u2191 26.0 26.1 26.2 26.3 26.4 26.5 26.6 26.7 26.8 26.9 Brink, J.; Moorman, P.; Boer, M.; Pruyn, J.; Verwoerd, C.; Bemmel, J. (2005). \"Involving the patient: a prospective study on use, appreciation and effectiveness of an information system in head and neck cancer care\". International Journal of Medical Informatics 74 (10): 839-849. doi:10.1016\/j.ijmedinf.2005.03.021. PMID 16043392.   \n\n\u2191 27.0 27.1 27.2 27.3 27.4 Jones, R.; McConville, J.; Mason, D.; Macpherson, L.; Navin, L.; McEwen, J. (1999). \"Attitudes towards, and utility of, an integrated medical-dental patient-held record in primary care\". British Journal of General Practice 49 (442): 368-73. PMC PMC1313422. PMID 10736887. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1313422 .   \n\n\u2191 28.0 28.1 28.2 28.3 28.4 28.5 28.6 28.7 Pyper, C.; Amery, J.; Watson, M.; Crook, C. (2004). \"Patients\u2019 experiences when accessing their online electronic patient records in primary care\". British Journal of General Practice 54 (498): 38-43. PMC PMC1314776. PMID 14965405. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314776 .   \n\n\u2191 29.0 29.1 29.2 29.3 Zayas-Caban, T.; Valdez, R. (2007). \"Do patients understand how PHRs work?\". American Medical Informatics Association Annual Symposium Proceedings 11: 1169. PMID 18694265.   \n\n\u2191 30.0 30.1 30.2 30.3 30.4 Cox, B.; Thornewill, J. (2008). \"The consumer\u2019s view of the electronic health record: engaging patients in EHR adoption\". Journal of Healthcare Information Management 22 (2): 43-7. PMID 19266994.   \n\n\u2191 31.0 31.1 Bright, B. (29 November 2007). \"Benefits of electronic health records seen as outweighing privacy risks\". The Wall Street Journal. http:\/\/www.wsj.com\/articles\/SB119565244262500549 .   \n\n\u2191 32.0 32.1 32.2 32.3 32.4 32.5 32.6 32.7 32.8 Keselman, A.; Slaughter, L.; Smith, C.A.; Kim, H.; Divita, G.; Brown, A.; Tsai, C.; Zeng-Treitler, Q. (2007). \"Towards consumer-friendly PHRs: patients\u2019 experience with reviewing their health records\". American Medical Informatics Association Annual Symposium Proceedings 2007: 399-403. PMC PMC2655877. PMID 18693866. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655877 .   \n\n\u2191 33.0 33.1 33.2 33.3 33.4 33.5 33.6 Hassol, A.; Walker, J.; Kidder, D.; Rokita, K.; Young, D.; Pierdon, S.; Deitz, D.; Kuck, S.; Ortiz, E. (2004). \"Patient experiences and attitudes about access to a patient electronic health care record and linked web messaging\". Journal of the American Medical Informatics Association 11 (6): 505\u201313. PMC PMC524631. PMID 15299001. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC524631 .   \n\n\u2191 34.0 34.1 34.2 34.3 34.4 34.5 34.6 34.7 34.8 34.9 Kim, E.; Stolyar, A.; Lober, W.; Herbaugh, A.; Shinstrom, S.; Zierler, B.; Soh, C.; Kim, Y. (2007). \"Usage patterns of a personal health record by elderly and disabled users\". American Medical Informatics Association Annual Symposium Proceedings 2007: 409-13. PMC PMC2655817. PMID 18693868. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655817 .   \n\n\u2191 35.0 35.1 35.2 35.3 35.4 35.5 35.6 35.7 35.8 35.9 Hess, R.; Bryce, C.; Paone, S.; Fischer, G.; McTigue, K.; Olshansky, E.; Zickmund, S.; Fitzgerald, K.; Siminerio, L. (2007). \"Exploring challenges and potentials of personal health records in diabetes self-management: implementation and initial assessment\". Telemedicine and e-Health 13 (5): 509-17. doi:10.1089\/tmj.2006.0089. PMID 17999613.   \n\n\u2191 36.0 36.1 36.2 36.3 Cimino, J.J.; Patel, V.L.; Kushniruk, A.W. (2002). \"The patient clinical information system (PatCIS): technical solutions for and experience with giving patients access to their electronic medical records\". International Journal of Medical Informatics 68 (1-3): 113-27. doi:10.1016\/S1386-5056(02)00070-9. PMID 12467796.   \n\n\u2191 37.0 37.1 37.2 37.3 37.4 Lober, W.; Zierler, B.; Herbaugh, A.; Stolyer, A.; Kim, E.H.; Kim, Y. (2006). \"Barriers to the use of a personal health record by an elderly population\". American Medical Informatics Association Annual Symposium Proceedings 2006: 514-518. PMC PMC1839577. PMID 17238394. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839577 .   \n\n\u2191 38.0 38.1 38.2 38.3 Wiljer, D.; Urowitz, S.; Apatu, E.; Leonard, D.; Quartey, N.K.; Catton, P. (2010). \"Understanding the support needs of patients accessing test results online\". Journal of Healthcare Information Management 24 (1): 57-63. PMID 20077927.   \n\n\u2191 Zeng-Treitler, Q.; Goryachev, S.; Hyeoneui, K.; Keselman, A.; Rosendale, D. (2007). \"Making texts in electronic health records comprehensible to consumers: a prototype translator\". American Medical Informatics Association Annual Symposium Proceedings 2007: 846-50. PMC PMC2655860. PMID 18693956. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655860 .   \n\n\u2191 40.0 40.1 Shcherbatykh, I.; Holbrook, A.; Thabane, L.; Dolovich, L. (2008). \"Methodologic issues in health informatics trials: the complexities of complex interventions\". Journal of the American Medical Informatics Association 15 (5): 575-80. doi:10.1197\/jamia.M2518. PMC PMC2528041. PMID 18579839. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2528041 .   \n\n\u2191 41.0 41.1 41.2 41.3 41.4 41.5 Grant, R.W.; Wald, J.S.; Schnipper, J.L.; Gandhi, T.K.; Poon, E.G.; Orav, E.J.; William, D.H.; Volk, L.A.; Middleton, B. (2008). \"Practice-linked online personal health records for type 2 diabetes mellitus: a randomized controlled trial\". Archives of Internal Medicine 168 (16): 1776-1782. doi:10.1001\/archinte.168.16.1776. PMC PMC3829635. PMID 18779465. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829635 .   \n\n\u2191 42.0 42.1 42.2 42.3 42.4 42.5 42.6 Holbrook, A.; Thabane, L.; Keshavjee, K.; Dolovich, L.; Bernstein, B.; Chan, D.; Troyan, S.; Foster, G.; Gerstein, H. (2009). \"Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial\". Canadian Medical Association Journal 181 (1\u20132): 37-44. doi:10.1503\/cmaj.081272. PMC PMC2704409. PMID 19581618. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2704409 .   \n\n\u2191 Brennan, P.F.; Downs, S.; Casper, G. (2010). \"Project HealthDesign: rethinking the power and potential of personal health records\". Journal of Biomedical Informatics 43 (5): S3\u20135. doi:10.1016\/j.jbi.2010.09.001. PMID 20937482.   \n\n\u2191 44.0 44.1 44.2 Ventres, W.; Kooienga, S.; Vuckovic, N.; Marlin, R.; Nygren, P.; Stewart, V. (2006). \"Physicians, patients, and the electronic health record: an ethnographic analysis\". Annals of Family Medicine 4 (2): 124-131. doi:10.1370\/afm.425. PMC PMC1467009. PMID 16569715. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1467009 .   \n\n\u2191 45.0 45.1 Weitzman, E.R.; Kaci, L.; Mandl, K.D. (2009). \"Acceptability of a personally controlled health record in a community-based setting: implications for policy and design\". Journal of Medical Internet Research 11 (2): e14. doi:10.2196\/jmir.1187. PMC PMC2762802. PMID 19403467. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2762802 .   \n\n\u2191 46.0 46.1 46.2 46.3 46.4 46.5 46.6 Kannry, J.; Beuria, P.; Wang, E.; Nissim, J. (2012). \"Personal health records: meaningful use, but for whom?\". Mount Sinai Journal of Medicine: A Journal of Translational and Personalized Medicine 79 (5): 593-602. doi:10.1002\/msj.21334. PMID 22976365.   \n\n\u2191 47.0 47.1 Krist, A.H.; Woolf, S.H. (2011). \"A vision for patient-centered health information systems\". The Journal of the American Medical Association 305 (3): 300-301. doi:10.1001\/jama.2010.2011. PMID 21245186.   \n\n\u2191 Gibbons, M.C.; Wilson, R.F.; Samal, L.; Lehmann, C.U.; Dickersin, K.; Lehmann, H.P.; Aboumatar, H.; Finkelstein, J.; Shelton, E.; Sharma, R.; Bass, E.B. (October 2009). \"Impact of Consumer Health Informatics Applications\" (PDF). U.S. Department of Health and Human Services. pp. 546. http:\/\/www.ahrq.gov\/research\/findings\/evidence-based-reports\/chiapp-evidence-report.pdf .   \n\n\u2191 Kim, J.; Jung, H.; Bates, D.W. (2011). \"History and trends of \"personal health record\" research in pubmed\". Healthcare Informatics Research 17 (1): 3-17. doi:10.4258\/hir.2011.17.1.3. PMC PMC3092992. PMID 21818452. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092992 .   \n\n\u2191 Certification Commission for Healthcare Information Technology (15 July 2008). \"Recommendations of the PHR advisory task force: certification of PHRs\" (PDF). https:\/\/hitanalyst.files.wordpress.com\/2008\/07\/cchitphratf.pdf .   \n\n\u2191 51.0 51.1 Pyper, C.; Amery, J.; Watson, M.; Crook, C. (2004). \"Access to electronic health records in primary care - a survey of patients' views\". Medical Science Monitor 10 (11): 17\u201322. PMID 15507869.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. Table 1\u20133 have been placed slightly differently. Some references didn't include links to PDF files or were missing issue numbers. Additionally, all journal references did not list DOIs and PubMed IDs; these have been added to the references to make them more useful. The authors also included reference item 40 (Kupchunas W. Personal health record new opportunity for patient education. Orthopaedic Nursing. 2006;26(3):185-191.) in their references but never used it as a citation in the text. As such, it has been omitted.\nFinally, in the PDF a couple of references were placed out of order by appearance, particularly references nine and ten in the introduction. Additionally, reference 40 is unused and removed. Because the wiki automatically assigns reference numbers based on order of appearance, the reference numbers here ultimately do not match up with those in the original paper. This was unavoidable.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\">https:\/\/www.limswiki.org\/index.php\/Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:52.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 4,381 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","5c6e2bf767feb103119262aff1c6473b_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_The_evolution_use_and_effects_of_integrated_personal_health_records_A_narrative_review skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:The evolution, use, and effects of integrated personal health records: A narrative review<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Objective: To present a summarized literature review of the evolution, use, and effects of Personal Health Records (PHRs). \n<\/p><p>Methods: Medline and PubMed were searched for \u2018personal health records\u2019. Seven hundred thirty-three references were initially screened resulting in 230 studies selected as relevant based on initial title and abstract review. After further review, a total of 52 articles provided relevant information and were included in this paper. These articles were reviewed by one author and grouped into the following categories: PHR evolution and adoption, patient user attitudes toward PHRs, patient reported barriers to use, and the role of PHRs in self-management. \n<\/p><p>Results: Eleven papers described evolution and adoption, 17 papers described PHR user attitudes, 10 papers described barriers to use, and 11 papers described PHR use in self-management. Three papers were not grouped into a category but were used to inform the Discussion. PHRs have evolved from patient-maintained paper health records to provider-linked <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a>. Patients report enthusiasm for the potential of modern PHRs, yet few patients actually use an electronic PHR. Low patient adoption of PHRs is associated with poor interface design and low health and computer literacy on the part of patient users. \n<\/p><p>Conclusion: PHR systems that account for patient\u2019s needs and skills can facilitate their adoption. Common barriers are avoidable when patients receive adequate guidance on useful features as well as technical support. When implemented effectively, PHRs can increase patient participation in health management, and improve patient-physician communication and health related decision making.\n<\/p><p><b>Keywords<\/b>: Personal Health Records; Electronic Health Records; Electronic Medical Records\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Involving patients in their health care using information technology (IT) such as interoperable personal health records (PHRs) may increase healthcare efficiency and improve quality while reducing medical errors.<sup id=\"rdp-ebb-cite_ref-HHSTheONC08_1-0\" class=\"reference\"><a href=\"#cite_note-HHSTheONC08-1\" rel=\"external_link\">[1]<\/a><\/sup> The idea behind a PHR has existed for decades.<sup id=\"rdp-ebb-cite_ref-HHSPersonal06_2-0\" class=\"reference\"><a href=\"#cite_note-HHSPersonal06-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-0\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup> Initially, PHRs were in the form of paper medical records created and maintained by patients and used to augment healthcare provided in person.<sup id=\"rdp-ebb-cite_ref-TangPers06_4-0\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup> These records, compiled and stored on paper by patients and families, allowed individuals to preserve their complete medical history.<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-1\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup> As IT evolved, patients and families began to electronically store many health-related documents, including clinical notes from different health care providers, <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> test results, and medication prescription records, essentially creating electronic versions of paper PHRs.<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-2\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TangPers06_4-1\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/p><p>Early patient-initiated PHRs have evolved into a wide variety of computer-based applications that allow patients to securely store health-related information such as laboratory test results; these can be maintained solely by patients or by both patients and clinicians.<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-3\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GearonPers07_5-0\" class=\"reference\"><a href=\"#cite_note-GearonPers07-5\" rel=\"external_link\">[5]<\/a><\/sup> Currently, employers, healthcare providers, and third party organizations have deployed a variety of electronic PHRs differing in architecture and function.<sup id=\"rdp-ebb-cite_ref-KaelberThe08_6-0\" class=\"reference\"><a href=\"#cite_note-KaelberThe08-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SunyaevComp10_7-0\" class=\"reference\"><a href=\"#cite_note-SunyaevComp10-7\" rel=\"external_link\">[7]<\/a><\/sup> These PHRs range from original, stand-alone applications where patients enter the majority of their medical information to those integrated into the clinical health record.<sup id=\"rdp-ebb-cite_ref-TangPers06_4-2\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup> In beginning to create standards for PHRs, the Markle Foundation\u2019s Connecting for Health Collaborative defined a PHR as \u201can electronic application through which individuals can access, manage and share their health information, and that of others for whom they are authorized, in a private, secure, and confidential environment\".<sup id=\"rdp-ebb-cite_ref-MarkleConnect03_8-0\" class=\"reference\"><a href=\"#cite_note-MarkleConnect03-8\" rel=\"external_link\">[8]<\/a><\/sup> Currently, an estimated 70 million insured patients have access to some form of electronic PHR and those who do not would like access.<sup id=\"rdp-ebb-cite_ref-KaelberARes08_9-0\" class=\"reference\"><a href=\"#cite_note-KaelberARes08-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HarrisFew06_10-0\" class=\"reference\"><a href=\"#cite_note-HarrisFew06-10\" rel=\"external_link\">[10]<\/a><\/sup>\n<\/p><p>When integrated with provider-maintained electronic health records (EHRs), PHRs are electronically linked to clinical information in the EHRs, which are repositories of all electronically available patient medical information from multiple sources, and which are updated by health care providers.<sup id=\"rdp-ebb-cite_ref-CarrionEval12_11-0\" class=\"reference\"><a href=\"#cite_note-CarrionEval12-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-StakicAgree10_12-0\" class=\"reference\"><a href=\"#cite_note-StakicAgree10-12\" rel=\"external_link\">[12]<\/a><\/sup> With integrated PHRs, patients can view automatically populated medical information, such as laboratory test results.<sup id=\"rdp-ebb-cite_ref-TangPers06_4-3\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KaelberThe08_6-1\" class=\"reference\"><a href=\"#cite_note-KaelberThe08-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CarrionEval12_11-1\" class=\"reference\"><a href=\"#cite_note-CarrionEval12-11\" rel=\"external_link\">[11]<\/a><\/sup> They have advantages over stand-alone PHRs by allowing securely linked patient-provider communication outside traditional clinical encounters. Integrated PHRs provide a means to create a shared patient record through evolving features including patient-physician collaborative tools and interactive decision-making tools, personalized management tools for chronic conditions, integrated and linked health information resources, and patient-entered information.<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-4\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TangPers06_4-4\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CarrionEval12_11-2\" class=\"reference\"><a href=\"#cite_note-CarrionEval12-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KristInter12_13-0\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PagliariPot07_14-0\" class=\"reference\"><a href=\"#cite_note-PagliariPot07-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-0\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WitryFam10_16-0\" class=\"reference\"><a href=\"#cite_note-WitryFam10-16\" rel=\"external_link\">[16]<\/a><\/sup>\n<\/p><p>To inform healthcare decision-making, PHR development, and future research on clinical outcomes, this literature review summarizes the evolution, use, and effects of PHRs, with a focus on integrated PHRs. Four broad areas are address: (1) the characteristics of PHR use, including their evolution and adoption, (2) patient and provider attitudes toward PHRs, (3) barriers to PHR adoption and use, and (4) the effects of PHRs on patient management and outcomes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>Medline and PubMed were searched to identify English-language articles focused on PHRs and published between 1970 and 2011. Studies were included based on the content of titles and abstracts, removing those for which the abbreviation PHR did not stand for \u2018personal health record\u2019. Of the 733 retrieved references initially identified in the search, 230 studies were selected as relevant based on initial title and abstract. Two independent reviewers (LRC and RH) screened these and a single reviewer (CRZ) verified the studies contained <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> regarding PHR evolution, consumer attitudes, barriers to use, and\/or self-management through in-depth investigation of study content. Information from the final 52 papers, representing both qualitative and quantitative studies, was placed into four main categories: (1) the characteristics of PHR use, including their evolution and adoption, (2) patient and provider attitudes toward PHRs, (3) barriers to PHR adoption and use, and (4) the effects of PHRs on patient management and outcomes. Papers could contribute information to multiple categories (Tables 1-3).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Evolution_and_adoption_of_PHRs\">Evolution and adoption of PHRs<\/span><\/h3>\n<p>Eleven papers described the evolution of PHRs and the settings in which they are used. Early examples of PHRs were paper-based and patient-maintained.<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-5\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup> Use of paper-based PHRs has continued even after computerized information systems had become available. For example, parents have routinely collected their children\u2019s basic medical information and tracked their child\u2019s development and immunizations using baby books or have carried wallet cards containing basic personal medical data (i.e., emergency contacts, blood types and allergy information).<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-6\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TobacmanUtil96_17-0\" class=\"reference\"><a href=\"#cite_note-TobacmanUtil96-17\" rel=\"external_link\">[17]<\/a><\/sup> Patients continue to keep some paper records despite the rise of electronic PHRs.<sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-0\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup> Currently almost half (42%) of Americans keep some form of a PHR, defined in this article as any single place where medical information is kept, and the majority (87%) are paper-based.<sup id=\"rdp-ebb-cite_ref-TaylorTwo04_19-0\" class=\"reference\"><a href=\"#cite_note-TaylorTwo04-19\" rel=\"external_link\">[19]<\/a><\/sup>\n<\/p><p>As IT developed, patients began creating digital rather than paper records. Electronic PHRs evolved as patients started entering their health information into computer-based applications.<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-7\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TangPers06_4-5\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup> PHR functionality expanded to give patients the ability to view personal health information stored in their health care provider\u2019s records.<sup id=\"rdp-ebb-cite_ref-TangPers06_4-6\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup> Web-based PHRs originated in the emergency room and included online emergency medical records.<sup id=\"rdp-ebb-cite_ref-DetmerInt08_3-8\" class=\"reference\"><a href=\"#cite_note-DetmerInt08-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GearonPers07_5-1\" class=\"reference\"><a href=\"#cite_note-GearonPers07-5\" rel=\"external_link\">[5]<\/a><\/sup> As practice and hospital-based EHRs evolved, they merged with PHRs and have become a major source of the information contained within integrated PHRs. Patients now have access to integrated PHRs through their insurers or healthcare providers; however, patient adoption of PHRs has lagged behind this access. Preliminary estimates from the National Ambulatory Medical Care Survey (NAMCS) found that 51% of physicians reported providing patients with access to an integrated or partially integrated PHR (i.e. an <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical record<\/a> (EMR) or EHR linked PHR).<sup id=\"rdp-ebb-cite_ref-HsiaoElec10_20-0\" class=\"reference\"><a href=\"#cite_note-HsiaoElec10-20\" rel=\"external_link\">[20]<\/a><\/sup> In a 2010 Consumer and Health Information Technology survey, however, only 7% of Americans reported having used either a stand-alone or an integrated PHR; a 2008 Markle Foundation survey reported fewer than 3%.<sup id=\"rdp-ebb-cite_ref-KristInter12_13-1\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-UndemCons10_21-0\" class=\"reference\"><a href=\"#cite_note-UndemCons10-21\" rel=\"external_link\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MarkleConnect08_22-0\" class=\"reference\"><a href=\"#cite_note-MarkleConnect08-22\" rel=\"external_link\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TenfordeThe11_23-0\" class=\"reference\"><a href=\"#cite_note-TenfordeThe11-23\" rel=\"external_link\">[23]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Attitudes_toward_adoption_and_use_of_PHRs\">Attitudes toward adoption and use of PHRs<\/span><\/h3>\n<p>Seventeen papers described patient or provider attitudes toward electronic and paper-based PHRs (Table 1). Patients are eager to use PHRs for their potential to improve health care delivery and outcomes but these positive attitudes do not translate into use.<sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-0\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-0\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-0\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JonesAtt99_27-0\" class=\"reference\"><a href=\"#cite_note-JonesAtt99-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperPat04_28-0\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZayasDo07_29-0\" class=\"reference\"><a href=\"#cite_note-ZayasDo07-29\" rel=\"external_link\">[29]<\/a><\/sup> Patients have mixed reasons for using or not using a PHR some of which are tied to their motivation to improve health outcomes and their relationship with their physician. Understanding patient motivation is important, particularly when designing and adopting PHRs.<sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-1\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup> Patients seek the ability to control access to their health information and believe they should have access.<sup id=\"rdp-ebb-cite_ref-CoxThe08_30-0\" class=\"reference\"><a href=\"#cite_note-CoxThe08-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BrightBen07_31-0\" class=\"reference\"><a href=\"#cite_note-BrightBen07-31\" rel=\"external_link\">[31]<\/a><\/sup> Patients view integrated PHRs favorably with one report finding that 60% of patients indicate they would use an integrated PHR to look up test results and record their medication, and another survey finding that 75% of patients would communicate with physicians electronically if given the option.<sup id=\"rdp-ebb-cite_ref-HarrisFew06_10-1\" class=\"reference\"><a href=\"#cite_note-HarrisFew06-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KristInter12_13-2\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup> Patients\u2019 motivation to participate in their health care fosters their interest in viewing their PHRs and viewing PHRs influences patients\u2019 care-related decision-making.<sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-2\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JonesAtt99_27-1\" class=\"reference\"><a href=\"#cite_note-JonesAtt99-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-0\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup> Diabetic patient users of an integrated PHR reported receiving care more quickly, and connecting with their doctor more easily.<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-1\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup> An integrated PHR with features such as secure patient-physician messaging, medication history updating, and online requests for medication renewals was highly valued by elderly and disabled patients, patients with chronic conditions, and middle-aged female patients.<sup id=\"rdp-ebb-cite_ref-HassolPat04_33-0\" class=\"reference\"><a href=\"#cite_note-HassolPat04-33\" rel=\"external_link\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KimUsage07_34-0\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HessExp07_35-0\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup> The ability to contact health care providers through secure messaging in an integrated PHR provided a feeling of security for patients in the Netherlands.<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-1\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup> Patients want to view their records in order to have detailed information about their health, and those using an integrated PHR reported feeling more in control of their chronic conditions and a sense of illness-ownership, which motivated them to contribute information to their EHR.<sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-3\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JonesAtt99_27-2\" class=\"reference\"><a href=\"#cite_note-JonesAtt99-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CiminoThe02_36-0\" class=\"reference\"><a href=\"#cite_note-CiminoThe02-36\" rel=\"external_link\">[36]<\/a><\/sup> Patients\u2019 satisfaction with their physicians influenced their use of an integrated PHR. Patients expressing satisfaction with their patient-provider relationship were less likely to use an integrated PHR than patients expressing dissatisfaction.<sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-1\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup> Those expressing dissatisfaction viewed access to their PHR as a means of gaining knowledge or control over their health. \n<\/p><p>Five studies reported on ease of use for patients accessing an integrated PHR.<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-2\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperPat04_28-1\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HassolPat04_33-1\" class=\"reference\"><a href=\"#cite_note-HassolPat04-33\" rel=\"external_link\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KimUsage07_34-1\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HessExp07_35-1\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup> More than 60% of patients with head or neck cancer in the Netherlands and the majority of middle-aged adult patients in the United Kingdom found an integrated PHR easy to navigate.<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-3\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperPat04_28-2\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup> Female patients who used an integrated PHR rated various functions easier to use than males.<sup id=\"rdp-ebb-cite_ref-HassolPat04_33-2\" class=\"reference\"><a href=\"#cite_note-HassolPat04-33\" rel=\"external_link\">[33]<\/a><\/sup> Several studies, however, reported that patients did not maintain health information in their PHRs despite ease of use. Elderly patients found value in using an integrated PHR for updating medications, health problems, and lab test information, yet failed to annotate certain health information such as immunizations and laboratory test results, which the authors attributed to difficulties with the user interface of the integrated PHR.<sup id=\"rdp-ebb-cite_ref-KimUsage07_34-2\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup>\n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 1<\/b>: Attitudes toward Adoption and Use of PHR<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Authors\/Year\n<\/th>\n<th> Population\n<\/th>\n<th> Design\n<\/th>\n<th> Results\n<\/th>\n<th> Implications According to Authors\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Krist et al. (2012)<sup id=\"rdp-ebb-cite_ref-KristInter12_13-3\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=4,500, patient users of an interactive PHR (IPHR) from 8 primary care practices\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">EMR and survey data were analyzed to determine IPHR effectiveness\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients (25% ) who used the IPHR were up-to-date on all services; double the rate for non-users\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Practices need to inform patients about PHR\u2019s value and relevance to care\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Witry et al. (2010)<sup id=\"rdp-ebb-cite_ref-WitryFam10_16-1\" class=\"reference\"><a href=\"#cite_note-WitryFam10-16\" rel=\"external_link\">[16]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=28, physicians from 4 family medicine practices\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">4 focus groups to explore physician views of PHRs\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Physicians thought PHRs beneficial for certain patient populations (e.g., mobile populations)<br \/>\u2022Physicians expressed doubt about the likelihood of patients updating their PHRs\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Providers are unfamiliar with electronic PHRs; physician use of electronic PHRs may increase with exposure\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tobacman et al. (1996) <sup id=\"rdp-ebb-cite_ref-TobacmanUtil96_17-1\" class=\"reference\"><a href=\"#cite_note-TobacmanUtil96-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=100, patient users of a standalone PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Telephone questionnaire to determine acceptance, usefulness, and impact of PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Adult patients were interested in PHRs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">There is an unmet need among patients for PHR access\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Winkelman et al. (2005)<sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-4\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">=12, inflammatory bowel disease patient users of an EMR linked PHR in Canada\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Qualitative study using in-depth interviews and focus groups\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022PHR access promoted a sense of illness ownership, healthy practices, and participation in illness management\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Direct patient participation is crucial for developing and designing a PHR system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Zickmund et al. (2007)<sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-2\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=39, diabetic patient users of an integrated PHR from 4 primary care practices\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10 focus groups conducted to determine impact of an electronic PHR on patient-physician relationship\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients appreciated having access to lab results<br \/>\u2022Patients appreciated using the PHR to communicate directly with their PCP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">A good patient-provider relationship may diminish the perceived utility of a PHR\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Brink et al. (2005)<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-4\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=36, cancer patient users of an electronic PHR and (n=36) general practitioners in the Netherlands\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Questionnaires completed on PHR use after 6 week period\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022PHR system was highly valued by patients<br \/>\u2022Patients (64%) reported increased knowledge of illness and treatment\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Using electronic PHR in conjunction with clinical practice is recommended\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Jones et al. (1999)<sup id=\"rdp-ebb-cite_ref-JonesAtt99_27-3\" class=\"reference\"><a href=\"#cite_note-JonesAtt99-27\" rel=\"external_link\">[27]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=783, general medicine and dental practitioners, and patient users of an electronic, stand-alone medical-dental PHR in the United Kingdom\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 part study: survey, randomized trial, and assessment of PHR records\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Patients felt positively about the PHR and recognized their role in maintaining information within<br \/>\u2022Dentists and doctors expressed positive attitudes towards the PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Patient\u2019s positive attitudes towards PHRs may increase after extended utilization\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Pyper et al. (2004)<sup id=\"rdp-ebb-cite_ref-PyperPat04_28-3\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=606, patient users of an integrated PHR in the United Kingdom\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Postal distribution of questionnaire\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patients concerned with the security and privacy of the PHR<br \/>\u2022Majority of patients wanted access to their records\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">The ideal PHR should be developed by a patient-physician partnership\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Zayas-Caban et al. (2007)<sup id=\"rdp-ebb-cite_ref-ZayasDo07_29-1\" class=\"reference\"><a href=\"#cite_note-ZayasDo07-29\" rel=\"external_link\">[29]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=7, patient users of a hypothetical PHR system\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2-part interviews conducted to determine how well patients understand PHRs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Adult patients expressed positive attitude concerning access to electronic PHR<br \/>\u2022Patients understood how they would use PHR to manage their health\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Patients\u2019 needs should be addressed in the design of and education about PHRs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Cox et al. (2008)<sup id=\"rdp-ebb-cite_ref-CoxThe08_30-1\" class=\"reference\"><a href=\"#cite_note-CoxThe08-30\" rel=\"external_link\">[30]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=1202, stakeholders, healthcare consumers, and patient users of an integrated PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Telephone, paper and web surveys plus focus groups assessed consumer\u2019s views\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Stakeholders trusted a PHR developed by a not-for-profit over one developed by government or private-sector organizations<br \/>\u2022Consumers expressed privacy concerns\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">A community outreach and communications program may inform the development of a PHR\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Keselman et al. (2007)<sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-1\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=103, chronic disease patient users of both paper-based and electronic PHRs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Survey to determine patient needs and experiences with their health records for optimizing PHR design\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients felt a secure, online PHR provided easy access to EHR<br \/>\u2022Patients felt medical records should be reviewed and discussed with health professionals to enhance patient comprehension\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHRs should offer the ability to view radiology reports, physicians\u2019 notes, and diagnostic images\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Hassol et al. (2004)<sup id=\"rdp-ebb-cite_ref-HassolPat04_33-3\" class=\"reference\"><a href=\"#cite_note-HassolPat04-33\" rel=\"external_link\">[33]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=1,421, patient users of an integrated PHR and (n=10) primary care physicians\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Online survey and focus groups with patients; one-on-one interviews with physicians\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Positive patient reports on ease of use and satisfaction with PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">PHRs improve efficiency and effectiveness of health care\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Kim et al. (2007)<sup id=\"rdp-ebb-cite_ref-KimUsage07_34-3\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=46, Elderly and disabled patient users of an electronic PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHR database log analyzed to determine patient usage patterns\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Amount of activity by participants suggests that users found the electronic PHR valuable\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Information for behavior of users\/nonusers should be studied to optimize electronic PHRs for underserved populations\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Hess et al. (2007)<sup id=\"rdp-ebb-cite_ref-HessExp07_35-2\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=39, patient users of an integrated PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Focus groups, patient\u2019s reactions to an electronic PHR in diabetes self-management\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Trusted, centralized health information was useful to patients<br \/>\u2022Participants most appreciated electronic communication feature and health-related management tools<br \/>\u2022Frequent users reported a sense of empowerment\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Patients highly valued the secure, electronic messaging feature of the electronic PHR\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cimino et al. (2002)<sup id=\"rdp-ebb-cite_ref-CiminoThe02_36-1\" class=\"reference\"><a href=\"#cite_note-CiminoThe02-36\" rel=\"external_link\">[36]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=13, patient and physician users of an integrated PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHR log file, online questionnaire, and telephone interview data analyzed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients felt increased ownership of their health<br \/>\u2022Patients were impressed with the PHR<br \/>\u2022Physicians reported improved communication with patients\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHR systems may improve health outcomes for select groups of patients\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Wiljer et al. (2010)<sup>[38]<\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=250, cancer patient users of an integrated PHR in Canada\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Log files analyzed to assess the impact of electronic PHR access\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patients preferred receiving PHR support by phone as opposed to email\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Optimizing PHR functionality may decrease human error\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pyper et al. (2004)<sup>[52]<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=100, patient users of an integrated PHR in the United Kingdom\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Interviews conducted to evaluate first-time patient-users\u2019 experience with an electronic PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients found the PHR useful and easy to use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHR access may improve patient care; requires additional resources to support future demand\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Barriers_toward_adoption_and_use_of_PHRs\">Barriers toward adoption and use of PHRs<\/span><\/h3>\n<p>Ten papers described patient or physician barriers to using electronic and paper-based PHRs (Table 2). A broad range of barriers to PHR adoption exists, many of which may be overcome by providing adequate technical support. Trends noted with paper-based PHRs, such as failure to document adult immunizations, laboratory test results, allergies and blood sugar, continued into electronic PHRs.<sup id=\"rdp-ebb-cite_ref-TobacmanUtil96_17-2\" class=\"reference\"><a href=\"#cite_note-TobacmanUtil96-17\" rel=\"external_link\">[17]<\/a><\/sup> Difficult concepts, unfamiliar medical terms, and unknown abbreviations are commonly cited barriers.<sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-2\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup> Low computer literacy, low health literacy, and computer anxiety are additional patient-reported barriers in accessing electronic PHRs.<sup id=\"rdp-ebb-cite_ref-LoberBar06_37-0\" class=\"reference\"><a href=\"#cite_note-LoberBar06-37\" rel=\"external_link\">[37]<\/a><\/sup> The time requirement for learning and, when information is not tightly linked between the PHR and EHR, entering personal health information into an electronic PHR system is problematic for patients as well as health care support staff.<sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-3\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LoberBar06_37-1\" class=\"reference\"><a href=\"#cite_note-LoberBar06-37\" rel=\"external_link\">[37]<\/a><\/sup> Nearly all breast cancer patients (98%) in one Canadian study required technical support when accessing their electronic PHR.<sup id=\"rdp-ebb-cite_ref-WiljerUnder10_38-0\" class=\"reference\"><a href=\"#cite_note-WiljerUnder10-38\" rel=\"external_link\">[38]<\/a><\/sup> Barriers to using an integrated PHR included lost or unknown user names and passwords, and patients\u2019 lack of awareness of useful features.<sup id=\"rdp-ebb-cite_ref-HessExp07_35-3\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup> \n<\/p><p>Patients are better able to access and maintain a PHR when given tailored education, technical assistance, self-management support, consumer-friendly PHR interface design, and access to trained staff.<sup id=\"rdp-ebb-cite_ref-TangPers06_4-7\" class=\"reference\"><a href=\"#cite_note-TangPers06-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZayasDo07_29-2\" class=\"reference\"><a href=\"#cite_note-ZayasDo07-29\" rel=\"external_link\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-3\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LoberBar06_37-2\" class=\"reference\"><a href=\"#cite_note-LoberBar06-37\" rel=\"external_link\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WiljerUnder10_38-1\" class=\"reference\"><a href=\"#cite_note-WiljerUnder10-38\" rel=\"external_link\">[38]<\/a><\/sup> Implementing these support mechanisms may require additional resources. Patients battling cancer found learning how to use an integrated PHR system was not difficult after receiving personal instruction.<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-5\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup> Integrated PHRs could alleviate comprehension barriers by providing online terminology support such as using a text translator to clarify medical terms.<sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-4\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZengMaking07_39-0\" class=\"reference\"><a href=\"#cite_note-ZengMaking07-39\" rel=\"external_link\">[39]<\/a><\/sup> In addition, offering patients emotional, informational and\/or tangible support when accessing EHR\/EMR data may increase the perceived and actual utility of an integrated PHR.<sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-5\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup>\n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 2<\/b>: Barriers toward Adoption and Use of PHRs<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Authors\/Year\n<\/th>\n<th> Population\n<\/th>\n<th> Design\n<\/th>\n<th> Results\n<\/th>\n<th> Implications According to Authors\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tobacman et al. (1996) <sup id=\"rdp-ebb-cite_ref-TobacmanUtil96_17-3\" class=\"reference\"><a href=\"#cite_note-TobacmanUtil96-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=100, patient users of a standalone PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Telephone questionnaire to determine acceptance, usefulness, and impact of PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients failed to record medical information\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Patient use of a PHR may increased over time\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Zickmund et al. (2007)<sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-4\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=39, diabetic patient users of an integrated PHR from 4 primary care practices\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">10 focus groups conducted to determine impact of an electronic PHR on patient-physician relationship\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patients expressed privacy concerns with email feature<br \/>\u2022Patients who expressed satisfaction with their physician were less likely to want PHR access\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Educating patients about e-mail security and offering direct provider e-mail may encourage patient use\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Brink et al. (2005)<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-6\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=36, cancer patient users of an electronic PHR and (n=36) general practitioners in the Netherlands\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Questionnaires completed on PHR use after 6 week period\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients (25%) encountered technical difficulties while using the PHR<br \/>\u2022General practitioners questioned PHR utility; the majority did not use the PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Integrated PHRs may increase patient and provider use\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Keselman et al. (2007)<sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-5\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=103, chronic disease patient users of both paper-based and electronic PHRs\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Survey to determine patient information needs and experiences with their health records\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Lack of patient comprehension of medical terms\/abbreviations, medical record information, and lab test results\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Patients need carefully designed PHRs and comprehensive patient data\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hassol et al. (2004)<sup id=\"rdp-ebb-cite_ref-HassolPat04_33-4\" class=\"reference\"><a href=\"#cite_note-HassolPat04-33\" rel=\"external_link\">[33]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=1,421, patient users of an integrated PHR and (n=10) primary care physicians\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Online survey and focus groups with patients; one-on-one interviews with physicians\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients reported problems with information accuracy and completeness in their medical records\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Patient and provider needs are best met when EHR data is accurate and complete\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Kim et al. (2007)<sup id=\"rdp-ebb-cite_ref-KimUsage07_34-4\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=46, Elderly and disabled patient users of an electronic PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">PHR database log analyzed to determine patient usage patterns\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patients did not fill certain fields in their PHRs with \u2019Immunizations\u2019 being the least updated information category\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Analyzing system usage logs led to improvements of the studied PHR\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hess et al. (2007)<sup id=\"rdp-ebb-cite_ref-HessExp07_35-4\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=39, patient users of an integrated PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Focus groups, patient\u2019s reactions to an electronic PHR in diabetes self-management\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients reported not knowing their username, password, and features included in access to PHR<br \/>\u2022Patients reported inaccurate information and missing lab results\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Patients may abandon PHRs if their expectations are not met\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Lober et al. (2006)<sup id=\"rdp-ebb-cite_ref-LoberBar06_37-3\" class=\"reference\"><a href=\"#cite_note-LoberBar06-37\" rel=\"external_link\">[37]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=38, elderly, disabled patient users of an integrated PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Surveys distributed to measure patient barriers with an electronic PHR system\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patients experienced computer literacy issues and computer anxiety<br \/>\u2022Cognitive impairments among participants affected their ability to maintain information in their PHRs\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Patients could create and maintain PHRs with assistance from registered nurses\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wiljer et al. (2010)<sup id=\"rdp-ebb-cite_ref-WiljerUnder10_38-2\" class=\"reference\"><a href=\"#cite_note-WiljerUnder10-38\" rel=\"external_link\">[38]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=250, cancer patient users of an integrated PHR in Canada\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Log files analyzed to assess the impact of access to an electronic PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients experienced technical difficulties and required technical support while using their PHRs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHR access requires technical and healthcare staff support\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Pyper et al. (2004)<sup>[52]<\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=100, patient users of an integrated PHR in the United Kingdom\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Interviews conducted to evaluate first-time patient-users\u2019 experience with an electronic PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patients requested explanation of medical terms and tests\/results<br \/>\u2022Patients (70%) found errors in their medical records\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Patients need time and assess to support staff when using their PHRs\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Role_of_PHRs_in_self-management\">Role of PHRs in self-management<\/span><\/h3>\n<p>Eleven papers examined patient use of electronic and paper-based PHRs in self-management (Table 3). PHR functions helping patients to better manage their health care allow them to enter, record, and track their own health information. Interactive features within EHR-linked PHRs have the potential to increase patient participation in illness management, improve patient-physician communication, and increase a patient\u2019s sense of illness-ownership.<sup id=\"rdp-ebb-cite_ref-PagliariPot07_14-1\" class=\"reference\"><a href=\"#cite_note-PagliariPot07-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-6\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CiminoThe02_36-2\" class=\"reference\"><a href=\"#cite_note-CiminoThe02-36\" rel=\"external_link\">[36]<\/a><\/sup> As it stands, there is currently a lack of concrete evidence that PHRs fulfilled the expectations set for them. While several studies have reported on improved clinical markers for patients with chronic conditions using integrated PHRs, few have demonstrated effects on clinical outcomes.<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-2\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-TenfordeThe11_23-1\" class=\"reference\"><a href=\"#cite_note-TenfordeThe11-23\" rel=\"external_link\">[23]<\/a><\/sup> This may be due to the inherent complexity of conducting health IT research, the variability in characteristics and features of electronic PHRs from practice to practice, or the absence of agreement about the standard definition of a PHR.<sup id=\"rdp-ebb-cite_ref-KimUsage07_34-5\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShcherbatykhMeth08_40-0\" class=\"reference\"><a href=\"#cite_note-ShcherbatykhMeth08-40\" rel=\"external_link\">[40]<\/a><\/sup>\n<\/p><p>Patients with chronic conditions may have the most to gain from using integrated PHRs. Improvements in diabetes-related clinical markers have been reported in studies using PHRs that incorporated systematic, active interactions between patients and providers, including improved glycemic control and blood pressure.<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-3\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrantPrac08_41-0\" class=\"reference\"><a href=\"#cite_note-GrantPrac08-41\" rel=\"external_link\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HolbrookIndi09_42-0\" class=\"reference\"><a href=\"#cite_note-HolbrookIndi09-42\" rel=\"external_link\">[42]<\/a><\/sup> Adult patients with type 2 diabetes using an integrated PHR were more optimistic about their chronic disease management and their relationship with their primary care physician.<sup id=\"rdp-ebb-cite_ref-HolbrookIndi09_42-1\" class=\"reference\"><a href=\"#cite_note-HolbrookIndi09-42\" rel=\"external_link\">[42]<\/a><\/sup> Although patients appreciate viewing their medical records, some have a greater need to record daily personal health information in a PHR.<sup id=\"rdp-ebb-cite_ref-BrennanProj10_43-0\" class=\"reference\"><a href=\"#cite_note-BrennanProj10-43\" rel=\"external_link\">[43]<\/a><\/sup> Providing patients with type 2 diabetes with the option to upload measurements such as blood glucose levels can enable physicians to make between-visit medication adjustments.<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-4\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup> Physicians and patients from primary care practices in mainly urban settings regarded the integrated PHR as an essential component of care and a means to establish active patient involvement.<sup id=\"rdp-ebb-cite_ref-VentresPhys06_44-0\" class=\"reference\"><a href=\"#cite_note-VentresPhys06-44\" rel=\"external_link\">[44]<\/a><\/sup>\n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 3<\/b>: Role of PHRs in Self-Management<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Authors\/Year\n<\/th>\n<th> Population\n<\/th>\n<th> Design\n<\/th>\n<th> Results\n<\/th>\n<th> Implications According to Authors\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Krist et al. (2012)<sup id=\"rdp-ebb-cite_ref-KristInter12_13-4\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=4,500, patient users of an interactive PHR (IPHR) from 8 primary care practices\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">EMR and survey data were analyzed to determine IPHR effectiveness\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients who used the IPHR increased uptake of preventative services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHRs with high functionality may improve patient outcome\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Ralston et al. (2009)<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-5\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=83, type 2 diabetes patients using an integrated PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Case study to evaluate the success of an initiative targeting patient access to care\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patients received care faster and easier<br \/>\u2022Connecting with physicians made easier<br \/>\u2022Patient glycated hemoglobin levels decreased\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Diabetes patients benefit from care management features in PHRs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Tenforde et al. (2011)<sup id=\"rdp-ebb-cite_ref-TenfordeThe11_23-2\" class=\"reference\"><a href=\"#cite_note-TenfordeThe11-23\" rel=\"external_link\">[23]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=21, published research articles on PHR adoption and attitudes, clinical outcomes, and existing PHRs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Literature review of MEDLINE articles on PHRs (published between 2000-2010)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Affects PHR access on patient outcomes difficult to determine<br \/>\u2022Study limitations included lack of patient blinding and generalizability<br \/>\u2022Studies show inconsistent improvement in diabetes care with PHR access\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Although evidence supporting PHRs, clinical value is limited; PHRs may empower patients in self-management\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Winkelman et al. (2005)<sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-6\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">=12, inflammatory bowel disease patient users of an EMR linked PHR in Canada\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Qualitative study using in-depth interviews and focus groups\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Having the patient\u2019s voice guide EMR structure promotes a sense of illness ownership<br \/>\u2022A PHR that supports self-care is most useful for patients\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">A useful PHR is customizable, provides informational, emotional, and tangible support\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Brink et al. (2005)<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-7\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=36, cancer patient users of an electronic PHR and (n=36) general practitioners in the Netherlands\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Questionnaires completed on PHR use after 6 week period\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022PHR detected patient problems undiscovered during regular visits<br \/>\u2022General practitioners (61%) expected health IT to play an important role in oncology care\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">An elderly, relatively computer illiterate patient group could use and appreciate the value of a PHR\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Keselman et al. (2007)<sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-7\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=103, chronic disease patient users of both paper-based and electronic PHRs\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Survey to determine patient needs and experiences with their health records for optimizing PHR design\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Desire for health self=management prompted patient interest in PHRs<br \/>\u2022Viewing PHRs resulted in patient engagement in care-related decisions and actions\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Optimization of PHR functionality will enable patient participation in health care\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Kim et al. (2007)<sup id=\"rdp-ebb-cite_ref-KimUsage07_34-6\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=46, Elderly and disabled patient users of an electronic PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">PHR database log analyzed to determine patient usage patterns\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients (96%-100%) updated their medications and recorded past and present health problems<br \/>\u2022Free-text field was used to record questions, note, and reminders\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Interactive features in integrated PHRs are important and useful for patients\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Cimino et al. (2002)<sup id=\"rdp-ebb-cite_ref-CiminoThe02_36-3\" class=\"reference\"><a href=\"#cite_note-CiminoThe02-36\" rel=\"external_link\">[36]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=13, patient and physician users of an integrated PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">PHR log file, online questionnaire, and telephone interview data analyzed\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Improved interactions with health care providers<br \/>\u2022Patients reported taking a more active role in their healthcare\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Access to PHRs enhanced patients\u2019 understanding of their conditions and improved communication with their physicians\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Grant et al. (2005)<sup id=\"rdp-ebb-cite_ref-GrantPrac08_41-1\" class=\"reference\"><a href=\"#cite_note-GrantPrac08-41\" rel=\"external_link\">[41]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n=244, diabetes patient users of an integrated, diabetes-specific PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Randomized control trial among 11 primary care practices\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022Patients who used the PHR were most likely to modify their medication regimens\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Integrated PHRs may facilitate medication initiation and dosage adjustment, leading to improved clinical outcomes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Holbrook et al. (2009)<sup id=\"rdp-ebb-cite_ref-HolbrookIndi09_42-2\" class=\"reference\"><a href=\"#cite_note-HolbrookIndi09-42\" rel=\"external_link\">[42]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">n=511, type 2 diabetes patients using an integrated PHR in Canada\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">Questionnaires to evaluate PHR use\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">\u2022Patient optimism increased in the intervention group compared to control<br \/>\u2022Patients in the intervention group were satisfied or more satisfied about their diabetes care after using the PHR\n<\/td>\n<td style=\"background-color:light grey; padding-left:10px; padding-right:10px;\">PHRs may contribute to improvements in blood pressure and glycated hemoglobin levels\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ventres et al. (2006)<sup id=\"rdp-ebb-cite_ref-VentresPhys06_44-1\" class=\"reference\"><a href=\"#cite_note-VentresPhys06-44\" rel=\"external_link\">[44]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Unknown, patients and physicians using an electronic PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">An ethnographic study including participant observation, interviews, and taped encounters, to identify factors influencing physician and patient use\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u2022PHR use enhanced patient-physician communication<br \/>\u2022Patient concerns were influenced by their level of comfort with computers<br \/>\u2022Physicians expressed frustration about the time needed to learn how to use the PHR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The effects of introducing PHRs into clinical practice may not be universally positive; consideration of factors influencing how PHRs are viewed and used in medical practice is advised\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Patients embrace the idea of PHRs, but adoption has been slow as patients lack the knowledge and training required to fully engage with integrated PHRs, and remain unaware of useful features.<sup id=\"rdp-ebb-cite_ref-WitryFam10_16-2\" class=\"reference\"><a href=\"#cite_note-WitryFam10-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HessExp07_35-5\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WeitzmanAccept09_45-0\" class=\"reference\"><a href=\"#cite_note-WeitzmanAccept09-45\" rel=\"external_link\">[45]<\/a><\/sup> Patients\u2019 understanding of integrative PHRs\u2019 value and relevance to care may increase patient adoption; therefore, practices may need to offer a wide range of materials and processes to inform patients.<sup id=\"rdp-ebb-cite_ref-KristInter12_13-5\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup> Overall, patients prefer integrated PHRs due to privacy concerns and their desire to be connected to their doctors.<sup id=\"rdp-ebb-cite_ref-KaelberARes08_9-1\" class=\"reference\"><a href=\"#cite_note-KaelberARes08-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-1\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KannryPers12_46-0\" class=\"reference\"><a href=\"#cite_note-KannryPers12-46\" rel=\"external_link\">[46]<\/a><\/sup> Patients report an interest in communicating with their providers and are less interested in adding health-related data.<sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-2\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KannryPers12_46-1\" class=\"reference\"><a href=\"#cite_note-KannryPers12-46\" rel=\"external_link\">[46]<\/a><\/sup> Specific features of integrated PHRs, such as the ability to look up test results and email physicians, appeal to patients, yet patients are deterred when confronted with unfamiliar medical terms and the need to memorize user names and passwords, or are lacking in health and computer literacy. PHRs will likely be more successful and effective when practices provide guidance to health care professionals in health record management and when patients receive both educational and technical support when accessing PHRs.<sup id=\"rdp-ebb-cite_ref-ZayasDo07_29-3\" class=\"reference\"><a href=\"#cite_note-ZayasDo07-29\" rel=\"external_link\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WeitzmanAccept09_45-1\" class=\"reference\"><a href=\"#cite_note-WeitzmanAccept09-45\" rel=\"external_link\">[45]<\/a><\/sup> When used effectively, PHRs can engage patients in their health care, resulting in increased illness ownership and positive health outcomes. However, conclusive evidence of the clinical value of using PHRs is needed as studies have yet to demonstrate the effects of PHR adoption on clinical outcomes.<sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-3\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/p><p>The ideal PHR actively engages patients in sharing and exchanging health information with their clinicians by offering a full spectrum of high functioning health IT capabilities to improve patient care and outcomes beyond the traditional clinical encounter. However, a consensus on what information to include in an integrated PHR has yet to be reached.<sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-4\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup> Some integrated PHR systems offer such functionality, but do not provide patients with all possible capabilities.<sup id=\"rdp-ebb-cite_ref-KristAVis11_47-0\" class=\"reference\"><a href=\"#cite_note-KristAVis11-47\" rel=\"external_link\">[47]<\/a><\/sup> Future integrated PHRs will likely offer secure patient-physician messaging, incorporate decision-support systems, evaluate patents\u2019 needs using evidence-based guidelines, and contain applications that offer behavioral feedback and the capabilities for individualized recommendations.<sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-5\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HessExp07_35-6\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HolbrookIndi09_42-3\" class=\"reference\"><a href=\"#cite_note-HolbrookIndi09-42\" rel=\"external_link\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KristAVis11_47-1\" class=\"reference\"><a href=\"#cite_note-KristAVis11-47\" rel=\"external_link\">[47]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GibbonsImpact09_48-0\" class=\"reference\"><a href=\"#cite_note-GibbonsImpact09-48\" rel=\"external_link\">[48]<\/a><\/sup> Furthermore, the adoption of national standards for integrated PHRs will be necessary to ensure the integrity and long-term sustainability of PHRs.<sup id=\"rdp-ebb-cite_ref-KimHist11_49-0\" class=\"reference\"><a href=\"#cite_note-KimHist11-49\" rel=\"external_link\">[49]<\/a><\/sup> The Certification Commission for Health IT has recommended certification for specific PHR features, including privacy, security, interoperability, and functionality.<sup id=\"rdp-ebb-cite_ref-CCHITRec08_50-0\" class=\"reference\"><a href=\"#cite_note-CCHITRec08-50\" rel=\"external_link\">[50]<\/a><\/sup>\n<\/p><p>Multiple studies have examined patient attitudes towards PHRs and usage among special populations. Specific patient populations, such as those with chronic or multiple illnesses, may benefit directly from integrated PHRs, which could motivate continued use.<sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-6\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup> However, current evidence demonstrating the benefits of PHRs in chronic disease management is lacking. The existing evidence showing that PHRs have value is restricted to diabetes management.<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-6\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HessExp07_35-7\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrantPrac08_41-2\" class=\"reference\"><a href=\"#cite_note-GrantPrac08-41\" rel=\"external_link\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HolbrookIndi09_42-4\" class=\"reference\"><a href=\"#cite_note-HolbrookIndi09-42\" rel=\"external_link\">[42]<\/a><\/sup> Patients in developed countries with diabetes tend to utilize PHRs more than healthy patients.<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-7\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HessExp07_35-8\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrantPrac08_41-3\" class=\"reference\"><a href=\"#cite_note-GrantPrac08-41\" rel=\"external_link\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HolbrookIndi09_42-5\" class=\"reference\"><a href=\"#cite_note-HolbrookIndi09-42\" rel=\"external_link\">[42]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KannryPers12_46-2\" class=\"reference\"><a href=\"#cite_note-KannryPers12-46\" rel=\"external_link\">[46]<\/a><\/sup>\n<\/p><p>Studies examining the effects of age, race, ethnicity, and sex on patient attitudes and usage have been inconclusive, demonstrating effects of these variables in some comparisons but not others.<sup id=\"rdp-ebb-cite_ref-KaelberARes08_9-2\" class=\"reference\"><a href=\"#cite_note-KaelberARes08-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-7\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KimUsage07_34-7\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LoberBar06_37-4\" class=\"reference\"><a href=\"#cite_note-LoberBar06-37\" rel=\"external_link\">[37]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KannryPers12_46-3\" class=\"reference\"><a href=\"#cite_note-KannryPers12-46\" rel=\"external_link\">[46]<\/a><\/sup> Factors such diagnosis, age, gender, and country of origin may account for the reported variation. Patient motivations for using a PHR are tied to and highly influenced by the patient-physician relationship, physician attitudes toward PHRs<sup id=\"rdp-ebb-cite_ref-WitryFam10_16-3\" class=\"reference\"><a href=\"#cite_note-WitryFam10-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-8\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-5\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KannryPers12_46-4\" class=\"reference\"><a href=\"#cite_note-KannryPers12-46\" rel=\"external_link\">[46]<\/a><\/sup>, and the desire for illness ownership<sup id=\"rdp-ebb-cite_ref-WitryFam10_16-4\" class=\"reference\"><a href=\"#cite_note-WitryFam10-16\" rel=\"external_link\">[16]<\/a><\/sup> and control over the dissemination of medical information.<sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-6\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperPat04_28-4\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoxThe08_30-2\" class=\"reference\"><a href=\"#cite_note-CoxThe08-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BrightBen07_31-1\" class=\"reference\"><a href=\"#cite_note-BrightBen07-31\" rel=\"external_link\">[31]<\/a><\/sup> Patients are likely to adopt, use, and value an integrated PHR that is user-friendly<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-8\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperPat04_28-5\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KimUsage07_34-8\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup>, provides a high level of privacy and security<sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-7\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoxThe08_30-3\" class=\"reference\"><a href=\"#cite_note-CoxThe08-30\" rel=\"external_link\">[30]<\/a><\/sup>, and offers advanced features such as messaging, editing, and medication renewal capabilities.<sup id=\"rdp-ebb-cite_ref-HassolPat04_33-5\" class=\"reference\"><a href=\"#cite_note-HassolPat04-33\" rel=\"external_link\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KimUsage07_34-9\" class=\"reference\"><a href=\"#cite_note-KimUsage07-34\" rel=\"external_link\">[34]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HessExp07_35-9\" class=\"reference\"><a href=\"#cite_note-HessExp07-35\" rel=\"external_link\">[35]<\/a><\/sup> Patient adoption and use will increase as health IT developers integrate patient feedback in the development process.\n<\/p><p>This review includes several limitations. First, the majority of studies included in this review were authored in the United States. However, 13 of the 52 papers selected were authored outside the United States, including five from Canada<sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-9\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WinkelmanPat05_24-7\" class=\"reference\"><a href=\"#cite_note-WinkelmanPat05-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WiljerUnder10_38-3\" class=\"reference\"><a href=\"#cite_note-WiljerUnder10-38\" rel=\"external_link\">[38]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShcherbatykhMeth08_40-1\" class=\"reference\"><a href=\"#cite_note-ShcherbatykhMeth08-40\" rel=\"external_link\">[40]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HolbrookIndi09_42-6\" class=\"reference\"><a href=\"#cite_note-HolbrookIndi09-42\" rel=\"external_link\">[42]<\/a><\/sup>, four from the United Kingdom<sup id=\"rdp-ebb-cite_ref-PagliariPot07_14-2\" class=\"reference\"><a href=\"#cite_note-PagliariPot07-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JonesAtt99_27-4\" class=\"reference\"><a href=\"#cite_note-JonesAtt99-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperPat04_28-6\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperAcc04_51-0\" class=\"reference\"><a href=\"#cite_note-PyperAcc04-51\" rel=\"external_link\">[51]<\/a><\/sup>, and the remaining from Spain<sup id=\"rdp-ebb-cite_ref-CarrionEval12_11-3\" class=\"reference\"><a href=\"#cite_note-CarrionEval12-11\" rel=\"external_link\">[11]<\/a><\/sup>, Serbia<sup id=\"rdp-ebb-cite_ref-StakicAgree10_12-1\" class=\"reference\"><a href=\"#cite_note-StakicAgree10-12\" rel=\"external_link\">[12]<\/a><\/sup>, Germany<sup id=\"rdp-ebb-cite_ref-SunyaevComp10_7-1\" class=\"reference\"><a href=\"#cite_note-SunyaevComp10-7\" rel=\"external_link\">[7]<\/a><\/sup>, and the Netherlands.<sup id=\"rdp-ebb-cite_ref-BrinkInv05_26-9\" class=\"reference\"><a href=\"#cite_note-BrinkInv05-26\" rel=\"external_link\">[26]<\/a><\/sup> Second, biases in the literature include small sample size<sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-8\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrantPrac08_41-4\" class=\"reference\"><a href=\"#cite_note-GrantPrac08-41\" rel=\"external_link\">[41]<\/a><\/sup>, non-random, self-selected samples<sup id=\"rdp-ebb-cite_ref-ZickmundInt05_25-8\" class=\"reference\"><a href=\"#cite_note-ZickmundInt05-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CoxThe08_30-4\" class=\"reference\"><a href=\"#cite_note-CoxThe08-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KeselmanTow07_32-8\" class=\"reference\"><a href=\"#cite_note-KeselmanTow07-32\" rel=\"external_link\">[32]<\/a><\/sup>, and limited ethnic, racial, and socioeconomic diversity.<sup id=\"rdp-ebb-cite_ref-KristInter12_13-6\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RalstonWeb09_15-9\" class=\"reference\"><a href=\"#cite_note-RalstonWeb09-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperPat04_28-7\" class=\"reference\"><a href=\"#cite_note-PyperPat04-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HassolPat04_33-6\" class=\"reference\"><a href=\"#cite_note-HassolPat04-33\" rel=\"external_link\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrantPrac08_41-5\" class=\"reference\"><a href=\"#cite_note-GrantPrac08-41\" rel=\"external_link\">[41]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PyperAcc04_51-1\" class=\"reference\"><a href=\"#cite_note-PyperAcc04-51\" rel=\"external_link\">[51]<\/a><\/sup> Only one paper included non-English speaking participants.<sup id=\"rdp-ebb-cite_ref-VentresPhys06_44-2\" class=\"reference\"><a href=\"#cite_note-VentresPhys06-44\" rel=\"external_link\">[44]<\/a><\/sup> Finally, although we have discussed many of the important findings and themes in the literature, it was not possible to detail every factor affecting PHR evolution and adoption, patient user attitudes toward PHRs, patient reported barriers to use, and the role of PHRs in self-management.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Integrated PHRs have the potential to improve the patient-provider relationship, enable shared decision-making, and allow the healthcare system to move toward a more personalized healthcare delivery system. Integrated PHRs will have a broader impact on public health as they evolve higher levels of functionality and as physicians increase PHR adoption.<sup id=\"rdp-ebb-cite_ref-KristInter12_13-7\" class=\"reference\"><a href=\"#cite_note-KristInter12-13\" rel=\"external_link\">[13]<\/a><\/sup> Since 2012, the federal government has given over $5 billion to providers and hospitals for the adoption and meaningful use of qualifying PHRs.<sup id=\"rdp-ebb-cite_ref-KannryPers12_46-5\" class=\"reference\"><a href=\"#cite_note-KannryPers12-46\" rel=\"external_link\">[46]<\/a><\/sup> Beginning in 2014, providers and hospital are required to make PHRs available to 50% of patients and achieve an adoption rate of 10%.<sup id=\"rdp-ebb-cite_ref-KannryPers12_46-6\" class=\"reference\"><a href=\"#cite_note-KannryPers12-46\" rel=\"external_link\">[46]<\/a><\/sup> Incentivizing providers and hospitals to adopt and become meaningful users of EHRs will provide more patients with the opportunity to use PHRs. However, the available evidence demonstrating that PHRs can support their intended use is insufficient as the existing literature on PHRs is limited and inconclusive.<sup id=\"rdp-ebb-cite_ref-KaelberARes08_9-3\" class=\"reference\"><a href=\"#cite_note-KaelberARes08-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArcherPers11_18-10\" class=\"reference\"><a href=\"#cite_note-ArcherPers11-18\" rel=\"external_link\">[18]<\/a><\/sup> Further patient- and physician-focused research on factors affecting PHR adoption and frequency of use is needed to improve PHR functionality, inform health IT development, and determine what motivates patients to not only adopt but to continue using PHRs.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h2>\n<p>No potential conflict of interest to report.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HHSTheONC08-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HHSTheONC08_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">US Department of Health and Human Services (June 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.healthit.gov\/sites\/default\/files\/hit-strategic-plan-summary-508-2.pdf\" target=\"_blank\">\"The ONC-Coordinated Federal Health Information Technology Strategic Plan: 2008U2012\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.healthit.gov\/sites\/default\/files\/hit-strategic-plan-summary-508-2.pdf\" target=\"_blank\">https:\/\/www.healthit.gov\/sites\/default\/files\/hit-strategic-plan-summary-508-2.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+ONC-Coordinated+Federal+Health+Information+Technology+Strategic+Plan%3A+2008U2012&rft.atitle=&rft.aulast=US+Department+of+Health+and+Human+Services&rft.au=US+Department+of+Health+and+Human+Services&rft.date=June+2008&rft_id=https%3A%2F%2Fwww.healthit.gov%2Fsites%2Fdefault%2Ffiles%2Fhit-strategic-plan-summary-508-2.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HHSPersonal06-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HHSPersonal06_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">US Department of Health and Human Services (February 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncvhs.hhs.gov\/wp-content\/uploads\/2014\/05\/0602nhiirpt.pdf\" target=\"_blank\">\"Personal Health Records and Personal Health Record Systems: A Report and Recommendation from the National Committee on Vital and Health Statistics\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ncvhs.hhs.gov\/wp-content\/uploads\/2014\/05\/0602nhiirpt.pdf\" target=\"_blank\">http:\/\/www.ncvhs.hhs.gov\/wp-content\/uploads\/2014\/05\/0602nhiirpt.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Personal+Health+Records+and+Personal+Health+Record+Systems%3A+A+Report+and+Recommendation+from+the+National+Committee+on+Vital+and+Health+Statistics&rft.atitle=&rft.aulast=US+Department+of+Health+and+Human+Services&rft.au=US+Department+of+Health+and+Human+Services&rft.date=February+2006&rft_id=http%3A%2F%2Fwww.ncvhs.hhs.gov%2Fwp-content%2Fuploads%2F2014%2F05%2F0602nhiirpt.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DetmerInt08-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DetmerInt08_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-1\" rel=\"external_link\">3.1<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-2\" rel=\"external_link\">3.2<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-3\" rel=\"external_link\">3.3<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-4\" rel=\"external_link\">3.4<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-5\" rel=\"external_link\">3.5<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-6\" rel=\"external_link\">3.6<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-7\" rel=\"external_link\">3.7<\/a><\/sup> <sup><a href=\"#cite_ref-DetmerInt08_3-8\" rel=\"external_link\">3.8<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Detmer, D.; Bloomrosen, M.; Raymond, B.; Tang, P. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2596104\" target=\"_blank\">\"Integrated personal health records: transformative tools for consumer-centric care\"<\/a>. <i>BioMed Central Medical Informatics and Decision Making<\/i> <b>8<\/b>: 45. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-8-45\" target=\"_blank\">10.1186\/1472-6947-8-45<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2596104\/\" target=\"_blank\">PMC2596104<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2596104\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2596104<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrated+personal+health+records%3A+transformative+tools+for+consumer-centric+care&rft.jtitle=BioMed+Central+Medical+Informatics+and+Decision+Making&rft.aulast=Detmer%2C+D.%3B+Bloomrosen%2C+M.%3B+Raymond%2C+B.%3B+Tang%2C+P.&rft.au=Detmer%2C+D.%3B+Bloomrosen%2C+M.%3B+Raymond%2C+B.%3B+Tang%2C+P.&rft.date=2008&rft.volume=8&rft.pages=45&rft_id=info:doi\/10.1186%2F1472-6947-8-45&rft_id=info:pmc\/PMC2596104&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2596104&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TangPers06-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TangPers06_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-TangPers06_4-1\" rel=\"external_link\">4.1<\/a><\/sup> <sup><a href=\"#cite_ref-TangPers06_4-2\" rel=\"external_link\">4.2<\/a><\/sup> <sup><a href=\"#cite_ref-TangPers06_4-3\" rel=\"external_link\">4.3<\/a><\/sup> <sup><a href=\"#cite_ref-TangPers06_4-4\" rel=\"external_link\">4.4<\/a><\/sup> <sup><a href=\"#cite_ref-TangPers06_4-5\" rel=\"external_link\">4.5<\/a><\/sup> <sup><a href=\"#cite_ref-TangPers06_4-6\" rel=\"external_link\">4.6<\/a><\/sup> <sup><a href=\"#cite_ref-TangPers06_4-7\" rel=\"external_link\">4.7<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tang, P.C.; Ash, J.S.; Bates, D.W.; Overhage, J.M.; Sands, D.Z. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551\" target=\"_blank\">\"Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>13<\/b> (2): 121-126. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2025\" target=\"_blank\">10.1197\/jamia.M2025<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1447551\/\" target=\"_blank\">PMC1447551<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16357345\" target=\"_blank\">16357345<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1447551<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Personal+health+records%3A+definitions%2C+benefits%2C+and+strategies+for+overcoming+barriers+to+adoption&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Tang%2C+P.C.%3B+Ash%2C+J.S.%3B+Bates%2C+D.W.%3B+Overhage%2C+J.M.%3B+Sands%2C+D.Z.&rft.au=Tang%2C+P.C.%3B+Ash%2C+J.S.%3B+Bates%2C+D.W.%3B+Overhage%2C+J.M.%3B+Sands%2C+D.Z.&rft.date=2006&rft.volume=13&rft.issue=2&rft.pages=121-126&rft_id=info:doi\/10.1197%2Fjamia.M2025&rft_id=info:pmc\/PMC1447551&rft_id=info:pmid\/16357345&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1447551&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GearonPers07-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GearonPers07_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-GearonPers07_5-1\" rel=\"external_link\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Gearon, C. (June 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/P\/PDF%20PHRPerspectives.pdf\" target=\"_blank\">\"Perspectives on the future of personal health records\"<\/a> (PDF). California HealthCare Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/P\/PDF%20PHRPerspectives.pdf\" target=\"_blank\">http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/P\/PDF%20PHRPerspectives.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Perspectives+on+the+future+of+personal+health+records&rft.atitle=&rft.aulast=Gearon%2C+C.&rft.au=Gearon%2C+C.&rft.date=June+2007&rft.pub=California+HealthCare+Foundation&rft_id=http%3A%2F%2Fwww.chcf.org%2F%7E%2Fmedia%2FMEDIA%2520LIBRARY%2520Files%2FPDF%2FP%2FPDF%2520PHRPerspectives.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KaelberThe08-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KaelberThe08_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-KaelberThe08_6-1\" rel=\"external_link\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kaelber, D.; Pan, E.C. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655982\" target=\"_blank\">\"The value of personal health records (PHR) systems\"<\/a>. <i>American Medical Informatics Association Annual Symposium Proceedings<\/i> <b>2008<\/b>: 343\u2013347. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2655982\/\" target=\"_blank\">PMC2655982<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655982\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655982<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+value+of+personal+health+records+%28PHR%29+systems&rft.jtitle=American+Medical+Informatics+Association+Annual+Symposium+Proceedings&rft.aulast=Kaelber%2C+D.%3B+Pan%2C+E.C.&rft.au=Kaelber%2C+D.%3B+Pan%2C+E.C.&rft.date=2008&rft.volume=2008&rft.pages=343%E2%80%93347&rft_id=info:pmc\/PMC2655982&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2655982&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SunyaevComp10-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SunyaevComp10_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-SunyaevComp10_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sunyaev, A.; Kaletsch, A.; Krcmar, H. (January 2010). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20130511163734\/http:\/\/www.alexander-kaletsch.de\/downloads\/api.pdf\" target=\"_blank\">\"Comparative evaluation of Google Health API vs. Microsoft Healthvault API\"<\/a> (PDF). <i>Proceedings of the Third International Conference on Health Informatics<\/i> <b>2010<\/b>: 195\u2013201. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.alexander-kaletsch.de\/downloads\/api.pdf\" target=\"_blank\">the original<\/a> on 11 May 2013<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20130511163734\/http:\/\/www.alexander-kaletsch.de\/downloads\/api.pdf\" target=\"_blank\">https:\/\/web.archive.org\/web\/20130511163734\/http:\/\/www.alexander-kaletsch.de\/downloads\/api.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparative+evaluation+of+Google+Health+API+vs.+Microsoft+Healthvault+API&rft.jtitle=Proceedings+of+the+Third+International+Conference+on+Health+Informatics&rft.aulast=Sunyaev%2C+A.%3B+Kaletsch%2C+A.%3B+Krcmar%2C+H.&rft.au=Sunyaev%2C+A.%3B+Kaletsch%2C+A.%3B+Krcmar%2C+H.&rft.date=January+2010&rft.volume=2010&rft.pages=195%E2%80%93201&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20130511163734%2Fhttp%3A%2F%2Fwww.alexander-kaletsch.de%2Fdownloads%2Fapi.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarkleConnect03-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarkleConnect03_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Markle Foundation (01 July 2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.policyarchive.org\/handle\/10207\/bitstreams\/15473.pdf\" target=\"_blank\">\"Connecting for Health: A Public-Private Collaborative\"<\/a> (PDF). p. 14<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.policyarchive.org\/handle\/10207\/bitstreams\/15473.pdf\" target=\"_blank\">http:\/\/www.policyarchive.org\/handle\/10207\/bitstreams\/15473.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Connecting+for+Health%3A+A+Public-Private+Collaborative&rft.atitle=&rft.aulast=Markle+Foundation&rft.au=Markle+Foundation&rft.date=01+July+2003&rft.pages=p.+14&rft_id=http%3A%2F%2Fwww.policyarchive.org%2Fhandle%2F10207%2Fbitstreams%2F15473.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KaelberARes08-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KaelberARes08_9-0\" rel=\"external_link\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-KaelberARes08_9-1\" rel=\"external_link\">9.1<\/a><\/sup> <sup><a href=\"#cite_ref-KaelberARes08_9-2\" rel=\"external_link\">9.2<\/a><\/sup> <sup><a href=\"#cite_ref-KaelberARes08_9-3\" rel=\"external_link\">9.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kaelber, D.C.; Jha, A.K.; Johnston, D.; Middleton, B.; Bates, D.W. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2585530\" target=\"_blank\">\"A research agenda for personal health records (PHRs)\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>15<\/b> (6): 729-736. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2547\" target=\"_blank\">10.1197\/jamia.M2547<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2585530\/\" target=\"_blank\">PMC2585530<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2585530\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2585530<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+research+agenda+for+personal+health+records+%28PHRs%29&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Kaelber%2C+D.C.%3B+Jha%2C+A.K.%3B+Johnston%2C+D.%3B+Middleton%2C+B.%3B+Bates%2C+D.W.&rft.au=Kaelber%2C+D.C.%3B+Jha%2C+A.K.%3B+Johnston%2C+D.%3B+Middleton%2C+B.%3B+Bates%2C+D.W.&rft.date=2008&rft.volume=15&rft.issue=6&rft.pages=729-736&rft_id=info:doi\/10.1197%2Fjamia.M2547&rft_id=info:pmc\/PMC2585530&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2585530&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarrisFew06-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HarrisFew06_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-HarrisFew06_10-1\" rel=\"external_link\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Harris Interactive (22 September 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.prnewswire.com\/news-releases\/few-patients-use-or-have-access-to-online-services-for-communicating-with-their-doctors-but-most-would-like-to-57121547.html\" target=\"_blank\">\"Few patients use or have access to online services for communicating with their doctors, but most would like to\"<\/a>. <i>The Wall Street Journal Online<\/i> <b>5<\/b> (16): 1\u20137<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.prnewswire.com\/news-releases\/few-patients-use-or-have-access-to-online-services-for-communicating-with-their-doctors-but-most-would-like-to-57121547.html\" target=\"_blank\">http:\/\/www.prnewswire.com\/news-releases\/few-patients-use-or-have-access-to-online-services-for-communicating-with-their-doctors-but-most-would-like-to-57121547.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Few+patients+use+or+have+access+to+online+services+for+communicating+with+their+doctors%2C+but+most+would+like+to&rft.jtitle=The+Wall+Street+Journal+Online&rft.aulast=Harris+Interactive&rft.au=Harris+Interactive&rft.date=22+September+2006&rft.volume=5&rft.issue=16&rft.pages=1%E2%80%937&rft_id=http%3A%2F%2Fwww.prnewswire.com%2Fnews-releases%2Ffew-patients-use-or-have-access-to-online-services-for-communicating-with-their-doctors-but-most-would-like-to-57121547.html&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CarrionEval12-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CarrionEval12_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-CarrionEval12_11-1\" rel=\"external_link\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-CarrionEval12_11-2\" rel=\"external_link\">11.2<\/a><\/sup> <sup><a href=\"#cite_ref-CarrionEval12_11-3\" rel=\"external_link\">11.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Carri\u00f3n, I.; Fern\u00e1ndez-Alem\u00e1n, J.L.; Jayne, C.; Palmer-Brown, D.; Toval, A.; Carrillo-de-Gea, J.M. (January 2012). \"Evaluation and neuronal network-based classification of the PHRs privacy policies\". <i>Proceedings of the 45th Hawaii International Conference on System Sciences<\/i> <b>2012<\/b>: 2840-2849. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FHICSS.2012.257\" target=\"_blank\">10.1109\/HICSS.2012.257<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluation+and+neuronal+network-based+classification+of+the+PHRs+privacy+policies&rft.jtitle=Proceedings+of+the+45th+Hawaii+International+Conference+on+System+Sciences&rft.aulast=Carri%C3%B3n%2C+I.%3B+Fern%C3%A1ndez-Alem%C3%A1n%2C+J.L.%3B+Jayne%2C+C.%3B+Palmer-Brown%2C+D.%3B+Toval%2C+A.%3B+Carrillo-de-Gea%2C+J.M.&rft.au=Carri%C3%B3n%2C+I.%3B+Fern%C3%A1ndez-Alem%C3%A1n%2C+J.L.%3B+Jayne%2C+C.%3B+Palmer-Brown%2C+D.%3B+Toval%2C+A.%3B+Carrillo-de-Gea%2C+J.M.&rft.date=January+2012&rft.volume=2012&rft.pages=2840-2849&rft_id=info:doi\/10.1109%2FHICSS.2012.257&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StakicAgree10-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-StakicAgree10_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-StakicAgree10_12-1\" rel=\"external_link\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stakic, S.B.; Teodosijevic, N. (August 2010). \"Agreement based distribution of responsibilities in national electronic health record systems\". <i>Proceedings of the IEEE International Conference on Social Computing \/ IEEE International Conference on Privacy, Security, Risk and Trust<\/i> <b>2010<\/b>: 607-614. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FSocialCom.2010.95\" target=\"_blank\">10.1109\/SocialCom.2010.95<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Agreement+based+distribution+of+responsibilities+in+national+electronic+health+record+systems&rft.jtitle=Proceedings+of+the+IEEE+International+Conference+on+Social+Computing+%2F+IEEE+International+Conference+on+Privacy%2C+Security%2C+Risk+and+Trust&rft.aulast=Stakic%2C+S.B.%3B+Teodosijevic%2C+N.&rft.au=Stakic%2C+S.B.%3B+Teodosijevic%2C+N.&rft.date=August+2010&rft.volume=2010&rft.pages=607-614&rft_id=info:doi\/10.1109%2FSocialCom.2010.95&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KristInter12-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KristInter12_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-KristInter12_13-1\" rel=\"external_link\">13.1<\/a><\/sup> <sup><a href=\"#cite_ref-KristInter12_13-2\" rel=\"external_link\">13.2<\/a><\/sup> <sup><a href=\"#cite_ref-KristInter12_13-3\" rel=\"external_link\">13.3<\/a><\/sup> <sup><a href=\"#cite_ref-KristInter12_13-4\" rel=\"external_link\">13.4<\/a><\/sup> <sup><a href=\"#cite_ref-KristInter12_13-5\" rel=\"external_link\">13.5<\/a><\/sup> <sup><a href=\"#cite_ref-KristInter12_13-6\" rel=\"external_link\">13.6<\/a><\/sup> <sup><a href=\"#cite_ref-KristInter12_13-7\" rel=\"external_link\">13.7<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krist, A.H.; Woolf, S.H.; Rothemich, S.F.; Johnson, R.E.; Peele, J.E.; Cunningham, T.D.; Longo, D.R.; Bello, G.A.; Matzke, G.R. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3392290\" target=\"_blank\">\"Interactive preventive health record to enhance delivery of recommended care: a randomized trial\"<\/a>. <i>Annals of Family Medicine<\/i> <b>10<\/b> (4): 312-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1370%2Fafm.1383\" target=\"_blank\">10.1370\/afm.1383<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3392290\/\" target=\"_blank\">PMC3392290<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22778119\" target=\"_blank\">22778119<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3392290\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3392290<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interactive+preventive+health+record+to+enhance+delivery+of+recommended+care%3A+a+randomized+trial&rft.jtitle=Annals+of+Family+Medicine&rft.aulast=Krist%2C+A.H.%3B+Woolf%2C+S.H.%3B+Rothemich%2C+S.F.%3B+Johnson%2C+R.E.%3B+Peele%2C+J.E.%3B+Cunningham%2C+T.D.%3B+Longo%2C+D.R.%3B+Bello%2C+G.A.%3B+Matzke%2C+G.R.&rft.au=Krist%2C+A.H.%3B+Woolf%2C+S.H.%3B+Rothemich%2C+S.F.%3B+Johnson%2C+R.E.%3B+Peele%2C+J.E.%3B+Cunningham%2C+T.D.%3B+Longo%2C+D.R.%3B+Bello%2C+G.A.%3B+Matzke%2C+G.R.&rft.date=2012&rft.volume=10&rft.issue=4&rft.pages=312-9&rft_id=info:doi\/10.1370%2Fafm.1383&rft_id=info:pmc\/PMC3392290&rft_id=info:pmid\/22778119&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3392290&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PagliariPot07-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PagliariPot07_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-PagliariPot07_14-1\" rel=\"external_link\">14.1<\/a><\/sup> <sup><a href=\"#cite_ref-PagliariPot07_14-2\" rel=\"external_link\">14.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pagliari, C.; Detmer, D.; Singleton, P. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1949437\" target=\"_blank\">\"Potential of electronic personal health records\"<\/a>. <i>British Medical Journal<\/i> <b>335<\/b> (7615): 330-333. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fbmj.39279.482963.AD\" target=\"_blank\">10.1136\/bmj.39279.482963.AD<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1949437\/\" target=\"_blank\">PMC1949437<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1949437\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1949437<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Potential+of+electronic+personal+health+records&rft.jtitle=British+Medical+Journal&rft.aulast=Pagliari%2C+C.%3B+Detmer%2C+D.%3B+Singleton%2C+P.&rft.au=Pagliari%2C+C.%3B+Detmer%2C+D.%3B+Singleton%2C+P.&rft.date=2007&rft.volume=335&rft.issue=7615&rft.pages=330-333&rft_id=info:doi\/10.1136%2Fbmj.39279.482963.AD&rft_id=info:pmc\/PMC1949437&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1949437&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RalstonWeb09-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RalstonWeb09_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-1\" rel=\"external_link\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-2\" rel=\"external_link\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-3\" rel=\"external_link\">15.3<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-4\" rel=\"external_link\">15.4<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-5\" rel=\"external_link\">15.5<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-6\" rel=\"external_link\">15.6<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-7\" rel=\"external_link\">15.7<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-8\" rel=\"external_link\">15.8<\/a><\/sup> <sup><a href=\"#cite_ref-RalstonWeb09_15-9\" rel=\"external_link\">15.9<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ralston, J.D.; Hirsch, I.B.; Hoath, J.; Mullen, M.; Cheadle, A.; Goldberg, H.I. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2628685\" target=\"_blank\">\"Web-based collaborative care for type 2 diabetes: a pilot randomized trial\"<\/a>. <i>Diabetes Care<\/i> <b>32<\/b> (2): 234-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2337%2Fdc08-1220\" target=\"_blank\">10.2337\/dc08-1220<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2628685\/\" target=\"_blank\">PMC2628685<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19017773\" target=\"_blank\">19017773<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2628685\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2628685<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Web-based+collaborative+care+for+type+2+diabetes%3A+a+pilot+randomized+trial&rft.jtitle=Diabetes+Care&rft.aulast=Ralston%2C+J.D.%3B+Hirsch%2C+I.B.%3B+Hoath%2C+J.%3B+Mullen%2C+M.%3B+Cheadle%2C+A.%3B+Goldberg%2C+H.I.&rft.au=Ralston%2C+J.D.%3B+Hirsch%2C+I.B.%3B+Hoath%2C+J.%3B+Mullen%2C+M.%3B+Cheadle%2C+A.%3B+Goldberg%2C+H.I.&rft.date=2009&rft.volume=32&rft.issue=2&rft.pages=234-9&rft_id=info:doi\/10.2337%2Fdc08-1220&rft_id=info:pmc\/PMC2628685&rft_id=info:pmid\/19017773&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2628685&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WitryFam10-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WitryFam10_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-WitryFam10_16-1\" rel=\"external_link\">16.1<\/a><\/sup> <sup><a href=\"#cite_ref-WitryFam10_16-2\" rel=\"external_link\">16.2<\/a><\/sup> <sup><a href=\"#cite_ref-WitryFam10_16-3\" rel=\"external_link\">16.3<\/a><\/sup> <sup><a href=\"#cite_ref-WitryFam10_16-4\" rel=\"external_link\">16.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Witry, M.J.; Doucette, W.R.; Daly, J.M.; Levy, B.T.; Chrischilles, E.A. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2805556\" target=\"_blank\">\"Family physician perceptions of personal health records\"<\/a>. <i>Perspectives in Health Information Management<\/i> <b>7<\/b> (Winter): 1d. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2805556\/\" target=\"_blank\">PMC2805556<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20697465\" target=\"_blank\">20697465<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2805556\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2805556<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Family+physician+perceptions+of+personal+health+records&rft.jtitle=Perspectives+in+Health+Information+Management&rft.aulast=Witry%2C+M.J.%3B+Doucette%2C+W.R.%3B+Daly%2C+J.M.%3B+Levy%2C+B.T.%3B+Chrischilles%2C+E.A.&rft.au=Witry%2C+M.J.%3B+Doucette%2C+W.R.%3B+Daly%2C+J.M.%3B+Levy%2C+B.T.%3B+Chrischilles%2C+E.A.&rft.date=2010&rft.volume=7&rft.issue=Winter&rft.pages=1d&rft_id=info:pmc\/PMC2805556&rft_id=info:pmid\/20697465&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2805556&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TobacmanUtil96-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TobacmanUtil96_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-TobacmanUtil96_17-1\" rel=\"external_link\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-TobacmanUtil96_17-2\" rel=\"external_link\">17.2<\/a><\/sup> <sup><a href=\"#cite_ref-TobacmanUtil96_17-3\" rel=\"external_link\">17.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tobacman, J.; Nolan, P. (1996). \"Utilization of a personal health record in a general medicine clinic\". <i>Journal of General Internal Medicine<\/i> <b>11<\/b> (6): 370-372. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/8803745\" target=\"_blank\">8803745<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Utilization+of+a+personal+health+record+in+a+general+medicine+clinic&rft.jtitle=Journal+of+General+Internal+Medicine&rft.aulast=Tobacman%2C+J.%3B+Nolan%2C+P.&rft.au=Tobacman%2C+J.%3B+Nolan%2C+P.&rft.date=1996&rft.volume=11&rft.issue=6&rft.pages=370-372&rft_id=info:pmid\/8803745&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArcherPers11-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ArcherPers11_18-0\" rel=\"external_link\">18.00<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-1\" rel=\"external_link\">18.01<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-2\" rel=\"external_link\">18.02<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-3\" rel=\"external_link\">18.03<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-4\" rel=\"external_link\">18.04<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-5\" rel=\"external_link\">18.05<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-6\" rel=\"external_link\">18.06<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-7\" rel=\"external_link\">18.07<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-8\" rel=\"external_link\">18.08<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-9\" rel=\"external_link\">18.09<\/a><\/sup> <sup><a href=\"#cite_ref-ArcherPers11_18-10\" rel=\"external_link\">18.10<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Archer, N.; Fevrier-Thomas, U.; Lokker, C.; McKibbon, K.A.; Straus, S.E. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3128401\" target=\"_blank\">\"Personal health records: a scoping review\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>18<\/b> (4): 515-522. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2011-000105\" target=\"_blank\">10.1136\/amiajnl-2011-000105<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3128401\/\" target=\"_blank\">PMC3128401<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21672914\" target=\"_blank\">21672914<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3128401\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3128401<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Personal+health+records%3A+a+scoping+review&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Archer%2C+N.%3B+Fevrier-Thomas%2C+U.%3B+Lokker%2C+C.%3B+McKibbon%2C+K.A.%3B+Straus%2C+S.E.&rft.au=Archer%2C+N.%3B+Fevrier-Thomas%2C+U.%3B+Lokker%2C+C.%3B+McKibbon%2C+K.A.%3B+Straus%2C+S.E.&rft.date=2011&rft.volume=18&rft.issue=4&rft.pages=515-522&rft_id=info:doi\/10.1136%2Famiajnl-2011-000105&rft_id=info:pmc\/PMC3128401&rft_id=info:pmid\/21672914&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3128401&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorTwo04-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorTwo04_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, H. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.prnewswire.com\/news-releases\/two-in-five-adults-keep-personal-or-family-health-records-and-almost-everybody-thinks-this-is-a-good-idea-71581677.html\" target=\"_blank\">\"Two in five adults keep personal or family health records and almost everybody thinks this is a good idea: electronic health records likely to grow rapidly\"<\/a>. <i>Health Care News<\/i> <b>4<\/b> (13)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.prnewswire.com\/news-releases\/two-in-five-adults-keep-personal-or-family-health-records-and-almost-everybody-thinks-this-is-a-good-idea-71581677.html\" target=\"_blank\">http:\/\/www.prnewswire.com\/news-releases\/two-in-five-adults-keep-personal-or-family-health-records-and-almost-everybody-thinks-this-is-a-good-idea-71581677.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Two+in+five+adults+keep+personal+or+family+health+records+and+almost+everybody+thinks+this+is+a+good+idea%3A+electronic+health+records+likely+to+grow+rapidly&rft.jtitle=Health+Care+News&rft.aulast=Taylor%2C+H.&rft.au=Taylor%2C+H.&rft.date=2004&rft.volume=4&rft.issue=13&rft_id=http%3A%2F%2Fwww.prnewswire.com%2Fnews-releases%2Ftwo-in-five-adults-keep-personal-or-family-health-records-and-almost-everybody-thinks-this-is-a-good-idea-71581677.html&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HsiaoElec10-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HsiaoElec10_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Hsiao, C-J.; Hing, E.; Socey, T.C.; Cai, B. (December 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cdc.gov\/nchs\/data\/hestat\/emr_ehr_09\/emr_ehr_09.pdf\" target=\"_blank\">\"Electronic medical record\/electronic health record systems of office-based physicians: United States, 2009 and preliminary 2010 state estimates\"<\/a> (PDF). National Center for Health Statistics<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cdc.gov\/nchs\/data\/hestat\/emr_ehr_09\/emr_ehr_09.pdf\" target=\"_blank\">http:\/\/www.cdc.gov\/nchs\/data\/hestat\/emr_ehr_09\/emr_ehr_09.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Electronic+medical+record%2Felectronic+health+record+systems+of+office-based+physicians%3A+United+States%2C+2009+and+preliminary+2010+state+estimates&rft.atitle=&rft.aulast=Hsiao%2C+C-J.%3B+Hing%2C+E.%3B+Socey%2C+T.C.%3B+Cai%2C+B.&rft.au=Hsiao%2C+C-J.%3B+Hing%2C+E.%3B+Socey%2C+T.C.%3B+Cai%2C+B.&rft.date=December+2010&rft.pub=National+Center+for+Health+Statistics&rft_id=http%3A%2F%2Fwww.cdc.gov%2Fnchs%2Fdata%2Fhestat%2Femr_ehr_09%2Femr_ehr_09.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UndemCons10-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UndemCons10_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Undem, T. (April 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/C\/PDF%20ConsumersHealthInfoTechnologyNationalSurvey.pdf\" target=\"_blank\">\"Consumers and health information technology: a national survey\"<\/a> (PDF). California HealthCare Foundation. pp. 40<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/C\/PDF%20ConsumersHealthInfoTechnologyNationalSurvey.pdf\" target=\"_blank\">http:\/\/www.chcf.org\/~\/media\/MEDIA%20LIBRARY%20Files\/PDF\/C\/PDF%20ConsumersHealthInfoTechnologyNationalSurvey.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Consumers+and+health+information+technology%3A+a+national+survey&rft.atitle=&rft.aulast=Undem%2C+T.&rft.au=Undem%2C+T.&rft.date=April+2010&rft.pages=pp.+40&rft.pub=California+HealthCare+Foundation&rft_id=http%3A%2F%2Fwww.chcf.org%2F%7E%2Fmedia%2FMEDIA%2520LIBRARY%2520Files%2FPDF%2FC%2FPDF%2520ConsumersHealthInfoTechnologyNationalSurvey.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarkleConnect08-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarkleConnect08_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Markle Foundation (June 2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.markle.org\/sites\/default\/files\/ResearchBrief-200806.pdf\" target=\"_blank\">\"Americans overwhelmingly believe electronic personal health records could improve their health\"<\/a> (PDF). pp. 7<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.markle.org\/sites\/default\/files\/ResearchBrief-200806.pdf\" target=\"_blank\">http:\/\/www.markle.org\/sites\/default\/files\/ResearchBrief-200806.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Americans+overwhelmingly+believe+electronic+personal+health+records+could+improve+their+health&rft.atitle=&rft.aulast=Markle+Foundation&rft.au=Markle+Foundation&rft.date=June+2008&rft.pages=pp.+7&rft_id=http%3A%2F%2Fwww.markle.org%2Fsites%2Fdefault%2Ffiles%2FResearchBrief-200806.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TenfordeThe11-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TenfordeThe11_23-0\" rel=\"external_link\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-TenfordeThe11_23-1\" rel=\"external_link\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-TenfordeThe11_23-2\" rel=\"external_link\">23.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tenforde, M.; Jain, A.; Hickner, J. (2011). \"The value of personal health records for chronic disease management: what do we know?\". <i>Family Medicine<\/i> <b>43<\/b> (5): 351-354. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21557106\" target=\"_blank\">21557106<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+value+of+personal+health+records+for+chronic+disease+management%3A+what+do+we+know%3F&rft.jtitle=Family+Medicine&rft.aulast=Tenforde%2C+M.%3B+Jain%2C+A.%3B+Hickner%2C+J.&rft.au=Tenforde%2C+M.%3B+Jain%2C+A.%3B+Hickner%2C+J.&rft.date=2011&rft.volume=43&rft.issue=5&rft.pages=351-354&rft_id=info:pmid\/21557106&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WinkelmanPat05-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WinkelmanPat05_24-0\" rel=\"external_link\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanPat05_24-1\" rel=\"external_link\">24.1<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanPat05_24-2\" rel=\"external_link\">24.2<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanPat05_24-3\" rel=\"external_link\">24.3<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanPat05_24-4\" rel=\"external_link\">24.4<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanPat05_24-5\" rel=\"external_link\">24.5<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanPat05_24-6\" rel=\"external_link\">24.6<\/a><\/sup> <sup><a href=\"#cite_ref-WinkelmanPat05_24-7\" rel=\"external_link\">24.7<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Winkelman, W.; Leonard, K.; Rossos, P. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090462\" target=\"_blank\">\"Patient-perceived usefulness of online electronic medical records: employing grounded theory in the development of information and communication technologies for use by patients living with chronic illness\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>12<\/b> (3): 306-314. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1090462\/\" target=\"_blank\">PMC1090462<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15684128\" target=\"_blank\">15684128<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090462\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090462<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patient-perceived+usefulness+of+online+electronic+medical+records%3A+employing+grounded+theory+in+the+development+of+information+and+communication+technologies+for+use+by+patients+living+with+chronic+illness&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Winkelman%2C+W.%3B+Leonard%2C+K.%3B+Rossos%2C+P.&rft.au=Winkelman%2C+W.%3B+Leonard%2C+K.%3B+Rossos%2C+P.&rft.date=2005&rft.volume=12&rft.issue=3&rft.pages=306-314&rft_id=info:pmc\/PMC1090462&rft_id=info:pmid\/15684128&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1090462&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZickmundInt05-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ZickmundInt05_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-1\" rel=\"external_link\">25.1<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-2\" rel=\"external_link\">25.2<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-3\" rel=\"external_link\">25.3<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-4\" rel=\"external_link\">25.4<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-5\" rel=\"external_link\">25.5<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-6\" rel=\"external_link\">25.6<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-7\" rel=\"external_link\">25.7<\/a><\/sup> <sup><a href=\"#cite_ref-ZickmundInt05_25-8\" rel=\"external_link\">25.8<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zickmund, S.; Hess, R.; Bryce, C.; McTigue, K.; Olshansky, E.; Fitzgerald, K.; Fischer, G. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2338160\" target=\"_blank\">\"Interest in the use of computerized patient portals: role of the provider-patient relationship\"<\/a>. <i>Journal of General Internal Medicine<\/i> <b>23<\/b> (S1): 20-26. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11606-007-0273-6\" target=\"_blank\">10.1007\/s11606-007-0273-6<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2338160\/\" target=\"_blank\">PMC2338160<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18095039\" target=\"_blank\">18095039<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2338160\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2338160<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interest+in+the+use+of+computerized+patient+portals%3A+role+of+the+provider-patient+relationship&rft.jtitle=Journal+of+General+Internal+Medicine&rft.aulast=Zickmund%2C+S.%3B+Hess%2C+R.%3B+Bryce%2C+C.%3B+McTigue%2C+K.%3B+Olshansky%2C+E.%3B+Fitzgerald%2C+K.%3B+Fischer%2C+G.&rft.au=Zickmund%2C+S.%3B+Hess%2C+R.%3B+Bryce%2C+C.%3B+McTigue%2C+K.%3B+Olshansky%2C+E.%3B+Fitzgerald%2C+K.%3B+Fischer%2C+G.&rft.date=2007&rft.volume=23&rft.issue=S1&rft.pages=20-26&rft_id=info:doi\/10.1007%2Fs11606-007-0273-6&rft_id=info:pmc\/PMC2338160&rft_id=info:pmid\/18095039&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2338160&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrinkInv05-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BrinkInv05_26-0\" rel=\"external_link\">26.0<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-1\" rel=\"external_link\">26.1<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-2\" rel=\"external_link\">26.2<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-3\" rel=\"external_link\">26.3<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-4\" rel=\"external_link\">26.4<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-5\" rel=\"external_link\">26.5<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-6\" rel=\"external_link\">26.6<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-7\" rel=\"external_link\">26.7<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-8\" rel=\"external_link\">26.8<\/a><\/sup> <sup><a href=\"#cite_ref-BrinkInv05_26-9\" rel=\"external_link\">26.9<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brink, J.; Moorman, P.; Boer, M.; Pruyn, J.; Verwoerd, C.; Bemmel, J. (2005). \"Involving the patient: a prospective study on use, appreciation and effectiveness of an information system in head and neck cancer care\". <i>International Journal of Medical Informatics<\/i> <b>74<\/b> (10): 839-849. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2005.03.021\" target=\"_blank\">10.1016\/j.ijmedinf.2005.03.021<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16043392\" target=\"_blank\">16043392<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Involving+the+patient%3A+a+prospective+study+on+use%2C+appreciation+and+effectiveness+of+an+information+system+in+head+and+neck+cancer+care&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Brink%2C+J.%3B+Moorman%2C+P.%3B+Boer%2C+M.%3B+Pruyn%2C+J.%3B+Verwoerd%2C+C.%3B+Bemmel%2C+J.&rft.au=Brink%2C+J.%3B+Moorman%2C+P.%3B+Boer%2C+M.%3B+Pruyn%2C+J.%3B+Verwoerd%2C+C.%3B+Bemmel%2C+J.&rft.date=2005&rft.volume=74&rft.issue=10&rft.pages=839-849&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2005.03.021&rft_id=info:pmid\/16043392&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JonesAtt99-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JonesAtt99_27-0\" rel=\"external_link\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-JonesAtt99_27-1\" rel=\"external_link\">27.1<\/a><\/sup> <sup><a href=\"#cite_ref-JonesAtt99_27-2\" rel=\"external_link\">27.2<\/a><\/sup> <sup><a href=\"#cite_ref-JonesAtt99_27-3\" rel=\"external_link\">27.3<\/a><\/sup> <sup><a href=\"#cite_ref-JonesAtt99_27-4\" rel=\"external_link\">27.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jones, R.; McConville, J.; Mason, D.; Macpherson, L.; Navin, L.; McEwen, J. (1999). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1313422\" target=\"_blank\">\"Attitudes towards, and utility of, an integrated medical-dental patient-held record in primary care\"<\/a>. <i>British Journal of General Practice<\/i> <b>49<\/b> (442): 368-73. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1313422\/\" target=\"_blank\">PMC1313422<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10736887\" target=\"_blank\">10736887<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1313422\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1313422<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Attitudes+towards%2C+and+utility+of%2C+an+integrated+medical-dental+patient-held+record+in+primary+care&rft.jtitle=British+Journal+of+General+Practice&rft.aulast=Jones%2C+R.%3B+McConville%2C+J.%3B+Mason%2C+D.%3B+Macpherson%2C+L.%3B+Navin%2C+L.%3B+McEwen%2C+J.&rft.au=Jones%2C+R.%3B+McConville%2C+J.%3B+Mason%2C+D.%3B+Macpherson%2C+L.%3B+Navin%2C+L.%3B+McEwen%2C+J.&rft.date=1999&rft.volume=49&rft.issue=442&rft.pages=368-73&rft_id=info:pmc\/PMC1313422&rft_id=info:pmid\/10736887&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1313422&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PyperPat04-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PyperPat04_28-0\" rel=\"external_link\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-PyperPat04_28-1\" rel=\"external_link\">28.1<\/a><\/sup> <sup><a href=\"#cite_ref-PyperPat04_28-2\" rel=\"external_link\">28.2<\/a><\/sup> <sup><a href=\"#cite_ref-PyperPat04_28-3\" rel=\"external_link\">28.3<\/a><\/sup> <sup><a href=\"#cite_ref-PyperPat04_28-4\" rel=\"external_link\">28.4<\/a><\/sup> <sup><a href=\"#cite_ref-PyperPat04_28-5\" rel=\"external_link\">28.5<\/a><\/sup> <sup><a href=\"#cite_ref-PyperPat04_28-6\" rel=\"external_link\">28.6<\/a><\/sup> <sup><a href=\"#cite_ref-PyperPat04_28-7\" rel=\"external_link\">28.7<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pyper, C.; Amery, J.; Watson, M.; Crook, C. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314776\" target=\"_blank\">\"Patients\u2019 experiences when accessing their online electronic patient records in primary care\"<\/a>. <i>British Journal of General Practice<\/i> <b>54<\/b> (498): 38-43. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1314776\/\" target=\"_blank\">PMC1314776<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/14965405\" target=\"_blank\">14965405<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314776\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1314776<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patients%E2%80%99+experiences+when+accessing+their+online+electronic+patient+records+in+primary+care&rft.jtitle=British+Journal+of+General+Practice&rft.aulast=Pyper%2C+C.%3B+Amery%2C+J.%3B+Watson%2C+M.%3B+Crook%2C+C.&rft.au=Pyper%2C+C.%3B+Amery%2C+J.%3B+Watson%2C+M.%3B+Crook%2C+C.&rft.date=2004&rft.volume=54&rft.issue=498&rft.pages=38-43&rft_id=info:pmc\/PMC1314776&rft_id=info:pmid\/14965405&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1314776&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZayasDo07-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ZayasDo07_29-0\" rel=\"external_link\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-ZayasDo07_29-1\" rel=\"external_link\">29.1<\/a><\/sup> <sup><a href=\"#cite_ref-ZayasDo07_29-2\" rel=\"external_link\">29.2<\/a><\/sup> <sup><a href=\"#cite_ref-ZayasDo07_29-3\" rel=\"external_link\">29.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zayas-Caban, T.; Valdez, R. (2007). \"Do patients understand how PHRs work?\". <i>American Medical Informatics Association Annual Symposium Proceedings<\/i> <b>11<\/b>: 1169. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18694265\" target=\"_blank\">18694265<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Do+patients+understand+how+PHRs+work%3F&rft.jtitle=American+Medical+Informatics+Association+Annual+Symposium+Proceedings&rft.aulast=Zayas-Caban%2C+T.%3B+Valdez%2C+R.&rft.au=Zayas-Caban%2C+T.%3B+Valdez%2C+R.&rft.date=2007&rft.volume=11&rft.pages=1169&rft_id=info:pmid\/18694265&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CoxThe08-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CoxThe08_30-0\" rel=\"external_link\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-CoxThe08_30-1\" rel=\"external_link\">30.1<\/a><\/sup> <sup><a href=\"#cite_ref-CoxThe08_30-2\" rel=\"external_link\">30.2<\/a><\/sup> <sup><a href=\"#cite_ref-CoxThe08_30-3\" rel=\"external_link\">30.3<\/a><\/sup> <sup><a href=\"#cite_ref-CoxThe08_30-4\" rel=\"external_link\">30.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cox, B.; Thornewill, J. (2008). \"The consumer\u2019s view of the electronic health record: engaging patients in EHR adoption\". <i>Journal of Healthcare Information Management<\/i> <b>22<\/b> (2): 43-7. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19266994\" target=\"_blank\">19266994<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+consumer%E2%80%99s+view+of+the+electronic+health+record%3A+engaging+patients+in+EHR+adoption&rft.jtitle=Journal+of+Healthcare+Information+Management&rft.aulast=Cox%2C+B.%3B+Thornewill%2C+J.&rft.au=Cox%2C+B.%3B+Thornewill%2C+J.&rft.date=2008&rft.volume=22&rft.issue=2&rft.pages=43-7&rft_id=info:pmid\/19266994&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrightBen07-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BrightBen07_31-0\" rel=\"external_link\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-BrightBen07_31-1\" rel=\"external_link\">31.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bright, B. (29 November 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wsj.com\/articles\/SB119565244262500549\" target=\"_blank\">\"Benefits of electronic health records seen as outweighing privacy risks\"<\/a>. <i>The Wall Street Journal<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.wsj.com\/articles\/SB119565244262500549\" target=\"_blank\">http:\/\/www.wsj.com\/articles\/SB119565244262500549<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Benefits+of+electronic+health+records+seen+as+outweighing+privacy+risks&rft.atitle=The+Wall+Street+Journal&rft.aulast=Bright%2C+B.&rft.au=Bright%2C+B.&rft.date=29+November+2007&rft_id=http%3A%2F%2Fwww.wsj.com%2Farticles%2FSB119565244262500549&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KeselmanTow07-32\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KeselmanTow07_32-0\" rel=\"external_link\">32.0<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-1\" rel=\"external_link\">32.1<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-2\" rel=\"external_link\">32.2<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-3\" rel=\"external_link\">32.3<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-4\" rel=\"external_link\">32.4<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-5\" rel=\"external_link\">32.5<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-6\" rel=\"external_link\">32.6<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-7\" rel=\"external_link\">32.7<\/a><\/sup> <sup><a href=\"#cite_ref-KeselmanTow07_32-8\" rel=\"external_link\">32.8<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Keselman, A.; Slaughter, L.; Smith, C.A.; Kim, H.; Divita, G.; Brown, A.; Tsai, C.; Zeng-Treitler, Q. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655877\" target=\"_blank\">\"Towards consumer-friendly PHRs: patients\u2019 experience with reviewing their health records\"<\/a>. <i>American Medical Informatics Association Annual Symposium Proceedings<\/i> <b>2007<\/b>: 399-403. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2655877\/\" target=\"_blank\">PMC2655877<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18693866\" target=\"_blank\">18693866<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655877\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655877<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+consumer-friendly+PHRs%3A+patients%E2%80%99+experience+with+reviewing+their+health+records&rft.jtitle=American+Medical+Informatics+Association+Annual+Symposium+Proceedings&rft.aulast=Keselman%2C+A.%3B+Slaughter%2C+L.%3B+Smith%2C+C.A.%3B+Kim%2C+H.%3B+Divita%2C+G.%3B+Brown%2C+A.%3B+Tsai%2C+C.%3B+Zeng-Treitler%2C+Q.&rft.au=Keselman%2C+A.%3B+Slaughter%2C+L.%3B+Smith%2C+C.A.%3B+Kim%2C+H.%3B+Divita%2C+G.%3B+Brown%2C+A.%3B+Tsai%2C+C.%3B+Zeng-Treitler%2C+Q.&rft.date=2007&rft.volume=2007&rft.pages=399-403&rft_id=info:pmc\/PMC2655877&rft_id=info:pmid\/18693866&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2655877&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HassolPat04-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HassolPat04_33-0\" rel=\"external_link\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-HassolPat04_33-1\" rel=\"external_link\">33.1<\/a><\/sup> <sup><a href=\"#cite_ref-HassolPat04_33-2\" rel=\"external_link\">33.2<\/a><\/sup> <sup><a href=\"#cite_ref-HassolPat04_33-3\" rel=\"external_link\">33.3<\/a><\/sup> <sup><a href=\"#cite_ref-HassolPat04_33-4\" rel=\"external_link\">33.4<\/a><\/sup> <sup><a href=\"#cite_ref-HassolPat04_33-5\" rel=\"external_link\">33.5<\/a><\/sup> <sup><a href=\"#cite_ref-HassolPat04_33-6\" rel=\"external_link\">33.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hassol, A.; Walker, J.; Kidder, D.; Rokita, K.; Young, D.; Pierdon, S.; Deitz, D.; Kuck, S.; Ortiz, E. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC524631\" target=\"_blank\">\"Patient experiences and attitudes about access to a patient electronic health care record and linked web messaging\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>11<\/b> (6): 505\u201313. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC524631\/\" target=\"_blank\">PMC524631<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15299001\" target=\"_blank\">15299001<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC524631\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC524631<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patient+experiences+and+attitudes+about+access+to+a+patient+electronic+health+care+record+and+linked+web+messaging&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Hassol%2C+A.%3B+Walker%2C+J.%3B+Kidder%2C+D.%3B+Rokita%2C+K.%3B+Young%2C+D.%3B+Pierdon%2C+S.%3B+Deitz%2C+D.%3B+Kuck%2C+S.%3B+Ortiz%2C+E.&rft.au=Hassol%2C+A.%3B+Walker%2C+J.%3B+Kidder%2C+D.%3B+Rokita%2C+K.%3B+Young%2C+D.%3B+Pierdon%2C+S.%3B+Deitz%2C+D.%3B+Kuck%2C+S.%3B+Ortiz%2C+E.&rft.date=2004&rft.volume=11&rft.issue=6&rft.pages=505%E2%80%9313&rft_id=info:pmc\/PMC524631&rft_id=info:pmid\/15299001&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC524631&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimUsage07-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KimUsage07_34-0\" rel=\"external_link\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-1\" rel=\"external_link\">34.1<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-2\" rel=\"external_link\">34.2<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-3\" rel=\"external_link\">34.3<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-4\" rel=\"external_link\">34.4<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-5\" rel=\"external_link\">34.5<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-6\" rel=\"external_link\">34.6<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-7\" rel=\"external_link\">34.7<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-8\" rel=\"external_link\">34.8<\/a><\/sup> <sup><a href=\"#cite_ref-KimUsage07_34-9\" rel=\"external_link\">34.9<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, E.; Stolyar, A.; Lober, W.; Herbaugh, A.; Shinstrom, S.; Zierler, B.; Soh, C.; Kim, Y. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655817\" target=\"_blank\">\"Usage patterns of a personal health record by elderly and disabled users\"<\/a>. <i>American Medical Informatics Association Annual Symposium Proceedings<\/i> <b>2007<\/b>: 409-13. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2655817\/\" target=\"_blank\">PMC2655817<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18693868\" target=\"_blank\">18693868<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655817\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655817<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Usage+patterns+of+a+personal+health+record+by+elderly+and+disabled+users&rft.jtitle=American+Medical+Informatics+Association+Annual+Symposium+Proceedings&rft.aulast=Kim%2C+E.%3B+Stolyar%2C+A.%3B+Lober%2C+W.%3B+Herbaugh%2C+A.%3B+Shinstrom%2C+S.%3B+Zierler%2C+B.%3B+Soh%2C+C.%3B+Kim%2C+Y.&rft.au=Kim%2C+E.%3B+Stolyar%2C+A.%3B+Lober%2C+W.%3B+Herbaugh%2C+A.%3B+Shinstrom%2C+S.%3B+Zierler%2C+B.%3B+Soh%2C+C.%3B+Kim%2C+Y.&rft.date=2007&rft.volume=2007&rft.pages=409-13&rft_id=info:pmc\/PMC2655817&rft_id=info:pmid\/18693868&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2655817&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HessExp07-35\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HessExp07_35-0\" rel=\"external_link\">35.0<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-1\" rel=\"external_link\">35.1<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-2\" rel=\"external_link\">35.2<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-3\" rel=\"external_link\">35.3<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-4\" rel=\"external_link\">35.4<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-5\" rel=\"external_link\">35.5<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-6\" rel=\"external_link\">35.6<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-7\" rel=\"external_link\">35.7<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-8\" rel=\"external_link\">35.8<\/a><\/sup> <sup><a href=\"#cite_ref-HessExp07_35-9\" rel=\"external_link\">35.9<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hess, R.; Bryce, C.; Paone, S.; Fischer, G.; McTigue, K.; Olshansky, E.; Zickmund, S.; Fitzgerald, K.; Siminerio, L. (2007). \"Exploring challenges and potentials of personal health records in diabetes self-management: implementation and initial assessment\". <i>Telemedicine and e-Health<\/i> <b>13<\/b> (5): 509-17. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Ftmj.2006.0089\" target=\"_blank\">10.1089\/tmj.2006.0089<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17999613\" target=\"_blank\">17999613<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+challenges+and+potentials+of+personal+health+records+in+diabetes+self-management%3A+implementation+and+initial+assessment&rft.jtitle=Telemedicine+and+e-Health&rft.aulast=Hess%2C+R.%3B+Bryce%2C+C.%3B+Paone%2C+S.%3B+Fischer%2C+G.%3B+McTigue%2C+K.%3B+Olshansky%2C+E.%3B+Zickmund%2C+S.%3B+Fitzgerald%2C+K.%3B+Siminerio%2C+L.&rft.au=Hess%2C+R.%3B+Bryce%2C+C.%3B+Paone%2C+S.%3B+Fischer%2C+G.%3B+McTigue%2C+K.%3B+Olshansky%2C+E.%3B+Zickmund%2C+S.%3B+Fitzgerald%2C+K.%3B+Siminerio%2C+L.&rft.date=2007&rft.volume=13&rft.issue=5&rft.pages=509-17&rft_id=info:doi\/10.1089%2Ftmj.2006.0089&rft_id=info:pmid\/17999613&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CiminoThe02-36\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CiminoThe02_36-0\" rel=\"external_link\">36.0<\/a><\/sup> <sup><a href=\"#cite_ref-CiminoThe02_36-1\" rel=\"external_link\">36.1<\/a><\/sup> <sup><a href=\"#cite_ref-CiminoThe02_36-2\" rel=\"external_link\">36.2<\/a><\/sup> <sup><a href=\"#cite_ref-CiminoThe02_36-3\" rel=\"external_link\">36.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cimino, J.J.; Patel, V.L.; Kushniruk, A.W. (2002). \"The patient clinical information system (PatCIS): technical solutions for and experience with giving patients access to their electronic medical records\". <i>International Journal of Medical Informatics<\/i> <b>68<\/b> (1-3): 113-27. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1386-5056%2802%2900070-9\" target=\"_blank\">10.1016\/S1386-5056(02)00070-9<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12467796\" target=\"_blank\">12467796<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+patient+clinical+information+system+%28PatCIS%29%3A+technical+solutions+for+and+experience+with+giving+patients+access+to+their+electronic+medical+records&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Cimino%2C+J.J.%3B+Patel%2C+V.L.%3B+Kushniruk%2C+A.W.&rft.au=Cimino%2C+J.J.%3B+Patel%2C+V.L.%3B+Kushniruk%2C+A.W.&rft.date=2002&rft.volume=68&rft.issue=1-3&rft.pages=113-27&rft_id=info:doi\/10.1016%2FS1386-5056%2802%2900070-9&rft_id=info:pmid\/12467796&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LoberBar06-37\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LoberBar06_37-0\" rel=\"external_link\">37.0<\/a><\/sup> <sup><a href=\"#cite_ref-LoberBar06_37-1\" rel=\"external_link\">37.1<\/a><\/sup> <sup><a href=\"#cite_ref-LoberBar06_37-2\" rel=\"external_link\">37.2<\/a><\/sup> <sup><a href=\"#cite_ref-LoberBar06_37-3\" rel=\"external_link\">37.3<\/a><\/sup> <sup><a href=\"#cite_ref-LoberBar06_37-4\" rel=\"external_link\">37.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lober, W.; Zierler, B.; Herbaugh, A.; Stolyer, A.; Kim, E.H.; Kim, Y. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839577\" target=\"_blank\">\"Barriers to the use of a personal health record by an elderly population\"<\/a>. <i>American Medical Informatics Association Annual Symposium Proceedings<\/i> <b>2006<\/b>: 514-518. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1839577\/\" target=\"_blank\">PMC1839577<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17238394\" target=\"_blank\">17238394<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839577\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839577<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Barriers+to+the+use+of+a+personal+health+record+by+an+elderly+population&rft.jtitle=American+Medical+Informatics+Association+Annual+Symposium+Proceedings&rft.aulast=Lober%2C+W.%3B+Zierler%2C+B.%3B+Herbaugh%2C+A.%3B+Stolyer%2C+A.%3B+Kim%2C+E.H.%3B+Kim%2C+Y.&rft.au=Lober%2C+W.%3B+Zierler%2C+B.%3B+Herbaugh%2C+A.%3B+Stolyer%2C+A.%3B+Kim%2C+E.H.%3B+Kim%2C+Y.&rft.date=2006&rft.volume=2006&rft.pages=514-518&rft_id=info:pmc\/PMC1839577&rft_id=info:pmid\/17238394&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1839577&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WiljerUnder10-38\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WiljerUnder10_38-0\" rel=\"external_link\">38.0<\/a><\/sup> <sup><a href=\"#cite_ref-WiljerUnder10_38-1\" rel=\"external_link\">38.1<\/a><\/sup> <sup><a href=\"#cite_ref-WiljerUnder10_38-2\" rel=\"external_link\">38.2<\/a><\/sup> <sup><a href=\"#cite_ref-WiljerUnder10_38-3\" rel=\"external_link\">38.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wiljer, D.; Urowitz, S.; Apatu, E.; Leonard, D.; Quartey, N.K.; Catton, P. (2010). \"Understanding the support needs of patients accessing test results online\". <i>Journal of Healthcare Information Management<\/i> <b>24<\/b> (1): 57-63. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20077927\" target=\"_blank\">20077927<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+the+support+needs+of+patients+accessing+test+results+online&rft.jtitle=Journal+of+Healthcare+Information+Management&rft.aulast=Wiljer%2C+D.%3B+Urowitz%2C+S.%3B+Apatu%2C+E.%3B+Leonard%2C+D.%3B+Quartey%2C+N.K.%3B+Catton%2C+P.&rft.au=Wiljer%2C+D.%3B+Urowitz%2C+S.%3B+Apatu%2C+E.%3B+Leonard%2C+D.%3B+Quartey%2C+N.K.%3B+Catton%2C+P.&rft.date=2010&rft.volume=24&rft.issue=1&rft.pages=57-63&rft_id=info:pmid\/20077927&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZengMaking07-39\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZengMaking07_39-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zeng-Treitler, Q.; Goryachev, S.; Hyeoneui, K.; Keselman, A.; Rosendale, D. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655860\" target=\"_blank\">\"Making texts in electronic health records comprehensible to consumers: a prototype translator\"<\/a>. <i>American Medical Informatics Association Annual Symposium Proceedings<\/i> <b>2007<\/b>: 846-50. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2655860\/\" target=\"_blank\">PMC2655860<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18693956\" target=\"_blank\">18693956<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655860\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655860<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Making+texts+in+electronic+health+records+comprehensible+to+consumers%3A+a+prototype+translator&rft.jtitle=American+Medical+Informatics+Association+Annual+Symposium+Proceedings&rft.aulast=Zeng-Treitler%2C+Q.%3B+Goryachev%2C+S.%3B+Hyeoneui%2C+K.%3B+Keselman%2C+A.%3B+Rosendale%2C+D.&rft.au=Zeng-Treitler%2C+Q.%3B+Goryachev%2C+S.%3B+Hyeoneui%2C+K.%3B+Keselman%2C+A.%3B+Rosendale%2C+D.&rft.date=2007&rft.volume=2007&rft.pages=846-50&rft_id=info:pmc\/PMC2655860&rft_id=info:pmid\/18693956&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2655860&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShcherbatykhMeth08-40\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShcherbatykhMeth08_40-0\" rel=\"external_link\">40.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShcherbatykhMeth08_40-1\" rel=\"external_link\">40.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shcherbatykh, I.; Holbrook, A.; Thabane, L.; Dolovich, L. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2528041\" target=\"_blank\">\"Methodologic issues in health informatics trials: the complexities of complex interventions\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>15<\/b> (5): 575-80. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2518\" target=\"_blank\">10.1197\/jamia.M2518<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2528041\/\" target=\"_blank\">PMC2528041<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18579839\" target=\"_blank\">18579839<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2528041\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2528041<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Methodologic+issues+in+health+informatics+trials%3A+the+complexities+of+complex+interventions&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Shcherbatykh%2C+I.%3B+Holbrook%2C+A.%3B+Thabane%2C+L.%3B+Dolovich%2C+L.&rft.au=Shcherbatykh%2C+I.%3B+Holbrook%2C+A.%3B+Thabane%2C+L.%3B+Dolovich%2C+L.&rft.date=2008&rft.volume=15&rft.issue=5&rft.pages=575-80&rft_id=info:doi\/10.1197%2Fjamia.M2518&rft_id=info:pmc\/PMC2528041&rft_id=info:pmid\/18579839&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2528041&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrantPrac08-41\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GrantPrac08_41-0\" rel=\"external_link\">41.0<\/a><\/sup> <sup><a href=\"#cite_ref-GrantPrac08_41-1\" rel=\"external_link\">41.1<\/a><\/sup> <sup><a href=\"#cite_ref-GrantPrac08_41-2\" rel=\"external_link\">41.2<\/a><\/sup> <sup><a href=\"#cite_ref-GrantPrac08_41-3\" rel=\"external_link\">41.3<\/a><\/sup> <sup><a href=\"#cite_ref-GrantPrac08_41-4\" rel=\"external_link\">41.4<\/a><\/sup> <sup><a href=\"#cite_ref-GrantPrac08_41-5\" rel=\"external_link\">41.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grant, R.W.; Wald, J.S.; Schnipper, J.L.; Gandhi, T.K.; Poon, E.G.; Orav, E.J.; William, D.H.; Volk, L.A.; Middleton, B. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829635\" target=\"_blank\">\"Practice-linked online personal health records for type 2 diabetes mellitus: a randomized controlled trial\"<\/a>. <i>Archives of Internal Medicine<\/i> <b>168<\/b> (16): 1776-1782. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Farchinte.168.16.1776\" target=\"_blank\">10.1001\/archinte.168.16.1776<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3829635\/\" target=\"_blank\">PMC3829635<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18779465\" target=\"_blank\">18779465<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829635\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3829635<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Practice-linked+online+personal+health+records+for+type+2+diabetes+mellitus%3A+a+randomized+controlled+trial&rft.jtitle=Archives+of+Internal+Medicine&rft.aulast=Grant%2C+R.W.%3B+Wald%2C+J.S.%3B+Schnipper%2C+J.L.%3B+Gandhi%2C+T.K.%3B+Poon%2C+E.G.%3B+Orav%2C+E.J.%3B+William%2C+D.H.%3B+Volk%2C+L.A.%3B+Middleton%2C+B.&rft.au=Grant%2C+R.W.%3B+Wald%2C+J.S.%3B+Schnipper%2C+J.L.%3B+Gandhi%2C+T.K.%3B+Poon%2C+E.G.%3B+Orav%2C+E.J.%3B+William%2C+D.H.%3B+Volk%2C+L.A.%3B+Middleton%2C+B.&rft.date=2008&rft.volume=168&rft.issue=16&rft.pages=1776-1782&rft_id=info:doi\/10.1001%2Farchinte.168.16.1776&rft_id=info:pmc\/PMC3829635&rft_id=info:pmid\/18779465&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3829635&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HolbrookIndi09-42\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HolbrookIndi09_42-0\" rel=\"external_link\">42.0<\/a><\/sup> <sup><a href=\"#cite_ref-HolbrookIndi09_42-1\" rel=\"external_link\">42.1<\/a><\/sup> <sup><a href=\"#cite_ref-HolbrookIndi09_42-2\" rel=\"external_link\">42.2<\/a><\/sup> <sup><a href=\"#cite_ref-HolbrookIndi09_42-3\" rel=\"external_link\">42.3<\/a><\/sup> <sup><a href=\"#cite_ref-HolbrookIndi09_42-4\" rel=\"external_link\">42.4<\/a><\/sup> <sup><a href=\"#cite_ref-HolbrookIndi09_42-5\" rel=\"external_link\">42.5<\/a><\/sup> <sup><a href=\"#cite_ref-HolbrookIndi09_42-6\" rel=\"external_link\">42.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Holbrook, A.; Thabane, L.; Keshavjee, K.; Dolovich, L.; Bernstein, B.; Chan, D.; Troyan, S.; Foster, G.; Gerstein, H. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2704409\" target=\"_blank\">\"Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial\"<\/a>. <i>Canadian Medical Association Journal<\/i> <b>181<\/b> (1\u20132): 37-44. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1503%2Fcmaj.081272\" target=\"_blank\">10.1503\/cmaj.081272<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2704409\/\" target=\"_blank\">PMC2704409<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19581618\" target=\"_blank\">19581618<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2704409\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2704409<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Individualized+electronic+decision+support+and+reminders+to+improve+diabetes+care+in+the+community%3A+COMPETE+II+randomized+trial&rft.jtitle=Canadian+Medical+Association+Journal&rft.aulast=Holbrook%2C+A.%3B+Thabane%2C+L.%3B+Keshavjee%2C+K.%3B+Dolovich%2C+L.%3B+Bernstein%2C+B.%3B+Chan%2C+D.%3B+Troyan%2C+S.%3B+Foster%2C+G.%3B+Gerstein%2C+H.&rft.au=Holbrook%2C+A.%3B+Thabane%2C+L.%3B+Keshavjee%2C+K.%3B+Dolovich%2C+L.%3B+Bernstein%2C+B.%3B+Chan%2C+D.%3B+Troyan%2C+S.%3B+Foster%2C+G.%3B+Gerstein%2C+H.&rft.date=2009&rft.volume=181&rft.issue=1%E2%80%932&rft.pages=37-44&rft_id=info:doi\/10.1503%2Fcmaj.081272&rft_id=info:pmc\/PMC2704409&rft_id=info:pmid\/19581618&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2704409&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrennanProj10-43\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrennanProj10_43-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brennan, P.F.; Downs, S.; Casper, G. (2010). \"Project HealthDesign: rethinking the power and potential of personal health records\". <i>Journal of Biomedical Informatics<\/i> <b>43<\/b> (5): S3\u20135. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2010.09.001\" target=\"_blank\">10.1016\/j.jbi.2010.09.001<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20937482\" target=\"_blank\">20937482<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Project+HealthDesign%3A+rethinking+the+power+and+potential+of+personal+health+records&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Brennan%2C+P.F.%3B+Downs%2C+S.%3B+Casper%2C+G.&rft.au=Brennan%2C+P.F.%3B+Downs%2C+S.%3B+Casper%2C+G.&rft.date=2010&rft.volume=43&rft.issue=5&rft.pages=S3%E2%80%935&rft_id=info:doi\/10.1016%2Fj.jbi.2010.09.001&rft_id=info:pmid\/20937482&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VentresPhys06-44\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VentresPhys06_44-0\" rel=\"external_link\">44.0<\/a><\/sup> <sup><a href=\"#cite_ref-VentresPhys06_44-1\" rel=\"external_link\">44.1<\/a><\/sup> <sup><a href=\"#cite_ref-VentresPhys06_44-2\" rel=\"external_link\">44.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ventres, W.; Kooienga, S.; Vuckovic, N.; Marlin, R.; Nygren, P.; Stewart, V. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1467009\" target=\"_blank\">\"Physicians, patients, and the electronic health record: an ethnographic analysis\"<\/a>. <i>Annals of Family Medicine<\/i> <b>4<\/b> (2): 124-131. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1370%2Fafm.425\" target=\"_blank\">10.1370\/afm.425<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1467009\/\" target=\"_blank\">PMC1467009<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16569715\" target=\"_blank\">16569715<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1467009\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1467009<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Physicians%2C+patients%2C+and+the+electronic+health+record%3A+an+ethnographic+analysis&rft.jtitle=Annals+of+Family+Medicine&rft.aulast=Ventres%2C+W.%3B+Kooienga%2C+S.%3B+Vuckovic%2C+N.%3B+Marlin%2C+R.%3B+Nygren%2C+P.%3B+Stewart%2C+V.&rft.au=Ventres%2C+W.%3B+Kooienga%2C+S.%3B+Vuckovic%2C+N.%3B+Marlin%2C+R.%3B+Nygren%2C+P.%3B+Stewart%2C+V.&rft.date=2006&rft.volume=4&rft.issue=2&rft.pages=124-131&rft_id=info:doi\/10.1370%2Fafm.425&rft_id=info:pmc\/PMC1467009&rft_id=info:pmid\/16569715&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1467009&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeitzmanAccept09-45\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WeitzmanAccept09_45-0\" rel=\"external_link\">45.0<\/a><\/sup> <sup><a href=\"#cite_ref-WeitzmanAccept09_45-1\" rel=\"external_link\">45.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Weitzman, E.R.; Kaci, L.; Mandl, K.D. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2762802\" target=\"_blank\">\"Acceptability of a personally controlled health record in a community-based setting: implications for policy and design\"<\/a>. <i>Journal of Medical Internet Research<\/i> <b>11<\/b> (2): e14. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fjmir.1187\" target=\"_blank\">10.2196\/jmir.1187<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2762802\/\" target=\"_blank\">PMC2762802<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19403467\" target=\"_blank\">19403467<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2762802\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2762802<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Acceptability+of+a+personally+controlled+health+record+in+a+community-based+setting%3A+implications+for+policy+and+design&rft.jtitle=Journal+of+Medical+Internet+Research&rft.aulast=Weitzman%2C+E.R.%3B+Kaci%2C+L.%3B+Mandl%2C+K.D.&rft.au=Weitzman%2C+E.R.%3B+Kaci%2C+L.%3B+Mandl%2C+K.D.&rft.date=2009&rft.volume=11&rft.issue=2&rft.pages=e14&rft_id=info:doi\/10.2196%2Fjmir.1187&rft_id=info:pmc\/PMC2762802&rft_id=info:pmid\/19403467&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2762802&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KannryPers12-46\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KannryPers12_46-0\" rel=\"external_link\">46.0<\/a><\/sup> <sup><a href=\"#cite_ref-KannryPers12_46-1\" rel=\"external_link\">46.1<\/a><\/sup> <sup><a href=\"#cite_ref-KannryPers12_46-2\" rel=\"external_link\">46.2<\/a><\/sup> <sup><a href=\"#cite_ref-KannryPers12_46-3\" rel=\"external_link\">46.3<\/a><\/sup> <sup><a href=\"#cite_ref-KannryPers12_46-4\" rel=\"external_link\">46.4<\/a><\/sup> <sup><a href=\"#cite_ref-KannryPers12_46-5\" rel=\"external_link\">46.5<\/a><\/sup> <sup><a href=\"#cite_ref-KannryPers12_46-6\" rel=\"external_link\">46.6<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kannry, J.; Beuria, P.; Wang, E.; Nissim, J. (2012). \"Personal health records: meaningful use, but for whom?\". <i>Mount Sinai Journal of Medicine: A Journal of Translational and Personalized Medicine<\/i> <b>79<\/b> (5): 593-602. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fmsj.21334\" target=\"_blank\">10.1002\/msj.21334<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22976365\" target=\"_blank\">22976365<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Personal+health+records%3A+meaningful+use%2C+but+for+whom%3F&rft.jtitle=Mount+Sinai+Journal+of+Medicine%3A+A+Journal+of+Translational+and+Personalized+Medicine&rft.aulast=Kannry%2C+J.%3B+Beuria%2C+P.%3B+Wang%2C+E.%3B+Nissim%2C+J.&rft.au=Kannry%2C+J.%3B+Beuria%2C+P.%3B+Wang%2C+E.%3B+Nissim%2C+J.&rft.date=2012&rft.volume=79&rft.issue=5&rft.pages=593-602&rft_id=info:doi\/10.1002%2Fmsj.21334&rft_id=info:pmid\/22976365&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KristAVis11-47\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KristAVis11_47-0\" rel=\"external_link\">47.0<\/a><\/sup> <sup><a href=\"#cite_ref-KristAVis11_47-1\" rel=\"external_link\">47.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krist, A.H.; Woolf, S.H. (2011). \"A vision for patient-centered health information systems\". <i>The Journal of the American Medical Association<\/i> <b>305<\/b> (3): 300-301. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2010.2011\" target=\"_blank\">10.1001\/jama.2010.2011<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21245186\" target=\"_blank\">21245186<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+vision+for+patient-centered+health+information+systems&rft.jtitle=The+Journal+of+the+American+Medical+Association&rft.aulast=Krist%2C+A.H.%3B+Woolf%2C+S.H.&rft.au=Krist%2C+A.H.%3B+Woolf%2C+S.H.&rft.date=2011&rft.volume=305&rft.issue=3&rft.pages=300-301&rft_id=info:doi\/10.1001%2Fjama.2010.2011&rft_id=info:pmid\/21245186&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GibbonsImpact09-48\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GibbonsImpact09_48-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Gibbons, M.C.; Wilson, R.F.; Samal, L.; Lehmann, C.U.; Dickersin, K.; Lehmann, H.P.; Aboumatar, H.; Finkelstein, J.; Shelton, E.; Sharma, R.; Bass, E.B. (October 2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ahrq.gov\/research\/findings\/evidence-based-reports\/chiapp-evidence-report.pdf\" target=\"_blank\">\"Impact of Consumer Health Informatics Applications\"<\/a> (PDF). U.S. Department of Health and Human Services. pp. 546<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ahrq.gov\/research\/findings\/evidence-based-reports\/chiapp-evidence-report.pdf\" target=\"_blank\">http:\/\/www.ahrq.gov\/research\/findings\/evidence-based-reports\/chiapp-evidence-report.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Impact+of+Consumer+Health+Informatics+Applications&rft.atitle=&rft.aulast=Gibbons%2C+M.C.%3B+Wilson%2C+R.F.%3B+Samal%2C+L.%3B+Lehmann%2C+C.U.%3B+Dickersin%2C+K.%3B+Lehmann%2C+H.P.%3B+Aboumatar%2C+H.%3B+Finkelstein%2C+J.%3B+Shelton%2C+E.%3B+Sharma%2C+R.%3B+Bass%2C+E.B.&rft.au=Gibbons%2C+M.C.%3B+Wilson%2C+R.F.%3B+Samal%2C+L.%3B+Lehmann%2C+C.U.%3B+Dickersin%2C+K.%3B+Lehmann%2C+H.P.%3B+Aboumatar%2C+H.%3B+Finkelstein%2C+J.%3B+Shelton%2C+E.%3B+Sharma%2C+R.%3B+Bass%2C+E.B.&rft.date=October+2009&rft.pages=pp.+546&rft.pub=U.S.+Department+of+Health+and+Human+Services&rft_id=http%3A%2F%2Fwww.ahrq.gov%2Fresearch%2Ffindings%2Fevidence-based-reports%2Fchiapp-evidence-report.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimHist11-49\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KimHist11_49-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, J.; Jung, H.; Bates, D.W. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092992\" target=\"_blank\">\"History and trends of \"personal health record\" research in pubmed\"<\/a>. <i>Healthcare Informatics Research<\/i> <b>17<\/b> (1): 3-17. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4258%2Fhir.2011.17.1.3\" target=\"_blank\">10.4258\/hir.2011.17.1.3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3092992\/\" target=\"_blank\">PMC3092992<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21818452\" target=\"_blank\">21818452<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092992\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092992<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=History+and+trends+of+%22personal+health+record%22+research+in+pubmed&rft.jtitle=Healthcare+Informatics+Research&rft.aulast=Kim%2C+J.%3B+Jung%2C+H.%3B+Bates%2C+D.W.&rft.au=Kim%2C+J.%3B+Jung%2C+H.%3B+Bates%2C+D.W.&rft.date=2011&rft.volume=17&rft.issue=1&rft.pages=3-17&rft_id=info:doi\/10.4258%2Fhir.2011.17.1.3&rft_id=info:pmc\/PMC3092992&rft_id=info:pmid\/21818452&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3092992&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CCHITRec08-50\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CCHITRec08_50-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Certification Commission for Healthcare Information Technology (15 July 2008). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/hitanalyst.files.wordpress.com\/2008\/07\/cchitphratf.pdf\" target=\"_blank\">\"Recommendations of the PHR advisory task force: certification of PHRs\"<\/a> (PDF)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/hitanalyst.files.wordpress.com\/2008\/07\/cchitphratf.pdf\" target=\"_blank\">https:\/\/hitanalyst.files.wordpress.com\/2008\/07\/cchitphratf.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Recommendations+of+the+PHR+advisory+task+force%3A+certification+of+PHRs&rft.atitle=&rft.aulast=Certification+Commission+for+Healthcare+Information+Technology&rft.au=Certification+Commission+for+Healthcare+Information+Technology&rft.date=15+July+2008&rft_id=https%3A%2F%2Fhitanalyst.files.wordpress.com%2F2008%2F07%2Fcchitphratf.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PyperAcc04-51\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PyperAcc04_51-0\" rel=\"external_link\">51.0<\/a><\/sup> <sup><a href=\"#cite_ref-PyperAcc04_51-1\" rel=\"external_link\">51.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pyper, C.; Amery, J.; Watson, M.; Crook, C. (2004). \"Access to electronic health records in primary care - a survey of patients' views\". <i>Medical Science Monitor<\/i> <b>10<\/b> (11): 17\u201322. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15507869\" target=\"_blank\">15507869<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Access+to+electronic+health+records+in+primary+care+-+a+survey+of+patients%27+views&rft.jtitle=Medical+Science+Monitor&rft.aulast=Pyper%2C+C.%3B+Amery%2C+J.%3B+Watson%2C+M.%3B+Crook%2C+C.&rft.au=Pyper%2C+C.%3B+Amery%2C+J.%3B+Watson%2C+M.%3B+Crook%2C+C.&rft.date=2004&rft.volume=10&rft.issue=11&rft.pages=17%E2%80%9322&rft_id=info:pmid\/15507869&rfr_id=info:sid\/en.wikipedia.org:Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. Table 1\u20133 have been placed slightly differently. Some references didn't include links to PDF files or were missing issue numbers. Additionally, all journal references did not list DOIs and PubMed IDs; these have been added to the references to make them more useful. The authors also included reference item 40 (Kupchunas W. Personal health record new opportunity for patient education. Orthopaedic Nursing. 2006;26(3):185-191.) in their references but never used it as a citation in the text. As such, it has been omitted.\n<\/p><p>Finally, in the PDF a couple of references were placed out of order by appearance, particularly references nine and ten in the introduction. Additionally, reference 40 is unused and removed. Because the wiki automatically assigns reference numbers based on order of appearance, the reference numbers here ultimately do not match up with those in the original paper. This was unavoidable.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210233\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 1.265 seconds\nReal time usage: 1.289 seconds\nPreprocessor visited node count: 41572\/1000000\nPreprocessor generated node count: 47689\/1000000\nPost\u2010expand include size: 366806\/2097152 bytes\nTemplate argument size: 135696\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 1247.949 1 - -total\n 79.85% 996.475 1 - Template:Reflist\n 69.84% 871.537 51 - Template:Citation\/core\n 60.67% 757.137 41 - Template:Cite_journal\n 12.50% 156.019 10 - Template:Cite_web\n 7.78% 97.137 78 - Template:Citation\/identifier\n 6.33% 78.963 1 - Template:Infobox_journal_article\n 6.11% 76.214 1 - Template:Infobox\n 4.14% 51.680 53 - Template:Citation\/make_link\n 3.35% 41.806 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7642-0!*!0!!en!*!* and timestamp 20181213210232 and revision id 23407\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review\">https:\/\/www.limswiki.org\/index.php\/Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","5c6e2bf767feb103119262aff1c6473b_images":[],"5c6e2bf767feb103119262aff1c6473b_timestamp":1544734952,"65b45b4f6a653c627bbc452940cf174f_type":"article","65b45b4f6a653c627bbc452940cf174f_title":"Support patient search on pathology reports with interactive online learning based data extraction (Zheng et al. 2015)","65b45b4f6a653c627bbc452940cf174f_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction","65b45b4f6a653c627bbc452940cf174f_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Support patient search on pathology reports with interactive online learning based data extraction\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nSupport patient search on pathology reports with interactive online learning based data extractionJournal\n \nJournal of Pathology InformaticsAuthor(s)\n \nZheng, Shuai; Lu, James J.; Appin, Christina; Brat, Daniel; Wang, FushengAuthor affiliation(s)\n \nEmory University, Stony Brook UniversityPrimary contact\n \nEmail: N\/AYear published\n \n2015Volume and issue\n \n6Page(s)\n \n51DOI\n \n10.4103\/2153-3539.166012ISSN\n \n2153-3539Distribution license\n \nCreative Commons Attribution-NonCommercial-ShareAlike 3.0 UnportedWebsite\n \nhttp:\/\/www.jpathinformatics.orgDownload\n \nhttp:\/\/www.jpathinformatics.org\/temp\/JPatholInform6151-4749188_131131.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods \n\n3.1 Interface of extraction and workflow \n3.2 Data extraction engine \n3.3 Adaptive online learning model \n3.4 Adaptive vocabularies \n3.5 Query interface and engine \n3.6 Evaluation metrics \n\n\n4 Results \n\n4.1 Datasets \n4.2 Experiment setup \n\n\n5 Discussions \n6 Conclusions \n\n6.1 Acknowledgments \n6.2 Financial support and sponsorship \n6.3 Conflicts of interest \n\n\n7 References \n8 Notes \n\n\n\nAbstract \nBackground: Structural reporting enables semantic understanding and prompt retrieval of clinical findings about patients. While synoptic pathology reporting provides templates for data entries, information in pathology reports remains primarily in narrative free text form. Extracting data of interest from narrative pathology reports could significantly improve the representation of the information and enable complex structured queries. However, manual extraction is tedious and error-prone, and automated tools are often constructed with a fixed training dataset and not easily adaptable. Our goal is to extract data from pathology reports to support advanced patient search with a highly adaptable semi-automated data extraction system, which can adjust and self-improve by learning from a user's interaction with minimal human effort. \nMethods: We have developed an online machine learning based information extraction system called IDEAL-X. With its graphical user interface, the system's data extraction engine automatically annotates values for users to review upon loading each report text. The system analyzes users' corrections regarding these annotations with online machine learning, and incrementally enhances and refines the learning model as reports are processed. The system also takes advantage of customized controlled vocabularies, which can be adaptively refined during the online learning process to further assist the data extraction. As the accuracy of automatic annotation improves overtime, the effort of human annotation is gradually reduced. After all reports are processed, a built-in query engine can be applied to conveniently define queries based on extracted structured data. \nResults: We have evaluated the system with a dataset of anatomic pathology reports from 50 patients. Extracted data elements include demographical data, diagnosis, genetic marker, and procedure. The system achieves F-1 scores of around 95% for the majority of tests. \nConclusions: Extracting data from pathology reports could enable more accurate knowledge to support biomedical research and clinical diagnosis. IDEAL-X provides a bridge that takes advantage of online machine learning based data extraction and the knowledge from human's feedback. By combining iterative online learning and adaptive controlled vocabularies, IDEAL-X can deliver highly adaptive and accurate data extraction to support patient search.\nKeywords: Controlled vocabularies, data extraction, online machine learning, pathology reports, patient search\n\nIntroduction \nPathology reports contain valuable research information embedded in narrative free text. The same information in structured format can be used to support clinical findings, decision making and biomedical research. Synoptic reporting[1][2][3] has become a powerful tool for providing summarized findings through predefined data element templates such as CAP Cancer Protocols.[4] Meanwhile, standard groups such as IHE are proposing structured reporting standards such as Anatomic Pathology Structured Reports[5] in Health Level Seven. While there is a major trend for structured reporting, the vast amount of pathology reports remain unstructured in legacy systems. And standardization efforts only capture major data elements, leaving a substantial amount of valuable information in free text that is difficult to process and search.\nInformation extraction is a technique that can generate structured representation of important information from pathology reports. The transformed data may be used to search easily for patient groups with certain traits as in, for example, find all patients with an age above 40 years old and that have a diagnosis glioma. Figure 1 shows a typical workflow of data extraction from pathology reports.\n\n\n\nFigure 1. Common pipeline of processing free text medical report\nPrevious work on data extraction from pathology reports addresses various tasks and different research problems. caTIES supports coding for surgical pathology reports.[6] A regular expression is used to mine specimens and related information in[7], MedTAS\/P extracts and represents cancer diseases from pathology reports with the hierarchical model.[8] Lupus represents extracted information with Semantic Web techniques.[9] NegEx is adopted to detect negation for annotating surgical pathology report.[10] These systems either employ rules engineered to specific topics and domains or they use statistical models learned in batch from manually annotated training data. The first approach lacks generalizability; new rules need to be designed and developed for each domain. The second approach based on machine learning is more flexible. But obtaining accurate training data can be costly and time-consuming.\nWe present a system, IDEAL-X, which combines online machine learning and customizable vocabularies to provide a generic, easy-to-use solution for clinical information extraction. Online machine learning[11][12][13] takes an iterative learning approach through interactive human intervention, the data extraction engine of IDEAL-X automatically predicts answers to annotate reports, gradually learns from human's feedback, and incrementally improves its accuracy. Compared to traditional batch training based algorithm, which requires pretraining with a reasonably large dataset, online learning based algorithms can significantly reduce human effort on labeling training data and provide the possibility of updating the learning models dynamically to fit a continually changing data environment. To enhance its performance, IDEAL-X supports adaptive vocabulary to support data extraction. A user can customize a controlled vocabulary, which could be continuously adjusted during online learning process. Once structured data elements are extracted, a query interface is provided to support patient search with filtering conditions on data elements.\n\nMethods \nIDEAL-X consists of five major parts: Extraction user interface, data extraction engine, online learning model, query engine, and interface. In general, the user interface resembles an ordinary data extraction and data entry system. It is unique, however, in its ability to transparently analyze and quickly learn from users' interactions the desired values for the data fields with online learning model. Additional user feedbacks incrementally refine the data extraction engine, as well as the vocabulary, in real-time, thus further reduce users' interaction effort thereafter. Processed reports are indexed by the query engine and made searchable by the query interface.\nA demo video can be found from the following link.[14]\n\nInterface of extraction and workflow \nThe workflow of IDEAL-X and the user interface are shown in Figure 2 and Figure 3, respectively. A user begins by specifying the input folder that contains the collection of report documents to be extracted. This is followed by an iterative process through the collection, in which for each document the value of interest is extracted, inspected, and verified. The resulting set of all processed documents are coalesced into a final output file. To work with each document, the left panel of the interface displays the report being processed and the right panel is the output of the extraction organized as a list of index-attribute-value triples. The \"index\" column uses colors to highlight locations of values in the report. The \"attribute\" and \"value\" columns show the data element names and the extracted values, respectively. The \"previous\" and \"next\" buttons at the bottom of the right side allow users to navigate through the document collection.\n\n\n\nFigure 2. A screen shot of IDEAL-X's interface\n\n\n\nFigure 3. The workflow of the IDEAL-X\nWhen a report is loaded, the system attempts to predict and prefill the values for as many data elements as possible. The user fills in any remaining data element (through click and drop) that the system leaves blank. The user may fill multiple terms if given data element is a multiple value field such as diagnosis. For a prefilled data element, the user may review and update its value if it is incorrect or incomplete. This simple and intuitive interface makes the workflow of the system easily accessible to any user.\nAt the beginning, the system is unable to predict values to most of the data elements. Through the combined manual extraction and revision process, the system learns the important contexts for the values and updates its decision model accordingly. As we will show in results later, the amount of information that the system is able to prefill correctly grows quickly.\n\nData extraction engine \nThe data extraction engine consists of the following major components: Preprocessing, vocabulary, answer predicting. Figure 4 shows relationships of the components with respect to the data flow. A parallelogram indicates results or inputs to components and procedures. The preprocessing component converts input texts and output forms into internal data structures used by the answer predicting component. The vocabulary component imports domain specific vocabulary to support information extraction. The answer predicting component extracts values from input texts to fill the output data elements. The online learning model utilizes judgments from users, in the form of edits on generated values, to update the decision model of the answer predicting component, which consists of vector space model, hidden Markov model (HMM) model, and rule induction model (see Adaptive Online Learning Model for details).\n\n\n\nFigure 4. System components and dataflow\nAutomatic population of the output form is performed in three steps (Figure 5). First, candidate sentences - those that are likely to contain values of interest, are detected by a combination of vector space model[15], keywords, and location matching. Candidate values, consisting of phrase chunks are then extracted with the HMM algorithm[16] or user-defined vocabulary. Lastly, constraints such as string patterns and numerical ranges are learned through rule induction[17], and applied to narrow the set of candidate values. For example, the system may only select candidate chunks with first letter capitalized. In this step, negation and uncertainty detection, which are performed based on predefined rules, are also applied to filter candidates. Candidates that receive confidence scores above the threshold are used to fill the output form, which could be later transformed into a single structured view compatible with a database table or excel spreadsheet. If required, extracted values may be normalized based on user-defined mappings.\n\n\n\nFigure 5. Modules of answer predicting component\nAdaptive online learning model \nIDEAL-X employs online supervised learning[12][13], in which updating system is conducted after processing each report in order to minimize the cumulative gap between prediction and correct answer. As input documents are processed, the algorithm incrementally improves its learning parameters based on user feedbacks. The feedbacks come in the form of user selection and correction to system predicted values. Text fragments, which are either highlighted by the system or selected by user are treated as answer values. The absence of any user action on a system-generated value is a positive feedback, and reinforces the learning model. If a revision occurred, user revised value is learned as positive feedback, which also indicates that the system's prediction is incorrect. Linguistic features associated with the value, such as part-of-speech tag, located section and co-occurring words in a sentence, are analyzed to improve the three steps performed by data extraction engine through vector space model[15], HMM model[16], and rule induction model[17] respectively. Through this interaction, IDEAL-X transparently learns the linguistic features of values to be extracted, and user doesn't have to predefine any constrains or thresholds.\n\nAdaptive vocabularies \nThe system allows the user to customize a domain-specific vocabulary such as drug names related to certain disease. In general, to create a seed vocabulary, a standard ontology such as the SNOMED clinical terms[18] or the National Cancer Institute (NCI) Thesaurus[19] is a useful starting point. But the vocabulary may not be complete and miss certain terminologies specific to local reporting domains. When a mismatch occurs between the vocabulary and an extracted value, IDEAL-X refines the vocabulary by adding the extracted terms and removing unneeded terms. This way, the vocabulary converges to a lexicon that is consistent with the extraction task. The vocabulary is also reusable: It can be exported for reproducibility, other extraction projects, and ontology construction. If desired, a postprocessing step to standardize the extracted values can be performed.\n\nQuery interface and engine \nThe extracted data are organized and indexed to facilitate querying. The system provides a built-in query interface (Figure 6) that allows the user to search for patients or reports based on user-specified conditions. The interface is split into three main panels. The right panel shows the search condition. For each attributes, the user may specify a value from the list of available values that the system has collected during extraction. The \"search\" button finds all reports that match all of the search criteria, and displays the results as a directory tree in the left panel. Selecting a node in this tree loads the content of the corresponding report into the text area of the second panel. When the user specifies multiple search conditions, the system searches for results that satisfy all criteria, in other words, the intersection set. When the user selects multiple values from the same condition, the union set will be generated.\n\n\n\nFigure 6. Query interface\nEvaluation metrics \nWe compared the system's output with the manually annotated ground truth with respect to precision, recall and F-1 measure. Precision estimates the correctness of extraction, recall estimates the completeness, and F-1 measure is the weighted average of these two.\n\nResults \nTo test the performance of information extraction, we perform two experiments: Experiment 1 to examine the effectiveness of online learning, and experiment 2 to examine the importance of adaptive vocabulary. The development of this system is driven by the demands of brain tumor research, in which pathology reports need to be queried based on demographic data, disease, procedure, among others, in order to locate patients with certain traits.\n\nDatasets \nWe randomly selected and annotated 50 anatomic pathology reports manually as a testing dataset for this study. The guideline for annotation is provided by a pathologist, who also verifies annotation results to resolve the disagreement. These pathology reports were from patients that had been diagnosed with a Grade II or Grade III infiltrating glioma and had their tumors resected at Emory University Hospitals. Another 50 reports, which are disjoint with the testing dataset, were used as development dataset.\n\nExperiment setup \nIn order to study the system's learning effectiveness, all experiments began with an empty model, without prior training or predefined constraints.\nWe perform tests on extracting demographic information such as age and gender, and commonly queried medical information such as diagnosis, genetic marker (both gene and protein) and therapy\/procedure (Table 1). Note that, these attributes may be available in the structured format in some reporting systems or databases. For the purpose of the experiment, here we assume structured data is not available and use these typical attributes to examine the effectiveness of IDEAL-X. To support extraction, we employ a seed vocabulary consisting of diagnosis, gene and procedure lexicons, obtained from the Human Disease Ontology[20] the Cell Cycle Ontology[21] and the NCI Thesaurus[19] Ontology respectively. We use these prevalence ontologies for experimental evaluation, other seed vocabulary may be more appropriate for other extraction tasks.\n\n\n\nTable 1. Test cases of data extraction\nResults of experiment 1 are shown in Table 2. Age and gender typically appear in report headers with limited contextual variation. For these, the system achieved very high precision and recall. Values related to diagnosis, genetic marker, and therapy appear in the text with larger structural and narrative variation. With the support of the seed vocabulary, the system achieved F1 scores of 88%, 93%, and 97%, respectively. To study the effectiveness of learning, for each test case, we divided the 50 reports into two groups in sequential order: The first 20 reports (as they appear in the directory), and the second 30 reports. The improvement of accuracy from the first group to the second group was significant, reflecting a high-rate of learning. For the four classes of attributes, F1 scores between the first and second groups increased from 94.7%, 82.1%, 90.0% and 95.3% to 100%, 91.2%, 95.3% and 99.5%, respectively.\n\n\n\nTable 2. Test result of experiment 1: Study of online learning\nThe results of experiment 2 show the ability to refine the vocabulary can have a major effect on the accuracy of data extraction. For diagnosis, genetic marker, therapy and procedure, Table 3 shows the difference of results with and without refining the seed vocabulary. When the system used the seed vocabulary directly without further refinement, the performance of the extraction relies on how closely the vocabulary content aligns with the extraction task. For a genetic marker, a very small difference in the F1 score was observed. For diagnosis and procedure, on the other hand, the downloaded ontology subsets contain considerable irrelevant information for pathology reports. This impacted the precision by 3.1% for diagnosis, and 10.3% for the procedure. Moreover, many terms were missing, which negatively affected the recall by 46% for diagnosis and 30% for procedure. Comparing these results with Table 2, we notice that when adaptive vocabulary is enabled, the system captures important terms quickly. If these terms could not be added in time, the system will keep on missing these values in following report therefore largely impairs recall rate in some cases. These results show that there could be a large discrepancy between standard ontology and the controlled vocabulary of a specific domain, and the benefits adaptable vocabularies could be substantial.\n\n\n\nTable 3. Test result of experiment 2: Study of adaptive vocabulary\nDiscussions \nIn the experiments, we have selected attributes that appear in different types of text that are found in pathology reports. Our goal is for testing the effectiveness of online machine learning and the importance of refining the seed vocabulary during the extraction process. Though extracting value in generic domain is challenging, given particular task or research topic, which has value of limited domain, the system could be customized easily to meet specific purpose of given task, for example, identify patient for brain tumor research. In follow-up studies, we will consider a broader set of attributes and enrich the supported data types. Values that the system can manage currently are limited to numerical and nominal values. Extracting temporal information, for example, will improve the utility of the system. In pathology research, medical events such as procedures are time sensitive. Augmenting the output with timelines would contextualize and help to connect the extracted values in important ways.\nA goal of IDEAL-X is to provide a generic solution for information extraction across medical domains. The system employs techniques that we believe are domain agnostic, which could be validated with use cases in other medical domains. We are collaborating with Emory Clinical Cardiovascular Research Institute and Rutgers University Radiation Oncology Department to validate the broader utility of IDEAL-X. One characteristic of the cardiology research project is the existence of multiple types of reports for each patient. This adds complexity to the interface and the learning algorithm. We also plan to study the advantage of using IDEAL-X, as a software assistant for annotation over manual annotation both on efficiency and accuracy.\nFinally, we will study the use of IDEAL-X to ease structured reporting such as providing machine generated automated hints to create synoptic pathology reports. Though answers may be inputted using a combo box to guarantee structured reporting, most existing medical report systems still allow for free-format text and uncontrolled vocabulary. In some cases, direct access to the structured database may not be available, therefore, requires extracting information from text pathology report directly.\n\nConclusions \nIDEAL-X employs iterative online machine learning and adaptive controlled vocabulary for information extraction from clinical reports. It automatically predicts annotations for values to be extracted and utilizes human feedback as knowledge to improve continuously the performance of extraction. Experimental results demonstrate that both online learning and adaptive vocabulary are highly effective. Extracted data are indexed by the built-in query engine and can be conveniently queried with a graphic interface. The adaptability and usability of the system make IDEAL-X a powerful data extraction tool to support patient search from pathology reports.\n\nAcknowledgments \nThis study is supported in part by grants from the Centers for Disease Control and Prevention 200-2014-M-59415.\n\nFinancial support and sponsorship \nNil.\n\nConflicts of interest \nThere are no conflicts of interest.\n\nReferences \n\n\n\u2191 Srigley, J.R.; McGowan, T.; Maclean, A.; Raby, M.; Ross, J.; Kramer, S.; Sawka, C. (2009). \"Standardized synoptic cancer pathology reporting: A population-based approach\". Journal of Surgical Oncology 99 (8): 517\u2013524. doi:10.1002\/jso.21282. PMID 19466743.   \n\n\u2191 Gill, A.J.; Johns, A.L.; Eckstein, R.; Samra, J.S.; Kaufman, A.; Chang, D.K.; Merrett N.D.; Cosman, P.H.; Smith, R.C.; Biankin, A.V.; Kench, J.G.; New South Wales Pancreatic Cancer Network (2009). \"Synoptic reporting improves histopathological assessment of pancreatic resection specimens\". Pathology 41 (2): 161\u2013167. PMID 19320058.   \n\n\u2191 Leslie, K.O.; Rosai, J. (1994). \"Standardization of the surgical pathology report: Formats, templates, and synoptic reports\". Seminars in Diagnostic Pathology 11 (4): 253\u20137. PMID 7878300.   \n\n\u2191 \"Cancer Protocols\". CAP.org. College of American Pathologists. 2015. http:\/\/www.cap.org\/web\/oracle\/webcenter\/portalapp\/pagehierarchy\/cancer_protocols.jspx . Retrieved 12 July 2015 .   \n\n\u2191 \"Anatomic Pathology Structured Reports\". IHE Wiki. Integrating the Healthcare Enterprise. 30 June 2010. http:\/\/wiki.ihe.net\/index.php?title=Anatomic_Pathology_Structured_Reports . Retrieved 12 July 2015 .   \n\n\u2191 Crowley, R.S.; Castine, M.; Mitchell, K.; Chavan, G.; McSherry, T.; Feldman, M. (2010). \"caTIES: A grid based system for coding and retrieval of surgical pathology reports and tissue specimens in support of translational research\". Journal of the American Medical Informatics Association 17 (3): 253\u201364. doi:10.1136\/jamia.2009.002295. PMC PMC2995710. PMID 20442142. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710 .   \n\n\u2191 Schadow, G.; McDonald, C.J. (2003). \"Extracting structured information from free text pathology reports\". AMIA Annual Symposium Proceedings 2003: 584\u20138. PMC PMC1480213. PMID 14728240. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1480213 .   \n\n\u2191 Coden, A.; Savova, G.; Sominsky, I.; Tanenblatt, M.; Masanz, J.; Schuler, K.; Cooper, J.; Guan, W.; de Groen, P.C. (2009). \"Automatically extracting cancer disease characteristics from pathology reports into a disease knowledge representation model\". Journal of Biomedical Informatics 42 (5): 937\u201349. doi:10.1016\/j.jbi.2008.12.005. PMID 19135551.   \n\n\u2191 Schlangen, David; Stede, Manfred; Bontas, Elena Paslaru (2004). \"Feeding Owl: Extracting and Representing the Content of Pathology Reports\". NLPXML '04 Proceeedings of the Workshop on NLP and XML (NLPXML-2004): RDF\/RDFS and OWL in Language Technology: 43\u201350. http:\/\/pub.uni-bielefeld.de\/publication\/1992186 .   \n\n\u2191 Mitchell, K.J.; Becich, M.J.; Berman, J.J.; Chapman, W.W.; Gilbertson, J.; Gupta, D.; Harrison, J.; Legowski, E.; Crowley, R.S. (2004). \"Implementation and evaluation of a negation tagger in a pipeline-based system for information extract from pathology reports\". Studies in Health Technology and Informatics 107 (Pt 1): 663-7. doi:10.3233\/978-1-60750-949-3-663. PMID 15360896.   \n\n\u2191 Smale, S.; Yao, Y. (2006). \"Online learning algorithms\". Foundations of Computational Mathematics 6 (2): 145\u2013170. doi:10.1007\/s10208-004-0160-z.   \n\n\u2191 12.0 12.1 Shalev-Shwartz, S. (2012). \"Online learning and online convex optimization\". Foundations and Trends in Machine Learning 4 (2): 107\u2013194. doi:10.1561\/2200000018.   \n\n\u2191 13.0 13.1 Shalev-Shwartz, S. (July 2007). \"Online Learning: Theory, Algorithms, and Applications\" (PDF). University of Chicago. http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf . Retrieved 12 July 2015 .   \n\n\u2191 Zheng, Shuai (07 September 2014). \"IDEAL-X: Information and Data Extraction using Adaptive Learning\". YouTube. http:\/\/youtu.be\/Q-DrWi31nv0 . Retrieved 12 July 2015 .   \n\n\u2191 15.0 15.1 Manning, C.D.; Raghavan, P.; Sch\u00fctze, H. (2008). Introduction to Information Retrieval (1st ed.). Cambridge, England: Cambridge University Press. ISBN 9780521865715.   \n\n\u2191 16.0 16.1 Elliott, R.J.; Aggoun, L.; Moore, J.B. (1995). Hidden Markov Models: Estimation and Control (1st ed.). New York, NY: Springer-Verlag New York. doi:10.1007\/978-0-387-84854-9. ISBN 9780387848549.   \n\n\u2191 17.0 17.1 F\u00fcrnkranz, J. (1999). \"Separate-and-conquer rule learning\". Artificial Intelligence Review 13 (1): 3\u201354. doi:10.1023\/A:1006524209794.   \n\n\u2191 \"SNOMED Clinical Terms (SNOMED CT)\". Unified Medical Language System (UMLS). U.S. National Library of Medicine. 25 February 2015. https:\/\/www.nlm.nih.gov\/research\/umls\/Snomed\/snomed_main.html . Retrieved 12 July 2015 .   \n\n\u2191 19.0 19.1 \"NCI Thesaurus\". NCI Thesaurus. National Cancer Institute. July 2015. https:\/\/ncit.nci.nih.gov\/ . Retrieved 12 July 2015 .   \n\n\u2191 \"Disease Ontology\". Institute for Genome Sciences, University of Maryland School of Medicine. 2015. http:\/\/www.disease-ontology.org\/ . Retrieved 12 July 2015 .   \n\n\u2191 Kuiper, Martin (2015). \"Semantic Systems Biology\". Institutt for biologi - NTNU. http:\/\/www.cellcycleontology.org\/ . Retrieved 12 July 2015 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases citation information was missing and was added to make the reference more useful.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\">https:\/\/www.limswiki.org\/index.php\/Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on reporting\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 21:01.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,410 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","65b45b4f6a653c627bbc452940cf174f_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Support patient search on pathology reports with interactive online learning based data extraction<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Structural reporting enables semantic understanding and prompt retrieval of clinical findings about patients. While <a href=\"https:\/\/www.limswiki.org\/index.php\/LIS_feature#Synoptic_reporting\" title=\"LIS feature\" target=\"_blank\" class=\"wiki-link\" data-key=\"ae461418eb0b8cde555298c3c2e74860\">synoptic pathology reporting<\/a> provides templates for data entries, information in <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_pathology\" title=\"Clinical pathology\" target=\"_blank\" class=\"wiki-link\" data-key=\"9b0b91aa13ae0209e073db6f30493daf\">pathology<\/a> reports remains primarily in narrative free text form. Extracting data of interest from narrative pathology reports could significantly improve the representation of the information and enable complex structured queries. However, manual extraction is tedious and error-prone, and automated tools are often constructed with a fixed training dataset and not easily adaptable. Our goal is to extract data from pathology reports to support advanced patient search with a highly adaptable semi-automated data extraction system, which can adjust and self-improve by learning from a user's interaction with minimal human effort. \n<\/p><p><b>Methods:<\/b> We have developed an online machine learning based information extraction system called IDEAL-X. With its graphical user interface, the system's data extraction engine automatically annotates values for users to review upon loading each report text. The system analyzes users' corrections regarding these annotations with online machine learning, and incrementally enhances and refines the learning model as reports are processed. The system also takes advantage of customized controlled vocabularies, which can be adaptively refined during the online learning process to further assist the data extraction. As the accuracy of automatic annotation improves overtime, the effort of human annotation is gradually reduced. After all reports are processed, a built-in query engine can be applied to conveniently define queries based on extracted structured data. \n<\/p><p><b>Results:<\/b> We have evaluated the system with a dataset of <a href=\"https:\/\/www.limswiki.org\/index.php\/Anatomical_pathology\" title=\"Anatomical pathology\" target=\"_blank\" class=\"wiki-link\" data-key=\"5668db6faf37e8c1432a1d7953f30cb7\">anatomic pathology<\/a> reports from 50 patients. Extracted data elements include demographical data, diagnosis, genetic marker, and procedure. The system achieves F-1 scores of around 95% for the majority of tests. \n<\/p><p><b>Conclusions:<\/b> Extracting data from pathology reports could enable more accurate knowledge to support biomedical research and clinical diagnosis. IDEAL-X provides a bridge that takes advantage of online machine learning based data extraction and the knowledge from human's feedback. By combining iterative online learning and adaptive controlled vocabularies, IDEAL-X can deliver highly adaptive and accurate data extraction to support patient search.\n<\/p><p><b>Keywords:<\/b> Controlled vocabularies, data extraction, online machine learning, pathology reports, patient search\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Pathology reports contain valuable research information embedded in narrative free text. The same information in structured format can be used to support clinical findings, decision making and biomedical research. Synoptic reporting<sup id=\"rdp-ebb-cite_ref-SrigleyStand_1-0\" class=\"reference\"><a href=\"#cite_note-SrigleyStand-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GillSyn_2-0\" class=\"reference\"><a href=\"#cite_note-GillSyn-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeslieStand_3-0\" class=\"reference\"><a href=\"#cite_note-LeslieStand-3\" rel=\"external_link\">[3]<\/a><\/sup> has become a powerful tool for providing summarized findings through predefined data element templates such as CAP Cancer Protocols.<sup id=\"rdp-ebb-cite_ref-CAPCancer_4-0\" class=\"reference\"><a href=\"#cite_note-CAPCancer-4\" rel=\"external_link\">[4]<\/a><\/sup> Meanwhile, standard groups such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Integrating_the_Healthcare_Enterprise\" title=\"Integrating the Healthcare Enterprise\" target=\"_blank\" class=\"wiki-link\" data-key=\"7ab927e321f8c301547e5b84f7ebbc14\">IHE<\/a> are proposing structured reporting standards such as Anatomic Pathology Structured Reports<sup id=\"rdp-ebb-cite_ref-IHEAna_5-0\" class=\"reference\"><a href=\"#cite_note-IHEAna-5\" rel=\"external_link\">[5]<\/a><\/sup> in <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Level_7\" title=\"Health Level 7\" target=\"_blank\" class=\"wiki-link\" data-key=\"e0bf845fb58d2bae05a846b47629e86f\">Health Level Seven<\/a>. While there is a major trend for structured reporting, the vast amount of pathology reports remain unstructured in legacy systems. And standardization efforts only capture major data elements, leaving a substantial amount of valuable information in free text that is difficult to process and search.\n<\/p><p>Information extraction is a technique that can generate structured representation of important information from pathology reports. The transformed data may be used to search easily for patient groups with certain traits as in, for example, find all patients with an age above 40 years old and that have a diagnosis glioma. Figure 1 shows a typical workflow of data extraction from pathology reports.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"e53662a8b9f7bf6c116f088d0ee43376\"><img alt=\"Fig1 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/4\/43\/Fig1_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Common pipeline of processing free text medical report<\/blockquote>\n<p>Previous work on data extraction from pathology reports addresses various tasks and different research problems. caTIES supports coding for surgical pathology reports.<sup id=\"rdp-ebb-cite_ref-CrowleycaTIES_6-0\" class=\"reference\"><a href=\"#cite_note-CrowleycaTIES-6\" rel=\"external_link\">[6]<\/a><\/sup> A regular expression is used to mine specimens and related information in<sup id=\"rdp-ebb-cite_ref-SchadowExtr_7-0\" class=\"reference\"><a href=\"#cite_note-SchadowExtr-7\" rel=\"external_link\">[7]<\/a><\/sup>, MedTAS\/P extracts and represents cancer diseases from pathology reports with the hierarchical model.<sup id=\"rdp-ebb-cite_ref-CodenAuto_8-0\" class=\"reference\"><a href=\"#cite_note-CodenAuto-8\" rel=\"external_link\">[8]<\/a><\/sup> Lupus represents extracted information with Semantic Web techniques.<sup id=\"rdp-ebb-cite_ref-SchlangenFeed_9-0\" class=\"reference\"><a href=\"#cite_note-SchlangenFeed-9\" rel=\"external_link\">[9]<\/a><\/sup> NegEx is adopted to detect negation for annotating surgical pathology report.<sup id=\"rdp-ebb-cite_ref-MitchellImp_10-0\" class=\"reference\"><a href=\"#cite_note-MitchellImp-10\" rel=\"external_link\">[10]<\/a><\/sup> These systems either employ rules engineered to specific topics and domains or they use statistical models learned in batch from manually annotated training data. The first approach lacks generalizability; new rules need to be designed and developed for each domain. The second approach based on machine learning is more flexible. But obtaining accurate training data can be costly and time-consuming.\n<\/p><p>We present a system, IDEAL-X, which combines online machine learning and customizable vocabularies to provide a generic, easy-to-use solution for clinical information extraction. Online machine learning<sup id=\"rdp-ebb-cite_ref-SmaleOnline_11-0\" class=\"reference\"><a href=\"#cite_note-SmaleOnline-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShalevOnline1_12-0\" class=\"reference\"><a href=\"#cite_note-ShalevOnline1-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShalevOnline2_13-0\" class=\"reference\"><a href=\"#cite_note-ShalevOnline2-13\" rel=\"external_link\">[13]<\/a><\/sup> takes an iterative learning approach through interactive human intervention, the data extraction engine of IDEAL-X automatically predicts answers to annotate reports, gradually learns from human's feedback, and incrementally improves its accuracy. Compared to traditional batch training based algorithm, which requires pretraining with a reasonably large dataset, online learning based algorithms can significantly reduce human effort on labeling training data and provide the possibility of updating the learning models dynamically to fit a continually changing data environment. To enhance its performance, IDEAL-X supports adaptive vocabulary to support data extraction. A user can customize a controlled vocabulary, which could be continuously adjusted during online learning process. Once structured data elements are extracted, a query interface is provided to support patient search with filtering conditions on data elements.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>IDEAL-X consists of five major parts: Extraction user interface, data extraction engine, online learning model, query engine, and interface. In general, the user interface resembles an ordinary data extraction and data entry system. It is unique, however, in its ability to transparently analyze and quickly learn from users' interactions the desired values for the data fields with online learning model. Additional user feedbacks incrementally refine the data extraction engine, as well as the vocabulary, in real-time, thus further reduce users' interaction effort thereafter. Processed reports are indexed by the query engine and made searchable by the query interface.\n<\/p><p>A demo video can be found from the following link.<sup id=\"rdp-ebb-cite_ref-IDEALVid_14-0\" class=\"reference\"><a href=\"#cite_note-IDEALVid-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Interface_of_extraction_and_workflow\">Interface of extraction and workflow<\/span><\/h3>\n<p>The workflow of IDEAL-X and the user interface are shown in Figure 2 and Figure 3, respectively. A user begins by specifying the input folder that contains the collection of report documents to be extracted. This is followed by an iterative process through the collection, in which for each document the value of interest is extracted, inspected, and verified. The resulting set of all processed documents are coalesced into a final output file. To work with each document, the left panel of the interface displays the report being processed and the right panel is the output of the extraction organized as a list of index-attribute-value triples. The \"index\" column uses colors to highlight locations of values in the report. The \"attribute\" and \"value\" columns show the data element names and the extracted values, respectively. The \"previous\" and \"next\" buttons at the bottom of the right side allow users to navigate through the document collection.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a8fd9b7feeb9e72c94c155f2c2dd247f\"><img alt=\"Fig2 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/9c\/Fig2_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> A screen shot of IDEAL-X's interface<\/blockquote>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a6c5eba4b1d708f56210b480eb158ff4\"><img alt=\"Fig3 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f7\/Fig3_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3.<\/b> The workflow of the IDEAL-X<\/blockquote>\n<p>When a report is loaded, the system attempts to predict and prefill the values for as many data elements as possible. The user fills in any remaining data element (through click and drop) that the system leaves blank. The user may fill multiple terms if given data element is a multiple value field such as diagnosis. For a prefilled data element, the user may review and update its value if it is incorrect or incomplete. This simple and intuitive interface makes the workflow of the system easily accessible to any user.\n<\/p><p>At the beginning, the system is unable to predict values to most of the data elements. Through the combined manual extraction and revision process, the system learns the important contexts for the values and updates its decision model accordingly. As we will show in results later, the amount of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> that the system is able to prefill correctly grows quickly.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_extraction_engine\">Data extraction engine<\/span><\/h3>\n<p>The data extraction engine consists of the following major components: Preprocessing, vocabulary, answer predicting. Figure 4 shows relationships of the components with respect to the data flow. A parallelogram indicates results or inputs to components and procedures. The preprocessing component converts input texts and output forms into internal data structures used by the answer predicting component. The vocabulary component imports domain specific vocabulary to support information extraction. The answer predicting component extracts values from input texts to fill the output data elements. The online learning model utilizes judgments from users, in the form of edits on generated values, to update the decision model of the answer predicting component, which consists of vector space model, hidden Markov model (HMM) model, and rule induction model (see Adaptive Online Learning Model for details).\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"ad4e4f5d271f4c0fef62c993b3e1b43c\"><img alt=\"Fig4 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f2\/Fig4_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4.<\/b> System components and dataflow<\/blockquote>\n<p>Automatic population of the output form is performed in three steps (Figure 5). First, candidate sentences - those that are likely to contain values of interest, are detected by a combination of vector space model<sup id=\"rdp-ebb-cite_ref-ManningIntro_15-0\" class=\"reference\"><a href=\"#cite_note-ManningIntro-15\" rel=\"external_link\">[15]<\/a><\/sup>, keywords, and location matching. Candidate values, consisting of phrase chunks are then extracted with the HMM algorithm<sup id=\"rdp-ebb-cite_ref-ElliotHid_16-0\" class=\"reference\"><a href=\"#cite_note-ElliotHid-16\" rel=\"external_link\">[16]<\/a><\/sup> or user-defined vocabulary. Lastly, constraints such as string patterns and numerical ranges are learned through rule induction<sup id=\"rdp-ebb-cite_ref-FurnkranzSep_17-0\" class=\"reference\"><a href=\"#cite_note-FurnkranzSep-17\" rel=\"external_link\">[17]<\/a><\/sup>, and applied to narrow the set of candidate values. For example, the system may only select candidate chunks with first letter capitalized. In this step, negation and uncertainty detection, which are performed based on predefined rules, are also applied to filter candidates. Candidates that receive confidence scores above the threshold are used to fill the output form, which could be later transformed into a single structured view compatible with a database table or excel spreadsheet. If required, extracted values may be normalized based on user-defined mappings.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"cc7359d6b946b400e1a00871e17d63ba\"><img alt=\"Fig5 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/b0\/Fig5_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 5.<\/b> Modules of answer predicting component<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Adaptive_online_learning_model\">Adaptive online learning model<\/span><\/h3>\n<p>IDEAL-X employs online supervised learning<sup id=\"rdp-ebb-cite_ref-ShalevOnline1_12-1\" class=\"reference\"><a href=\"#cite_note-ShalevOnline1-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShalevOnline2_13-1\" class=\"reference\"><a href=\"#cite_note-ShalevOnline2-13\" rel=\"external_link\">[13]<\/a><\/sup>, in which updating system is conducted after processing each report in order to minimize the cumulative gap between prediction and correct answer. As input documents are processed, the algorithm incrementally improves its learning parameters based on user feedbacks. The feedbacks come in the form of user selection and correction to system predicted values. Text fragments, which are either highlighted by the system or selected by user are treated as answer values. The absence of any user action on a system-generated value is a positive feedback, and reinforces the learning model. If a revision occurred, user revised value is learned as positive feedback, which also indicates that the system's prediction is incorrect. Linguistic features associated with the value, such as part-of-speech tag, located section and co-occurring words in a sentence, are analyzed to improve the three steps performed by data extraction engine through vector space model<sup id=\"rdp-ebb-cite_ref-ManningIntro_15-1\" class=\"reference\"><a href=\"#cite_note-ManningIntro-15\" rel=\"external_link\">[15]<\/a><\/sup>, HMM model<sup id=\"rdp-ebb-cite_ref-ElliotHid_16-1\" class=\"reference\"><a href=\"#cite_note-ElliotHid-16\" rel=\"external_link\">[16]<\/a><\/sup>, and rule induction model<sup id=\"rdp-ebb-cite_ref-FurnkranzSep_17-1\" class=\"reference\"><a href=\"#cite_note-FurnkranzSep-17\" rel=\"external_link\">[17]<\/a><\/sup> respectively. Through this interaction, IDEAL-X transparently learns the linguistic features of values to be extracted, and user doesn't have to predefine any constrains or thresholds.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Adaptive_vocabularies\">Adaptive vocabularies<\/span><\/h3>\n<p>The system allows the user to customize a domain-specific vocabulary such as drug names related to certain disease. In general, to create a seed vocabulary, a standard ontology such as the SNOMED clinical terms<sup id=\"rdp-ebb-cite_ref-NIHSNOMED_18-0\" class=\"reference\"><a href=\"#cite_note-NIHSNOMED-18\" rel=\"external_link\">[18]<\/a><\/sup> or the National Cancer Institute (NCI) Thesaurus<sup id=\"rdp-ebb-cite_ref-NCIThes_19-0\" class=\"reference\"><a href=\"#cite_note-NCIThes-19\" rel=\"external_link\">[19]<\/a><\/sup> is a useful starting point. But the vocabulary may not be complete and miss certain terminologies specific to local reporting domains. When a mismatch occurs between the vocabulary and an extracted value, IDEAL-X refines the vocabulary by adding the extracted terms and removing unneeded terms. This way, the vocabulary converges to a lexicon that is consistent with the extraction task. The vocabulary is also reusable: It can be exported for reproducibility, other extraction projects, and ontology construction. If desired, a postprocessing step to standardize the extracted values can be performed.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Query_interface_and_engine\">Query interface and engine<\/span><\/h3>\n<p>The extracted data are organized and indexed to facilitate querying. The system provides a built-in query interface (Figure 6) that allows the user to search for patients or reports based on user-specified conditions. The interface is split into three main panels. The right panel shows the search condition. For each attributes, the user may specify a value from the list of available values that the system has collected during extraction. The \"search\" button finds all reports that match all of the search criteria, and displays the results as a directory tree in the left panel. Selecting a node in this tree loads the content of the corresponding report into the text area of the second panel. When the user specifies multiple search conditions, the system searches for results that satisfy all criteria, in other words, the intersection set. When the user selects multiple values from the same condition, the union set will be generated.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"990452495a779a8d7173e25580a97fe3\"><img alt=\"Fig6 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/c7\/Fig6_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 6.<\/b> Query interface<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Evaluation_metrics\">Evaluation metrics<\/span><\/h3>\n<p>We compared the system's output with the manually annotated ground truth with respect to precision, recall and F-1 measure. Precision estimates the correctness of extraction, recall estimates the completeness, and F-1 measure is the weighted average of these two.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>To test the performance of information extraction, we perform two experiments: Experiment 1 to examine the effectiveness of online learning, and experiment 2 to examine the importance of adaptive vocabulary. The development of this system is driven by the demands of brain tumor research, in which pathology reports need to be queried based on demographic data, disease, procedure, among others, in order to locate patients with certain traits.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Datasets\">Datasets<\/span><\/h3>\n<p>We randomly selected and annotated 50 anatomic pathology reports manually as a testing dataset for this study. The guideline for annotation is provided by a pathologist, who also verifies annotation results to resolve the disagreement. These pathology reports were from patients that had been diagnosed with a Grade II or Grade III infiltrating glioma and had their tumors resected at Emory University Hospitals. Another 50 reports, which are disjoint with the testing dataset, were used as development dataset.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Experiment_setup\">Experiment setup<\/span><\/h3>\n<p>In order to study the system's learning effectiveness, all experiments began with an empty model, without prior training or predefined constraints.\n<\/p><p>We perform tests on extracting demographic information such as age and gender, and commonly queried medical information such as diagnosis, genetic marker (both gene and protein) and therapy\/procedure (Table 1). Note that, these attributes may be available in the structured format in some reporting systems or databases. For the purpose of the experiment, here we assume structured data is not available and use these typical attributes to examine the effectiveness of IDEAL-X. To support extraction, we employ a seed vocabulary consisting of diagnosis, gene and procedure lexicons, obtained from the Human Disease Ontology<sup id=\"rdp-ebb-cite_ref-IGSDisease_20-0\" class=\"reference\"><a href=\"#cite_note-IGSDisease-20\" rel=\"external_link\">[20]<\/a><\/sup> the Cell Cycle Ontology<sup id=\"rdp-ebb-cite_ref-_21-0\" class=\"reference\"><a href=\"#cite_note--21\" rel=\"external_link\">[21]<\/a><\/sup> and the NCI Thesaurus<sup id=\"rdp-ebb-cite_ref-NCIThes_19-1\" class=\"reference\"><a href=\"#cite_note-NCIThes-19\" rel=\"external_link\">[19]<\/a><\/sup> Ontology respectively. We use these prevalence ontologies for experimental evaluation, other seed vocabulary may be more appropriate for other extraction tasks.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b6a7f86c8b8d8febde8233ad3b3f22d2\"><img alt=\"Tab1 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/0\/01\/Tab1_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 1.<\/b> Test cases of data extraction<\/blockquote>\n<p>Results of experiment 1 are shown in Table 2. Age and gender typically appear in report headers with limited contextual variation. For these, the system achieved very high precision and recall. Values related to diagnosis, genetic marker, and therapy appear in the text with larger structural and narrative variation. With the support of the seed vocabulary, the system achieved F1 scores of 88%, 93%, and 97%, respectively. To study the effectiveness of learning, for each test case, we divided the 50 reports into two groups in sequential order: The first 20 reports (as they appear in the directory), and the second 30 reports. The improvement of accuracy from the first group to the second group was significant, reflecting a high-rate of learning. For the four classes of attributes, F1 scores between the first and second groups increased from 94.7%, 82.1%, 90.0% and 95.3% to 100%, 91.2%, 95.3% and 99.5%, respectively.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab2_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"83f89060a8af9b1252a2b6cc30eebd15\"><img alt=\"Tab2 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/5\/58\/Tab2_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 2.<\/b> Test result of experiment 1: Study of online learning<\/blockquote>\n<p>The results of experiment 2 show the ability to refine the vocabulary can have a major effect on the accuracy of data extraction. For diagnosis, genetic marker, therapy and procedure, Table 3 shows the difference of results with and without refining the seed vocabulary. When the system used the seed vocabulary directly without further refinement, the performance of the extraction relies on how closely the vocabulary content aligns with the extraction task. For a genetic marker, a very small difference in the F1 score was observed. For diagnosis and procedure, on the other hand, the downloaded ontology subsets contain considerable irrelevant information for pathology reports. This impacted the precision by 3.1% for diagnosis, and 10.3% for the procedure. Moreover, many terms were missing, which negatively affected the recall by 46% for diagnosis and 30% for procedure. Comparing these results with Table 2, we notice that when adaptive vocabulary is enabled, the system captures important terms quickly. If these terms could not be added in time, the system will keep on missing these values in following report therefore largely impairs recall rate in some cases. These results show that there could be a large discrepancy between standard ontology and the controlled vocabulary of a specific domain, and the benefits adaptable vocabularies could be substantial.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab3_Zheng_JPathInfo2015_6.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"06b42013a558260b7a7ad41a2bd4057a\"><img alt=\"Tab3 Zheng JPathInfo2015 6.jpg\" src=\"https:\/\/www.limswiki.org\/images\/7\/78\/Tab3_Zheng_JPathInfo2015_6.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 3.<\/b> Test result of experiment 2: Study of adaptive vocabulary<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Discussions\">Discussions<\/span><\/h2>\n<p>In the experiments, we have selected attributes that appear in different types of text that are found in pathology reports. Our goal is for testing the effectiveness of online machine learning and the importance of refining the seed vocabulary during the extraction process. Though extracting value in generic domain is challenging, given particular task or research topic, which has value of limited domain, the system could be customized easily to meet specific purpose of given task, for example, identify patient for brain tumor research. In follow-up studies, we will consider a broader set of attributes and enrich the supported data types. Values that the system can manage currently are limited to numerical and nominal values. Extracting temporal information, for example, will improve the utility of the system. In pathology research, medical events such as procedures are time sensitive. Augmenting the output with timelines would contextualize and help to connect the extracted values in important ways.\n<\/p><p>A goal of IDEAL-X is to provide a generic solution for information extraction across medical domains. The system employs techniques that we believe are domain agnostic, which could be validated with use cases in other medical domains. We are collaborating with Emory Clinical Cardiovascular Research Institute and Rutgers University Radiation Oncology Department to validate the broader utility of IDEAL-X. One characteristic of the cardiology research project is the existence of multiple types of reports for each patient. This adds complexity to the interface and the learning algorithm. We also plan to study the advantage of using IDEAL-X, as a software assistant for annotation over manual annotation both on efficiency and accuracy.\n<\/p><p>Finally, we will study the use of IDEAL-X to ease structured reporting such as providing machine generated automated hints to create synoptic pathology reports. Though answers may be inputted using a combo box to guarantee structured reporting, most existing medical report systems still allow for free-format text and uncontrolled vocabulary. In some cases, direct access to the structured database may not be available, therefore, requires extracting information from text pathology report directly.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>IDEAL-X employs iterative online machine learning and adaptive controlled vocabulary for information extraction from clinical reports. It automatically predicts annotations for values to be extracted and utilizes human feedback as knowledge to improve continuously the performance of extraction. Experimental results demonstrate that both online learning and adaptive vocabulary are highly effective. Extracted data are indexed by the built-in query engine and can be conveniently queried with a graphic interface. The adaptability and usability of the system make IDEAL-X a powerful data extraction tool to support patient search from pathology reports.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Acknowledgments\">Acknowledgments<\/span><\/h3>\n<p>This study is supported in part by grants from the Centers for Disease Control and Prevention 200-2014-M-59415.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Financial_support_and_sponsorship\">Financial support and sponsorship<\/span><\/h3>\n<p>Nil.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h3>\n<p>There are no conflicts of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-SrigleyStand-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SrigleyStand_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Srigley, J.R.; McGowan, T.; Maclean, A.; Raby, M.; Ross, J.; Kramer, S.; Sawka, C. (2009). \"Standardized synoptic cancer pathology reporting: A population-based approach\". <i>Journal of Surgical Oncology<\/i> <b>99<\/b> (8): 517\u2013524. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fjso.21282\" target=\"_blank\">10.1002\/jso.21282<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19466743\" target=\"_blank\">19466743<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Standardized+synoptic+cancer+pathology+reporting%3A+A+population-based+approach&rft.jtitle=Journal+of+Surgical+Oncology&rft.aulast=Srigley%2C+J.R.%3B+McGowan%2C+T.%3B+Maclean%2C+A.%3B+Raby%2C+M.%3B+Ross%2C+J.%3B+Kramer%2C+S.%3B+Sawka%2C+C.&rft.au=Srigley%2C+J.R.%3B+McGowan%2C+T.%3B+Maclean%2C+A.%3B+Raby%2C+M.%3B+Ross%2C+J.%3B+Kramer%2C+S.%3B+Sawka%2C+C.&rft.date=2009&rft.volume=99&rft.issue=8&rft.pages=517%E2%80%93524&rft_id=info:doi\/10.1002%2Fjso.21282&rft_id=info:pmid\/19466743&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GillSyn-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GillSyn_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gill, A.J.; Johns, A.L.; Eckstein, R.; Samra, J.S.; Kaufman, A.; Chang, D.K.; Merrett N.D.; Cosman, P.H.; Smith, R.C.; Biankin, A.V.; Kench, J.G.; New South Wales Pancreatic Cancer Network (2009). \"Synoptic reporting improves histopathological assessment of pancreatic resection specimens\". <i>Pathology<\/i> <b>41<\/b> (2): 161\u2013167. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19320058\" target=\"_blank\">19320058<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Synoptic+reporting+improves+histopathological+assessment+of+pancreatic+resection+specimens&rft.jtitle=Pathology&rft.aulast=Gill%2C+A.J.%3B+Johns%2C+A.L.%3B+Eckstein%2C+R.%3B+Samra%2C+J.S.%3B+Kaufman%2C+A.%3B+Chang%2C+D.K.%3B+Merrett+N.D.%3B+Cosman%2C+P.H.%3B+Smith%2C+R.C.%3B+Biankin%2C+A.V.%3B+Kench%2C+J.G.%3B+New+South+Wales+Pancreatic+Cancer+Network&rft.au=Gill%2C+A.J.%3B+Johns%2C+A.L.%3B+Eckstein%2C+R.%3B+Samra%2C+J.S.%3B+Kaufman%2C+A.%3B+Chang%2C+D.K.%3B+Merrett+N.D.%3B+Cosman%2C+P.H.%3B+Smith%2C+R.C.%3B+Biankin%2C+A.V.%3B+Kench%2C+J.G.%3B+New+South+Wales+Pancreatic+Cancer+Network&rft.date=2009&rft.volume=41&rft.issue=2&rft.pages=161%E2%80%93167&rft_id=info:pmid\/19320058&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeslieStand-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeslieStand_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leslie, K.O.; Rosai, J. (1994). \"Standardization of the surgical pathology report: Formats, templates, and synoptic reports\". <i>Seminars in Diagnostic Pathology<\/i> <b>11<\/b> (4): 253\u20137. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/7878300\" target=\"_blank\">7878300<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Standardization+of+the+surgical+pathology+report%3A+Formats%2C+templates%2C+and+synoptic+reports&rft.jtitle=Seminars+in+Diagnostic+Pathology&rft.aulast=Leslie%2C+K.O.%3B+Rosai%2C+J.&rft.au=Leslie%2C+K.O.%3B+Rosai%2C+J.&rft.date=1994&rft.volume=11&rft.issue=4&rft.pages=253%E2%80%937&rft_id=info:pmid\/7878300&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CAPCancer-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CAPCancer_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cap.org\/web\/oracle\/webcenter\/portalapp\/pagehierarchy\/cancer_protocols.jspx\" target=\"_blank\">\"Cancer Protocols\"<\/a>. <i>CAP.org<\/i>. College of American Pathologists. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cap.org\/web\/oracle\/webcenter\/portalapp\/pagehierarchy\/cancer_protocols.jspx\" target=\"_blank\">http:\/\/www.cap.org\/web\/oracle\/webcenter\/portalapp\/pagehierarchy\/cancer_protocols.jspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Cancer+Protocols&rft.atitle=CAP.org&rft.date=2015&rft.pub=College+of+American+Pathologists&rft_id=http%3A%2F%2Fwww.cap.org%2Fweb%2Foracle%2Fwebcenter%2Fportalapp%2Fpagehierarchy%2Fcancer_protocols.jspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IHEAna-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IHEAna_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/wiki.ihe.net\/index.php?title=Anatomic_Pathology_Structured_Reports\" target=\"_blank\">\"Anatomic Pathology Structured Reports\"<\/a>. <i>IHE Wiki<\/i>. Integrating the Healthcare Enterprise. 30 June 2010<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/wiki.ihe.net\/index.php?title=Anatomic_Pathology_Structured_Reports\" target=\"_blank\">http:\/\/wiki.ihe.net\/index.php?title=Anatomic_Pathology_Structured_Reports<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Anatomic+Pathology+Structured+Reports&rft.atitle=IHE+Wiki&rft.date=30+June+2010&rft.pub=Integrating+the+Healthcare+Enterprise&rft_id=http%3A%2F%2Fwiki.ihe.net%2Findex.php%3Ftitle%3DAnatomic_Pathology_Structured_Reports&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CrowleycaTIES-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CrowleycaTIES_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Crowley, R.S.; Castine, M.; Mitchell, K.; Chavan, G.; McSherry, T.; Feldman, M. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710\" target=\"_blank\">\"caTIES: A grid based system for coding and retrieval of surgical pathology reports and tissue specimens in support of translational research\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>17<\/b> (3): 253\u201364. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Fjamia.2009.002295\" target=\"_blank\">10.1136\/jamia.2009.002295<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2995710\/\" target=\"_blank\">PMC2995710<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20442142\" target=\"_blank\">20442142<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2995710<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=caTIES%3A+A+grid+based+system+for+coding+and+retrieval+of+surgical+pathology+reports+and+tissue+specimens+in+support+of+translational+research&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Crowley%2C+R.S.%3B+Castine%2C+M.%3B+Mitchell%2C+K.%3B+Chavan%2C+G.%3B+McSherry%2C+T.%3B+Feldman%2C+M.&rft.au=Crowley%2C+R.S.%3B+Castine%2C+M.%3B+Mitchell%2C+K.%3B+Chavan%2C+G.%3B+McSherry%2C+T.%3B+Feldman%2C+M.&rft.date=2010&rft.volume=17&rft.issue=3&rft.pages=253%E2%80%9364&rft_id=info:doi\/10.1136%2Fjamia.2009.002295&rft_id=info:pmc\/PMC2995710&rft_id=info:pmid\/20442142&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2995710&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchadowExtr-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchadowExtr_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schadow, G.; McDonald, C.J. (2003). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1480213\" target=\"_blank\">\"Extracting structured information from free text pathology reports\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>2003<\/b>: 584\u20138. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1480213\/\" target=\"_blank\">PMC1480213<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/14728240\" target=\"_blank\">14728240<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1480213\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1480213<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Extracting+structured+information+from+free+text+pathology+reports&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Schadow%2C+G.%3B+McDonald%2C+C.J.&rft.au=Schadow%2C+G.%3B+McDonald%2C+C.J.&rft.date=2003&rft.volume=2003&rft.pages=584%E2%80%938&rft_id=info:pmc\/PMC1480213&rft_id=info:pmid\/14728240&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1480213&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CodenAuto-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CodenAuto_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Coden, A.; Savova, G.; Sominsky, I.; Tanenblatt, M.; Masanz, J.; Schuler, K.; Cooper, J.; Guan, W.; de Groen, P.C. (2009). \"Automatically extracting cancer disease characteristics from pathology reports into a disease knowledge representation model\". <i>Journal of Biomedical Informatics<\/i> <b>42<\/b> (5): 937\u201349. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2008.12.005\" target=\"_blank\">10.1016\/j.jbi.2008.12.005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19135551\" target=\"_blank\">19135551<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Automatically+extracting+cancer+disease+characteristics+from+pathology+reports+into+a+disease+knowledge+representation+model&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Coden%2C+A.%3B+Savova%2C+G.%3B+Sominsky%2C+I.%3B+Tanenblatt%2C+M.%3B+Masanz%2C+J.%3B+Schuler%2C+K.%3B+Cooper%2C+J.%3B+Guan%2C+W.%3B+de+Groen%2C+P.C.&rft.au=Coden%2C+A.%3B+Savova%2C+G.%3B+Sominsky%2C+I.%3B+Tanenblatt%2C+M.%3B+Masanz%2C+J.%3B+Schuler%2C+K.%3B+Cooper%2C+J.%3B+Guan%2C+W.%3B+de+Groen%2C+P.C.&rft.date=2009&rft.volume=42&rft.issue=5&rft.pages=937%E2%80%9349&rft_id=info:doi\/10.1016%2Fj.jbi.2008.12.005&rft_id=info:pmid\/19135551&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchlangenFeed-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchlangenFeed_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schlangen, David; Stede, Manfred; Bontas, Elena Paslaru (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/pub.uni-bielefeld.de\/publication\/1992186\" target=\"_blank\">\"Feeding Owl: Extracting and Representing the Content of Pathology Reports\"<\/a>. <i>NLPXML '04 Proceeedings of the Workshop on NLP and XML (NLPXML-2004): RDF\/RDFS and OWL in Language Technology<\/i>: 43\u201350<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/pub.uni-bielefeld.de\/publication\/1992186\" target=\"_blank\">http:\/\/pub.uni-bielefeld.de\/publication\/1992186<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Feeding+Owl%3A+Extracting+and+Representing+the+Content+of+Pathology+Reports&rft.jtitle=NLPXML+%2704+Proceeedings+of+the+Workshop+on+NLP+and+XML+%28NLPXML-2004%29%3A+RDF%2FRDFS+and+OWL+in+Language+Technology&rft.aulast=Schlangen%2C+David%3B+Stede%2C+Manfred%3B+Bontas%2C+Elena+Paslaru&rft.au=Schlangen%2C+David%3B+Stede%2C+Manfred%3B+Bontas%2C+Elena+Paslaru&rft.date=2004&rft.pages=43%E2%80%9350&rft_id=http%3A%2F%2Fpub.uni-bielefeld.de%2Fpublication%2F1992186&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MitchellImp-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MitchellImp_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mitchell, K.J.; Becich, M.J.; Berman, J.J.; Chapman, W.W.; Gilbertson, J.; Gupta, D.; Harrison, J.; Legowski, E.; Crowley, R.S. (2004). \"Implementation and evaluation of a negation tagger in a pipeline-based system for information extract from pathology reports\". <i>Studies in Health Technology and Informatics<\/i> <b>107<\/b> (Pt 1): 663-7. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2F978-1-60750-949-3-663\" target=\"_blank\">10.3233\/978-1-60750-949-3-663<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15360896\" target=\"_blank\">15360896<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementation+and+evaluation+of+a+negation+tagger+in+a+pipeline-based+system+for+information+extract+from+pathology+reports&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Mitchell%2C+K.J.%3B+Becich%2C+M.J.%3B+Berman%2C+J.J.%3B+Chapman%2C+W.W.%3B+Gilbertson%2C+J.%3B+Gupta%2C+D.%3B+Harrison%2C+J.%3B+Legowski%2C+E.%3B+Crowley%2C+R.S.&rft.au=Mitchell%2C+K.J.%3B+Becich%2C+M.J.%3B+Berman%2C+J.J.%3B+Chapman%2C+W.W.%3B+Gilbertson%2C+J.%3B+Gupta%2C+D.%3B+Harrison%2C+J.%3B+Legowski%2C+E.%3B+Crowley%2C+R.S.&rft.date=2004&rft.volume=107&rft.issue=Pt+1&rft.pages=663-7&rft_id=info:doi\/10.3233%2F978-1-60750-949-3-663&rft_id=info:pmid\/15360896&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmaleOnline-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SmaleOnline_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Smale, S.; Yao, Y. (2006). \"Online learning algorithms\". <i>Foundations of Computational Mathematics<\/i> <b>6<\/b> (2): 145\u2013170. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10208-004-0160-z\" target=\"_blank\">10.1007\/s10208-004-0160-z<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Online+learning+algorithms&rft.jtitle=Foundations+of+Computational+Mathematics&rft.aulast=Smale%2C+S.%3B+Yao%2C+Y.&rft.au=Smale%2C+S.%3B+Yao%2C+Y.&rft.date=2006&rft.volume=6&rft.issue=2&rft.pages=145%E2%80%93170&rft_id=info:doi\/10.1007%2Fs10208-004-0160-z&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShalevOnline1-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShalevOnline1_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShalevOnline1_12-1\" rel=\"external_link\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shalev-Shwartz, S. (2012). \"Online learning and online convex optimization\". <i>Foundations and Trends in Machine Learning<\/i> <b>4<\/b> (2): 107\u2013194. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1561%2F2200000018\" target=\"_blank\">10.1561\/2200000018<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Online+learning+and+online+convex+optimization&rft.jtitle=Foundations+and+Trends+in+Machine+Learning&rft.aulast=Shalev-Shwartz%2C+S.&rft.au=Shalev-Shwartz%2C+S.&rft.date=2012&rft.volume=4&rft.issue=2&rft.pages=107%E2%80%93194&rft_id=info:doi\/10.1561%2F2200000018&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShalevOnline2-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShalevOnline2_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShalevOnline2_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Shalev-Shwartz, S. (July 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf\" target=\"_blank\">\"Online Learning: Theory, Algorithms, and Applications\"<\/a> (PDF). University of Chicago<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf\" target=\"_blank\">http:\/\/ttic.uchicago.edu\/~shai\/papers\/ShalevThesis07.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Online+Learning%3A+Theory%2C+Algorithms%2C+and+Applications&rft.atitle=&rft.aulast=Shalev-Shwartz%2C+S.&rft.au=Shalev-Shwartz%2C+S.&rft.date=July+2007&rft.pub=University+of+Chicago&rft_id=http%3A%2F%2Fttic.uchicago.edu%2F%7Eshai%2Fpapers%2FShalevThesis07.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IDEALVid-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IDEALVid_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Zheng, Shuai (07 September 2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/youtu.be\/Q-DrWi31nv0\" target=\"_blank\">\"IDEAL-X: Information and Data Extraction using Adaptive Learning\"<\/a>. YouTube<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/youtu.be\/Q-DrWi31nv0\" target=\"_blank\">http:\/\/youtu.be\/Q-DrWi31nv0<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=IDEAL-X%3A+Information+and+Data+Extraction+using+Adaptive+Learning&rft.atitle=&rft.aulast=Zheng%2C+Shuai&rft.au=Zheng%2C+Shuai&rft.date=07+September+2014&rft.pub=YouTube&rft_id=http%3A%2F%2Fyoutu.be%2FQ-DrWi31nv0&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ManningIntro-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ManningIntro_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-ManningIntro_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Manning, C.D.; Raghavan, P.; Sch\u00fctze, H. (2008). <i>Introduction to Information Retrieval<\/i> (1st ed.). Cambridge, England: Cambridge University Press. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780521865715.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Introduction+to+Information+Retrieval&rft.aulast=Manning%2C+C.D.%3B+Raghavan%2C+P.%3B+Sch%C3%BCtze%2C+H.&rft.au=Manning%2C+C.D.%3B+Raghavan%2C+P.%3B+Sch%C3%BCtze%2C+H.&rft.date=2008&rft.edition=1st&rft.place=Cambridge%2C+England&rft.pub=Cambridge+University+Press&rft.isbn=9780521865715&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElliotHid-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ElliotHid_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-ElliotHid_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Elliott, R.J.; Aggoun, L.; Moore, J.B. (1995). <i>Hidden Markov Models: Estimation and Control<\/i> (1st ed.). New York, NY: Springer-Verlag New York. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-0-387-84854-9\" target=\"_blank\">10.1007\/978-0-387-84854-9<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780387848549.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Hidden+Markov+Models%3A+Estimation+and+Control&rft.aulast=Elliott%2C+R.J.%3B+Aggoun%2C+L.%3B+Moore%2C+J.B.&rft.au=Elliott%2C+R.J.%3B+Aggoun%2C+L.%3B+Moore%2C+J.B.&rft.date=1995&rft.edition=1st&rft.place=New+York%2C+NY&rft.pub=Springer-Verlag+New+York&rft_id=info:doi\/10.1007%2F978-0-387-84854-9&rft.isbn=9780387848549&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FurnkranzSep-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FurnkranzSep_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-FurnkranzSep_17-1\" rel=\"external_link\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">F\u00fcrnkranz, J. (1999). \"Separate-and-conquer rule learning\". <i>Artificial Intelligence Review<\/i> <b>13<\/b> (1): 3\u201354. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1023%2FA%3A1006524209794\" target=\"_blank\">10.1023\/A:1006524209794<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Separate-and-conquer+rule+learning&rft.jtitle=Artificial+Intelligence+Review&rft.aulast=F%C3%BCrnkranz%2C+J.&rft.au=F%C3%BCrnkranz%2C+J.&rft.date=1999&rft.volume=13&rft.issue=1&rft.pages=3%E2%80%9354&rft_id=info:doi\/10.1023%2FA%3A1006524209794&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NIHSNOMED-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NIHSNOMED_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.nlm.nih.gov\/research\/umls\/Snomed\/snomed_main.html\" target=\"_blank\">\"SNOMED Clinical Terms (SNOMED CT)\"<\/a>. <i>Unified Medical Language System (UMLS)<\/i>. U.S. National Library of Medicine. 25 February 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.nlm.nih.gov\/research\/umls\/Snomed\/snomed_main.html\" target=\"_blank\">https:\/\/www.nlm.nih.gov\/research\/umls\/Snomed\/snomed_main.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SNOMED+Clinical+Terms+%28SNOMED+CT%29&rft.atitle=Unified+Medical+Language+System+%28UMLS%29&rft.date=25+February+2015&rft.pub=U.S.+National+Library+of+Medicine&rft_id=https%3A%2F%2Fwww.nlm.nih.gov%2Fresearch%2Fumls%2FSnomed%2Fsnomed_main.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NCIThes-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-NCIThes_19-0\" rel=\"external_link\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-NCIThes_19-1\" rel=\"external_link\">19.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/ncit.nci.nih.gov\/\" target=\"_blank\">\"NCI Thesaurus\"<\/a>. <i>NCI Thesaurus<\/i>. National Cancer Institute. July 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/ncit.nci.nih.gov\/\" target=\"_blank\">https:\/\/ncit.nci.nih.gov\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NCI+Thesaurus&rft.atitle=NCI+Thesaurus&rft.date=July+2015&rft.pub=National+Cancer+Institute&rft_id=https%3A%2F%2Fncit.nci.nih.gov%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IGSDisease-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IGSDisease_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.disease-ontology.org\/\" target=\"_blank\">\"Disease Ontology\"<\/a>. Institute for Genome Sciences, University of Maryland School of Medicine. 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.disease-ontology.org\/\" target=\"_blank\">http:\/\/www.disease-ontology.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Disease+Ontology&rft.atitle=&rft.date=2015&rft.pub=Institute+for+Genome+Sciences%2C+University+of+Maryland+School+of+Medicine&rft_id=http%3A%2F%2Fwww.disease-ontology.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note--21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Kuiper, Martin (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cellcycleontology.org\/\" target=\"_blank\">\"Semantic Systems Biology\"<\/a>. Institutt for biologi - NTNU<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cellcycleontology.org\/\" target=\"_blank\">http:\/\/www.cellcycleontology.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 12 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Semantic+Systems+Biology&rft.atitle=&rft.aulast=Kuiper%2C+Martin&rft.au=Kuiper%2C+Martin&rft.date=2015&rft.pub=Institutt+for+biologi+-+NTNU&rft_id=http%3A%2F%2Fwww.cellcycleontology.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases citation information was missing and was added to make the reference more useful.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210231\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.600 seconds\nReal time usage: 0.651 seconds\nPreprocessor visited node count: 16398\/1000000\nPreprocessor generated node count: 35226\/1000000\nPost\u2010expand include size: 118220\/2097152 bytes\nTemplate argument size: 40371\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 584.986 1 - -total\n 81.12% 474.525 1 - Template:Reflist\n 69.20% 404.813 21 - Template:Citation\/core\n 43.91% 256.872 11 - Template:Cite_journal\n 23.87% 139.658 8 - Template:Cite_web\n 13.92% 81.416 1 - Template:Infobox_journal_article\n 13.39% 78.311 1 - Template:Infobox\n 7.55% 44.157 80 - Template:Infobox\/row\n 6.77% 39.617 2 - Template:Cite_book\n 5.97% 34.941 19 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7827-0!*!0!!en!5!* and timestamp 20181213210231 and revision id 23421\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction\">https:\/\/www.limswiki.org\/index.php\/Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","65b45b4f6a653c627bbc452940cf174f_images":["https:\/\/www.limswiki.org\/images\/4\/43\/Fig1_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/9\/9c\/Fig2_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/f\/f7\/Fig3_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/f\/f2\/Fig4_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/b\/b0\/Fig5_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/c\/c7\/Fig6_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/0\/01\/Tab1_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/5\/58\/Tab2_Zheng_JPathInfo2015_6.jpg","https:\/\/www.limswiki.org\/images\/7\/78\/Tab3_Zheng_JPathInfo2015_6.jpg"],"65b45b4f6a653c627bbc452940cf174f_timestamp":1544734951,"5d49b7c0866e1210c1696240ca7f59fd_type":"article","5d49b7c0866e1210c1696240ca7f59fd_title":"Return on investment in electronic health records in primary care practices: A mixed-methods study (Jang et al. 2014)","5d49b7c0866e1210c1696240ca7f59fd_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study","5d49b7c0866e1210c1696240ca7f59fd_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Return on investment in electronic health records in primary care practices: A mixed-methods study\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nReturn on investment in electronic health records in primary care practices: A mixed-methods studyJournal\n \nJMIR Medical InformaticsAuthor(s)\n \nJang, Yeona; Lortie, Michel A.; Sanche, StevenAuthor affiliation(s)\n \nMcGill University, Desautels Faculty of Management; St Mary's Research Centre, MontrealPrimary contact\n \nEmail: yeona.jang@mcgill.ca; Phone: 1.514.398.8489Editors\n \nEysenbach, G.Year published\n \n2014Volume and issue\n \n2 (2)Page(s)\n \ne25DOI\n \n10.2196\/medinform.3631ISSN\n \n2291-9694Distribution license\n \nCreative Commons Attribution 2.0Website\n \nhttp:\/\/medinform.jmir.org\/2014\/2\/e25\/Download\n \nhttp:\/\/medinform.jmir.org\/article\/download\/medinform_v2i2e25\/2 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n\n2.1 Context \n2.2 Measurement \n2.3 Objectives \n\n\n3 Methods \n\n3.1 Sample \n3.2 Methodology \n\n\n4 Results \n\n4.1 Analysis of break-even point as indicator of return on investment \n\n4.1.1 Overview \n4.1.2 Change in clinic net revenue after implementation of electronic health records \n4.1.3 Percent changes of counts after implementation \u2014 Not linearly proportional to one another \n4.1.4 Percent changes in number of active patients and revenue after implementation \u2014 Not linearly proportional to one another \n4.1.5 Sign test results \n4.1.6 Analysis of variance in realizing financial performance \u2014 Key factors \n\n\n4.2 Analysis of electronic health record functionalities used in primary care clinics \n\n\n5 Discussion \n\n5.1 Principal findings \n5.2 Implications for practitioners and managers in primary care \n5.3 Implications for policy makers \n5.4 Future research \n5.5 Limitations \n\n\n6 Acknowledgements \n7 Conflicts of interest \n8 References \n9 Abbreviations \n10 Notes \n\n\n\nAbstract \nBackground: The use of electronic health records (EHR) in clinical settings is considered pivotal to a patient-centered health care delivery system. However, uncertainty in cost recovery from EHR investments remains a significant concern in primary care practices.\nObjective: Guided by the question of \u201cWhen implemented in primary care practices, what will be the return on investment (ROI) from an EHR implementation?\u201d, the objectives of this study are two-fold: (1) to assess ROI from EHR in primary care practices and (2) to identify principal factors affecting the realization of positive ROI from EHR. We used a break-even point, that is, the time required to achieve cost recovery from an EHR investment, as an ROI indicator of an EHR investment.\nMethods: Given the complexity exhibited by most EHR implementation projects, this study adopted a retrospective mixed-method research approach, particularly a multiphase study design approach. For this study, data were collected from community-based primary care clinics using EHR systems.\nResults: We collected data from 17 primary care clinics using EHR systems. Our data show that the sampled primary care clinics recovered their EHR investments within an average period of 10 months (95% CI 6.2-17.4 months), seeing more patients with an average increase of 27% in the active-patients-to-clinician-FTE (full time equivalent) ratio and an average increase of 10% in the active-patients-to-clinical-support-staff-FTE ratio after an EHR implementation. Our analysis suggests, with a 95% confidence level, that the increase in the number of active patients (P=.006), the increase in the active-patients-to-clinician-FTE ratio (P<.001), and the increase in the clinic net revenue (P<.001) are positively associated with the EHR implementation, likely contributing substantially to an average break-even point of 10 months.\nConclusions: We found that primary care clinics can realize a positive ROI with EHR. Our analysis of the variances in the time required to achieve cost recovery from EHR investments suggests that a positive ROI does not appear automatically upon implementing an EHR and that a clinic\u2019s ability to leverage EHR for process changes seems to play a role. Policies that provide support to help primary care practices successfully make EHR-enabled changes, such as support of clinic workflow optimization with an EHR system, could facilitate the realization of positive ROI from EHR in primary care practices.\nKeywords: return on investment in electronic health records; cost recovery from EHR implementation; ROI indicator; physician satisfaction with EHR; primary care practices \n\nIntroduction \nContext \nThe use of electronic health records (EHR) in clinical settings is widely recommended as an innovation enabler with potential benefits of reducing health care costs, while improving quality and safety, and is considered central to achieving patient-centered health care.[1][2][3][4] As a wide array of EHR projects have been implemented within various health care settings, the health care field is rich with volumes of work examining the benefits of EHR. However, the existing literature reports mixed results in benefits realized from EHR implementation.[5][6] Such mixed results suggest that the implementation of EHR systems does not automatically guarantee the conversion of potential benefits into realized benefits.\nThe implementation of EHR systems within primary care practices is seen as particularly complex[7][8][9][10], with physicians and other staff in primary care practices citing obstacles such as difficulty in adapting to the significant changes in workflow and the time commitment required to learn to use the new software while prioritizing patient care.[11][12][13][14] While there is a growing body of evidence that EHR can be a valuable tool for improving quality of care and patient safety with relatively positive perceptions about EHR benefits[15][16][17], uncertainty about cost recovery of an EHR investment remains a significant concern in primary care practices.[7][8][18][19] Various studies on EHR impact and adoption also raise the need for cost-benefit analysis of EHR investments.[5][20] Thus, this study seeks to assess the return on investment (ROI) from an EHR implementation in primary care settings, aiming to complement the current insights on cost recovery concerns in existing literature.\n\nMeasurement \nReturn on investment is a common approach to measuring rates of return on money invested, in terms of increased profit attributable to the investment. A standard ROI is defined as follows:\nROI = (Gain from investment - Cost of investment)\/Cost of investment\nResults reported by various studies regarding ROI from EHR systems in primary care settings are mixed.[21][22][23][24][25] Most of the existing literature used a bottom-up approach identifying specific cost-saving areas and collecting the data on financial savings made in these areas attributable to EHR systems. However, EHR is a process-enabling information technology (IT) that offers the opportunity to streamline information-intensive workflow, remove manual hand-off of data and information, and facilitate coordination \u2014 thus facilitating the execution of entire business processes rather than individual tasks. Due mainly to the context-sensitive nature of benefits realization from a process-enabling IT such as EHR and the scarcity of detailed financial data relating to gains and\/or savings directly attributable to an EHR system in primary care clinics, this study used break-even-point analysis as an indicator of ROI, instead of standard ROI analysis.\nThe break-even point of an EHR investment is defined as the number of months it takes a clinic to recover the cost of the EHR system and other associated implementation costs, with increased revenues and\/or decreased expenses. Increases in revenues and\/or decreases in expenses are assessed by considering net revenues during three distinct periods of time: pre-EHR, peri-EHR, and post-EHR. The pre-EHR period is defined as the full fiscal year before the implementation of an EHR system started. The peri-EHR period is defined as the fiscal year(s) containing the EHR implementation period (ie, during EHR implementation). If the peri-EHR period covers more than one fiscal year, the net revenue is averaged over these fiscal years. The post-EHR period is defined as the full fiscal year following the end of the peri-EHR period.\nTo calculate the break-even point of implementing an EHR system in a clinic, the cost of EHR implementation is set equal to the difference in the clinic\u2019s net revenue between the pre-EHR and peri-EHR periods, plus the difference in the clinic\u2019s net revenue between the pre-EHR and post-EHR periods, as summarized in the following formula:\nC EHR= [(NR peri\u2212NR pre)\/12]*M imp + [(NR post\u2212NR pre)\/12] * (M break-even\u2212M imp)\nIn this formula: CEHR=cost of EHR implementation, NRperi=annual clinic net revenue in the peri-EHR period, NRpre=annual clinic net revenue in the pre-EHR period, NRpost=annual clinic net revenue in the post-EHR period, Mimp=the number of months taken to complete the EHR implementation in the clinic, and Mbreak-even=months to break even. The net revenue of a clinic is defined as the sum of the physicians\u2019 billings for work done in the clinic minus any expenses that the clinic pays to maintain its ongoing practice. In the case where the months to break even were less than the months of EHR implementation, in other words, the net revenue difference between pre-EHR and peri-EHR periods is large enough to recover the cost of EHR implementation, the formula was adjusted by setting the cost of EHR implementation equal to the difference in a clinic\u2019s net revenue between the pre-EHR and peri-EHR periods, or:\nCEHR = [(NRperi \u2013 NRpre)\/12]* Mbreak-even\n\nObjectives \nGuided by the research question \u201cWhen implemented in primary care practices, what will be the ROI from an EHR implementation?\u201d, the objectives of this research are twofold: (1) to assess the ROI from an EHR implementation in primary care practices by measuring the time required to recover the cost of converting a clinic from a paper-based environment to an EHR-enabled environment and (2) to identify principal factors affecting the realization of a positive ROI from an EHR implementation in primary care practices. Such ROI information related to cost recovery of an EHR investment would be helpful to both clinics considering implementing EHR systems and to policy makers designing EHR-adoption funding programs and policies.\n\nMethods \nSample \nCommunity-based, primary care clinics meeting the following four eligibility criteria were recruited for this study on ROI from EHR in primary-care settings. First, this study focused on community-based, primary care clinics. Thus, specialty care clinics and walk-in clinics were excluded. Second, clinics were required to have implemented EHR systems. Third, clinics were required to have been paper-based in the past, in order to ensure that the comparison between pre-EHR and post-EHR implementation performance was possible for the ROI calculation. Fourth, clinics were required to provide operational and financial data necessary to calculate ROI, as well as the information on challenges and opportunities that they had experienced both during and after the EHR implementation.\nThe research team contacted 132 randomly selected community-based, primary care clinics in Canada that met the first two eligibility criteria for recruitment to the study. Of the 132 clinics, 62 clinics declined to participate, mostly citing time constraints. Of the 70 clinics remaining, 34 clinics were not eligible, mainly because they were unable to provide the operational and financial data necessary to calculate ROI. Of the 36 eligible clinics, 19 clinics later declined to participate, due mainly to time constraints. Thus, data were collected from a total of the 17 eligible clinics, resulting in the study participation rate of 13%, which is relatively consistent with typical participation rates of family physicians reported in other studies involving interviews and observations.[26] No statistically significant differences were observed between participating and non-participating primary care clinics in terms of geographic location (P=.315), the number of physicians or other clinicians (P<.001), or the number of patients per physician (P=.192). Table 1 summarizes the basic statistics on the size of the sampled clinics. We used Full Time Equivalent (FTE) in comparing the size of primary care clinics.\n\n\n\n\n\n\n\nTable 1. Basic statistics on the size of a primary care clinic in the study\n\n\n\n\n Average\n\n SD\n\n Median\n\n Minimum\n\n Maximum\n\n\nClinic size: clinician FTEs\n\n\nPre-EHR period\n\n3.4\n\n2.6\n\n3.0\n\n1.0\n\n8.5\n\n\nPost-EHR period\n\n3.6\n\n2.4\n\n3.0\n\n0.8\n\n8.0\n\n\nClinic size: clinical support staff FTEs\n\n\nPre-EHR period\n\n3.4\n\n2.9\n\n2.8\n\n1.0\n\n12.0\n\n\nPost-EHR period\n\n4.2\n\n3.1\n\n3.0\n\n0.9\n\n12.0\n\n\n\nMethodology \nGiven the complexity exhibited by most EHR implementation projects, this study used a mixed-method research approach, particularly a multiphase study design.[27] By combining quantitative and qualitative data, mixed-method research can provide a fuller understanding of the complex and multidimensional world of primary care clinics than would otherwise be achieved using either approach alone.\nIn the quantitative study phase, questionnaire modules were designed, based on prior research in the existing literature[28][29][30][31][32][33], to collect data on EHR implementation costs, EHR functionalities in use, physician satisfaction with EHR, and physicians\u2019 perceptions about the impact of EHR on operational efficiency and on quality of care. Each clinic respondent was required to complete the study instruments using the online questionnaires or researcher-assisted telephone questionnaires. The minimum financial data required for the study include clinic revenue and clinic net revenue as well as EHR implementation cost that consisted of EHR software costs, hardware costs, support costs, and labor costs associated with EHR system implementation and training, in the three different periods\u2014before EHR implementation, during EHR implementation, and after EHR implementation. The minimum operational data required for this ROI study include the number of active patients, clinician FTEs, and clinical support staff FTEs, in the same three periods. The lead researcher served as dedicated support liaison for clinics, in order to ensure that the costs of the EHR implementation, as well as other financial and operational data before and after EHR implementation, were abstracted from clinic records in a consistent fashion. In the subsequent qualitative study, semistructured interviews and observations were conducted with clinic staff and physicians identified as responsible for such functions as patient appointment management, patient record management, test results management, patient encounters, and billing, to assess factors affecting the realization of a positive ROI from an EHR implementation in primary care practices.\nThe data collected from 17 sampled primary care clinics were documented and analyzed using statistical analysis and grounded theory.[34] As break-even points were analyzed, we compared those clinics that were successful in realizing a positive ROI from EHR implementation to those that were less successful, in an attempt to identify principal factors impacting the realization of positive ROI from EHR. In particular, we used linear regression analysis to estimate the relationships of the outcome variable \u201cbreak-even point\u201d with the explanatory variables that include the codes identified from the qualitative data through the coding process.\n\nResults \nAnalysis of break-even point as indicator of return on investment \nOverview \nOur analysis suggests that the sampled primary care clinics typically recovered their investment in EHR within an average of 10 months (95% confidence interval: 6.2 months, 17.4 months), seeing more patients with improved active-patients-to-clinician-FTE and active-patients-to-clinical-support-staff-FTE ratios in the post-EHR implementation period.\n\nChange in clinic net revenue after implementation of electronic health records \nOnce an EHR system is implemented, a key factor that impacts the time required to achieve cost recovery from the EHR investments is clinic net revenue. With respect to how clinics fared financially upon adopting EHR systems, all but one of the primary care clinics in our study achieved an increase in clinic net revenue in the post-EHR period, as shown in Table 2.\n\n\n\n\n\n\n\nTable 2. Percent changes in clinic revenues, net revenues, and clinician FTEs between the pre-EHR and post-EHR periods\n\n\nClinic #\n\nPercent change between the pre-EHR and post-EHR periods (in ascending order by percent change in number of clinician FTEs), %\n\n\n In number of clinician FTEs\n\n In clinic revenue\n\n In clinic\u2019s net revenue\n\n\nClinic 1\n\n-29\n\n23\n\n23\n\n\nClinic 2\n\n-20\n\n-28\n\n22\n\n\nClinic 3\n\n-14\n\n27\n\n4\n\n\nClinic 4\n\n-2\n\n29\n\n26\n\n\nClinic 7\n\n0\n\n55\n\n9\n\n\nClinic 5\n\n0\n\n50\n\n63\n\n\nClinic 9\n\n0\n\n33\n\n8\n\n\nClinic 10\n\n0\n\n31\n\n28\n\n\nClinic 8\n\n0\n\n23\n\n28\n\n\nClinic 11\n\n0\n\n19\n\n16\n\n\nClinic 6\n\n0\n\n3\n\n15\n\n\nClinic 12\n\n0\n\n-10\n\n20\n\n\nClinic 13\n\n0\n\n-15\n\n-30\n\n\nClinic 14\n\n10\n\n120\n\n116\n\n\nClinic 15\n\n47\n\n223\n\n227\n\n\nClinic 16\n\n53\n\n103\n\n98\n\n\nClinic 17\n\n329\n\n603\n\n845\n\n\nAverage\n\n22\n\n76\n\n89\n\n\n\nIn addition to clinic net revenue, the sampled clinics showed, on average, positive increases in active patient count, clinician count, clinical support staff count, and clinic revenue in the post-EHR implementation period. These increases are summarized in Figure 1.\n\n\n\nFigure 1. Average percent changes in active patient count, clinician FTE count, clinical support staff\r\nFTE count, clinic revenue, and clinic net revenue between the pre-EHR and post-EHR periods\nPercent increase in clinic net revenue between the pre-EHR and post-EHR periods showed a very strong positive correlation with percent increase in clinic revenue in the same periods (r=.99). Percent increase in clinic revenue showed a strong positive correlation with percent increase in the number of active patients (r=.87). It also showed a strong positive correlation with percent increase in the number of clinician FTEs, as well as with the number of clinical-support-staff FTEs (r=.96 and r=.97, respectively). These correlation coefficients (r values) are summarized in Figure 2.\n\n\n\nFigure 2. Correlations (r-values): clinic net revenue, clinic revenue, active patient count,\r\nclinician FTE count, and clinical support staff FTE count\nPercent changes of counts after implementation \u2014 Not linearly proportional to one another \nInterestingly, the percent increases in active patient count, clinician FTE count, and clinical support staff FTE count are not linearly proportional to one another. An average active-patient-count increase of 56% was handled by an average 22% increase for clinician FTEs and an average 39% increase for clinical-support-staff FTEs. This finding suggests change in operational efficiency after EHR implementation, with respect to the active-patients-to-clinician-FTE ratio and the active-patients-to-clinical-support-staff-FTE ratio. The sampled clinics showed an average increase of 27% in the active-patients-to-clinician-FTE ratio and an average increase of 10% in the active-patients-to-clinical-support-staff-FTE ratio, as illustrated in Figure 3.\n\n\n\nFigure 3. Average percent changes in a clinic\u2019s operational efficiency and financial\r\nperformance between the pre-EHR and post-EHR periods\nPercent increase in the number of active patients showed strong positive correlations with percent increases in active-patients-to-clinician-FTE ratio (r=.64) and in active-patients-to-clinical-support-staff-FTE ratio (r=.70), as shown in Figure 4.\n\n\n\nFigure 4. Correlations (r-values): clinic net revenue, clinic revenue, active patient count,\r\nclinician FTE count, clinical support staff FTE count, active-patients-to-clinician-FTE ratio,\r\nand active-patients-to-clinical-support-staff-FTE ratio\nThese correlations, together with the nonlinear percent changes summarized in Figure 3, suggest that the increased efficiency in the post-EHR period contributed to a clinic\u2019s ability to accommodate the increased number of active patients.\n\nPercent changes in number of active patients and revenue after implementation \u2014 Not linearly proportional to one another \nThe percent increase in clinic revenue was also not linearly proportional to the percent increase in the number of active patients \u2014 an average increase of 76% versus an average increase of 56%, respectively. In addition to the increase in the number of active patients, there seem to be other factors that contributed to clinic revenue increase in the post-EHR period (detailed analysis on the impact of EHR on the sampled clinics\u2019 billing patterns and revenue management processes, required to identify the contributing elements of the greater than linear increase in clinic revenue over the increase in patient count, is beyond the scope of the study).\nThe study also finds that percent increase in clinic net revenue was not linearly proportional to percent increase in clinic revenue. The average additional 13% increase in clinic net revenue (89%, which is 13% above the clinic average revenue of 76%) is attributable to the enhanced operational efficiency in the post-EHR period, which suggests the relative cost-savings effect after the EHR implementation.\n\nSign test results \nWe further tested the financial and operational impact of EHR in the post-EHR period, in order to assess the degree to which these findings could be extended to the population of clinics implementing EHR. The sign test, as opposed to t test, was adopted because the sample size was less than 30 and because the distributions shown in the data were not normal, with a high degree of skewness in most cases. The sign test results shown in Table 3 suggest, with a 95% confidence level, that the median percent change in clinic net revenue between the pre-EHR and post-EHR periods is positive in the population of the primary care clinics implementing EHR (sign test M=7.5 with P<.001).\n\n\n\n\n\n\n\nTable 3. Summary of statistical analysis of change in a clinic\u2019s operational efficiency and financial performance between between the pre-EHR and post-EHR periods\n\n\n Percent changes between the pre-EHR and post-EHR periods\n\n Average\n\n SD\n\n Median\n\n M\n\n Sign test, P value\n\n\nPercent change in clinic net revenue\n\n89%\n\n203%\n\n23%\n\n7.5\n\n<.001\n\n\nPercent change in the number of active patients\n\n56%\n\n119%\n\n10%\n\n5.0\n\n.006\n\n\nPercent change in active-patients-to-clinician-FTE ratio\n\n27%\n\n53%\n\n9%\n\n6.5\n\n<.001\n\n\nPercent change in the number of clinician FTEs\n\n22%\n\n82%\n\n0%\n\n0.0\n\n1.00\n\n\nPercent change in the number of clinical support staff FTEs\n\n37%\n\n75%\n\n0%\n\n3.0\n\n.07\n\n\nPercent change in active-patients-to-clinical-support-staff-FTE ratio\n\n10%\n\n29%\n\n4%\n\n2.5\n\n.277\n\n\n\nThe same conclusions can be made for the median percent changes in the active-patients-to-clinician-FTE ratio and in the number of active patients in the same periods (M=6.5 and M=5, respectively). However, for the median percent changes with respect to the number of clinician FTEs, the number of clinical support staff FTEs, and the active-patients-to-clinical-support-staff-FTE ratio, we could not reject with a 95% confidence level the null hypothesis of no change after EHR implementation.\nThe correlation coefficients shown in Figure 4 and sign test results summarized in Table 4 suggest that the increase in the active patient count may not be the only factor that contributed to an average break-even point of 10 months upon EHR implementation. Percent increases in the number of active patients, in the active-patients-to-physician-FTE ratio, and in clinic net revenue appear to be positively associated with the EHR implementation, likely contributing substantially to an average break-even point of 10 months.\n\nAnalysis of variance in realizing financial performance \u2014 Key factors \nStudy participants reported improvements in their ability to manage patient information after the implementation of EHR systems, citing improved ability to manage results such as obtaining test results from laboratories and following the results of an investigation over time (64%, 11\/17 clinics). Respondents also reported an improved ability to seek out specific information from patient records (57%, 10\/17 clinics), and access complete, up-to-date patient charts and review patient problems (43%, 6\/15 clinics). See Table 4 for key EHR impacts expressed by study participants.\n\n\n\n\n\n\n\nTable 4. Impact of EHR on clinic practices identified by study participants\n\n\n Categories\n\n Participant comments\n\n\nA. Impact of EHR on a clinic\u2019s ability to manage results\n\n\u201cWe receive results electronically and can graph them; graphs help \u2018engage\u2019 the patient.\u201d\r\n \r\n\u201cDirect to physician lab results has very positive effect on physician efficiency and patient care.\u201d\n\n\nB. Impact of EHR on a clinic\u2019s ability to seek out specific information from patient records\n\n\u201cComplete chart is always available, anywhere which affects patient safety and means better care.\u201d\r\n \r\n\u201cIntegration of information for referral requests is a great benefit.\u201d\r\n \r\n\u201cBilling codes are up-to-date. (And) billing is automatic by the doctor inside encounter note, which simplifies billing and is easier to manage reconciliation. No missed billing opportunities.\u201d\n\n\nC. Impact of EHR on a clinic\u2019s ability to prepare patient encounter\n\n\u201cReview of patient information prior to encounter is greatly facilitated.\u201d\r\n \r\n\u201cEasier to prepare for encounter; maintenance of problem list \/summary is much easier\u201d\r\n \r\n\u201cImmediate access to patient information\u2014no lost files.\u201d\n\n\n\nSome primary care clinics did better than others in using EHR and achieving faster break-even from EHR investment, which can be observed in Tables 1 and 3. To gain insights into key differences between those clinics that were highly successful and those less successful in realizing a positive ROI from EHR, we conducted regression analysis on break-even point as the outcome variable. We used the codes identified through the coding process of the qualitative data as a part of the explanatory variables to estimate their relationships with the outcome variable \u201cbreak-even point\u201d. As summarized in Table 5, the regression analysis suggests four statistically significant factors impacting the return on EHR investment, that is, the time required to achieve cost recovery from an investment in EHR.\n\n\n\n\n\n\n\nTable 5. Significant linear regression results of the outcome \u201cbreak-even point\u201d with explanatory variables (break-even point was log-transformed to approach a Normal distribution)\n\n\n Explanatory variable\n\n Variable values\n\n Regression coefficient\n\n Standard error of coefficient\n\n P value\n\n r2\n\n\n(a) Age of EHR: Months between Jan 1, 2013, and EHR implementation start date\n\nNumber of months\n\n0.03\n\n0.01\n\n.049\n\n.64\n\n\n(b) e-Prescriptions complying with national standards\n\n0 (No) to 1 (Yes)\n\n-1.32\n\n0.34\n\n.006\n\n.50\n\n\n(c) Extent to which EHR complies to national standards\n\nContinuous (from 0 to 10)\n\n-0.19\n\n0.07\n\n.038\n\n.54\n\n\n(d) Process change: Use of flow sheets\n\n0 (No) to 1 (Yes)\n\n-1.29\n\n0.46\n\n.022\n\n.68\n\n\n\nNote that in Table 5, the regression coefficient of an explanatory variable with a negative value indicates faster recovery of the EHR investment (ie, a shorter time required to achieve cost recovery from an EHR investment), while a positive value implies slower recovery of the EHR investment (ie, a longer time required to achieve cost recovery from an EHR investment).\nAge of electronic health record systems\nThe first result to note in item (a) of Table 5 is that older EHR implementations, in particular those implemented in 2004-2005, were slower to recover their investment, even though they still achieved a break-even point. This result suggests that the newer the EHR, the sooner a positive ROI can be achieved. The earlier EHR systems used by these clinics were less user-friendly and required longer training cycles for the users, which may explain why clinics with these earlier systems took longer to recoup their financial investment.\nCompliance with national standards\nThe second and third results, shown in items (b) and (c) of Table 5, suggest a positive link between the ROI indicator and the compliance with national standards such as codes representing prescription drugs. There was an improvement in clinics\u2019 compliance with national standards and ability to comply with evidence-based medicine. This improvement was related to the age of the EHR system used by the clinics. Newer EHR implementations may be more likely to comply with national standards, given that the newer EHRs are likely to support national standards better.\nUse of flow sheets and ability to manage patient information\nFinally, clinics using EHR flow sheets scored consistently better times to break even, shown in item (d) of Table 5. Clinics reported the use of flow sheets, or structured data collection forms, as a mechanism for compliance to evidence-based medicine. The use of flow sheets in EHRs provides advanced features such as those related to the automatic maintenance of patient problem lists and pharmacological profiles. These enhanced features contribute directly to the physician\u2019s efficiency by eliminating the time that would otherwise be spent manually maintaining these lists\u2014a task that can be time-consuming, highly repetitive, and labor-intensive to maintain with consistency in a paper-based environment. The availability of up-to-date lists makes patient encounter preparation easier and more rapid, as the necessary information is available at a glance.\n\nAnalysis of electronic health record functionalities used in primary care clinics \nOur study finds that despite the limited use of EHR functionalities and limited interoperability, the sampled clinics achieved overall positive operational and financial performance. Table 6 summarizes the data we gathered on EHR functionalities, frequency of use, and ease of use.\nMost frequently and routinely used EHR functionalities were related to medication management. Health information exchange and patient engagement portal functionalities saw no significant use (the investigation of why these functionalities were not used is beyond the scope of this study).\nRespondents stressed that it typically takes a few months to understand any particular EHR function sufficiently to effectively introduce it in their clinical practices. This finding, coupled with the finding that despite the limited use of EHR functionalities the clinics achieved overall positive improvement in operational and financial performance in the post-EHR period, suggests that a clinic\u2019s ability to embed particular EHR functionalities in their workflow and make use of these functionalities in their day-to-day clinical practices is of more importance in realizing a positive ROI from EHR implementation than implementing an EHR software package with the maximum number of features and functionalities.\n\n\n\n\n\n\n\nTable 6. EHR functionalities and utilization reported during the study period\n\n\n EHR functionalities\n\n % of clinics answering in the affirmative\n\n\nUser Interface: Does the EHR system currently in use at this clinic have any of the following user interface technologies? (N=17)\n\n\nAlternative presentation formats for clinical information\n\n100.0\n\n\nSupport for guideline-based data collection and treatment\n\n94.1\n\n\nSupport for multiple platform access\n\n88.2\n\n\nSupport for context sensitive alerts, warnings, and guidance\n\n70.6\n\n\nClinical notes capture in narrative form\n\n23.5\n\n\nListing functionality: With the EHR system you currently have, how easy is it for you (or staff in your practice) to generate the following information about your patients? (N=17)\n\n\nList of all medications taken by an individual patient\n\n100.0\n\n\nProvide patients with clinical summaries for each visit\n\n88.2\n\n\nList of all laboratory results for an individual patient\n\n88.2\n\n\nList of patients by diagnosis (eg, diabetes or cancer)\n\n82.4\n\n\nList of patients who are due or overdue for tests or preventive care\n\n76.5\n\n\nList of all patients taking a particular medication\n\n76.5\n\n\nList of patients by laboratory result (eg, HbA1C>9.0)\n\n52.3\n\n\nReminder functionality: Are the tasks routinely performed for patients at your site using EHR? (N=17)\n\n\nClinicians receive a reminder for guideline-based interventions and\/or screening tests\n\n58.8\n\n\nClinicians receive an alert or prompt to provide patients with test results\n\n41.2\n\n\nPatients are sent reminder notices when it is time for regular preventive or follow-up care\n\n35.3\n\n\nAll laboratory tests ordered are tracked until results reach clinicians\n\n29.4\n\n\nDoes your site and the clinicians that practice in your site use the EHR system to facilitate any of the following workflow activities (N=16)\n\n\nElectronic prescribing of medication\n\n93.3\n\n\nElectronic prompts about a potential problem with drug dose or drug interaction\n\n87.5\n\n\nElectronic receipt of laboratory results integrated into the EHR system (not scanned)\n\n62.5\n\n\nElectronic ordering of laboratory tests\n\n43.8\n\n\nElectronic referring to specialists\n\n37.5\n\n\nElectronic transferring of prescriptions to a pharmacy\n\n6.7\n\n\nHealth information exchange functionalities: Can you electronically exchange the following with any doctors outside your practice? (N=16)\n\n\nElectronic exchange outside practice: patient clinical summaries\n\n25.0\n\n\nElectronic exchange outside practice: laboratory and diagnostic tests\n\n18.8\n\n\nPatient engagement functionality: Please indicate whether the EHR system in use at your site allows patients to\u2026 (N=17)\n\n\nAccess alcohol consumption advice online\n\n11.8\n\n\nAccess advice for informal caregivers online\n\n11.8\n\n\nEmail about a medical question or concern\n\n11.8\n\n\nAccess dietary advice online\n\n11.8\n\n\nAccess advice on physical activity online\n\n11.8\n\n\nAccess advice on self-management of chronic conditions online\n\n11.8\n\n\nAccess smoking cessation advice online\n\n11.8\n\n\nRequest appointments online\n\n5.9\n\n\nView a list of medications (current and past) online\n\n5.9\n\n\nView other components of their chart (current and past) online\n\n0.0\n\n\nView medical imaging results (current and past) online\n\n0.0\n\n\nRequest refills for prescriptions online\n\n0.0\n\n\nView test results (current and past) online\n\n0.0\n\n\nInteroperability: Were any of the following INTEROPERABILITY technologies implemented in the EHR system currently in use at this site? (N=17)\n\n\nDiagnoses are coded using international standards\n\n94.1\n\n\nMedications and pharmacological profiles are coded to national standards\n\n82.4\n\n\nPatient records are supported by standards-based data migration technology\n\n50.0\n\n\nePrescriptions comply with national standards\n\n52.9\n\n\nPatient Identifier is based on national or jurisdictional standard\n\n58.8\n\n\nPatient Identifier is supported by aliasing technology to achieve positive ID across systems\n\n37.5\n\n\nFindings are coded using international standards\n\n58.8\n\n\nCommunications with other clinics and institutions use international standards\n\n31.3\n\n\nInvestigations, referrals, and imaging requests make use of order tracking technology\n\n35.3\n\n\n\nDiscussion \nPrincipal findings \nThis study aimed to complement current insights into the cost recovery concerns related to EHR investments by considering the research question \u201cWhen implemented in primary care practices, what will be the ROI from EHR?\u201d. The study finds that primary care clinics can realize a positive ROI from the implementation of EHR. Our analysis offers evidence that the increases in net revenue, in the active-patients-to-clinician-FTE ratio, and in the number of active patients are positively associated with the EHR implementation, likely contributing substantially to an average break-even point of 10 months.\nIn addition, the analysis conducted to understand the variances in financial and operational performance among the sampled clinics provides insights into key differences between those clinics that were highly successful and those less successful in realizing a positive ROI from EHR. Some clinics seem to be more innovative than others in using EHR in their practices to achieve significantly better operational and financial results. The analysis suggests that a clinic\u2019s ability to take advantage of EHR to support process changes has a significant effect on the time required to achieve cost recovery from an investment in EHR. In particular, the clinics that were successful in realizing faster time to break even were better at using EHR in workflow areas involving patient information \u2014 such as maintaining patient problem lists, managing test results, and complying with national coding standards, all of which make patient encounter preparation easier and more rapid. We also find that the clinics achieved positive financial performance, even though not all EHR functionalities were used. The alignment of EHR functionalities with clinic workflow plays an important role in achieving positive operational and financial results with EHR. Identified as particularly important EHR-product improvements that would ease adoption of workflow changes are automations that assist clinicians, clinical support staff, and administrative staff both in the overall management of the practice and within the patient encounter, as well as consistent and comprehensive compliance with national standards such as national drug coding standards.\n\nImplications for practitioners and managers in primary care \nThe knowledge gained from this ROI study on EHR is important to practicing primary care physicians who are concerned about how they will fare financially upon investing in EHR, as they face ever increasing pressure to transition from their paper-based records to electronic systems. This study provides evidence to practitioners in primary care that investment in EHR can be a sound decision with a reasonable cost recovery time frame, while providing immediate opportunities for increased operational efficiency and the potential for further improvements in clinic performance and benefits realization from EHR. Practitioners in primary care who are considering the investment in EHR should note the important relationship between EHR functionality, clinic workflow change, and a positive ROI from EHR implementation. Positive ROI does not happen automatically upon implementing an EHR package, and a clinic\u2019s ability to leverage EHR for process changes plays a role in achieving a positive ROI.\n\nImplications for policy makers \nThis study\u2019s finding on increased active patient count and clinic operational efficiency after the EHR implementation, in particular with respect to improvement in the active-patients-to-clinician-FTE ratio, offers the possibility that EHR can play a role in addressing the shortage in family physicians. As primary care clinics implement EHR systems and discover better ways to take advantage of EHR in their practices, a key question will be how to incorporate such learnings and deliver enhanced EHR products back into the clinics to realize the full potential of EHR. Policies that enable the establishment of a closed-loop feedback mechanism between EHR vendors and health care providers could facilitate targeted enhancements to EHR systems. In addition, policies that provide support to help primary care practices successfully make EHR-enabled changes, such as support of workflow optimization with an EHR system that would ease adoption, could not only facilitate the realization of positive ROI but also help address the shortage in family physicians.\n\nFuture research \nSome of the factors identified in this research as key factors impacting the realization of a positive ROI from EHR implementation, such as improved access to up-to-date patient charts and improved ability to obtain test results from laboratories and follow the results of an investigation over time, have implications to quality of care and patient safety. Thus, future research will be to investigate the relationship between financial ROI and realization of clinical benefits of EHR such as quality, safety, and patient outcomes, as depicted in Figure 5. Other research should include a study to identify best practices for implementing and using EHR, with concrete examples of success factors and failure factors as well as ways to tailor these best practices relevant to particular clinic situations. In addition, panel analysis, which deals with two-dimensional panel data (cross sectional and times series)[35], can be conducted with the cohort of primary care clinics to understand the effect of learning curve on a clinic\u2019s ability to realize positive ROI and non-financial, clinical benefits from EHR implementation. Knowledge gained from such studies could facilitate EHR adoption and subsequent benefits realization in primary care practices.\n\n\n\nFigure 5. Future research: investigate the relationship between return on EHR investment and\r\nclinical benefits realization from EHR implementation\nLimitations \nThe principal limitation of this study is that the number of primary care clinics examined was limited, due mainly to time constraints of clinics to participate in the study and scarcity of suitably detailed operational and financial data necessary for ROI calculation. For the clinics recruited to the study, the most limiting factor was that of collecting a complete picture of the cost and benefits needed to assess an ROI from EHR implementation. This was due mainly to the absence of standardized financial and business-case approaches to the governance of these independent organizations. The insights gained from the participants in our study, however, provide salient insights into the impact of EHR investment to facilitate the EHR adoption across practicing primary care physicians, with information on time required to achieve cost recovery from an EHR investment and on principal factors impacting cost-recovery performance.\n\nAcknowledgements \nThe authors wish to acknowledge the valuable contributions of Liette Lapointe, PhD, Isabelle Vedele, PhD, John Hughes, MD, Raymond Simkus, MD, and Susan Law, PhD, in all phases of this study.\nThis study was funded by Canada Health Infoway Inc, an independent, not-for-profit corporation established in Canada to accelerate the adoption of electronic health records and related technologies on a pan-Canadian basis. The opinions, results, and conclusions reported in this manuscript are those of the authors. No endorsement by Canada Health Infoway is intended or should be inferred.\n\nConflicts of interest \nNone declared.\n\nReferences \n\n\n\u2191 Hillestad, R.; Bigelow, J.; Bower, A.; Girosi, F.; Meili, R.; Scoville, R.; Taylor, R. (2005). \"Can electronic medical record systems transform health care? Potential health benefits, savings, and costs\". Heath Affairs 24 (5): 1103-1117. doi:10.1377\/hlthaff.24.5.1103. PMID 16162551.   \n\n\u2191 Blumenthal, D. (2009). \"Stimulating the adoption of health information technology\". The New England Journal of Medicine 360 (15): 1477-1479. doi:10.1056\/NEJMp0901592. PMID 19321856.   \n\n\u2191 Bates, D.W. (2010). \"Getting in step: electronic health records and their role in care coordination\". Journal of General Internal Medicine 25 (3): 174-176. doi:10.1007\/s11606-010-1252-x. PMC PMC2839327. PMID 20127195. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2839327 .   \n\n\u2191 Saleem, J.J.; Flanagan, M.E.; Wilck, N.R.; Demetriades, J.; Doebbeling, B.N. (2013). \"The next-generation electronic health record: perspectives of key leaders from the US Department of Veterans Affairs\". Journal of the American Medical Informatics Association 20 (e1): e175-e177. doi:10.1136\/amiajnl-2013-001748. PMC PMC3715365. PMID 23599227. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3715365 .   \n\n\u2191 5.0 5.1 Black, A.D.; Car, J.; Pagliari, C.; Anandan, C.; Cresswell, K. Bokun, T.; NcKinstry, B.; Procter, R.; Majeed, A.; Sheikh, A. (2013). \"The impact of eHealth on the Quality and Safety of Health Care: a Systematic Overview\". PLoS Medicine 8 (1). doi:10.1371\/journal.pmed.1000387. PMC PMC3022523. PMID 21267058. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3022523 .   \n\n\u2191 Holroyd-Leduc, J.M.; Lorenzetti, D.; Straus, S.E.; Sykes, L.; Quan, H. (2011). \"The impact of the electronic medical record on structure, process, and outcomes within primary care: a systematic review of the evidence\". Journal of the American Medical Informatics Association 18 (6). doi:10.1136\/amiajnl-2010-000019. PMC PMC3197985. PMID 21659445. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985 .   \n\n\u2191 7.0 7.1 Gans, D.; Kralewski, J.; Hammons, T.; Dowd, B. (2005). \"Medical groups' adoption of electronic health records and information systems\". Health Affairs 24 (5): 1323-1333. doi:10.1377\/hlthaff.24.5.1323. PMC PMC3197985. PMID 21659445. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985 .   \n\n\u2191 8.0 8.1 DesRoches, C.M.; Campbell, E.G.; Rao, S.R.; Donelan, K.; Ferris, T.G.; Jha, A.; et al. (2008). \"Electronic health records in ambulatory care \u2014 a national survey of physicians\". The New England Journal of Medicine 359 (1): 50-60. doi:10.1056\/NEJMsa0802005. PMID 18565855.   \n\n\u2191 El-Kareh, R.; Gandhi, T.K.; Poon, E.G.; Newmark, L.P.; Ungar, J.; Lipsitz, S.; et al. (2009). \"Trends in primary care clinician perceptions of a new electronic health record\". Journal of General Internal Medicine 24 (4): 464-468. doi:10.1007\/s11606-009-0906-z. PMC PMC2659149. PMID 19156468. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2659149 .   \n\n\u2191 Bassi, J.; Lau, F.; Lesperance, M. (2012). \"Perceived impact of electronic medical records in physician office practices: a review of survey-based research\". Interactive Journal of Medical Research 1 (2): e3. doi:10.2196\/ijmr.2113. PMC PMC3626136. PMID 23611832. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626136 .   \n\n\u2191 Keshavjee, K.; Bosomworth, J.; Copen, J.; Lai, J.; Kucukyazici, B.; Lilani, R.; et al. (2006). \"Best practices in EMR implementation: a systematic review\" (PDF). AMIA Annual Symposium Proceedings 2006: 982. PMC PMC1839412. PMID 17238601. http:\/\/www.infoclin.ca\/assets\/7e474_best%20practices%20in%20emr%20implementation%20-%20july,%202006.pdf .   \n\n\u2191 Ilie, V.; Van Slyke, C.; Parikh, M.A.; Courtney, J.F. (2009). \"Paper Versus Electronic Medical Records: The Effects of Access on Physicians' Decisions to Use Complex Information Technologies\". Decision Sciences 40 (2): 213-241. doi:10.1111\/j.1540-5915.2009.00227.x.   \n\n\u2191 Valdes, I.; Kibbe, D.C.; Tolleson, G.; Kunik, M.E.; Petersen, L.A. (2004). \"Barriers to proliferation of electronic medical records\". Informatics in Primary Care 12 (1): 3\u20139. PMID 15140347. http:\/\/www.ingentaconnect.com\/content\/bcs\/ipc\/2004\/00000012\/00000001\/art00002 .   \n\n\u2191 Archer, N.; Cocosila, M. (2011). \"A comparison of physician pre-adoption and adoption views on electronic health records in Canadian medical practices\". Journal of Medical Internet Research 13 (3): e57. doi:10.2196\/jmir.1726. PMC PMC3222163. PMID 21840835. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3222163 .   \n\n\u2191 Terry, A.L.; Thorpe, C.F.; Giles, G.; Brown, J.B.; Harris, S.B.; Reid, G.J.; Thind, A.; Stewart, M. (2008). \"Implementing electronic health records: Key factors in primary care\". Canadian Family Physician 54 (5): 730-736. PMC PMC2377228. PMID 18474707. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2377228 .   \n\n\u2191 Ludwick, D.A.; Doucette, J. (2009). \"Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries\". International Journal of Medical Informatics 78 (1): 22-31. doi:10.1016\/j.ijmedinf.2008.06.005. PMID 18644745.   \n\n\u2191 Boonstra, A.; Broekhuis, M. (2010). \"Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions\". BMC Health Services Research 10: 231. doi:10.1186\/1472-6963-10-231. PMC PMC2924334. PMID 20691097. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334 .   \n\n\u2191 Simon, S.R.; Kaushal, R.; Cleary, P.D.; Jenter, C.A.; Volk, L.A.; Poon, E.G.; et al. (2007). \"Correlates of electronic health record adoption in office practices: a statewide survey\". Journal of the American Medical Informatics Association 14: 110-117. PMC PMC2215070. PMID 17068351. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2215070 .   \n\n\u2191 Kemper, A.R.; Uren, R.L.; Clark, S.J. (2006). \"Adoption of electronic health records in primary care pediatric practices\". Pediatrics 118 (1): e20-4. PMID 16818534.   \n\n\u2191 Rozenblum, R.; Jang, Y.; Zimlichman, E.; Zalzberg, C.; Tamblyn, M.; Buckeridge, D.; et al. (2011). \"A qualitative study of Canada\u2019s experience with the implementation of electronic health information technology\". CMAJ 183 (5): e281-8. doi:10.1503\/cmaj.100856. PMC PMC3060213. PMID 21343262. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3060213 .   \n\n\u2191 Pifer, E.A.; Smith, S.; Keever, G.W. (2001). \"EMR to the rescue. An ambulatory care pilot project shows that data sharing equals cost shaving\". Healthcare Informatics 18 (2): 111\u2013114. PMID 11225061.   \n\n\u2191 Wang, S.J.; Middleton, B.; Prosser, L.A.; Bardon, C.G.; Spurr, C.D.; Carchidi, P.J.; et al. (2003). \"A cost-benefit analysis of electronic medical records in primary care\". American Journal of Medicine 114 (5): 397-403. PMID 12714130.   \n\n\u2191 Miller, R.H.; West, C.; Brown, T.M.; Sim, I.; Ganchoff, C. (2005). \"The value of electronic health records in solo or small group practices\". Health Affairs 24 (5): 1127-1137. doi:10.1377\/hlthaff.24.5.1127. PMID 16162555.   \n\n\u2191 Grieger, D.L.; Cohen, S.H.; Krusch, D.A. (2007). \"A pilot study to document the return on investment for implementing an ambulatory electronic health record at an academic medical center\". Journal of the American College of Surgeons 205 (1): 89-96. doi:10.1016\/j.jamcollsurg.2007.02.074. PMID 17617337.   \n\n\u2191 Adler-Milstein, J.; Green, C.E.; Bates, D.W. (2013). \"A survey analysis suggests that electronic health records will yield revenue gains for some practices and losses for many\". Health Affairs 32 (3): 562-570. doi:10.1377\/hlthaff.2012.0306. PMID 23459736.   \n\n\u2191 Sahin, D.; Yaffe, M.J.; Sussman, T.; McCusker, J. (2014). \"A mixed studies literature review of family physicians' participation in research\". Family Medicine 46 (7): 503-514. PMID 25058542.   \n\n\u2191 Creswell, J.; Plano Clark, V. (2010). Designing and conducting mixed methods research. SAGE Publications, Inc. ISBN 9781412975179.   \n\n\u2191 L\u00e6rum, H.; Faxvaag, A. (2004). \"Task-oriented evaluation of electronic medical records systems: development and validation of a questionnaire for physicians\". BMC Medical Informatics and Decision Making 4 (1). doi:10.1186\/1472-6947-4-1.   \n\n\u2191 Joos, D.; Chen, Q.; Jirjis, J.; Johnson, K.B. (2006). \"An electronic medical record in primary care: impact on satisfaction, work efficiency and clinic processes\". AMIA Annual Symposium Proceedings 2006: 394-8. PMC PMC1839545. PMID 17238370. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839545 .   \n\n\u2191 Aaronson, J.W.; Murphy-Cullen, C.L.; Chop, W.M.; Frey, R.D. (2001). \"Electronic medical records: the family practice resident perspective\". Family Medicine 33 (2): 128-132. PMID 11271741.   \n\n\u2191 Audet, A.M.; Doty, M.M.; Shamasdin, J.; Schoenbaum, S.C. (2005). \"Measure, learn, and improve: physicians' involvement in quality improvement\". Health Affairs 24 (3): 843-853. doi:10.1377\/hlthaff.24.3.843. PMID 15886180.   \n\n\u2191 Otieno, O.G.; Toyama, H.; Asonuma, M.; Kanai-Pak, M.; Naitoh, K. (2007). \"Nurses' views on the use, quality and user satisfaction with electronic medical records: questionnaire development\". Journal of Advanced Nursing 60 (2): 209-219. doi:10.1111\/j.1365-2648.2007.04384.x. PMID 17877568.   \n\n\u2191 Sittig, D.F.; Kuperman, G.J.; Fiskio, J. (1999). \"Evaluating physician satisfaction regarding user interactions with an electronic medical record system\". AMIA Annual Symposium Proceedings 1999: 400-404. PMC PMC2232602. PMID 10566389. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2232602 .   \n\n\u2191 Glaser, B.G.; Strauss, A.L. (1999). The discovery of grounded theory: Strategies for qualitative research. Aldine Transaction. ISBN 9780202302607.   \n\n\u2191 Hsiao, C. (2003). Analysis of Panel Data (2nd ed.). Cambridge University Press. ISBN 9780521522717.   \n\n\nAbbreviations \nEHR: electronic health records\nFTE: full time equivalent\nIT: information technology\nROI: return on investment\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful.\nPer the distribution agreement, the following copyright information is also being added: \n\u00a9Yeona Jang, Michel A Lortie, Steven Sanche. Originally published in JMIR Medical Informatics (http:\/\/medinform.jmir.org), 29.09.2014.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\">https:\/\/www.limswiki.org\/index.php\/Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:59.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,963 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","5d49b7c0866e1210c1696240ca7f59fd_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Return_on_investment_in_electronic_health_records_in_primary_care_practices_A_mixed-methods_study skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Return on investment in electronic health records in primary care practices: A mixed-methods study<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> The use of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> (EHR) in clinical settings is considered pivotal to a patient-centered health care delivery system. However, uncertainty in cost recovery from EHR investments remains a significant concern in primary care practices.\n<\/p><p><b>Objective:<\/b> Guided by the question of \u201cWhen implemented in primary care practices, what will be the return on investment (ROI) from an EHR implementation?\u201d, the objectives of this study are two-fold: (1) to assess ROI from EHR in primary care practices and (2) to identify principal factors affecting the realization of positive ROI from EHR. We used a break-even point, that is, the time required to achieve cost recovery from an EHR investment, as an ROI indicator of an EHR investment.\n<\/p><p><b>Methods:<\/b> Given the complexity exhibited by most EHR implementation projects, this study adopted a retrospective mixed-method research approach, particularly a multiphase study design approach. For this study, data were collected from community-based primary care clinics using EHR systems.\n<\/p><p><b>Results:<\/b> We collected data from 17 primary care clinics using EHR systems. Our data show that the sampled primary care clinics recovered their EHR investments within an average period of 10 months (95% CI 6.2-17.4 months), seeing more patients with an average increase of 27% in the active-patients-to-clinician-FTE (full time equivalent) ratio and an average increase of 10% in the active-patients-to-clinical-support-staff-FTE ratio after an EHR implementation. Our analysis suggests, with a 95% confidence level, that the increase in the number of active patients (<i>P<\/i>=.006), the increase in the active-patients-to-clinician-FTE ratio (<i>P<\/i><.001), and the increase in the clinic net revenue (<i>P<\/i><.001) are positively associated with the EHR implementation, likely contributing substantially to an average break-even point of 10 months.\n<\/p><p><b>Conclusions:<\/b> We found that primary care clinics can realize a positive ROI with EHR. Our analysis of the variances in the time required to achieve cost recovery from EHR investments suggests that a positive ROI does not appear automatically upon implementing an EHR and that a clinic\u2019s ability to leverage EHR for process changes seems to play a role. Policies that provide support to help primary care practices successfully make EHR-enabled changes, such as support of clinic workflow optimization with an EHR system, could facilitate the realization of positive ROI from EHR in primary care practices.\n<\/p><p><b>Keywords:<\/b> return on investment in electronic health records; cost recovery from EHR implementation; ROI indicator; physician satisfaction with EHR; primary care practices \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Context\">Context<\/span><\/h3>\n<p>The use of electronic health records (EHR) in clinical settings is widely recommended as an innovation enabler with potential benefits of reducing health care costs, while improving quality and safety, and is considered central to achieving patient-centered health care.<sup id=\"rdp-ebb-cite_ref-HillestadCan05_1-0\" class=\"reference\"><a href=\"#cite_note-HillestadCan05-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BlumenthalStim09_2-0\" class=\"reference\"><a href=\"#cite_note-BlumenthalStim09-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BatesGet10_3-0\" class=\"reference\"><a href=\"#cite_note-BatesGet10-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SaleemThe13_4-0\" class=\"reference\"><a href=\"#cite_note-SaleemThe13-4\" rel=\"external_link\">[4]<\/a><\/sup> As a wide array of EHR projects have been implemented within various health care settings, the health care field is rich with volumes of work examining the benefits of EHR. However, the existing literature reports mixed results in benefits realized from EHR implementation.<sup id=\"rdp-ebb-cite_ref-BlackThe11_5-0\" class=\"reference\"><a href=\"#cite_note-BlackThe11-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HolroydThe11_6-0\" class=\"reference\"><a href=\"#cite_note-HolroydThe11-6\" rel=\"external_link\">[6]<\/a><\/sup> Such mixed results suggest that the implementation of EHR systems does not automatically guarantee the conversion of potential benefits into realized benefits.\n<\/p><p>The implementation of EHR systems within primary care practices is seen as particularly complex<sup id=\"rdp-ebb-cite_ref-GansMed05_7-0\" class=\"reference\"><a href=\"#cite_note-GansMed05-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DesRochesElec08_8-0\" class=\"reference\"><a href=\"#cite_note-DesRochesElec08-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-El-KarehTrends09_9-0\" class=\"reference\"><a href=\"#cite_note-El-KarehTrends09-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BassiPerc12_10-0\" class=\"reference\"><a href=\"#cite_note-BassiPerc12-10\" rel=\"external_link\">[10]<\/a><\/sup>, with physicians and other staff in primary care practices citing obstacles such as difficulty in adapting to the significant changes in workflow and the time commitment required to learn to use the new software while prioritizing patient care.<sup id=\"rdp-ebb-cite_ref-KeshavjeeBest06_11-0\" class=\"reference\"><a href=\"#cite_note-KeshavjeeBest06-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-IliePaper09_12-0\" class=\"reference\"><a href=\"#cite_note-IliePaper09-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ValdesInfo04_13-0\" class=\"reference\"><a href=\"#cite_note-ValdesInfo04-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArcherAComp11_14-0\" class=\"reference\"><a href=\"#cite_note-ArcherAComp11-14\" rel=\"external_link\">[14]<\/a><\/sup> While there is a growing body of evidence that EHR can be a valuable tool for improving quality of care and patient safety with relatively positive perceptions about EHR benefits<sup id=\"rdp-ebb-cite_ref-TerryImp08_15-0\" class=\"reference\"><a href=\"#cite_note-TerryImp08-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LudwickAdopt09_16-0\" class=\"reference\"><a href=\"#cite_note-LudwickAdopt09-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BoonstraBar10_17-0\" class=\"reference\"><a href=\"#cite_note-BoonstraBar10-17\" rel=\"external_link\">[17]<\/a><\/sup>, uncertainty about cost recovery of an EHR investment remains a significant concern in primary care practices.<sup id=\"rdp-ebb-cite_ref-GansMed05_7-1\" class=\"reference\"><a href=\"#cite_note-GansMed05-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DesRochesElec08_8-1\" class=\"reference\"><a href=\"#cite_note-DesRochesElec08-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SimonCorr07_18-0\" class=\"reference\"><a href=\"#cite_note-SimonCorr07-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KemperAdop06_19-0\" class=\"reference\"><a href=\"#cite_note-KemperAdop06-19\" rel=\"external_link\">[19]<\/a><\/sup> Various studies on EHR impact and adoption also raise the need for cost-benefit analysis of EHR investments.<sup id=\"rdp-ebb-cite_ref-BlackThe11_5-1\" class=\"reference\"><a href=\"#cite_note-BlackThe11-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RozenblumAQual11_20-0\" class=\"reference\"><a href=\"#cite_note-RozenblumAQual11-20\" rel=\"external_link\">[20]<\/a><\/sup> Thus, this study seeks to assess the return on investment (ROI) from an EHR implementation in primary care settings, aiming to complement the current insights on cost recovery concerns in existing literature.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Measurement\">Measurement<\/span><\/h3>\n<p>Return on investment is a common approach to measuring rates of return on money invested, in terms of increased profit attributable to the investment. A standard ROI is defined as follows:\n<\/p><p>ROI = (Gain from investment - Cost of investment)\/Cost of investment\n<\/p><p>Results reported by various studies regarding ROI from EHR systems in primary care settings are mixed.<sup id=\"rdp-ebb-cite_ref-PiferEMR01_21-0\" class=\"reference\"><a href=\"#cite_note-PiferEMR01-21\" rel=\"external_link\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WangACost03_22-0\" class=\"reference\"><a href=\"#cite_note-WangACost03-22\" rel=\"external_link\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MillerThe05_23-0\" class=\"reference\"><a href=\"#cite_note-MillerThe05-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GriegerAPilot07_24-0\" class=\"reference\"><a href=\"#cite_note-GriegerAPilot07-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AdlerASurv13_25-0\" class=\"reference\"><a href=\"#cite_note-AdlerASurv13-25\" rel=\"external_link\">[25]<\/a><\/sup> Most of the existing literature used a bottom-up approach identifying specific cost-saving areas and collecting the data on financial savings made in these areas attributable to EHR systems. However, EHR is a process-enabling information technology (IT) that offers the opportunity to streamline information-intensive workflow, remove manual hand-off of data and information, and facilitate coordination \u2014 thus facilitating the execution of entire business processes rather than individual tasks. Due mainly to the context-sensitive nature of benefits realization from a process-enabling IT such as EHR and the scarcity of detailed financial data relating to gains and\/or savings directly attributable to an EHR system in primary care clinics, this study used break-even-point analysis as an indicator of ROI, instead of standard ROI analysis.\n<\/p><p>The break-even point of an EHR investment is defined as the number of months it takes a clinic to recover the cost of the EHR system and other associated implementation costs, with increased revenues and\/or decreased expenses. Increases in revenues and\/or decreases in expenses are assessed by considering net revenues during three distinct periods of time: pre-EHR, peri-EHR, and post-EHR. The pre-EHR period is defined as the full fiscal year before the implementation of an EHR system started. The peri-EHR period is defined as the fiscal year(s) containing the EHR implementation period (ie, during EHR implementation). If the peri-EHR period covers more than one fiscal year, the net revenue is averaged over these fiscal years. The post-EHR period is defined as the full fiscal year following the end of the peri-EHR period.\n<\/p><p>To calculate the break-even point of implementing an EHR system in a clinic, the cost of EHR implementation is set equal to the difference in the clinic\u2019s net revenue between the pre-EHR and peri-EHR periods, plus the difference in the clinic\u2019s net revenue between the pre-EHR and post-EHR periods, as summarized in the following formula:\n<\/p><p>C <i><sub>EHR<\/sub><\/i>= [(NR <i><sub>peri<\/sub><\/i>\u2212NR <i><sub>pre<\/sub><\/i>)\/12]*M <i><sub>imp<\/sub><\/i> + [(NR <i><sub>post<\/sub><\/i>\u2212NR <i><sub>pre<\/sub><\/i>)\/12] * (M <i><sub>break-even<\/sub><\/i>\u2212M <i><sub>imp<\/sub><\/i>)\n<\/p><p>In this formula: <i>C<sub>EHR<\/sub><\/i>=cost of EHR implementation, <i>NR<sub>peri<\/sub><\/i>=annual clinic net revenue in the peri-EHR period, <i>NR<sub>pre<\/sub><\/i>=annual clinic net revenue in the pre-EHR period, <i>NR<sub>post<\/sub><\/i>=annual clinic net revenue in the post-EHR period, <i>M<sub>imp<\/sub><\/i>=the number of months taken to complete the EHR implementation in the clinic, and <i>M<sub>break-even<\/sub><\/i>=months to break even. The net revenue of a clinic is defined as the sum of the physicians\u2019 billings for work done in the clinic minus any expenses that the clinic pays to maintain its ongoing practice. In the case where the months to break even were less than the months of EHR implementation, in other words, the net revenue difference between pre-EHR and peri-EHR periods is large enough to recover the cost of EHR implementation, the formula was adjusted by setting the cost of EHR implementation equal to the difference in a clinic\u2019s net revenue between the pre-EHR and peri-EHR periods, or:\n<\/p><p><i>C<sub>EHR<\/sub><\/i> = [(<i>NR<sub>peri<\/sub><\/i> \u2013 <i>NR<sub>pre<\/sub><\/i>)\/12]* <i>M<sub>break-even<\/sub><\/i>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Objectives\">Objectives<\/span><\/h3>\n<p>Guided by the research question \u201cWhen implemented in primary care practices, what will be the ROI from an EHR implementation?\u201d, the objectives of this research are twofold: (1) to assess the ROI from an EHR implementation in primary care practices by measuring the time required to recover the cost of converting a clinic from a paper-based environment to an EHR-enabled environment and (2) to identify principal factors affecting the realization of a positive ROI from an EHR implementation in primary care practices. Such ROI information related to cost recovery of an EHR investment would be helpful to both clinics considering implementing EHR systems and to policy makers designing EHR-adoption funding programs and policies.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Sample\">Sample<\/span><\/h3>\n<p>Community-based, primary care clinics meeting the following four eligibility criteria were recruited for this study on ROI from EHR in primary-care settings. First, this study focused on community-based, primary care clinics. Thus, specialty care clinics and walk-in clinics were excluded. Second, clinics were required to have implemented EHR systems. Third, clinics were required to have been paper-based in the past, in order to ensure that the comparison between pre-EHR and post-EHR implementation performance was possible for the ROI calculation. Fourth, clinics were required to provide operational and financial data necessary to calculate ROI, as well as the information on challenges and opportunities that they had experienced both during and after the EHR implementation.\n<\/p><p>The research team contacted 132 randomly selected community-based, primary care clinics in Canada that met the first two eligibility criteria for recruitment to the study. Of the 132 clinics, 62 clinics declined to participate, mostly citing time constraints. Of the 70 clinics remaining, 34 clinics were not eligible, mainly because they were unable to provide the operational and financial data necessary to calculate ROI. Of the 36 eligible clinics, 19 clinics later declined to participate, due mainly to time constraints. Thus, data were collected from a total of the 17 eligible clinics, resulting in the study participation rate of 13%, which is relatively consistent with typical participation rates of family physicians reported in other studies involving interviews and observations.<sup id=\"rdp-ebb-cite_ref-SahinAMix14_26-0\" class=\"reference\"><a href=\"#cite_note-SahinAMix14-26\" rel=\"external_link\">[26]<\/a><\/sup> No statistically significant differences were observed between participating and non-participating primary care clinics in terms of geographic location (<i>P<\/i>=.315), the number of physicians or other clinicians (<i>P<\/i><.001), or the number of patients per physician (<i>P<\/i>=.192). Table 1 summarizes the basic statistics on the size of the sampled clinics. We used Full Time Equivalent (FTE) in comparing the size of primary care clinics.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 1. Basic statistics on the size of a primary care clinic in the study<\/b>\n<\/td><\/tr>\n<tr>\n<th>\n<\/th>\n<th> Average\n<\/th>\n<th> SD\n<\/th>\n<th> Median\n<\/th>\n<th> Minimum\n<\/th>\n<th> Maximum\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold\">Clinic size: clinician FTEs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Pre-EHR period\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Post-EHR period\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold\">Clinic size: clinical support staff FTEs\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Pre-EHR period\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Post-EHR period\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12.0\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Methodology\">Methodology<\/span><\/h3>\n<p>Given the complexity exhibited by most EHR implementation projects, this study used a mixed-method research approach, particularly a multiphase study design.<sup id=\"rdp-ebb-cite_ref-CresswellDes10_27-0\" class=\"reference\"><a href=\"#cite_note-CresswellDes10-27\" rel=\"external_link\">[27]<\/a><\/sup> By combining quantitative and qualitative data, mixed-method research can provide a fuller understanding of the complex and multidimensional world of primary care clinics than would otherwise be achieved using either approach alone.\n<\/p><p>In the quantitative study phase, questionnaire modules were designed, based on prior research in the existing literature<sup id=\"rdp-ebb-cite_ref-LaerumTask04_28-0\" class=\"reference\"><a href=\"#cite_note-LaerumTask04-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JoosAnElec06_29-0\" class=\"reference\"><a href=\"#cite_note-JoosAnElec06-29\" rel=\"external_link\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AaronsonElec01_30-0\" class=\"reference\"><a href=\"#cite_note-AaronsonElec01-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AudetMeas05_31-0\" class=\"reference\"><a href=\"#cite_note-AudetMeas05-31\" rel=\"external_link\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OtienoNur07_32-0\" class=\"reference\"><a href=\"#cite_note-OtienoNur07-32\" rel=\"external_link\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SittigEval99_33-0\" class=\"reference\"><a href=\"#cite_note-SittigEval99-33\" rel=\"external_link\">[33]<\/a><\/sup>, to collect data on EHR implementation costs, EHR functionalities in use, physician satisfaction with EHR, and physicians\u2019 perceptions about the impact of EHR on operational efficiency and on quality of care. Each clinic respondent was required to complete the study instruments using the online questionnaires or researcher-assisted telephone questionnaires. The minimum financial data required for the study include clinic revenue and clinic net revenue as well as EHR implementation cost that consisted of EHR software costs, hardware costs, support costs, and labor costs associated with EHR system implementation and training, in the three different periods\u2014before EHR implementation, during EHR implementation, and after EHR implementation. The minimum operational data required for this ROI study include the number of active patients, clinician FTEs, and clinical support staff FTEs, in the same three periods. The lead researcher served as dedicated support liaison for clinics, in order to ensure that the costs of the EHR implementation, as well as other financial and operational data before and after EHR implementation, were abstracted from clinic records in a consistent fashion. In the subsequent qualitative study, semistructured interviews and observations were conducted with clinic staff and physicians identified as responsible for such functions as patient appointment management, patient record management, test results management, patient encounters, and billing, to assess factors affecting the realization of a positive ROI from an EHR implementation in primary care practices.\n<\/p><p>The data collected from 17 sampled primary care clinics were documented and analyzed using statistical analysis and grounded theory.<sup id=\"rdp-ebb-cite_ref-GlaserThe99_34-0\" class=\"reference\"><a href=\"#cite_note-GlaserThe99-34\" rel=\"external_link\">[34]<\/a><\/sup> As break-even points were analyzed, we compared those clinics that were successful in realizing a positive ROI from EHR implementation to those that were less successful, in an attempt to identify principal factors impacting the realization of positive ROI from EHR. In particular, we used linear regression analysis to estimate the relationships of the outcome variable \u201cbreak-even point\u201d with the explanatory variables that include the codes identified from the qualitative data through the coding process.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Analysis_of_break-even_point_as_indicator_of_return_on_investment\">Analysis of break-even point as indicator of return on investment<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Overview\">Overview<\/span><\/h4>\n<p>Our analysis suggests that the sampled primary care clinics typically recovered their investment in EHR within an average of 10 months (95% confidence interval: 6.2 months, 17.4 months), seeing more patients with improved active-patients-to-clinician-FTE and active-patients-to-clinical-support-staff-FTE ratios in the post-EHR implementation period.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Change_in_clinic_net_revenue_after_implementation_of_electronic_health_records\">Change in clinic net revenue after implementation of electronic health records<\/span><\/h4>\n<p>Once an EHR system is implemented, a key factor that impacts the time required to achieve cost recovery from the EHR investments is clinic net revenue. With respect to how clinics fared financially upon adopting EHR systems, all but one of the primary care clinics in our study achieved an increase in clinic net revenue in the post-EHR period, as shown in Table 2.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 2. Percent changes in clinic revenues, net revenues, and clinician FTEs between the pre-EHR and post-EHR periods<\/b>\n<\/td><\/tr>\n<tr>\n<th rowspan=\"2\">Clinic #\n<\/th>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" rowspan=\"1\" colspan=\"3\">Percent change between the pre-EHR and post-EHR periods (in ascending order by percent change in number of clinician FTEs), %\n<\/td><\/tr>\n<tr>\n<th> In number of clinician FTEs\n<\/th>\n<th> In clinic revenue\n<\/th>\n<th> In clinic\u2019s net revenue\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">26\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">55\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">63\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">16\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">20\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-30\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">120\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">116\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">47\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">223\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">227\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">53\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">103\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinic 17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">329\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">603\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">845\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Average\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">89\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>In addition to clinic net revenue, the sampled clinics showed, on average, positive increases in active patient count, clinician count, clinical support staff count, and clinic revenue in the post-EHR implementation period. These increases are summarized in Figure 1.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Jang_JMIRMedInfo2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2fa2bfdadf2e040d5b4bfea44614eb57\"><img alt=\"Fig1 Jang JMIRMedInfo2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/99\/Fig1_Jang_JMIRMedInfo2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Average percent changes in active patient count, clinician FTE count, clinical support staff<br \/>FTE count, clinic revenue, and clinic net revenue between the pre-EHR and post-EHR periods<\/blockquote>\n<p>Percent increase in clinic net revenue between the pre-EHR and post-EHR periods showed a very strong positive correlation with percent increase in clinic revenue in the same periods (<i>r<\/i>=.99). Percent increase in clinic revenue showed a strong positive correlation with percent increase in the number of active patients (<i>r<\/i>=.87). It also showed a strong positive correlation with percent increase in the number of clinician FTEs, as well as with the number of clinical-support-staff FTEs (<i>r<\/i>=.96 and <i>r<\/i>=.97, respectively). These correlation coefficients (<i>r<\/i> values) are summarized in Figure 2.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Jang_JMIRMedInfo2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"d23c106a2f3e286ca8cc47eec8d1fe3a\"><img alt=\"Fig2 Jang JMIRMedInfo2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/6\/6d\/Fig2_Jang_JMIRMedInfo2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> Correlations (r-values): clinic net revenue, clinic revenue, active patient count,<br \/>clinician FTE count, and clinical support staff FTE count<\/blockquote>\n<h4><span class=\"mw-headline\" id=\"Percent_changes_of_counts_after_implementation_.E2.80.94_Not_linearly_proportional_to_one_another\">Percent changes of counts after implementation \u2014 Not linearly proportional to one another<\/span><\/h4>\n<p>Interestingly, the percent increases in active patient count, clinician FTE count, and clinical support staff FTE count are not linearly proportional to one another. An average active-patient-count increase of 56% was handled by an average 22% increase for clinician FTEs and an average 39% increase for clinical-support-staff FTEs. This finding suggests change in operational efficiency after EHR implementation, with respect to the active-patients-to-clinician-FTE ratio and the active-patients-to-clinical-support-staff-FTE ratio. The sampled clinics showed an average increase of 27% in the active-patients-to-clinician-FTE ratio and an average increase of 10% in the active-patients-to-clinical-support-staff-FTE ratio, as illustrated in Figure 3.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Jang_JMIRMedInfo2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"bb93f716e536ab0dd249537e20784fc5\"><img alt=\"Fig3 Jang JMIRMedInfo2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f4\/Fig3_Jang_JMIRMedInfo2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3.<\/b> Average percent changes in a clinic\u2019s operational efficiency and financial<br \/>performance between the pre-EHR and post-EHR periods<\/blockquote>\n<p>Percent increase in the number of active patients showed strong positive correlations with percent increases in active-patients-to-clinician-FTE ratio (<i>r<\/i>=.64) and in active-patients-to-clinical-support-staff-FTE ratio (<i>r<\/i>=.70), as shown in Figure 4.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Jang_JMIRMedInfo2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"038bc9e4c3c6fc880a7d0163ecbc1046\"><img alt=\"Fig4 Jang JMIRMedInfo2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/4\/44\/Fig4_Jang_JMIRMedInfo2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4.<\/b> Correlations (r-values): clinic net revenue, clinic revenue, active patient count,<br \/>clinician FTE count, clinical support staff FTE count, active-patients-to-clinician-FTE ratio,<br \/>and active-patients-to-clinical-support-staff-FTE ratio<\/blockquote>\n<p>These correlations, together with the nonlinear percent changes summarized in Figure 3, suggest that the increased efficiency in the post-EHR period contributed to a clinic\u2019s ability to accommodate the increased number of active patients.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Percent_changes_in_number_of_active_patients_and_revenue_after_implementation_.E2.80.94_Not_linearly_proportional_to_one_another\">Percent changes in number of active patients and revenue after implementation \u2014 Not linearly proportional to one another<\/span><\/h4>\n<p>The percent increase in clinic revenue was also not linearly proportional to the percent increase in the number of active patients \u2014 an average increase of 76% versus an average increase of 56%, respectively. In addition to the increase in the number of active patients, there seem to be other factors that contributed to clinic revenue increase in the post-EHR period (detailed analysis on the impact of EHR on the sampled clinics\u2019 billing patterns and revenue management processes, required to identify the contributing elements of the greater than linear increase in clinic revenue over the increase in patient count, is beyond the scope of the study).\n<\/p><p>The study also finds that percent increase in clinic net revenue was not linearly proportional to percent increase in clinic revenue. The average additional 13% increase in clinic net revenue (89%, which is 13% above the clinic average revenue of 76%) is attributable to the enhanced operational efficiency in the post-EHR period, which suggests the relative cost-savings effect after the EHR implementation.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Sign_test_results\">Sign test results<\/span><\/h4>\n<p>We further tested the financial and operational impact of EHR in the post-EHR period, in order to assess the degree to which these findings could be extended to the population of clinics implementing EHR. The sign test, as opposed to t test, was adopted because the sample size was less than 30 and because the distributions shown in the data were not normal, with a high degree of skewness in most cases. The sign test results shown in Table 3 suggest, with a 95% confidence level, that the median percent change in clinic net revenue between the pre-EHR and post-EHR periods is positive in the population of the primary care clinics implementing EHR (sign test M=7.5 with <i>P<\/i><.001).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 3. Summary of statistical analysis of change in a clinic\u2019s operational efficiency and financial performance between between the pre-EHR and post-EHR periods<\/b>\n<\/td><\/tr>\n<tr>\n<th> Percent changes between the pre-EHR and post-EHR periods\n<\/th>\n<th> Average\n<\/th>\n<th> SD\n<\/th>\n<th> Median\n<\/th>\n<th> M\n<\/th>\n<th> Sign test, <i>P<\/i> value\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Percent change in clinic net revenue\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">89%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">203%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><.001\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Percent change in the number of active patients\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">56%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">119%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.006\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Percent change in active-patients-to-clinician-FTE ratio\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">27%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">53%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><.001\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Percent change in the number of clinician FTEs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.00\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Percent change in the number of clinical support staff FTEs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">37%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.07\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Percent change in active-patients-to-clinical-support-staff-FTE ratio\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.277\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The same conclusions can be made for the median percent changes in the active-patients-to-clinician-FTE ratio and in the number of active patients in the same periods (M=6.5 and M=5, respectively). However, for the median percent changes with respect to the number of clinician FTEs, the number of clinical support staff FTEs, and the active-patients-to-clinical-support-staff-FTE ratio, we could not reject with a 95% confidence level the null hypothesis of no change after EHR implementation.\n<\/p><p>The correlation coefficients shown in Figure 4 and sign test results summarized in Table 4 suggest that the increase in the active patient count may not be the only factor that contributed to an average break-even point of 10 months upon EHR implementation. Percent increases in the number of active patients, in the active-patients-to-physician-FTE ratio, and in clinic net revenue appear to be positively associated with the EHR implementation, likely contributing substantially to an average break-even point of 10 months.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Analysis_of_variance_in_realizing_financial_performance_.E2.80.94_Key_factors\">Analysis of variance in realizing financial performance \u2014 Key factors<\/span><\/h4>\n<p>Study participants reported improvements in their ability to manage patient information after the implementation of EHR systems, citing improved ability to manage results such as obtaining test results from laboratories and following the results of an investigation over time (64%, 11\/17 clinics). Respondents also reported an improved ability to seek out specific information from patient records (57%, 10\/17 clinics), and access complete, up-to-date patient charts and review patient problems (43%, 6\/15 clinics). See Table 4 for key EHR impacts expressed by study participants.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 4. Impact of EHR on clinic practices identified by study participants<\/b>\n<\/td><\/tr>\n<tr>\n<th> Categories\n<\/th>\n<th> Participant comments\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">A. Impact of EHR on a clinic\u2019s ability to manage results\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cWe receive results electronically and can graph them; graphs help \u2018engage\u2019 the patient.\u201d<br \/> <br \/>\u201cDirect to physician lab results has very positive effect on physician efficiency and patient care.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">B. Impact of EHR on a clinic\u2019s ability to seek out specific information from patient records\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cComplete chart is always available, anywhere which affects patient safety and means better care.\u201d<br \/> <br \/>\u201cIntegration of information for referral requests is a great benefit.\u201d<br \/> <br \/>\u201cBilling codes are up-to-date. (And) billing is automatic by the doctor inside encounter note, which simplifies billing and is easier to manage reconciliation. No missed billing opportunities.\u201d\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">C. Impact of EHR on a clinic\u2019s ability to prepare patient encounter\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\u201cReview of patient information prior to encounter is greatly facilitated.\u201d<br \/> <br \/>\u201cEasier to prepare for encounter; maintenance of problem list \/summary is much easier\u201d<br \/> <br \/>\u201cImmediate access to patient information\u2014no lost files.\u201d\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Some primary care clinics did better than others in using EHR and achieving faster break-even from EHR investment, which can be observed in Tables 1 and 3. To gain insights into key differences between those clinics that were highly successful and those less successful in realizing a positive ROI from EHR, we conducted regression analysis on break-even point as the outcome variable. We used the codes identified through the coding process of the qualitative data as a part of the explanatory variables to estimate their relationships with the outcome variable \u201cbreak-even point\u201d. As summarized in Table 5, the regression analysis suggests four statistically significant factors impacting the return on EHR investment, that is, the time required to achieve cost recovery from an investment in EHR.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 5. Significant linear regression results of the outcome \u201cbreak-even point\u201d with explanatory variables (break-even point was log-transformed to approach a Normal distribution)<\/b>\n<\/td><\/tr>\n<tr>\n<th> Explanatory variable\n<\/th>\n<th> Variable values\n<\/th>\n<th> Regression coefficient\n<\/th>\n<th> Standard error of coefficient\n<\/th>\n<th> <i>P<\/i> value\n<\/th>\n<th> <i>r<\/i><sup>2<\/sup>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">(a) Age of EHR: Months between Jan 1, 2013, and EHR implementation start date\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Number of months\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.01\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.049\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.64\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">(b) e-Prescriptions complying with national standards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0 (No) to 1 (Yes)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-1.32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.006\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.50\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">(c) Extent to which EHR complies to national standards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Continuous (from 0 to 10)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-0.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.038\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.54\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">(d) Process change: Use of flow sheets\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0 (No) to 1 (Yes)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-1.29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.022\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">.68\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Note that in Table 5, the regression coefficient of an explanatory variable with a negative value indicates faster recovery of the EHR investment (ie, a shorter time required to achieve cost recovery from an EHR investment), while a positive value implies slower recovery of the EHR investment (ie, a longer time required to achieve cost recovery from an EHR investment).\n<\/p><p><b>Age of electronic health record systems<\/b>\n<\/p><p>The first result to note in item (a) of Table 5 is that older EHR implementations, in particular those implemented in 2004-2005, were slower to recover their investment, even though they still achieved a break-even point. This result suggests that the newer the EHR, the sooner a positive ROI can be achieved. The earlier EHR systems used by these clinics were less user-friendly and required longer training cycles for the users, which may explain why clinics with these earlier systems took longer to recoup their financial investment.\n<\/p><p><b>Compliance with national standards<\/b>\n<\/p><p>The second and third results, shown in items (b) and (c) of Table 5, suggest a positive link between the ROI indicator and the compliance with national standards such as codes representing prescription drugs. There was an improvement in clinics\u2019 compliance with national standards and ability to comply with evidence-based medicine. This improvement was related to the age of the EHR system used by the clinics. Newer EHR implementations may be more likely to comply with national standards, given that the newer EHRs are likely to support national standards better.\n<\/p><p><b>Use of flow sheets and ability to manage patient information<\/b>\n<\/p><p>Finally, clinics using EHR flow sheets scored consistently better times to break even, shown in item (d) of Table 5. Clinics reported the use of flow sheets, or structured data collection forms, as a mechanism for compliance to evidence-based medicine. The use of flow sheets in EHRs provides advanced features such as those related to the automatic maintenance of patient problem lists and pharmacological profiles. These enhanced features contribute directly to the physician\u2019s efficiency by eliminating the time that would otherwise be spent manually maintaining these lists\u2014a task that can be time-consuming, highly repetitive, and labor-intensive to maintain with consistency in a paper-based environment. The availability of up-to-date lists makes patient encounter preparation easier and more rapid, as the necessary information is available at a glance.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Analysis_of_electronic_health_record_functionalities_used_in_primary_care_clinics\">Analysis of electronic health record functionalities used in primary care clinics<\/span><\/h3>\n<p>Our study finds that despite the limited use of EHR functionalities and limited interoperability, the sampled clinics achieved overall positive operational and financial performance. Table 6 summarizes the data we gathered on EHR functionalities, frequency of use, and ease of use.\n<\/p><p>Most frequently and routinely used EHR functionalities were related to medication management. Health information exchange and patient engagement portal functionalities saw no significant use (the investigation of why these functionalities were not used is beyond the scope of this study).\n<\/p><p>Respondents stressed that it typically takes a few months to understand any particular EHR function sufficiently to effectively introduce it in their clinical practices. This finding, coupled with the finding that despite the limited use of EHR functionalities the clinics achieved overall positive improvement in operational and financial performance in the post-EHR period, suggests that a clinic\u2019s ability to embed particular EHR functionalities in their workflow and make use of these functionalities in their day-to-day clinical practices is of more importance in realizing a positive ROI from EHR implementation than implementing an EHR software package with the maximum number of features and functionalities.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 6. EHR functionalities and utilization reported during the study period<\/b>\n<\/td><\/tr>\n<tr>\n<th> EHR functionalities\n<\/th>\n<th> % of clinics answering in the affirmative\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-top:10px; padding-left:30px; padding-right:10px;\" colspan=\"2\"><b>User Interface: Does the EHR system currently in use at this clinic have any of the following user interface technologies? (N=17)<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Alternative presentation formats for clinical information\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Support for guideline-based data collection and treatment\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">94.1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Support for multiple platform access\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Support for context sensitive alerts, warnings, and guidance\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">70.6\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinical notes capture in narrative form\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-top:10px; padding-left:30px; padding-right:10px;\" colspan=\"2\"><b>Listing functionality: With the EHR system you currently have, how easy is it for you (or staff in your practice) to generate the following information about your patients? (N=17)<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">List of all medications taken by an individual patient\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">100.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Provide patients with clinical summaries for each visit\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">List of all laboratory results for an individual patient\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">List of patients by diagnosis (eg, diabetes or cancer)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">List of patients who are due or overdue for tests or preventive care\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">List of all patients taking a particular medication\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">76.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">List of patients by laboratory result (eg, HbA1C>9.0)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">52.3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-top:10px; padding-left:30px; padding-right:10px;\" colspan=\"2\"><b>Reminder functionality: Are the tasks routinely performed for patients at your site using EHR? (N=17)<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinicians receive a reminder for guideline-based interventions and\/or screening tests\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">58.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Clinicians receive an alert or prompt to provide patients with test results\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">41.2\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Patients are sent reminder notices when it is time for regular preventive or follow-up care\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">35.3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">All laboratory tests ordered are tracked until results reach clinicians\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">29.4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-top:10px; padding-left:30px; padding-right:10px;\" colspan=\"2\"><b>Does your site and the clinicians that practice in your site use the EHR system to facilitate any of the following workflow activities (N=16)<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic prescribing of medication\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">93.3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic prompts about a potential problem with drug dose or drug interaction\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">87.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic receipt of laboratory results integrated into the EHR system (not scanned)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">62.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic ordering of laboratory tests\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">43.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic referring to specialists\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">37.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic transferring of prescriptions to a pharmacy\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.7\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-top:10px; padding-left:30px; padding-right:10px;\" colspan=\"2\"><b>Health information exchange functionalities: Can you electronically exchange the following with any doctors outside your practice? (N=16)<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic exchange outside practice: patient clinical summaries\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">25.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Electronic exchange outside practice: laboratory and diagnostic tests\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">18.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-top:10px; padding-left:30px; padding-right:10px;\" colspan=\"2\"><b>Patient engagement functionality: Please indicate whether the EHR system in use at your site allows patients to\u2026 (N=17)<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Access alcohol consumption advice online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Access advice for informal caregivers online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Email about a medical question or concern\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Access dietary advice online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Access advice on physical activity online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Access advice on self-management of chronic conditions online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Access smoking cessation advice online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Request appointments online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.9\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">View a list of medications (current and past) online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5.9\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">View other components of their chart (current and past) online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">View medical imaging results (current and past) online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Request refills for prescriptions online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">View test results (current and past) online\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-top:10px; padding-left:30px; padding-right:10px;\" colspan=\"2\"><b>Interoperability: Were any of the following INTEROPERABILITY technologies implemented in the EHR system currently in use at this site? (N=17)<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Diagnoses are coded using international standards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">94.1\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Medications and pharmacological profiles are coded to national standards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.4\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Patient records are supported by standards-based data migration technology\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">50.0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">ePrescriptions comply with national standards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">52.9\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Patient Identifier is based on national or jurisdictional standard\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">58.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Patient Identifier is supported by aliasing technology to achieve positive ID across systems\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">37.5\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Findings are coded using international standards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">58.8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Communications with other clinics and institutions use international standards\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">31.3\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:30px; padding-right:10px;\">Investigations, referrals, and imaging requests make use of order tracking technology\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">35.3\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Principal_findings\">Principal findings<\/span><\/h3>\n<p>This study aimed to complement current insights into the cost recovery concerns related to EHR investments by considering the research question \u201cWhen implemented in primary care practices, what will be the ROI from EHR?\u201d. The study finds that primary care clinics can realize a positive ROI from the implementation of EHR. Our analysis offers evidence that the increases in net revenue, in the active-patients-to-clinician-FTE ratio, and in the number of active patients are positively associated with the EHR implementation, likely contributing substantially to an average break-even point of 10 months.\n<\/p><p>In addition, the analysis conducted to understand the variances in financial and operational performance among the sampled clinics provides insights into key differences between those clinics that were highly successful and those less successful in realizing a positive ROI from EHR. Some clinics seem to be more innovative than others in using EHR in their practices to achieve significantly better operational and financial results. The analysis suggests that a clinic\u2019s ability to take advantage of EHR to support process changes has a significant effect on the time required to achieve cost recovery from an investment in EHR. In particular, the clinics that were successful in realizing faster time to break even were better at using EHR in workflow areas involving patient information \u2014 such as maintaining patient problem lists, managing test results, and complying with national coding standards, all of which make patient encounter preparation easier and more rapid. We also find that the clinics achieved positive financial performance, even though not all EHR functionalities were used. The alignment of EHR functionalities with clinic workflow plays an important role in achieving positive operational and financial results with EHR. Identified as particularly important EHR-product improvements that would ease adoption of workflow changes are automations that assist clinicians, clinical support staff, and administrative staff both in the overall management of the practice and within the patient encounter, as well as consistent and comprehensive compliance with national standards such as national drug coding standards.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implications_for_practitioners_and_managers_in_primary_care\">Implications for practitioners and managers in primary care<\/span><\/h3>\n<p>The knowledge gained from this ROI study on EHR is important to practicing primary care physicians who are concerned about how they will fare financially upon investing in EHR, as they face ever increasing pressure to transition from their paper-based records to electronic systems. This study provides evidence to practitioners in primary care that investment in EHR can be a sound decision with a reasonable cost recovery time frame, while providing immediate opportunities for increased operational efficiency and the potential for further improvements in clinic performance and benefits realization from EHR. Practitioners in primary care who are considering the investment in EHR should note the important relationship between EHR functionality, clinic workflow change, and a positive ROI from EHR implementation. Positive ROI does not happen automatically upon implementing an EHR package, and a clinic\u2019s ability to leverage EHR for process changes plays a role in achieving a positive ROI.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implications_for_policy_makers\">Implications for policy makers<\/span><\/h3>\n<p>This study\u2019s finding on increased active patient count and clinic operational efficiency after the EHR implementation, in particular with respect to improvement in the active-patients-to-clinician-FTE ratio, offers the possibility that EHR can play a role in addressing the shortage in family physicians. As primary care clinics implement EHR systems and discover better ways to take advantage of EHR in their practices, a key question will be how to incorporate such learnings and deliver enhanced EHR products back into the clinics to realize the full potential of EHR. Policies that enable the establishment of a closed-loop feedback mechanism between EHR vendors and health care providers could facilitate targeted enhancements to EHR systems. In addition, policies that provide support to help primary care practices successfully make EHR-enabled changes, such as support of workflow optimization with an EHR system that would ease adoption, could not only facilitate the realization of positive ROI but also help address the shortage in family physicians.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Future_research\">Future research<\/span><\/h3>\n<p>Some of the factors identified in this research as key factors impacting the realization of a positive ROI from EHR implementation, such as improved access to up-to-date patient charts and improved ability to obtain test results from laboratories and follow the results of an investigation over time, have implications to quality of care and patient safety. Thus, future research will be to investigate the relationship between financial ROI and realization of clinical benefits of EHR such as quality, safety, and patient outcomes, as depicted in Figure 5. Other research should include a study to identify best practices for implementing and using EHR, with concrete examples of success factors and failure factors as well as ways to tailor these best practices relevant to particular clinic situations. In addition, panel analysis, which deals with two-dimensional panel data (cross sectional and times series)<sup id=\"rdp-ebb-cite_ref-HsiaoAnal03_35-0\" class=\"reference\"><a href=\"#cite_note-HsiaoAnal03-35\" rel=\"external_link\">[35]<\/a><\/sup>, can be conducted with the cohort of primary care clinics to understand the effect of learning curve on a clinic\u2019s ability to realize positive ROI and non-financial, clinical benefits from EHR implementation. Knowledge gained from such studies could facilitate EHR adoption and subsequent benefits realization in primary care practices.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Jang_JMIRMedInfo2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"03be06af7955873b8ed8a9ad3d8d5c4e\"><img alt=\"Fig5 Jang JMIRMedInfo2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/fe\/Fig5_Jang_JMIRMedInfo2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 5.<\/b> Future research: investigate the relationship between return on EHR investment and<br \/>clinical benefits realization from EHR implementation<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Limitations\">Limitations<\/span><\/h3>\n<p>The principal limitation of this study is that the number of primary care clinics examined was limited, due mainly to time constraints of clinics to participate in the study and scarcity of suitably detailed operational and financial data necessary for ROI calculation. For the clinics recruited to the study, the most limiting factor was that of collecting a complete picture of the cost and benefits needed to assess an ROI from EHR implementation. This was due mainly to the absence of standardized financial and business-case approaches to the governance of these independent organizations. The insights gained from the participants in our study, however, provide salient insights into the impact of EHR investment to facilitate the EHR adoption across practicing primary care physicians, with information on time required to achieve cost recovery from an EHR investment and on principal factors impacting cost-recovery performance.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors wish to acknowledge the valuable contributions of Liette Lapointe, PhD, Isabelle Vedele, PhD, John Hughes, MD, Raymond Simkus, MD, and Susan Law, PhD, in all phases of this study.\nThis study was funded by Canada Health Infoway Inc, an independent, not-for-profit corporation established in Canada to accelerate the adoption of electronic health records and related technologies on a pan-Canadian basis. The opinions, results, and conclusions reported in this manuscript are those of the authors. No endorsement by Canada Health Infoway is intended or should be inferred.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h2>\n<p>None declared.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HillestadCan05-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HillestadCan05_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hillestad, R.; Bigelow, J.; Bower, A.; Girosi, F.; Meili, R.; Scoville, R.; Taylor, R. (2005). \"Can electronic medical record systems transform health care? Potential health benefits, savings, and costs\". <i>Heath Affairs<\/i> <b>24<\/b> (5): 1103-1117. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.24.5.1103\" target=\"_blank\">10.1377\/hlthaff.24.5.1103<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16162551\" target=\"_blank\">16162551<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Can+electronic+medical+record+systems+transform+health+care%3F+Potential+health+benefits%2C+savings%2C+and+costs&rft.jtitle=Heath+Affairs&rft.aulast=Hillestad%2C+R.%3B+Bigelow%2C+J.%3B+Bower%2C+A.%3B+Girosi%2C+F.%3B+Meili%2C+R.%3B+Scoville%2C+R.%3B+Taylor%2C+R.&rft.au=Hillestad%2C+R.%3B+Bigelow%2C+J.%3B+Bower%2C+A.%3B+Girosi%2C+F.%3B+Meili%2C+R.%3B+Scoville%2C+R.%3B+Taylor%2C+R.&rft.date=2005&rft.volume=24&rft.issue=5&rft.pages=1103-1117&rft_id=info:doi\/10.1377%2Fhlthaff.24.5.1103&rft_id=info:pmid\/16162551&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlumenthalStim09-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlumenthalStim09_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blumenthal, D. (2009). \"Stimulating the adoption of health information technology\". <i>The New England Journal of Medicine<\/i> <b>360<\/b> (15): 1477-1479. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMp0901592\" target=\"_blank\">10.1056\/NEJMp0901592<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19321856\" target=\"_blank\">19321856<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Stimulating+the+adoption+of+health+information+technology&rft.jtitle=The+New+England+Journal+of+Medicine&rft.aulast=Blumenthal%2C+D.&rft.au=Blumenthal%2C+D.&rft.date=2009&rft.volume=360&rft.issue=15&rft.pages=1477-1479&rft_id=info:doi\/10.1056%2FNEJMp0901592&rft_id=info:pmid\/19321856&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BatesGet10-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BatesGet10_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bates, D.W. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2839327\" target=\"_blank\">\"Getting in step: electronic health records and their role in care coordination\"<\/a>. <i>Journal of General Internal Medicine<\/i> <b>25<\/b> (3): 174-176. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11606-010-1252-x\" target=\"_blank\">10.1007\/s11606-010-1252-x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2839327\/\" target=\"_blank\">PMC2839327<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20127195\" target=\"_blank\">20127195<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2839327\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2839327<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Getting+in+step%3A+electronic+health+records+and+their+role+in+care+coordination&rft.jtitle=Journal+of+General+Internal+Medicine&rft.aulast=Bates%2C+D.W.&rft.au=Bates%2C+D.W.&rft.date=2010&rft.volume=25&rft.issue=3&rft.pages=174-176&rft_id=info:doi\/10.1007%2Fs11606-010-1252-x&rft_id=info:pmc\/PMC2839327&rft_id=info:pmid\/20127195&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2839327&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaleemThe13-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SaleemThe13_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saleem, J.J.; Flanagan, M.E.; Wilck, N.R.; Demetriades, J.; Doebbeling, B.N. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3715365\" target=\"_blank\">\"The next-generation electronic health record: perspectives of key leaders from the US Department of Veterans Affairs\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>20<\/b> (e1): e175-e177. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2013-001748\" target=\"_blank\">10.1136\/amiajnl-2013-001748<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3715365\/\" target=\"_blank\">PMC3715365<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23599227\" target=\"_blank\">23599227<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3715365\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3715365<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+next-generation+electronic+health+record%3A+perspectives+of+key+leaders+from+the+US+Department+of+Veterans+Affairs&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Saleem%2C+J.J.%3B+Flanagan%2C+M.E.%3B+Wilck%2C+N.R.%3B+Demetriades%2C+J.%3B+Doebbeling%2C+B.N.&rft.au=Saleem%2C+J.J.%3B+Flanagan%2C+M.E.%3B+Wilck%2C+N.R.%3B+Demetriades%2C+J.%3B+Doebbeling%2C+B.N.&rft.date=2013&rft.volume=20&rft.issue=e1&rft.pages=e175-e177&rft_id=info:doi\/10.1136%2Famiajnl-2013-001748&rft_id=info:pmc\/PMC3715365&rft_id=info:pmid\/23599227&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3715365&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlackThe11-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BlackThe11_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-BlackThe11_5-1\" rel=\"external_link\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Black, A.D.; Car, J.; Pagliari, C.; Anandan, C.; Cresswell, K. Bokun, T.; NcKinstry, B.; Procter, R.; Majeed, A.; Sheikh, A. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3022523\" target=\"_blank\">\"The impact of eHealth on the Quality and Safety of Health Care: a Systematic Overview\"<\/a>. <i>PLoS Medicine<\/i> <b>8<\/b> (1). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1000387\" target=\"_blank\">10.1371\/journal.pmed.1000387<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3022523\/\" target=\"_blank\">PMC3022523<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21267058\" target=\"_blank\">21267058<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3022523\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3022523<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+eHealth+on+the+Quality+and+Safety+of+Health+Care%3A+a+Systematic+Overview&rft.jtitle=PLoS+Medicine&rft.aulast=Black%2C+A.D.%3B+Car%2C+J.%3B+Pagliari%2C+C.%3B+Anandan%2C+C.%3B+Cresswell%2C+K.+Bokun%2C+T.%3B+NcKinstry%2C+B.%3B+Procter%2C+R.%3B+Majeed%2C+A.%3B+Sheikh%2C+A.&rft.au=Black%2C+A.D.%3B+Car%2C+J.%3B+Pagliari%2C+C.%3B+Anandan%2C+C.%3B+Cresswell%2C+K.+Bokun%2C+T.%3B+NcKinstry%2C+B.%3B+Procter%2C+R.%3B+Majeed%2C+A.%3B+Sheikh%2C+A.&rft.date=2013&rft.volume=8&rft.issue=1&rft_id=info:doi\/10.1371%2Fjournal.pmed.1000387&rft_id=info:pmc\/PMC3022523&rft_id=info:pmid\/21267058&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3022523&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HolroydThe11-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HolroydThe11_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Holroyd-Leduc, J.M.; Lorenzetti, D.; Straus, S.E.; Sykes, L.; Quan, H. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985\" target=\"_blank\">\"The impact of the electronic medical record on structure, process, and outcomes within primary care: a systematic review of the evidence\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>18<\/b> (6). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2010-000019\" target=\"_blank\">10.1136\/amiajnl-2010-000019<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3197985\/\" target=\"_blank\">PMC3197985<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21659445\" target=\"_blank\">21659445<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+the+electronic+medical+record+on+structure%2C+process%2C+and+outcomes+within+primary+care%3A+a+systematic+review+of+the+evidence&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Holroyd-Leduc%2C+J.M.%3B+Lorenzetti%2C+D.%3B+Straus%2C+S.E.%3B+Sykes%2C+L.%3B+Quan%2C+H.&rft.au=Holroyd-Leduc%2C+J.M.%3B+Lorenzetti%2C+D.%3B+Straus%2C+S.E.%3B+Sykes%2C+L.%3B+Quan%2C+H.&rft.date=2011&rft.volume=18&rft.issue=6&rft_id=info:doi\/10.1136%2Famiajnl-2010-000019&rft_id=info:pmc\/PMC3197985&rft_id=info:pmid\/21659445&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3197985&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GansMed05-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GansMed05_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-GansMed05_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gans, D.; Kralewski, J.; Hammons, T.; Dowd, B. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985\" target=\"_blank\">\"Medical groups' adoption of electronic health records and information systems\"<\/a>. <i>Health Affairs<\/i> <b>24<\/b> (5): 1323-1333. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.24.5.1323\" target=\"_blank\">10.1377\/hlthaff.24.5.1323<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3197985\/\" target=\"_blank\">PMC3197985<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21659445\" target=\"_blank\">21659445<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3197985<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+groups%27+adoption+of+electronic+health+records+and+information+systems&rft.jtitle=Health+Affairs&rft.aulast=Gans%2C+D.%3B+Kralewski%2C+J.%3B+Hammons%2C+T.%3B+Dowd%2C+B.&rft.au=Gans%2C+D.%3B+Kralewski%2C+J.%3B+Hammons%2C+T.%3B+Dowd%2C+B.&rft.date=2005&rft.volume=24&rft.issue=5&rft.pages=1323-1333&rft_id=info:doi\/10.1377%2Fhlthaff.24.5.1323&rft_id=info:pmc\/PMC3197985&rft_id=info:pmid\/21659445&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3197985&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DesRochesElec08-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DesRochesElec08_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-DesRochesElec08_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DesRoches, C.M.; Campbell, E.G.; Rao, S.R.; Donelan, K.; Ferris, T.G.; Jha, A.; et al. (2008). \"Electronic health records in ambulatory care \u2014 a national survey of physicians\". <i>The New England Journal of Medicine<\/i> <b>359<\/b> (1): 50-60. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMsa0802005\" target=\"_blank\">10.1056\/NEJMsa0802005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18565855\" target=\"_blank\">18565855<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+records+in+ambulatory+care+%E2%80%94+a+national+survey+of+physicians&rft.jtitle=The+New+England+Journal+of+Medicine&rft.aulast=DesRoches%2C+C.M.%3B+Campbell%2C+E.G.%3B+Rao%2C+S.R.%3B+Donelan%2C+K.%3B+Ferris%2C+T.G.%3B+Jha%2C+A.%3B+et+al.&rft.au=DesRoches%2C+C.M.%3B+Campbell%2C+E.G.%3B+Rao%2C+S.R.%3B+Donelan%2C+K.%3B+Ferris%2C+T.G.%3B+Jha%2C+A.%3B+et+al.&rft.date=2008&rft.volume=359&rft.issue=1&rft.pages=50-60&rft_id=info:doi\/10.1056%2FNEJMsa0802005&rft_id=info:pmid\/18565855&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-El-KarehTrends09-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-El-KarehTrends09_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">El-Kareh, R.; Gandhi, T.K.; Poon, E.G.; Newmark, L.P.; Ungar, J.; Lipsitz, S.; et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2659149\" target=\"_blank\">\"Trends in primary care clinician perceptions of a new electronic health record\"<\/a>. <i>Journal of General Internal Medicine<\/i> <b>24<\/b> (4): 464-468. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11606-009-0906-z\" target=\"_blank\">10.1007\/s11606-009-0906-z<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2659149\/\" target=\"_blank\">PMC2659149<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19156468\" target=\"_blank\">19156468<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2659149\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2659149<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Trends+in+primary+care+clinician+perceptions+of+a+new+electronic+health+record&rft.jtitle=Journal+of+General+Internal+Medicine&rft.aulast=El-Kareh%2C+R.%3B+Gandhi%2C+T.K.%3B+Poon%2C+E.G.%3B+Newmark%2C+L.P.%3B+Ungar%2C+J.%3B+Lipsitz%2C+S.%3B+et+al.&rft.au=El-Kareh%2C+R.%3B+Gandhi%2C+T.K.%3B+Poon%2C+E.G.%3B+Newmark%2C+L.P.%3B+Ungar%2C+J.%3B+Lipsitz%2C+S.%3B+et+al.&rft.date=2009&rft.volume=24&rft.issue=4&rft.pages=464-468&rft_id=info:doi\/10.1007%2Fs11606-009-0906-z&rft_id=info:pmc\/PMC2659149&rft_id=info:pmid\/19156468&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2659149&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BassiPerc12-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BassiPerc12_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bassi, J.; Lau, F.; Lesperance, M. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626136\" target=\"_blank\">\"Perceived impact of electronic medical records in physician office practices: a review of survey-based research\"<\/a>. <i>Interactive Journal of Medical Research<\/i> <b>1<\/b> (2): e3. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fijmr.2113\" target=\"_blank\">10.2196\/ijmr.2113<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3626136\/\" target=\"_blank\">PMC3626136<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23611832\" target=\"_blank\">23611832<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626136\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626136<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Perceived+impact+of+electronic+medical+records+in+physician+office+practices%3A+a+review+of+survey-based+research&rft.jtitle=Interactive+Journal+of+Medical+Research&rft.aulast=Bassi%2C+J.%3B+Lau%2C+F.%3B+Lesperance%2C+M.&rft.au=Bassi%2C+J.%3B+Lau%2C+F.%3B+Lesperance%2C+M.&rft.date=2012&rft.volume=1&rft.issue=2&rft.pages=e3&rft_id=info:doi\/10.2196%2Fijmr.2113&rft_id=info:pmc\/PMC3626136&rft_id=info:pmid\/23611832&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3626136&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KeshavjeeBest06-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KeshavjeeBest06_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Keshavjee, K.; Bosomworth, J.; Copen, J.; Lai, J.; Kucukyazici, B.; Lilani, R.; et al. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.infoclin.ca\/assets\/7e474_best%20practices%20in%20emr%20implementation%20-%20july,%202006.pdf\" target=\"_blank\">\"Best practices in EMR implementation: a systematic review\"<\/a> (PDF). <i>AMIA Annual Symposium Proceedings<\/i> <b>2006<\/b>: 982. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1839412\/\" target=\"_blank\">PMC1839412<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17238601\" target=\"_blank\">17238601<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.infoclin.ca\/assets\/7e474_best%20practices%20in%20emr%20implementation%20-%20july,%202006.pdf\" target=\"_blank\">http:\/\/www.infoclin.ca\/assets\/7e474_best%20practices%20in%20emr%20implementation%20-%20july,%202006.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Best+practices+in+EMR+implementation%3A+a+systematic+review&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Keshavjee%2C+K.%3B+Bosomworth%2C+J.%3B+Copen%2C+J.%3B+Lai%2C+J.%3B+Kucukyazici%2C+B.%3B+Lilani%2C+R.%3B+et+al.&rft.au=Keshavjee%2C+K.%3B+Bosomworth%2C+J.%3B+Copen%2C+J.%3B+Lai%2C+J.%3B+Kucukyazici%2C+B.%3B+Lilani%2C+R.%3B+et+al.&rft.date=2006&rft.volume=2006&rft.pages=982&rft_id=info:pmc\/PMC1839412&rft_id=info:pmid\/17238601&rft_id=http%3A%2F%2Fwww.infoclin.ca%2Fassets%2F7e474_best%2520practices%2520in%2520emr%2520implementation%2520-%2520july%2C%25202006.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IliePaper09-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IliePaper09_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ilie, V.; Van Slyke, C.; Parikh, M.A.; Courtney, J.F. (2009). \"Paper Versus Electronic Medical Records: The Effects of Access on Physicians' Decisions to Use Complex Information Technologies\". <i>Decision Sciences<\/i> <b>40<\/b> (2): 213-241. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1540-5915.2009.00227.x\" target=\"_blank\">10.1111\/j.1540-5915.2009.00227.x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Paper+Versus+Electronic+Medical+Records%3A+The+Effects+of+Access+on+Physicians%27+Decisions+to+Use+Complex+Information+Technologies&rft.jtitle=Decision+Sciences&rft.aulast=Ilie%2C+V.%3B+Van+Slyke%2C+C.%3B+Parikh%2C+M.A.%3B+Courtney%2C+J.F.&rft.au=Ilie%2C+V.%3B+Van+Slyke%2C+C.%3B+Parikh%2C+M.A.%3B+Courtney%2C+J.F.&rft.date=2009&rft.volume=40&rft.issue=2&rft.pages=213-241&rft_id=info:doi\/10.1111%2Fj.1540-5915.2009.00227.x&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ValdesInfo04-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ValdesInfo04_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Valdes, I.; Kibbe, D.C.; Tolleson, G.; Kunik, M.E.; Petersen, L.A. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ingentaconnect.com\/content\/bcs\/ipc\/2004\/00000012\/00000001\/art00002\" target=\"_blank\">\"Barriers to proliferation of electronic medical records\"<\/a>. <i>Informatics in Primary Care<\/i> <b>12<\/b> (1): 3\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15140347\" target=\"_blank\">15140347<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ingentaconnect.com\/content\/bcs\/ipc\/2004\/00000012\/00000001\/art00002\" target=\"_blank\">http:\/\/www.ingentaconnect.com\/content\/bcs\/ipc\/2004\/00000012\/00000001\/art00002<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Barriers+to+proliferation+of+electronic+medical+records&rft.jtitle=Informatics+in+Primary+Care&rft.aulast=Valdes%2C+I.%3B+Kibbe%2C+D.C.%3B+Tolleson%2C+G.%3B+Kunik%2C+M.E.%3B+Petersen%2C+L.A.&rft.au=Valdes%2C+I.%3B+Kibbe%2C+D.C.%3B+Tolleson%2C+G.%3B+Kunik%2C+M.E.%3B+Petersen%2C+L.A.&rft.date=2004&rft.volume=12&rft.issue=1&rft.pages=3%E2%80%939&rft_id=info:pmid\/15140347&rft_id=http%3A%2F%2Fwww.ingentaconnect.com%2Fcontent%2Fbcs%2Fipc%2F2004%2F00000012%2F00000001%2Fart00002&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArcherAComp11-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArcherAComp11_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Archer, N.; Cocosila, M. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3222163\" target=\"_blank\">\"A comparison of physician pre-adoption and adoption views on electronic health records in Canadian medical practices\"<\/a>. <i>Journal of Medical Internet Research<\/i> <b>13<\/b> (3): e57. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fjmir.1726\" target=\"_blank\">10.2196\/jmir.1726<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3222163\/\" target=\"_blank\">PMC3222163<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21840835\" target=\"_blank\">21840835<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3222163\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3222163<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comparison+of+physician+pre-adoption+and+adoption+views+on+electronic+health+records+in+Canadian+medical+practices&rft.jtitle=Journal+of+Medical+Internet+Research&rft.aulast=Archer%2C+N.%3B+Cocosila%2C+M.&rft.au=Archer%2C+N.%3B+Cocosila%2C+M.&rft.date=2011&rft.volume=13&rft.issue=3&rft.pages=e57&rft_id=info:doi\/10.2196%2Fjmir.1726&rft_id=info:pmc\/PMC3222163&rft_id=info:pmid\/21840835&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3222163&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TerryImp08-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TerryImp08_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Terry, A.L.; Thorpe, C.F.; Giles, G.; Brown, J.B.; Harris, S.B.; Reid, G.J.; Thind, A.; Stewart, M. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2377228\" target=\"_blank\">\"Implementing electronic health records: Key factors in primary care\"<\/a>. <i>Canadian Family Physician<\/i> <b>54<\/b> (5): 730-736. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2377228\/\" target=\"_blank\">PMC2377228<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18474707\" target=\"_blank\">18474707<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2377228\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2377228<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementing+electronic+health+records%3A+Key+factors+in+primary+care&rft.jtitle=Canadian+Family+Physician&rft.aulast=Terry%2C+A.L.%3B+Thorpe%2C+C.F.%3B+Giles%2C+G.%3B+Brown%2C+J.B.%3B+Harris%2C+S.B.%3B+Reid%2C+G.J.%3B+Thind%2C+A.%3B+Stewart%2C+M.&rft.au=Terry%2C+A.L.%3B+Thorpe%2C+C.F.%3B+Giles%2C+G.%3B+Brown%2C+J.B.%3B+Harris%2C+S.B.%3B+Reid%2C+G.J.%3B+Thind%2C+A.%3B+Stewart%2C+M.&rft.date=2008&rft.volume=54&rft.issue=5&rft.pages=730-736&rft_id=info:pmc\/PMC2377228&rft_id=info:pmid\/18474707&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2377228&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LudwickAdopt09-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LudwickAdopt09_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ludwick, D.A.; Doucette, J. (2009). \"Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries\". <i>International Journal of Medical Informatics<\/i> <b>78<\/b> (1): 22-31. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2008.06.005\" target=\"_blank\">10.1016\/j.ijmedinf.2008.06.005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18644745\" target=\"_blank\">18644745<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adopting+electronic+medical+records+in+primary+care%3A+lessons+learned+from+health+information+systems+implementation+experience+in+seven+countries&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Ludwick%2C+D.A.%3B+Doucette%2C+J.&rft.au=Ludwick%2C+D.A.%3B+Doucette%2C+J.&rft.date=2009&rft.volume=78&rft.issue=1&rft.pages=22-31&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2008.06.005&rft_id=info:pmid\/18644745&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BoonstraBar10-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BoonstraBar10_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Boonstra, A.; Broekhuis, M. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334\" target=\"_blank\">\"Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions\"<\/a>. <i>BMC Health Services Research<\/i> <b>10<\/b>: 231. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6963-10-231\" target=\"_blank\">10.1186\/1472-6963-10-231<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2924334\/\" target=\"_blank\">PMC2924334<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20691097\" target=\"_blank\">20691097<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2924334<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Barriers+to+the+acceptance+of+electronic+medical+records+by+physicians+from+systematic+review+to+taxonomy+and+interventions&rft.jtitle=BMC+Health+Services+Research&rft.aulast=Boonstra%2C+A.%3B+Broekhuis%2C+M.&rft.au=Boonstra%2C+A.%3B+Broekhuis%2C+M.&rft.date=2010&rft.volume=10&rft.pages=231&rft_id=info:doi\/10.1186%2F1472-6963-10-231&rft_id=info:pmc\/PMC2924334&rft_id=info:pmid\/20691097&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2924334&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SimonCorr07-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SimonCorr07_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Simon, S.R.; Kaushal, R.; Cleary, P.D.; Jenter, C.A.; Volk, L.A.; Poon, E.G.; et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2215070\" target=\"_blank\">\"Correlates of electronic health record adoption in office practices: a statewide survey\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>14<\/b>: 110-117. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2215070\/\" target=\"_blank\">PMC2215070<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17068351\" target=\"_blank\">17068351<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2215070\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2215070<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Correlates+of+electronic+health+record+adoption+in+office+practices%3A+a+statewide+survey&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Simon%2C+S.R.%3B+Kaushal%2C+R.%3B+Cleary%2C+P.D.%3B+Jenter%2C+C.A.%3B+Volk%2C+L.A.%3B+Poon%2C+E.G.%3B+et+al.&rft.au=Simon%2C+S.R.%3B+Kaushal%2C+R.%3B+Cleary%2C+P.D.%3B+Jenter%2C+C.A.%3B+Volk%2C+L.A.%3B+Poon%2C+E.G.%3B+et+al.&rft.date=2007&rft.volume=14&rft.pages=110-117&rft_id=info:pmc\/PMC2215070&rft_id=info:pmid\/17068351&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2215070&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KemperAdop06-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KemperAdop06_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kemper, A.R.; Uren, R.L.; Clark, S.J. (2006). \"Adoption of electronic health records in primary care pediatric practices\". <i>Pediatrics<\/i> <b>118<\/b> (1): e20-4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16818534\" target=\"_blank\">16818534<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adoption+of+electronic+health+records+in+primary+care+pediatric+practices&rft.jtitle=Pediatrics&rft.aulast=Kemper%2C+A.R.%3B+Uren%2C+R.L.%3B+Clark%2C+S.J.&rft.au=Kemper%2C+A.R.%3B+Uren%2C+R.L.%3B+Clark%2C+S.J.&rft.date=2006&rft.volume=118&rft.issue=1&rft.pages=e20-4&rft_id=info:pmid\/16818534&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RozenblumAQual11-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RozenblumAQual11_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rozenblum, R.; Jang, Y.; Zimlichman, E.; Zalzberg, C.; Tamblyn, M.; Buckeridge, D.; et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3060213\" target=\"_blank\">\"A qualitative study of Canada\u2019s experience with the implementation of electronic health information technology\"<\/a>. <i>CMAJ<\/i> <b>183<\/b> (5): e281-8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1503%2Fcmaj.100856\" target=\"_blank\">10.1503\/cmaj.100856<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3060213\/\" target=\"_blank\">PMC3060213<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21343262\" target=\"_blank\">21343262<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3060213\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3060213<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+qualitative+study+of+Canada%E2%80%99s+experience+with+the+implementation+of+electronic+health+information+technology&rft.jtitle=CMAJ&rft.aulast=Rozenblum%2C+R.%3B+Jang%2C+Y.%3B+Zimlichman%2C+E.%3B+Zalzberg%2C+C.%3B+Tamblyn%2C+M.%3B+Buckeridge%2C+D.%3B+et+al.&rft.au=Rozenblum%2C+R.%3B+Jang%2C+Y.%3B+Zimlichman%2C+E.%3B+Zalzberg%2C+C.%3B+Tamblyn%2C+M.%3B+Buckeridge%2C+D.%3B+et+al.&rft.date=2011&rft.volume=183&rft.issue=5&rft.pages=e281-8&rft_id=info:doi\/10.1503%2Fcmaj.100856&rft_id=info:pmc\/PMC3060213&rft_id=info:pmid\/21343262&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3060213&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PiferEMR01-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PiferEMR01_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pifer, E.A.; Smith, S.; Keever, G.W. (2001). \"EMR to the rescue. An ambulatory care pilot project shows that data sharing equals cost shaving\". <i>Healthcare Informatics<\/i> <b>18<\/b> (2): 111\u2013114. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11225061\" target=\"_blank\">11225061<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=EMR+to+the+rescue.+An+ambulatory+care+pilot+project+shows+that+data+sharing+equals+cost+shaving&rft.jtitle=Healthcare+Informatics&rft.aulast=Pifer%2C+E.A.%3B+Smith%2C+S.%3B+Keever%2C+G.W.&rft.au=Pifer%2C+E.A.%3B+Smith%2C+S.%3B+Keever%2C+G.W.&rft.date=2001&rft.volume=18&rft.issue=2&rft.pages=111%E2%80%93114&rft_id=info:pmid\/11225061&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangACost03-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WangACost03_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, S.J.; Middleton, B.; Prosser, L.A.; Bardon, C.G.; Spurr, C.D.; Carchidi, P.J.; et al. (2003). \"A cost-benefit analysis of electronic medical records in primary care\". <i>American Journal of Medicine<\/i> <b>114<\/b> (5): 397-403. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12714130\" target=\"_blank\">12714130<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+cost-benefit+analysis+of+electronic+medical+records+in+primary+care&rft.jtitle=American+Journal+of+Medicine&rft.aulast=Wang%2C+S.J.%3B+Middleton%2C+B.%3B+Prosser%2C+L.A.%3B+Bardon%2C+C.G.%3B+Spurr%2C+C.D.%3B+Carchidi%2C+P.J.%3B+et+al.&rft.au=Wang%2C+S.J.%3B+Middleton%2C+B.%3B+Prosser%2C+L.A.%3B+Bardon%2C+C.G.%3B+Spurr%2C+C.D.%3B+Carchidi%2C+P.J.%3B+et+al.&rft.date=2003&rft.volume=114&rft.issue=5&rft.pages=397-403&rft_id=info:pmid\/12714130&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MillerThe05-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MillerThe05_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Miller, R.H.; West, C.; Brown, T.M.; Sim, I.; Ganchoff, C. (2005). \"The value of electronic health records in solo or small group practices\". <i>Health Affairs<\/i> <b>24<\/b> (5): 1127-1137. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.24.5.1127\" target=\"_blank\">10.1377\/hlthaff.24.5.1127<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16162555\" target=\"_blank\">16162555<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+value+of+electronic+health+records+in+solo+or+small+group+practices&rft.jtitle=Health+Affairs&rft.aulast=Miller%2C+R.H.%3B+West%2C+C.%3B+Brown%2C+T.M.%3B+Sim%2C+I.%3B+Ganchoff%2C+C.&rft.au=Miller%2C+R.H.%3B+West%2C+C.%3B+Brown%2C+T.M.%3B+Sim%2C+I.%3B+Ganchoff%2C+C.&rft.date=2005&rft.volume=24&rft.issue=5&rft.pages=1127-1137&rft_id=info:doi\/10.1377%2Fhlthaff.24.5.1127&rft_id=info:pmid\/16162555&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GriegerAPilot07-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GriegerAPilot07_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grieger, D.L.; Cohen, S.H.; Krusch, D.A. (2007). \"A pilot study to document the return on investment for implementing an ambulatory electronic health record at an academic medical center\". <i>Journal of the American College of Surgeons<\/i> <b>205<\/b> (1): 89-96. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jamcollsurg.2007.02.074\" target=\"_blank\">10.1016\/j.jamcollsurg.2007.02.074<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17617337\" target=\"_blank\">17617337<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+pilot+study+to+document+the+return+on+investment+for+implementing+an+ambulatory+electronic+health+record+at+an+academic+medical+center&rft.jtitle=Journal+of+the+American+College+of+Surgeons&rft.aulast=Grieger%2C+D.L.%3B+Cohen%2C+S.H.%3B+Krusch%2C+D.A.&rft.au=Grieger%2C+D.L.%3B+Cohen%2C+S.H.%3B+Krusch%2C+D.A.&rft.date=2007&rft.volume=205&rft.issue=1&rft.pages=89-96&rft_id=info:doi\/10.1016%2Fj.jamcollsurg.2007.02.074&rft_id=info:pmid\/17617337&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AdlerASurv13-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AdlerASurv13_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Adler-Milstein, J.; Green, C.E.; Bates, D.W. (2013). \"A survey analysis suggests that electronic health records will yield revenue gains for some practices and losses for many\". <i>Health Affairs<\/i> <b>32<\/b> (3): 562-570. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.2012.0306\" target=\"_blank\">10.1377\/hlthaff.2012.0306<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23459736\" target=\"_blank\">23459736<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+survey+analysis+suggests+that+electronic+health+records+will+yield+revenue+gains+for+some+practices+and+losses+for+many&rft.jtitle=Health+Affairs&rft.aulast=Adler-Milstein%2C+J.%3B+Green%2C+C.E.%3B+Bates%2C+D.W.&rft.au=Adler-Milstein%2C+J.%3B+Green%2C+C.E.%3B+Bates%2C+D.W.&rft.date=2013&rft.volume=32&rft.issue=3&rft.pages=562-570&rft_id=info:doi\/10.1377%2Fhlthaff.2012.0306&rft_id=info:pmid\/23459736&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SahinAMix14-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SahinAMix14_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sahin, D.; Yaffe, M.J.; Sussman, T.; McCusker, J. (2014). \"A mixed studies literature review of family physicians' participation in research\". <i>Family Medicine<\/i> <b>46<\/b> (7): 503-514. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25058542\" target=\"_blank\">25058542<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+mixed+studies+literature+review+of+family+physicians%27+participation+in+research&rft.jtitle=Family+Medicine&rft.aulast=Sahin%2C+D.%3B+Yaffe%2C+M.J.%3B+Sussman%2C+T.%3B+McCusker%2C+J.&rft.au=Sahin%2C+D.%3B+Yaffe%2C+M.J.%3B+Sussman%2C+T.%3B+McCusker%2C+J.&rft.date=2014&rft.volume=46&rft.issue=7&rft.pages=503-514&rft_id=info:pmid\/25058542&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CresswellDes10-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CresswellDes10_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Creswell, J.; Plano Clark, V. (2010). <i>Designing and conducting mixed methods research<\/i>. SAGE Publications, Inc. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781412975179.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Designing+and+conducting+mixed+methods+research&rft.aulast=Creswell%2C+J.%3B+Plano+Clark%2C+V.&rft.au=Creswell%2C+J.%3B+Plano+Clark%2C+V.&rft.date=2010&rft.pub=SAGE+Publications%2C+Inc&rft.isbn=9781412975179&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LaerumTask04-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LaerumTask04_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">L\u00e6rum, H.; Faxvaag, A. (2004). \"Task-oriented evaluation of electronic medical records systems: development and validation of a questionnaire for physicians\". <i>BMC Medical Informatics and Decision Making<\/i> <b>4<\/b> (1). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-4-1\" target=\"_blank\">10.1186\/1472-6947-4-1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Task-oriented+evaluation+of+electronic+medical+records+systems%3A+development+and+validation+of+a+questionnaire+for+physicians&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=L%C3%A6rum%2C+H.%3B+Faxvaag%2C+A.&rft.au=L%C3%A6rum%2C+H.%3B+Faxvaag%2C+A.&rft.date=2004&rft.volume=4&rft.issue=1&rft_id=info:doi\/10.1186%2F1472-6947-4-1&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JoosAnElec06-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JoosAnElec06_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Joos, D.; Chen, Q.; Jirjis, J.; Johnson, K.B. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839545\" target=\"_blank\">\"An electronic medical record in primary care: impact on satisfaction, work efficiency and clinic processes\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>2006<\/b>: 394-8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1839545\/\" target=\"_blank\">PMC1839545<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17238370\" target=\"_blank\">17238370<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839545\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1839545<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+electronic+medical+record+in+primary+care%3A+impact+on+satisfaction%2C+work+efficiency+and+clinic+processes&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Joos%2C+D.%3B+Chen%2C+Q.%3B+Jirjis%2C+J.%3B+Johnson%2C+K.B.&rft.au=Joos%2C+D.%3B+Chen%2C+Q.%3B+Jirjis%2C+J.%3B+Johnson%2C+K.B.&rft.date=2006&rft.volume=2006&rft.pages=394-8&rft_id=info:pmc\/PMC1839545&rft_id=info:pmid\/17238370&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1839545&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AaronsonElec01-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AaronsonElec01_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aaronson, J.W.; Murphy-Cullen, C.L.; Chop, W.M.; Frey, R.D. (2001). \"Electronic medical records: the family practice resident perspective\". <i>Family Medicine<\/i> <b>33<\/b> (2): 128-132. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11271741\" target=\"_blank\">11271741<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+medical+records%3A+the+family+practice+resident+perspective&rft.jtitle=Family+Medicine&rft.aulast=Aaronson%2C+J.W.%3B+Murphy-Cullen%2C+C.L.%3B+Chop%2C+W.M.%3B+Frey%2C+R.D.&rft.au=Aaronson%2C+J.W.%3B+Murphy-Cullen%2C+C.L.%3B+Chop%2C+W.M.%3B+Frey%2C+R.D.&rft.date=2001&rft.volume=33&rft.issue=2&rft.pages=128-132&rft_id=info:pmid\/11271741&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AudetMeas05-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AudetMeas05_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Audet, A.M.; Doty, M.M.; Shamasdin, J.; Schoenbaum, S.C. (2005). \"Measure, learn, and improve: physicians' involvement in quality improvement\". <i>Health Affairs<\/i> <b>24<\/b> (3): 843-853. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.24.3.843\" target=\"_blank\">10.1377\/hlthaff.24.3.843<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15886180\" target=\"_blank\">15886180<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Measure%2C+learn%2C+and+improve%3A+physicians%27+involvement+in+quality+improvement&rft.jtitle=Health+Affairs&rft.aulast=Audet%2C+A.M.%3B+Doty%2C+M.M.%3B+Shamasdin%2C+J.%3B+Schoenbaum%2C+S.C.&rft.au=Audet%2C+A.M.%3B+Doty%2C+M.M.%3B+Shamasdin%2C+J.%3B+Schoenbaum%2C+S.C.&rft.date=2005&rft.volume=24&rft.issue=3&rft.pages=843-853&rft_id=info:doi\/10.1377%2Fhlthaff.24.3.843&rft_id=info:pmid\/15886180&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OtienoNur07-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OtienoNur07_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Otieno, O.G.; Toyama, H.; Asonuma, M.; Kanai-Pak, M.; Naitoh, K. (2007). \"Nurses' views on the use, quality and user satisfaction with electronic medical records: questionnaire development\". <i>Journal of Advanced Nursing<\/i> <b>60<\/b> (2): 209-219. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1365-2648.2007.04384.x\" target=\"_blank\">10.1111\/j.1365-2648.2007.04384.x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17877568\" target=\"_blank\">17877568<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Nurses%27+views+on+the+use%2C+quality+and+user+satisfaction+with+electronic+medical+records%3A+questionnaire+development&rft.jtitle=Journal+of+Advanced+Nursing&rft.aulast=Otieno%2C+O.G.%3B+Toyama%2C+H.%3B+Asonuma%2C+M.%3B+Kanai-Pak%2C+M.%3B+Naitoh%2C+K.&rft.au=Otieno%2C+O.G.%3B+Toyama%2C+H.%3B+Asonuma%2C+M.%3B+Kanai-Pak%2C+M.%3B+Naitoh%2C+K.&rft.date=2007&rft.volume=60&rft.issue=2&rft.pages=209-219&rft_id=info:doi\/10.1111%2Fj.1365-2648.2007.04384.x&rft_id=info:pmid\/17877568&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SittigEval99-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SittigEval99_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sittig, D.F.; Kuperman, G.J.; Fiskio, J. (1999). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2232602\" target=\"_blank\">\"Evaluating physician satisfaction regarding user interactions with an electronic medical record system\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>1999<\/b>: 400-404. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2232602\/\" target=\"_blank\">PMC2232602<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10566389\" target=\"_blank\">10566389<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2232602\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2232602<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Evaluating+physician+satisfaction+regarding+user+interactions+with+an+electronic+medical+record+system&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Sittig%2C+D.F.%3B+Kuperman%2C+G.J.%3B+Fiskio%2C+J.&rft.au=Sittig%2C+D.F.%3B+Kuperman%2C+G.J.%3B+Fiskio%2C+J.&rft.date=1999&rft.volume=1999&rft.pages=400-404&rft_id=info:pmc\/PMC2232602&rft_id=info:pmid\/10566389&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2232602&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GlaserThe99-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GlaserThe99_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Glaser, B.G.; Strauss, A.L. (1999). <i>The discovery of grounded theory: Strategies for qualitative research<\/i>. Aldine Transaction. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780202302607.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+discovery+of+grounded+theory%3A+Strategies+for+qualitative+research&rft.aulast=Glaser%2C+B.G.%3B+Strauss%2C+A.L.&rft.au=Glaser%2C+B.G.%3B+Strauss%2C+A.L.&rft.date=1999&rft.pub=Aldine+Transaction&rft.isbn=9780202302607&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HsiaoAnal03-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HsiaoAnal03_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Hsiao, C. (2003). <i>Analysis of Panel Data<\/i> (2nd ed.). Cambridge University Press. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780521522717.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Analysis+of+Panel+Data&rft.aulast=Hsiao%2C+C.&rft.au=Hsiao%2C+C.&rft.date=2003&rft.edition=2nd&rft.pub=Cambridge+University+Press&rft.isbn=9780521522717&rfr_id=info:sid\/en.wikipedia.org:Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>EHR<\/b>: electronic health records\n<\/p><p><b>FTE<\/b>: full time equivalent\n<\/p><p><b>IT<\/b>: information technology\n<\/p><p><b>ROI<\/b>: return on investment\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful.\n<\/p><p>Per the distribution agreement, the following copyright information is also being added: \n<\/p><p>\u00a9Yeona Jang, Michel A Lortie, Steven Sanche. Originally published in JMIR Medical Informatics (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/medinform.jmir.org\" target=\"_blank\">http:\/\/medinform.jmir.org<\/a>), 29.09.2014.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210231\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.895 seconds\nReal time usage: 0.932 seconds\nPreprocessor visited node count: 28661\/1000000\nPreprocessor generated node count: 35713\/1000000\nPost\u2010expand include size: 252968\/2097152 bytes\nTemplate argument size: 83145\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 863.706 1 - -total\n 88.20% 761.767 1 - Template:Reflist\n 78.61% 678.937 35 - Template:Citation\/core\n 76.51% 660.818 32 - Template:Cite_journal\n 10.83% 93.524 69 - Template:Citation\/identifier\n 7.32% 63.184 1 - Template:Infobox_journal_article\n 6.89% 59.540 1 - Template:Infobox\n 5.98% 51.638 3 - Template:Cite_book\n 4.08% 35.259 80 - Template:Infobox\/row\n 4.08% 35.237 35 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7660-0!*!0!!en!5!* and timestamp 20181213210230 and revision id 23419\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study\">https:\/\/www.limswiki.org\/index.php\/Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","5d49b7c0866e1210c1696240ca7f59fd_images":["https:\/\/www.limswiki.org\/images\/9\/99\/Fig1_Jang_JMIRMedInfo2014_2-2.jpg","https:\/\/www.limswiki.org\/images\/6\/6d\/Fig2_Jang_JMIRMedInfo2014_2-2.jpg","https:\/\/www.limswiki.org\/images\/f\/f4\/Fig3_Jang_JMIRMedInfo2014_2-2.jpg","https:\/\/www.limswiki.org\/images\/4\/44\/Fig4_Jang_JMIRMedInfo2014_2-2.jpg","https:\/\/www.limswiki.org\/images\/f\/fe\/Fig5_Jang_JMIRMedInfo2014_2-2.jpg"],"5d49b7c0866e1210c1696240ca7f59fd_timestamp":1544734950,"80b7c208090cc5cb594724fd09a469c6_type":"article","80b7c208090cc5cb594724fd09a469c6_title":"Medical informatics specialty in the developed English-speaking countries: The terminology comparative analysis (Kobryn 2015)","80b7c208090cc5cb594724fd09a469c6_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis","80b7c208090cc5cb594724fd09a469c6_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Medical informatics specialty in the developed English-speaking countries: The terminology comparative analysis\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nMedical informatics specialty in the developed English-speaking countries: The terminology comparative analysisJournal\n \nComparative Professional PedagogyAuthor(s)\n \nKobryn, NadiaAuthor affiliation(s)\n \nDanylo Halytsky Lviv National Medical University, UkrainePrimary contact\n \nE-mail: nadiyak83@e-mail.uaYear published\n \n2015Volume and issue\n \n5 (1)Page(s)\n \n86\u201391DOI\n \n10.1515\/rpp-2015-0026ISSN\n \n2353-9518Distribution license\n \nCreative Commons Attribution-NonCommercial-NoDerivatives 3.0Website\n \nhttp:\/\/www.degruyter.com\/view\/j\/rpp.2015.5.issue-1\/rpp-2015-0026\/rpp-2015-0026.xmlDownload\n \nhttp:\/\/www.degruyter.com (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 The aim of the study \n4 Theoretical framework and research methods \n5 Results \n6 Conclusions \n7 References \n8 Notes \n\n\n\nAbstract \nThe article studies the development process of medical informatics specialty terminology as the ground for further research into foreign countries\u2019 experience, including the Canadian one, of specialists\u2019 professional training in the field of MI. The study determines the origin and chief stages of the formation and development of the medical informatics terminological system. The author performs the comparative analysis of terms used by the world organizations on health care informatisation issues, particularly International Medical Informatics Association as well as medical informatics associations of the USA and Canada as the leading countries where qualified workforce in the medical informatics specialty is trained. The European and Ukrainian experience has also been taken into consideration. The results of the comparative study have shown that the English terms \u2018medical informatics\u2019, \u2018biomedical informatics\u2019 and \u2018health informatics\u2019 serve as the umbrella terms for professional training programs and include a set of subspecialties that identify diverse spheres of information technology applications to medical science and practice, namely \u2018clinical informatics\u2019, \u2018bioinformatics\u2019, \u2018health care informatics\u2019, \u2018nursing informatics\u2019, \u2018imaging informatics\u2019, etc.\nKey words: professional education, medical informatics specialty, professional training, practical skills, health informatics, biomedical informatics.\n\nIntroduction \nIn the middle of the 20th century, the penetration of information technologies into medicine and health care led to the appearance of a new applied science, academic discipline and later on a specialty that is presently termed medical informatics (MI). As the field is relatively young and develops quickly, there is no consistency in its definitions and nomenclature. One of the illustrative examples here is the existence of a set of terms in English scientific literature that are concerned with the notion of MI. The most common among them are medical informatics, bioinformatics, biomedical informatics, health informatics, clinical informatics, etc. The American scientist W. Hersh defines the phenomenon of diverse and unstandardised usage of determining adjectives in the MI terminology as \u201cadjective problem\u201d, thus explaining ambiguities and confusion of terms in the nominative system of the field.[1]\nSo, today there is the obvious necessity for detailed analysis and concretization of the main notions that deal with computer technologies application to medicine, health care system informatisation and workforce professional training in the MI specialty. \n\nThe aim of the study \nThe research is performed in order to analyze English terminological system denoting the MI field and to determine those terminological units that serve as umbrella terms for the specialty as well as those considered the names of the MI subspecialties.\n\nTheoretical framework and research methods \nThe methodology of our research implies applying a number of theoretical research methods. We have used the comparative and historical method for studying the origin, formation and development of MI as a science and a specialty; the logical method for finding objective laws of the concept formation and reasoning in MI; induction and deduction for collecting theoretical and factual material with its generalization, analysis and synthesis for concretizing the subject matter of the main notions and terms; comparison and collation of the terminological system used by the leading world MI organizations that are engaged in studying the problems of health care system informatisation and specialists\u2019 professional training in the field of MI. In particular, we compared the terminology operated in the activities of International Medical Informatics Association IMIA, American Medical Informatics Association AMIA, Canada\u2019s Health Informatics Association COACH as well as European Federation of Medical Informatics EFMI and Ukrainian Association for \u201cComputer Medicine\u201d.\n\nResults \nThe retrospective analysis of the MI terminological system development is of great importance in the context of studying specialists\u2019 professional training in the field of MI. The results of the conducted research have distinguished three main stages in formation and development of the specialty terminology.\nThe first stage is related to the generation of the notion of MI (the 1950s-1970s). The term \u2018medical informatics\u2019 appeared only in the 1970s although the emergence of the new science dates back to 1959 when the American scientists and radiologists R. Ledley and L. Lusted published the article in which the authors gave reasons for using computers to optimise statistical data processing in medical diagnostics. It was the first time when the terms \u2018medical computer science\u2019 and \u2018medical information science\u2019 were used in English scientific literature.[2] Later on, however, it became apparent that the terms proposed in 1959 did not fully convey the essence of the notion. \nThe second stage reflects the purposeful formation of the terminology in the MI field (the 1970s-1990s). In 1974 the book \u201cEducation in Informatics of Health Personal\u201d was published. The collective authors led by the British researcher J. Anderson proposed the term \u2018medical informatics\u2019 as a basic one to denote the practice of computers application to medicine. Although the work did not provide a scientifically approved definition of the given term, it proposed the equivalent fully reflecting the content of the new scientific field. As the word \u2018informatics\u2019 has polyfunctional nature and gives the name to the science dealing with information processing, its proper storage and effective usage, the term is also associated with computer technologies, engineering and telecommunication. The adjective \u2018medical\u2019, in its turn, indicates the relationship of the science to researches and practical activities of specialists in the field of MI that are aimed at disease prevention, diagnosis and treatment.[3] Therefore, since 1974 till now the term MI has been widely used throughout the world and is present in the titles of such representative MI organizations as International Medical Informatics Association, American Medical Informatics Association or European Federation of Medical Informatics.\nIn the 1990s, however, the nominative word-combination \u2018biomedical informatics\u2019 (BMI) was more often collaterally used with the term MI in the scientific literature. Such a tendency in the terminology shift can have historically-grounded explanations. In particular, the American scientist E. Shortliffe considers that changes in the terminology and ways of its interpretations happened gradually and were induced by modern advanced technologies appearance and their integration into the society life-sustaining activities in the 1990s. They enabled to perform scientific experiments that gave the understanding of human disease nature at molecular and genetic levels.[2] Thus, at the end of the 20th century the term \u2018biomedical informatics\u2019 appeared indicating the connection of the specialty not only with medicine but also with biological chemistry, histology, genetics and embryology as the subfields of biomedicine.\nThe third stage (the present one) is characterized by outlining certain specifics in the terminological system usage within the studied specialty due to information technology penetration into various areas of biomedicine and health care service. It is exactly that period when such terms as \u2018clinical informatics\u2019, \u2018nursing informatics\u2019, \u2018dental informatics\u2019, \u2018health care informatics\u2019, etc. appeared specifying the specialists\u2019 professional functioning. Undoubtedly, hierarchy of the given terms within the system of notions concerned with MI is not questioned. On the other hand, the analysis of the terminology operated by the leading world MI organizations enables to conclude about substantial peculiarities of their terminology interpretation.\nThe distinct hierarchy within the field is noticed in the terminological system developed by the American school, which is represented by American Medical Informatics Association. The American association uses the term \u2018biomedical informatics\u2019 (BMI) in its operation as the basic one for determination of the interdisciplinary specialty that involves biomedicine and informatics.[4] Hereby, BMI is considered as the applied branch of informatics that develops methodology, technical means and theoretical basis for biomedical information application in order to promote research conduction, health improvement as well as qualitative health care service delivery.[5]\nAccording to the Glossary found on the AMIA\u2019s site, trained specialists regard BMI as a general term for the scientific discipline with many other related terms to be the spheres of its practical application. Bioinformatics, in particular, indicates the specialty aimed at studying the usage of technical means of information visualization and presentation in biological systems, especially in molecular biology, genomics and proteomics. The usage of the word-combination \u2018health informatics\u2019 is specific to the applied investigations and practical activities of BMI specialists carried out in the two other subfields \u2013 clinical informatics and public health informatics. The goal of the two latter ones consists in ensuring quality of medical service delivery and promoting public health care efficacy.[6] In its turn, the terminological unit \u2018medical informatics\u2019 is used in the context of clinical informatics dealing with disease diagnosis and treatment and has been designed for doctors. \u2018Nursing informatics\u2019 and \u2018dental informatics\u2019 should also be mentioned here as clinical informatics branches concerned with the development of information processing systems that are intended to improve work efficiency of the nursing and dentist staff.[4]\nIn contrast to the USA experience in the MI terminology usage, the term \u2018health informatics\u2019 (HI) is commonly used in Canada. It is also worth mentioning that Canada\u2019s Health Informatics Association (COACH) employs it in wider context if compared to the interpretation of the same term by AMIA. The term determines the specialty which is the convergence of clinical practice, biomedical information management, information technologies as well as management practice in medicine and health care system aimed at delivery of quality medical service, disease prevention, popularization of healthy lifestyle, etc.[7]\nIn view of the above said, the HI specialists in Canada should:\n\n be knowledgeable about the key aspects of the clinical practice that covers the wide range of activities intended for disease diagnosis, prophylaxis and patient treatment;\n possess practical skills in information management that implies biomedical information collection, processing, storage and interpretation;\n be proficient in applying information technologies for the effective information storage or processing;\n acquire competency of management practice that provides a HI specialist with means for medical information system design and monitoring in order to use biomedical information rationally. \nIt should also be noted that the terms \u2018health informatics\u2019 and \u2018medical informatics\u2019 are practically used in the same contexts which is apparent from both the sites of the Canadian higher educational institutions proposing programs of professional training in HI and from publications of the Canadian scientists J. Moehr and A. Grant. Besides, some researchers suppose the adjective \u2018medical\u2019 to be connected with disease diagnosis, prevention or treatment, thus forming the field\u2019s \u2018negative\u2019 profile. The word \u2018health\u2019 used before \u2018informatics\u2019, to the contrary, shifts the accent and points that the main goal of HI as a specialty is optimal information usage by the field\u2019s specialists for improving health of both the individual patient and the population as a whole.[8]\nThe analysis of the terminology formation by European Federation of MI (EFMI) as the formal representative of national MI organizations of the European countries has revealed the preference to \u2018medical informatics\u2019 as the terminological unit for common use. In fact, the term denotes MI both as the science and umbrella term for a number of specialties related to health care and informatics, thus indicating that the terminological system used by EFMI is, to some extent, similar to the above-mentioned AMIA\u2019s terminology interpretation.\nThere is also no coherence in the MI terminology usage by Ukrainian Association for \u201cComputer Medicine\u201d, a national member of the international MI organization IMIA. On its official web site, the word-combination \u2018\u043c\u0435\u0434\u0438\u0447\u043d\u0430 \u0456\u043d\u0444\u043e\u0440\u043c\u0430\u0442\u0438\u043a\u0430\u2019 is widely used without any definitions proposed for the term. Nevertheless, the association clearly states that its goals are directed at the implementation of new medical information systems, the design of electronic medical history, reformation of health care system in Ukraine as well as the development of biotechnical systems and new software for medical institutions.[9] Thus, the Ukrainian MI specialists mostly operate the term medical informatics (\u2018\u043c\u0435\u0434\u0438\u0447\u043d\u0430 \u0456\u043d\u0444\u043e\u0440\u043c\u0430\u0442\u0438\u043a\u0430\u2019 in Ukrainian) as a general one. Moreover, the terminological unit is found in the specialty title \u201cMedical and Biological Informatics and Cybernetics\u201d (\u201c\u041c\u0435\u0434\u0438\u0447\u043d\u0430 \u0442\u0430 \u0411\u0456\u043e\u043b\u043e\u0433\u0456\u0447\u043d\u0430 \u0456\u043d\u0444\u043e\u0440\u043c\u0430\u0442\u0438\u043a\u0430 \u0456 \u043a\u0456\u0411\u0435\u0440\u043d\u0435\u0442\u0438\u043a\u0430\u201d in Ukrainian) compiled by the Ukrainian MI association for the Ukrainian Higher Attestation Commission as well as in the names of academic disciplines in the curricula of the Ukrainian higher medical educational establishments.\nAll the above-mentioned MI national organizations are the members of International MI Association (IMIA). It plays a crucial role in information technology implementation into medicine and supports researches aimed at effective informatisation of the health care system as well as development of educational policy in professional training of the MI workforce. IMIA uses the term \u2018biomedical informatics\u2019 in its widest meaning covering all possible determining adjectives including \u2018health\u2019, \u2018medical\u2019, \u2018nursing\u2019, \u2018bio\u2019, etc.[10] Besides, in IMIA recommendations on professional education in MI, the word-combination \u2018biomedical and health informatics\u2019 is used throughout the text[11] We consider that such a broad interpretation of the term may indicate that IMIA as the international MI organization makes efforts to unify the field\u2019s terminology in order to avoid further misunderstandings and ambiguities. Secondly, simultaneous usage of the terms \u2018biomedical and health informatics\u2019 proves their equivalency, with difference consisting mainly in the term preferences restricted to a specific geographic area.\n\nConclusions \nThe MI specialty is relatively young and is continuously changing, therefore the unified strategy of using the associated terminology has not been developed yet. It is known that there is a branching system of terms that determine the MI subfields. There are vague boundaries between them and it is not always possible to track where one field finishes and the other one begins. Moreover, the list of adjectives preceding the word \u2018informatics\u2019 is not limited only to such determinants as \u2018biomedical\u2019, \u2018medical\u2019, \u2018health\u2019, \u2018public health care\u2019 or \u2018clinical\u2019. As the science constantly develops and its further specializations continues, new subspecialties within the existing ones appear, for example imaging informatics, pathology informatics, consumer health informatics etc.[1] In future, this list may be prolonged with new terms due to advancement of information technologies and the further MI specialization.\nIn summary, the analysis of the terminological system related to the notion of MI in the context of professional education has enabled to distinguish three terms that are used as equivalents and serve as umbrella terms \u2013 \u2018biomedical informatics\u2019, \u2018medical informatics\u2019 and \u2018health informatics\u2019. They determine the field which specialists are concerned with practical aspects of biomedical information storage, collection, circulation, mining, retrieval, exchange and security, etc. One of the noticed differences among them is the preferences in term usage according to the principle of geographical area. In particular,\n\u2018biomedical informatics\u2019 is of common usage in the USA, \u2018health informatics\u2019 \u2013 in Canada and \u2018medical informatics\u2019 is mostly used in the European countries as well as in Ukraine. Consequently, the term \u2018medical informatics\u2019 has historically become widely used by scientists and teaching staff and serves as generally applicable terminological unit to denote the key theories, concepts and techniques for information technology usage in health care system, biomedicine and hence in the system of professional training of the MI specialists.\nThe comparative analysis of the MI specialty terminology in the developed English-speaking countries is considered an essential ground for further research into the foreign countries\u2019 experience in professional training of the MI specialists for the purposes of its progressive ideas implementation into the educational system of Ukraine.\n\nReferences \n\n\u2191 1.0 1.1 Hersh, W. R. (2009). \"A Stimulus to Define Informatics and Health Information Technology\". BMC Medical Informatics and Decision Making 9 (24): 6. doi:10.1186\/1472-6947-9-24.   \n\n\u2191 2.0 2.1 Shortliffe, E. H. (16 April 2013). \"Biomedical Informatics: Assessing its Evolving Role as both Science and Application\". NYU Langone Medical Center, Department of Medicine. http:\/\/www.med.nyu.edu\/medicine\/education\/grand-rounds\/mgr-edward-shortliffe-md-phd-video . Retrieved 19 November 2014 .   \n\n\u2191 Collen, M. F. (1986). \"Origins of Medical Informatics\". The Western Journal of Medicine (145): 778\u2013785. PMC PMC1307150. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1307150 .   \n\n\u2191 4.0 4.1 Kulikowski, C. A.; Shortliffe, E. H.; Currie, L. M. (2012). \"AMIA Board White Paper: Definition of Biomedical Informatics and Specification of Core Competences for Graduate Education in the Discipline\". Journal of the American Medical Informatics Association (19): 931\u2013938. doi:10.1136\/amiajnl-2012-001053.   \n\n\u2191 \"AMIA: Informatics Professionals. Leading the Way.\". AMIA. 2014. http:\/\/www.amia.org . Retrieved 17 November 2014 .   \n\n\u2191 \"Glossary of Acronyms and Terms Commonly Used in Informatics\". AMIA. 2014. http:\/\/www.amia.org\/glossary . Retrieved 14 August 2014 .   \n\n\u2191 Canada\u2019s Health Informatics Association (November 2012). \"Health Informatics Professional Core Competencies\" (PDF). p. 22. https:\/\/www.coachorg.com\/en\/resourcecentre\/resources\/Health-Informatics-Core-Competencies.pdf . Retrieved 25 July 2014 .   \n\n\u2191 Dalrymple, P. W. (2011). \"Data, Information, Knowledge: The Emerging Field of Health Informatics\". Bulletin of the American Society for Information Science and Technology 37 (5): 41\u201344. doi:10.1002\/bult.2011.1720370512.   \n\n\u2191 \"\u0423\u041a\u0420\u0410\u0407\u041d\u0421\u042c\u041a\u0410 \u0410\u0421\u041e\u0426\u0406\u0410\u0426\u0406\u042f \"\u041a\u041e\u041c\u041f'\u042e\u0422\u0415\u0420\u041d\u0410 \u041c\u0415\u0414\u0418\u0426\u0418\u041d\u0410\"\". \u0423\u0410\u041a\u041c. 2014. http:\/\/uacm.kharkov.ua\/ukr\/index.shtml?uabout.htm . Retrieved 18 November 2014 .   \n\n\u2191 International Medical Informatics Association (11 June 2007). \"Towards IMIA 2015 \u2013 The IMIA Strategic Plan\" (PDF). p. 15. http:\/\/www.imia-medinfo.org\/new2\/sites\/default\/files\/IMIA_Strategic_Plan_2007final.pdf . Retrieved 27 July 2014 .   \n\n\u2191 Mantas, J.; Ammenwerth, E.; Demiris, G. (2010). \"Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics \u2013 1st revision\". Methods of Information in Medicine 49 (2): 105\u2013120. doi:10.3414\/ME5119.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. The original article puts the references in alphabetical order; however, we've ordered them in the order they appear in the article. We also added PMCID and DOI when they were missing from the original reference. No other modifications were made in accordance with the \"no derivatives\" portion of the distribution license.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:58.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,042 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","80b7c208090cc5cb594724fd09a469c6_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Medical_informatics_specialty_in_the_developed_English-speaking_countries_The_terminology_comparative_analysis skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Medical informatics specialty in the developed English-speaking countries: The terminology comparative analysis<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>The article studies the development process of <a href=\"https:\/\/www.limswiki.org\/index.php\/Medical_informatics\" title=\"Medical informatics\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"f89ecb3b26617b8c6e09bc5e050cfd5d\">medical informatics<\/a> specialty terminology as the ground for further research into foreign countries\u2019 experience, including the Canadian one, of specialists\u2019 professional training in the field of MI. The study determines the origin and chief stages of the formation and development of the medical informatics terminological system. The author performs the comparative analysis of terms used by the world organizations on health care informatisation issues, particularly International Medical Informatics Association as well as medical informatics associations of the USA and Canada as the leading countries where qualified workforce in the medical informatics specialty is trained. The European and Ukrainian experience has also been taken into consideration. The results of the comparative study have shown that the English terms \u2018medical informatics\u2019, \u2018biomedical informatics\u2019 and \u2018health informatics\u2019 serve as the umbrella terms for professional training programs and include a set of subspecialties that identify diverse spheres of information technology applications to medical science and practice, namely \u2018clinical informatics\u2019, \u2018<a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a>\u2019, \u2018health care informatics\u2019, \u2018nursing informatics\u2019, \u2018<a href=\"https:\/\/www.limswiki.org\/index.php\/Imaging_informatics\" title=\"Imaging informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"fc0ae6a154d8896767defefdb6d14d0e\">imaging informatics<\/a>\u2019, etc.\n<\/p><p><b>Key words<\/b>: professional education, medical informatics specialty, professional training, practical skills, health informatics, biomedical informatics.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>In the middle of the 20th century, the penetration of information technologies into medicine and health care led to the appearance of a new applied science, academic discipline and later on a specialty that is presently termed medical informatics (MI). As the field is relatively young and develops quickly, there is no consistency in its definitions and nomenclature. One of the illustrative examples here is the existence of a set of terms in English scientific literature that are concerned with the notion of MI. The most common among them are medical informatics, bioinformatics, biomedical informatics, health informatics, clinical informatics, etc. The American scientist W. Hersh defines the phenomenon of diverse and unstandardised usage of determining adjectives in the MI terminology as \u201cadjective problem\u201d, thus explaining ambiguities and confusion of terms in the nominative system of the field.<sup id=\"rdp-ebb-cite_ref-Hersh09_1-0\" class=\"reference\"><a href=\"#cite_note-Hersh09-1\" rel=\"external_link\">[1]<\/a><\/sup>\n<\/p><p>So, today there is the obvious necessity for detailed analysis and concretization of the main notions that deal with computer technologies application to medicine, health care system informatisation and workforce professional training in the MI specialty. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_aim_of_the_study\">The aim of the study<\/span><\/h2>\n<p>The research is performed in order to analyze English terminological system denoting the MI field and to determine those terminological units that serve as umbrella terms for the specialty as well as those considered the names of the MI subspecialties.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Theoretical_framework_and_research_methods\">Theoretical framework and research methods<\/span><\/h2>\n<p>The methodology of our research implies applying a number of theoretical research methods. We have used the comparative and historical method for studying the origin, formation and development of MI as a science and a specialty; the logical method for finding objective laws of the concept formation and reasoning in MI; induction and deduction for collecting theoretical and factual material with its generalization, analysis and synthesis for concretizing the subject matter of the main notions and terms; comparison and collation of the terminological system used by the leading world MI organizations that are engaged in studying the problems of health care system informatisation and specialists\u2019 professional training in the field of MI. In particular, we compared the terminology operated in the activities of International Medical Informatics Association IMIA, American Medical Informatics Association AMIA, Canada\u2019s Health Informatics Association COACH as well as European Federation of Medical Informatics EFMI and Ukrainian Association for \u201cComputer Medicine\u201d.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>The retrospective analysis of the MI terminological system development is of great importance in the context of studying specialists\u2019 professional training in the field of MI. The results of the conducted research have distinguished three main stages in formation and development of the specialty terminology.\n<\/p><p><i>The first stage<\/i> is related to the generation of the notion of MI (the 1950s-1970s). The term \u2018medical informatics\u2019 appeared only in the 1970s although the emergence of the new science dates back to 1959 when the American scientists and radiologists R. Ledley and L. Lusted published the article in which the authors gave reasons for using computers to optimise statistical data processing in medical diagnostics. It was the first time when the terms \u2018medical computer science\u2019 and \u2018medical information science\u2019 were used in English scientific literature.<sup id=\"rdp-ebb-cite_ref-Shortliffe13_2-0\" class=\"reference\"><a href=\"#cite_note-Shortliffe13-2\" rel=\"external_link\">[2]<\/a><\/sup> Later on, however, it became apparent that the terms proposed in 1959 did not fully convey the essence of the notion. \n<\/p><p><i>The second stage<\/i> reflects the purposeful formation of the terminology in the MI field (the 1970s-1990s). In 1974 the book \u201cEducation in <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">Informatics<\/a> of Health Personal\u201d was published. The collective authors led by the British researcher J. Anderson proposed the term \u2018medical informatics\u2019 as a basic one to denote the practice of computers application to medicine. Although the work did not provide a scientifically approved definition of the given term, it proposed the equivalent fully reflecting the content of the new scientific field. As the word \u2018informatics\u2019 has polyfunctional nature and gives the name to the science dealing with <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> processing, its proper storage and effective usage, the term is also associated with computer technologies, engineering and telecommunication. The adjective \u2018medical\u2019, in its turn, indicates the relationship of the science to researches and practical activities of specialists in the field of MI that are aimed at disease prevention, diagnosis and treatment.<sup id=\"rdp-ebb-cite_ref-Collen86_3-0\" class=\"reference\"><a href=\"#cite_note-Collen86-3\" rel=\"external_link\">[3]<\/a><\/sup> Therefore, since 1974 till now the term MI has been widely used throughout the world and is present in the titles of such representative MI organizations as International Medical Informatics Association, American Medical Informatics Association or European Federation of Medical Informatics.\n<\/p><p>In the 1990s, however, the nominative word-combination \u2018biomedical informatics\u2019 (BMI) was more often collaterally used with the term MI in the scientific literature. Such a tendency in the terminology shift can have historically-grounded explanations. In particular, the American scientist E. Shortliffe considers that changes in the terminology and ways of its interpretations happened gradually and were induced by modern advanced technologies appearance and their integration into the society life-sustaining activities in the 1990s. They enabled to perform scientific experiments that gave the understanding of human disease nature at molecular and genetic levels.<sup id=\"rdp-ebb-cite_ref-Shortliffe13_2-1\" class=\"reference\"><a href=\"#cite_note-Shortliffe13-2\" rel=\"external_link\">[2]<\/a><\/sup> Thus, at the end of the 20th century the term \u2018biomedical informatics\u2019 appeared indicating the connection of the specialty not only with medicine but also with biological chemistry, histology, genetics and embryology as the subfields of biomedicine.\n<\/p><p><i>The third stage<\/i> (the present one) is characterized by outlining certain specifics in the terminological system usage within the studied specialty due to information technology penetration into various areas of biomedicine and health care service. It is exactly that period when such terms as \u2018clinical informatics\u2019, \u2018nursing informatics\u2019, \u2018dental informatics\u2019, \u2018health care informatics\u2019, etc. appeared specifying the specialists\u2019 professional functioning. Undoubtedly, hierarchy of the given terms within the system of notions concerned with MI is not questioned. On the other hand, the analysis of the terminology operated by the leading world MI organizations enables to conclude about substantial peculiarities of their terminology interpretation.\n<\/p><p>The distinct hierarchy within the field is noticed in the terminological system developed by the American school, which is represented by American Medical Informatics Association. The American association uses the term \u2018biomedical informatics\u2019 (BMI) in its operation as the basic one for determination of the interdisciplinary specialty that involves biomedicine and informatics.<sup id=\"rdp-ebb-cite_ref-Kulikowski12_4-0\" class=\"reference\"><a href=\"#cite_note-Kulikowski12-4\" rel=\"external_link\">[4]<\/a><\/sup> Hereby, BMI is considered as the applied branch of informatics that develops methodology, technical means and theoretical basis for biomedical information application in order to promote research conduction, health improvement as well as qualitative health care service delivery.<sup id=\"rdp-ebb-cite_ref-AMIA14_5-0\" class=\"reference\"><a href=\"#cite_note-AMIA14-5\" rel=\"external_link\">[5]<\/a><\/sup>\n<\/p><p>According to the Glossary found on the AMIA\u2019s site, trained specialists regard BMI as a general term for the scientific discipline with many other related terms to be the spheres of its practical application. Bioinformatics, in particular, indicates the specialty aimed at studying the usage of technical means of information visualization and presentation in biological systems, especially in molecular biology, genomics and proteomics. The usage of the word-combination \u2018health informatics\u2019 is specific to the applied investigations and practical activities of BMI specialists carried out in the two other subfields \u2013 <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics#Clinical_Informatics\" title=\"Health informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8758a5f8f4034718e18d576916e2a7dc\">clinical informatics<\/a> and <a href=\"https:\/\/www.limswiki.org\/index.php\/Public_health_informatics\" title=\"Public health informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"f0372a80f101e9f6fd00490dc1ebcedd\">public health informatics<\/a>. The goal of the two latter ones consists in ensuring quality of medical service delivery and promoting public health care efficacy.<sup id=\"rdp-ebb-cite_ref-AMIAGloss14_6-0\" class=\"reference\"><a href=\"#cite_note-AMIAGloss14-6\" rel=\"external_link\">[6]<\/a><\/sup> In its turn, the terminological unit \u2018medical informatics\u2019 is used in the context of clinical informatics dealing with disease diagnosis and treatment and has been designed for doctors. \u2018Nursing informatics\u2019 and \u2018dental informatics\u2019 should also be mentioned here as clinical informatics branches concerned with the development of information processing systems that are intended to improve work efficiency of the nursing and dentist staff.<sup id=\"rdp-ebb-cite_ref-Kulikowski12_4-1\" class=\"reference\"><a href=\"#cite_note-Kulikowski12-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/p><p>In contrast to the USA experience in the MI terminology usage, the term \u2018health informatics\u2019 (HI) is commonly used in Canada. It is also worth mentioning that Canada\u2019s Health Informatics Association (COACH) employs it in wider context if compared to the interpretation of the same term by AMIA. The term determines the specialty which is the convergence of clinical practice, biomedical information management, information technologies as well as management practice in medicine and health care system aimed at delivery of quality medical service, disease prevention, popularization of healthy lifestyle, etc.<sup id=\"rdp-ebb-cite_ref-CHIA12_7-0\" class=\"reference\"><a href=\"#cite_note-CHIA12-7\" rel=\"external_link\">[7]<\/a><\/sup>\n<\/p><p>In view of the above said, the HI specialists in Canada should:\n<\/p>\n<ul><li> be knowledgeable about the key aspects of the clinical practice that covers the wide range of activities intended for disease diagnosis, prophylaxis and patient treatment;<\/li>\n<li> possess practical skills in information management that implies biomedical information collection, processing, storage and interpretation;<\/li>\n<li> be proficient in applying information technologies for the effective information storage or processing;<\/li>\n<li> acquire competency of management practice that provides a HI specialist with means for medical information system design and monitoring in order to use biomedical information rationally. <\/li><\/ul>\n<p>It should also be noted that the terms \u2018health informatics\u2019 and \u2018medical informatics\u2019 are practically used in the same contexts which is apparent from both the sites of the Canadian higher educational institutions proposing programs of professional training in HI and from publications of the Canadian scientists J. Moehr and A. Grant. Besides, some researchers suppose the adjective \u2018medical\u2019 to be connected with disease diagnosis, prevention or treatment, thus forming the field\u2019s \u2018negative\u2019 profile. The word \u2018health\u2019 used before \u2018informatics\u2019, to the contrary, shifts the accent and points that the main goal of HI as a specialty is optimal information usage by the field\u2019s specialists for improving health of both the individual patient and the population as a whole.<sup id=\"rdp-ebb-cite_ref-Dalrymple11_8-0\" class=\"reference\"><a href=\"#cite_note-Dalrymple11-8\" rel=\"external_link\">[8]<\/a><\/sup>\n<\/p><p>The analysis of the terminology formation by European Federation of MI (EFMI) as the formal representative of national MI organizations of the European countries has revealed the preference to \u2018medical informatics\u2019 as the terminological unit for common use. In fact, the term denotes MI both as the science and umbrella term for a number of specialties related to health care and informatics, thus indicating that the terminological system used by EFMI is, to some extent, similar to the above-mentioned AMIA\u2019s terminology interpretation.\n<\/p><p>There is also no coherence in the MI terminology usage by Ukrainian Association for \u201cComputer Medicine\u201d, a national member of the international MI organization IMIA. On its official web site, the word-combination \u2018\u043c\u0435\u0434\u0438\u0447\u043d\u0430 \u0456\u043d\u0444\u043e\u0440\u043c\u0430\u0442\u0438\u043a\u0430\u2019 is widely used without any definitions proposed for the term. Nevertheless, the association clearly states that its goals are directed at the implementation of new medical information systems, the design of electronic medical history, reformation of health care system in Ukraine as well as the development of biotechnical systems and new software for medical institutions.<sup id=\"rdp-ebb-cite_ref-UACM14_9-0\" class=\"reference\"><a href=\"#cite_note-UACM14-9\" rel=\"external_link\">[9]<\/a><\/sup> Thus, the Ukrainian MI specialists mostly operate the term medical informatics (\u2018\u043c\u0435\u0434\u0438\u0447\u043d\u0430 \u0456\u043d\u0444\u043e\u0440\u043c\u0430\u0442\u0438\u043a\u0430\u2019 in Ukrainian) as a general one. Moreover, the terminological unit is found in the specialty title \u201cMedical and Biological Informatics and Cybernetics\u201d (\u201c\u041c\u0435\u0434\u0438\u0447\u043d\u0430 \u0442\u0430 \u0411\u0456\u043e\u043b\u043e\u0433\u0456\u0447\u043d\u0430 \u0456\u043d\u0444\u043e\u0440\u043c\u0430\u0442\u0438\u043a\u0430 \u0456 \u043a\u0456\u0411\u0435\u0440\u043d\u0435\u0442\u0438\u043a\u0430\u201d in Ukrainian) compiled by the Ukrainian MI association for the Ukrainian Higher Attestation Commission as well as in the names of academic disciplines in the curricula of the Ukrainian higher medical educational establishments.\n<\/p><p>All the above-mentioned MI national organizations are the members of International MI Association (IMIA). It plays a crucial role in information technology implementation into medicine and supports researches aimed at effective informatisation of the health care system as well as development of educational policy in professional training of the MI workforce. IMIA uses the term \u2018biomedical informatics\u2019 in its widest meaning covering all possible determining adjectives including \u2018health\u2019, \u2018medical\u2019, \u2018nursing\u2019, \u2018bio\u2019, etc.<sup id=\"rdp-ebb-cite_ref-IMIA07_10-0\" class=\"reference\"><a href=\"#cite_note-IMIA07-10\" rel=\"external_link\">[10]<\/a><\/sup> Besides, in IMIA recommendations on professional education in MI, the word-combination \u2018biomedical and health informatics\u2019 is used throughout the text<sup id=\"rdp-ebb-cite_ref-Mantas10_11-0\" class=\"reference\"><a href=\"#cite_note-Mantas10-11\" rel=\"external_link\">[11]<\/a><\/sup> We consider that such a broad interpretation of the term may indicate that IMIA as the international MI organization makes efforts to unify the field\u2019s terminology in order to avoid further misunderstandings and ambiguities. Secondly, simultaneous usage of the terms \u2018biomedical and health informatics\u2019 proves their equivalency, with difference consisting mainly in the term preferences restricted to a specific geographic area.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>The MI specialty is relatively young and is continuously changing, therefore the unified strategy of using the associated terminology has not been developed yet. It is known that there is a branching system of terms that determine the MI subfields. There are vague boundaries between them and it is not always possible to track where one field finishes and the other one begins. Moreover, the list of adjectives preceding the word \u2018informatics\u2019 is not limited only to such determinants as \u2018biomedical\u2019, \u2018medical\u2019, \u2018health\u2019, \u2018public health care\u2019 or \u2018clinical\u2019. As the science constantly develops and its further specializations continues, new subspecialties within the existing ones appear, for example imaging informatics, pathology informatics, consumer health informatics etc.<sup id=\"rdp-ebb-cite_ref-Hersh09_1-1\" class=\"reference\"><a href=\"#cite_note-Hersh09-1\" rel=\"external_link\">[1]<\/a><\/sup> In future, this list may be prolonged with new terms due to advancement of information technologies and the further MI specialization.\n<\/p><p>In summary, the analysis of the terminological system related to the notion of MI in the context of professional education has enabled to distinguish three terms that are used as equivalents and serve as umbrella terms \u2013 \u2018biomedical informatics\u2019, \u2018medical informatics\u2019 and \u2018health informatics\u2019. They determine the field which specialists are concerned with practical aspects of biomedical information storage, collection, circulation, mining, retrieval, exchange and security, etc. One of the noticed differences among them is the preferences in term usage according to the principle of geographical area. In particular,\n\u2018biomedical informatics\u2019 is of common usage in the USA, \u2018health informatics\u2019 \u2013 in Canada and \u2018medical informatics\u2019 is mostly used in the European countries as well as in Ukraine. Consequently, the term \u2018medical informatics\u2019 has historically become widely used by scientists and teaching staff and serves as generally applicable terminological unit to denote the key theories, concepts and techniques for information technology usage in health care system, biomedicine and hence in the system of professional training of the MI specialists.\n<\/p><p>The comparative analysis of the MI specialty terminology in the developed English-speaking countries is considered an essential ground for further research into the foreign countries\u2019 experience in professional training of the MI specialists for the purposes of its progressive ideas implementation into the educational system of Ukraine.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<ol class=\"references\">\n<li id=\"cite_note-Hersh09-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Hersh09_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-Hersh09_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hersh, W. R. (2009). \"A Stimulus to Define Informatics and Health Information Technology\". <i>BMC Medical Informatics and Decision Making<\/i> <b>9<\/b> (24): 6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-9-24\" target=\"_blank\">10.1186\/1472-6947-9-24<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Stimulus+to+Define+Informatics+and+Health+Information+Technology&rft.jtitle=BMC+Medical+Informatics+and+Decision+Making&rft.aulast=Hersh%2C+W.+R.&rft.au=Hersh%2C+W.+R.&rft.date=2009&rft.volume=9&rft.issue=24&rft.pages=6&rft_id=info:doi\/10.1186%2F1472-6947-9-24&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Shortliffe13-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Shortliffe13_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-Shortliffe13_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Shortliffe, E. H. (16 April 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.med.nyu.edu\/medicine\/education\/grand-rounds\/mgr-edward-shortliffe-md-phd-video\" target=\"_blank\">\"Biomedical Informatics: Assessing its Evolving Role as both Science and Application\"<\/a>. NYU Langone Medical Center, Department of Medicine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.med.nyu.edu\/medicine\/education\/grand-rounds\/mgr-edward-shortliffe-md-phd-video\" target=\"_blank\">http:\/\/www.med.nyu.edu\/medicine\/education\/grand-rounds\/mgr-edward-shortliffe-md-phd-video<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 19 November 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Biomedical+Informatics%3A+Assessing+its+Evolving+Role+as+both+Science+and+Application&rft.atitle=&rft.aulast=Shortliffe%2C+E.+H.&rft.au=Shortliffe%2C+E.+H.&rft.date=16+April+2013&rft.pub=NYU+Langone+Medical+Center%2C+Department+of+Medicine&rft_id=http%3A%2F%2Fwww.med.nyu.edu%2Fmedicine%2Feducation%2Fgrand-rounds%2Fmgr-edward-shortliffe-md-phd-video&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Collen86-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Collen86_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Collen, M. F. (1986). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1307150\" target=\"_blank\">\"Origins of Medical Informatics\"<\/a>. <i>The Western Journal of Medicine<\/i> (145): 778\u2013785. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1307150\/\" target=\"_blank\">PMC1307150<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1307150\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1307150<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Origins+of+Medical+Informatics&rft.jtitle=The+Western+Journal+of+Medicine&rft.aulast=Collen%2C+M.+F.&rft.au=Collen%2C+M.+F.&rft.date=1986&rft.issue=145&rft.pages=778%E2%80%93785&rft_id=info:pmc\/PMC1307150&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1307150&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Kulikowski12-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Kulikowski12_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-Kulikowski12_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kulikowski, C. A.; Shortliffe, E. H.; Currie, L. M. (2012). \"AMIA Board White Paper: Definition of Biomedical Informatics and Specification of Core Competences for Graduate Education in the Discipline\". <i>Journal of the American Medical Informatics Association<\/i> (19): 931\u2013938. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2012-001053\" target=\"_blank\">10.1136\/amiajnl-2012-001053<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=AMIA+Board+White+Paper%3A+Definition+of+Biomedical+Informatics+and+Specification+of+Core+Competences+for+Graduate+Education+in+the+Discipline&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Kulikowski%2C+C.+A.%3B+Shortliffe%2C+E.+H.%3B+Currie%2C+L.+M.&rft.au=Kulikowski%2C+C.+A.%3B+Shortliffe%2C+E.+H.%3B+Currie%2C+L.+M.&rft.date=2012&rft.issue=19&rft.pages=931%E2%80%93938&rft_id=info:doi\/10.1136%2Famiajnl-2012-001053&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AMIA14-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AMIA14_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.amia.org\" target=\"_blank\">\"AMIA: Informatics Professionals. Leading the Way.\"<\/a>. AMIA. 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.amia.org\" target=\"_blank\">http:\/\/www.amia.org<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 November 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AMIA%3A+Informatics+Professionals.+Leading+the+Way.&rft.atitle=&rft.date=2014&rft.pub=AMIA&rft_id=http%3A%2F%2Fwww.amia.org&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AMIAGloss14-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AMIAGloss14_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.amia.org\/glossary\" target=\"_blank\">\"Glossary of Acronyms and Terms Commonly Used in Informatics\"<\/a>. AMIA. 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.amia.org\/glossary\" target=\"_blank\">http:\/\/www.amia.org\/glossary<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 14 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Glossary+of+Acronyms+and+Terms+Commonly+Used+in+Informatics&rft.atitle=&rft.date=2014&rft.pub=AMIA&rft_id=http%3A%2F%2Fwww.amia.org%2Fglossary&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CHIA12-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CHIA12_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Canada\u2019s Health Informatics Association (November 2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.coachorg.com\/en\/resourcecentre\/resources\/Health-Informatics-Core-Competencies.pdf\" target=\"_blank\">\"Health Informatics Professional Core Competencies\"<\/a> (PDF). p. 22<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.coachorg.com\/en\/resourcecentre\/resources\/Health-Informatics-Core-Competencies.pdf\" target=\"_blank\">https:\/\/www.coachorg.com\/en\/resourcecentre\/resources\/Health-Informatics-Core-Competencies.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 25 July 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Health+Informatics+Professional+Core+Competencies&rft.atitle=&rft.aulast=Canada%E2%80%99s+Health+Informatics+Association&rft.au=Canada%E2%80%99s+Health+Informatics+Association&rft.date=November+2012&rft.pages=p.+22&rft_id=https%3A%2F%2Fwww.coachorg.com%2Fen%2Fresourcecentre%2Fresources%2FHealth-Informatics-Core-Competencies.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Dalrymple11-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Dalrymple11_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dalrymple, P. W. (2011). \"Data, Information, Knowledge: The Emerging Field of Health Informatics\". <i>Bulletin of the American Society for Information Science and Technology<\/i> <b>37<\/b> (5): 41\u201344. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fbult.2011.1720370512\" target=\"_blank\">10.1002\/bult.2011.1720370512<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data%2C+Information%2C+Knowledge%3A+The+Emerging+Field+of+Health+Informatics&rft.jtitle=Bulletin+of+the+American+Society+for+Information+Science+and+Technology&rft.aulast=Dalrymple%2C+P.+W.&rft.au=Dalrymple%2C+P.+W.&rft.date=2011&rft.volume=37&rft.issue=5&rft.pages=41%E2%80%9344&rft_id=info:doi\/10.1002%2Fbult.2011.1720370512&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UACM14-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UACM14_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/uacm.kharkov.ua\/ukr\/index.shtml?uabout.htm\" target=\"_blank\">\"\u0423\u041a\u0420\u0410\u0407\u041d\u0421\u042c\u041a\u0410 \u0410\u0421\u041e\u0426\u0406\u0410\u0426\u0406\u042f \"\u041a\u041e\u041c\u041f'\u042e\u0422\u0415\u0420\u041d\u0410 \u041c\u0415\u0414\u0418\u0426\u0418\u041d\u0410\"\"<\/a>. \u0423\u0410\u041a\u041c. 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/uacm.kharkov.ua\/ukr\/index.shtml?uabout.htm\" target=\"_blank\">http:\/\/uacm.kharkov.ua\/ukr\/index.shtml?uabout.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 November 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=%D0%A3%D0%9A%D0%A0%D0%90%D0%87%D0%9D%D0%A1%D0%AC%D0%9A%D0%90+%D0%90%D0%A1%D0%9E%D0%A6%D0%86%D0%90%D0%A6%D0%86%D0%AF+%22%D0%9A%D0%9E%D0%9C%D0%9F%27%D0%AE%D0%A2%D0%95%D0%A0%D0%9D%D0%90+%D0%9C%D0%95%D0%94%D0%98%D0%A6%D0%98%D0%9D%D0%90%22&rft.atitle=&rft.date=2014&rft.pub=%D0%A3%D0%90%D0%9A%D0%9C&rft_id=http%3A%2F%2Fuacm.kharkov.ua%2Fukr%2Findex.shtml%3Fuabout.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IMIA07-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IMIA07_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">International Medical Informatics Association (11 June 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.imia-medinfo.org\/new2\/sites\/default\/files\/IMIA_Strategic_Plan_2007final.pdf\" target=\"_blank\">\"Towards IMIA 2015 \u2013 The IMIA Strategic Plan\"<\/a> (PDF). p. 15<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.imia-medinfo.org\/new2\/sites\/default\/files\/IMIA_Strategic_Plan_2007final.pdf\" target=\"_blank\">http:\/\/www.imia-medinfo.org\/new2\/sites\/default\/files\/IMIA_Strategic_Plan_2007final.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 27 July 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Towards+IMIA+2015+%E2%80%93+The+IMIA+Strategic+Plan&rft.atitle=&rft.aulast=International+Medical+Informatics+Association&rft.au=International+Medical+Informatics+Association&rft.date=11+June+2007&rft.pages=p.+15&rft_id=http%3A%2F%2Fwww.imia-medinfo.org%2Fnew2%2Fsites%2Fdefault%2Ffiles%2FIMIA_Strategic_Plan_2007final.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mantas10-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Mantas10_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mantas, J.; Ammenwerth, E.; Demiris, G. (2010). \"Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics \u2013 1st revision\". <i>Methods of Information in Medicine<\/i> <b>49<\/b> (2): 105\u2013120. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3414%2FME5119\" target=\"_blank\">10.3414\/ME5119<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Recommendations+of+the+International+Medical+Informatics+Association+%28IMIA%29+on+Education+in+Biomedical+and+Health+Informatics+%E2%80%93+1st+revision&rft.jtitle=Methods+of+Information+in+Medicine&rft.aulast=Mantas%2C+J.%3B+Ammenwerth%2C+E.%3B+Demiris%2C+G.&rft.au=Mantas%2C+J.%3B+Ammenwerth%2C+E.%3B+Demiris%2C+G.&rft.date=2010&rft.volume=49&rft.issue=2&rft.pages=105%E2%80%93120&rft_id=info:doi\/10.3414%2FME5119&rfr_id=info:sid\/en.wikipedia.org:Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. The original article puts the references in alphabetical order; however, we've ordered them in the order they appear in the article. We also added PMCID and DOI when they were missing from the original reference. No other modifications were made in accordance with the \"no derivatives\" portion of the distribution license.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210229\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.298 seconds\nReal time usage: 0.321 seconds\nPreprocessor visited node count: 9106\/1000000\nPreprocessor generated node count: 29906\/1000000\nPost\u2010expand include size: 62383\/2097152 bytes\nTemplate argument size: 22091\/2097152 bytes\nHighest expansion depth: 17\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 297.890 1 - -total\n 62.65% 186.639 11 - Template:Citation\/core\n 38.17% 113.707 5 - Template:Cite_journal\n 29.92% 89.122 6 - Template:Cite_web\n 22.47% 66.942 1 - Template:Infobox_journal_article\n 21.63% 64.438 1 - Template:Infobox\n 13.17% 39.242 80 - Template:Infobox\/row\n 4.50% 13.399 5 - Template:Citation\/identifier\n 3.55% 10.587 11 - Template:Citation\/make_link\n 1.54% 4.602 11 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7630-0!*!0!!en!*!* and timestamp 20181213210229 and revision id 23415\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","80b7c208090cc5cb594724fd09a469c6_images":[],"80b7c208090cc5cb594724fd09a469c6_timestamp":1544734949,"bcdf1ad9d7bcb8661bc008ff0ca9fb02_type":"article","bcdf1ad9d7bcb8661bc008ff0ca9fb02_title":"Making big data useful for health care: A summary of the inaugural MIT Critical Data Conference (Badawi et al. 2014)","bcdf1ad9d7bcb8661bc008ff0ca9fb02_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference","bcdf1ad9d7bcb8661bc008ff0ca9fb02_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Making big data useful for health care: A summary of the inaugural MIT Critical Data Conference\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nMaking big data useful for health care: A summary of the inaugural MIT Critical Data ConferenceJournal\n \nJMIR Medical InformaticsAuthor(s)\n \nBadawi, O.; Brennan, T.; Celi, L.A.; Feng, M.; Ghassemi, M.; Ippolito, A.; Johnson, A.; Mark, R.G.; Mayaud, L.; Moody, G.;\r\nMoses, C.; Naumann, T.; Nikore, V.; Pimentel, M.; Pollard, T.J.; Santos, M.; Stone, D.J.; Zimolzak, A.Author affiliation(s)\n \nMassachusetts Institute of TechnologyPrimary contact\n \nLeo Anthony Celi, MD, MPH, MS - Phone: 1.617.253.7937; Email: lceli@mit.eduEditors\n \nEysenbach, G.Year published\n \n2014Volume and issue\n \n2 (2)Page(s)\n \ne22DOI\n \n10.2196\/medinform.3447ISSN\n \n2291-9694Distribution license\n \nCreative Commons Attribution 2.0Website\n \nhttp:\/\/medinform.jmir.org\/2014\/2\/e22\/Download\n \nhttp:\/\/medinform.jmir.org\/article\/download\/medinform_v2i2e22\/2 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 The problem \n4 Secondary usage of clinical data \n5 Connecting patients, providers, and payers \n6 Creating a data-driven learning system \n7 Physician culture as a barrier to spread of innovation \n8 The role of industry in the data revolution in health care \n9 The unreasonable effectiveness of data \n10 The story of MIMIC: Open-access critical care data \n11 Opportunities and challenges in wearable sensor datasets \n12 Big data, genomics, and public health \n13 The pitfalls and potential of big data in health care \n14 Comparative effectiveness using big data \n15 Conclusions \n16 Acknowledgements \n17 Conflict of interests \n18 References \n19 Abbreviations \n20 Notes \n\n\n\nAbstract \nWith growing concerns that big data will only augment the problem of unreliable research, the Laboratory of Computational Physiology at the Massachusetts Institute of Technology organized the Critical Data Conference in January 2014. Thought leaders from academia, government, and industry across disciplines \u2014 including clinical medicine, computer science, public health, informatics, biomedical research, health technology, statistics, and epidemiology \u2014 gathered and discussed the pitfalls and challenges of big data in health care. The key message from the conference is that the value of large amounts of data hinges on the ability of researchers to share data, methodologies, and findings in an open setting. If empirical value is to be from the analysis of retrospective data, groups must continuously work together on similar problems to create more effective peer review. This will lead to improvement in methodology and quality, with each iteration of analysis resulting in more reliability.\nKeywords: big data; open data; unreliable research; machine learning; knowledge creation\n\nIntroduction \nFailure to store, analyze, and utilize the vast amount of data generated during clinical care has restricted both quality of care and advances in the practice of medicine. Other industries, such as finance and energy, have already embraced data analytics for the purpose of learning. While such innovations remain relatively limited in the clinical domain, interest in \u201cbig data in clinical care\u201d has dramatically increased. This is due partly to the widespread adoption of electronic medical record (EMR) systems and partly to the growing awareness that better data analytics are required to manage the complex enterprise of the health care system. For the most part, however, the clinical enterprise has not had to address the problems particular to \u201cbig data\u201d because it has not yet satisfactorily addressed more fundamental data management issues. It is now becoming apparent that we are on the cusp of a great transformation that will incorporate data and data science integrally within the health care domain. In addition to the necessary major digital enhancements of the retrospective analyses that have variably been in place, real time and predictive analytics will also become ubiquitous core functionalities in the more firmly data-based environment of the (near) future. The initial Massachusetts Institute of Technology (MIT) Critical Data Conference was conceived and conducted to address the many data issues involved in this important transformation.[1][2]\nIncreasing interest in creating the clinical analog of \u201cbusiness intelligence\u201d has made evident the necessity of developing and nurturing a clinical culture that can manage and translate data-based findings, including those from \u201cbig data\u201d studies. Combining this improved secondary use of clinical data with a data-driven approach to learning will enable this new culture to close the clinical data feedback loop facilitating better and more personalized care. Authors have noted several hallmarks of \u201cbig data\u201d: very large datasets, a large number of unrelated and\/or unstructured datasets, or high speed or low latency of data creation.[3][4] The intensive care unit (ICU) provides a potent example of a particularly data rich clinical domain with the potential for both clinical and financial benefits if these large amounts of data can be harnessed and systematically leveraged into guiding practice. Thus, we use the term \u201cCritical Data\u201d to refer to big data in the setting of the ICU.\nThis paper summarizes the lectures and group discussions that took place during the recent Critical Data Conference at MIT, Cambridge MA, on January 7, 2014. The conference was the second part of a two-part event that brought together clinicians, data scientists, statisticians and epidemiologists.\nThe event opened with a \u201cdata marathon\u201d on January 3-5, 2014 (Figure 1), which brought together teams of data scientists and clinicians to mine the Multiparameter Intelligent Monitoring in Intensive Care (MIMIC) database (version II). MIMIC II is an open-access database consisting of over 60,000 recorded ICU stays from the adult intensive care units at the Beth Israel Deaconess Medical Center (BIDMC) in Boston, MA.[5] Over 100 people participated in the two-day data marathon, and posters of the projects were displayed at the Critical Data Conference.\nThe Critical Data Conference on January 7 was an approximately ten-hour program comprising two keynote addresses (Jeffrey Drazen, MD and John Ioannidis, MD, PhD), seven individual lectures, three panel discussions, and two poster sessions (Figure 2). The overall conference theme was meaningful secondary use of big data from critical care settings. Materials from the conference (program, slides, and videos) are available online at the MIT Critical Data conference site.[6]\n\n\n\nFigure 1. Presentation at the Critical Data Marathon. Photo credit: Andrew Zimolzak.\n\n\n\nFigure 2. Critical Data poster session. Photo credit: Andrew Zimolzak.\nThe problem \nIn his keynote address, Jeffrey Drazen, MD, Editor-in-Chief of the New England Journal of Medicine, noted that the number of evidence-based recommendations built on randomized controlled trials (RCTs), the current gold standard for data quality, is insufficient to address the majority of clinical decisions. Subsequently, clinicians are often left to practice medicine \u201cblindly.\u201d Without the knowledge generation required to capture the decisional factors involved in realistic clinical scenarios, clinical decision making is often less data-driven than determined by the \u201cplay of chance\u201d buttressed by past experience. Historically, a doctor took a history, performed a physical examination and made a diagnosis based on what he or she observed. As technology and medical theory progressed, knowledge such as laboratory and imaging modalities helped mitigate chance in the diagnosis of disease. Rote application of existing knowledge is not enough, as physicians want to establish causality. Until now this has been done with theories, but moving forward, theories will be inadequate unless they are confirmed, translated to practice, and systematically disseminated in clinical practice.\nThis trial-and-error process continues today because data generated from routine care is most often not captured and is rarely disseminated for the purpose of improving population health. Even in information-rich care settings like the ICU, the knowledge necessary to mitigate the play of chance is lacking.[7][8] As such, the ICU provides a fertile ground for potential improvement. Specifically, Drazen suggested a potential role for clinical data mining to answer questions that cannot be answered using RCTs.[9] This approach would likely yield benefits both more quickly and with fewer resources.\nDrazen concluded with the question \u201cAt what point is data good enough?\u201d Documented associations may be strong but not sufficiently \u201cproven\u201d to establish causality. Drazen drew a comparison with experimental physicists who hone future studies on the work of theorists as well as on prior experimental results: biomedical informaticians can identify meaningful associations that can then guide design of new RCTs where data quality can be increased by controlling for potential confounders. This will require cross-disciplinary collaboration of frontline clinicians, medical staff, database engineers and biomedical informaticians, in addition to strong partnerships with health information system vendors in order to close the loop from knowledge discovery during routine care to the real time application of best care for populations.\n\nSecondary usage of clinical data \nCharles Safran, MD, MS, Chief of the Division of Clinical Computing at the BIDMC and Harvard Medical School, spoke next, sharing the dream of evidence-based medicine (EBM): the ideal situation in which quality evidence would exist to guide clinicians through all the conundrums faced on a near-daily basis (eg, which test to order, how to interpret the test results, and what therapy to institute). For the last half-century, prospective RCTs have been the gold standard in EBM. Safran noted, as Drazen had, that such trials suffer from a number of limitations including economic burdens and design limitations. An RCT can only address a severely limited bundle of particularly well-posed clinical questions. For many clinical situations, it is either unethical or even impossible to proceed with an RCT. Furthermore, the inclusion and exclusion criteria often limit the generalizability of an RCT study and, given the time it normally takes to run an RCT, it is very difficult for these studies to remain current with the rapidly evolving practice of medicine.\nCan another approach avoid at least some of the limitations of RCTs? Safran suggested that retrospective observational studies (ROS) utilizing EMR data are a promising avenue for generating EBM. Digital records contain extensive clinical information including medical history, diagnoses, medications, immunization dates, allergies, radiology images, and laboratory and test results. Consequently, routinely collected EMR data contains the rich, continuous and time-sensitive information needed to support clinical decision making and evidence generation.[10] However, despite the many potential benefits, Safran pointed out that secondary use of EMR data is still subject to limitations: EMR data were not collected primarily for the purpose of evidence generation and data analytics but for real-time and longitudinal patient care.[11] As a result, EMR data are often poorly structured, disorganized, unstandardized, and contaminated with errors, artifacts, and missing values.\nSafran echoed one of Drazen\u2019s points by proposing that we should combine the usage of prospective RCTs and ROS in such a way that each complements the limitations of the other. Furthermore, he suggested the possibility of incorporating additional novel sources of data such as social media data, health data from portable sensors, and genetic data. While there are many barriers to establishing such a comprehensive framework, a big data picture of clinical, genetic, and treatment variables holds promise in revolutionizing diagnosis and treatment.\n\nConnecting patients, providers, and payers \nFor John Halamka, MD, MS, the Chief Information Officer of BIDMC, working with big data in hospital systems is hugely challenging but at the same time holds tremendous promise in providing more meaningful information to help clinicians treat patients across the continuum of care. In his position, Halamka has been tasked to aggregate data in novel ways in order to provide better care for BIDMC\u2019s patient population. One opportunity for furthering \u201cbig data in health care\u201d is to normalize the data collected via their EMR system and store it in large, centralized databases. In turn, analytic tools can then be applied to identify and isolate the quality data reporting measures required to participate as an Accountable Care Organization (ACO) under the Affordable Care Act.\nHalamka emphasized that building these large datasets does not intrinsically provide value from the start, stating that \u201cworkflow is disparate, the vocabulary is disparate, and the people are disparate.\u201d Therefore, the normalization of data and its distillation into standard schemas are difficult due to discrepancies across longitudinal data. Further, since each vendor models concepts differently, there must be an emphasis on developing a \u201cleast common denominator\u201d concept map across vendors\u2019 offerings.\nNevertheless, through this normalization effort, doctors can utilize \u201cscorecards\u201d to evaluate their own patient population within and across the different payment models, such as Blue Cross Blue Shield\u2019s Alternative Quality Contact measures, the Centers for Medicare and Medicaid (CMS) Physician Quality Reporting System measures, and the CMS ACO measures. In addition, physicians can query this dataset to identify the most effective treatment regimes. However, such queries do pose privacy and security issues in the hospital setting, and these risks are further complicated by hospital staff utilizing personal mobile devices such as cell phones, laptops, and tablets.\n\nCreating a data-driven learning system \nThe problem posed to the first panel (Figure 3), comprising Gari Clifford, PhD, Perren Cobb, MD, and Joseph Frassica, MD, and moderated by Leo Anthony Celi, MD, MS, MPH, was how to create a data-driven learning system in clinical practice.[8] Privacy concerns were cited as the central barrier, as there is a tradeoff between re-identification risk and the value of sharing. Furthermore, recent work shows patients are reluctant to share for certain purposes such as marketing, pharmaceutical, and quality improvement measures, indicating a need for public education about the benefits of data sharing and that shared data can be utilized without being used for marketing and other unwanted purposes.[2]\nThere is also tension between intellectual property rights and transparency. Resolution of this may require collaboration between government, industry and academic institutions, as seen with the US Critical Illness and Injury Trials Group.[12] There is also a risk that data sharing will make authors reluctant to write audacious or unconventional papers (as did Reinhart and Rogoff[13]), if data sharing puts such papers at perceived higher risk of refutation (such as the refutation of Herndon et al[14]).\nFinally, the panel raised concerns about the high quantity but perceived low quality of the data that is actually captured. While there is hope that automatically captured data may be more accurate than manually entered data, there is also some risk that doing so will introduce additional noise, furthering the problem of quantity over quality. This concern poses the challenge of capturing more and higher quality data in order to promote reproducibility. Panelists observed that multidisciplinary conferences like the Critical Data Conference are especially beneficial in this regard, as they provide an opportunity for clinicians and data scientists to better understand the relation between real-world activity and the data that such activity generates.\n\n\n\nFigure 3. Data-driven learning system panel. Photo credit: Andrew Zimolzak.\nPhysician culture as a barrier to spread of innovation \nIn the following panel moderated by critical care physician Leo Anthony Celi, MD, MS, MPH, fellow intensivists Djillali Annane, MD, PhD, Peter Clardy, MD and Taylor Thompson, MD reflected on the barriers presented by the current clinician culture toward the goal of data-driven innovation in medicine (Figure 4). The panelists observed that historically, EBM was perceived to be incompatible with well-established observational trials and experience, perhaps instilling a residual degree of resistance. Consequently, echoing Safran\u2019s sentiments, it will be increasingly important that \u201cbig data\u201d is understood as a complement to RCTs and (patho)physiologic studies. Furthermore, condensing and filtering the vast quantity of data to make it applicable at the bedside will be key to adoption. The specific inclusion of clinicians during the design process will help to deter the creation of tools that inundate staff with extraneous information and burdensome extra tasks. Likewise the incorporation of \u201cbig data\u201d into medical education, in a way that students and resident trainees will be able to understand its importance in both everyday care and expediting research, is vital.\nWhile the panel agreed that more evidence is required to determine whether big data can facilitate comparative effectiveness research, it was acknowledged that it is necessary to investigate this alternative since RCTs do not, and will not, provide answers to an important fraction of the decisions required on a daily basis. Scaling up RCTs to account for the thousands of decisions each day is not feasible, so big data approaches may provide the most effective way to fill these gaps. For example, three groups currently leading clinical trials research in the analysis of fluid resuscitation in critically ill patients have collaborated to create a common database architecture to allow for individual patient meta-analysis and for these trials to be evaluated in aggregate by an external monitoring committee.\n\n\n\nFigure 4. Physician culture panel. Photo credit: Andrew Zimolzak.\nThe role of industry in the data revolution in health care \nThere is concern that industry continues to view data as a potential source of revenue and would therefore be opposed to providing open access to what they consider to be proprietary data with business value. In the last panel discussion of the day, moderated by Ambar Bhattacharyya, MBA of Bessem Venture Partners, industry panelists Josh Gray, MBA of AthenaResearch, Enakshi Singh, MS of SAP, and Omar Badawi, PharmD, MPH of Philips Healthcare provided their insights on the topic.\nThey first addressed the issue of sharing databases freely, noting that concerns associated with data ownership are not restricted to industry. Similar problems and conflicts are observed among most stakeholders in health care data ownership: patients, hospitals, providers, payers, vendors, and academia. Generally speaking, industrial data owners want to protect their data from those who may use it competitively against them, share or sell the data to derive direct clinical value, or profit from possible insights. They also wish to avoid the overhead costs associated with sharing. They are interested in allowing society to leverage their data in order to make gains if, and only if, these other interests remain unaffected.\nThe costs for responsibly sharing secondary clinical data are not trivial. Although understanding the complexity of the data presents a significant challenge, understanding the workflow for entering data in the primary system is often even more complicated, requiring extensive support. Therefore, sharing secondary clinical data can be a costly initiative for industry, lowering its priority as a business objective. These challenges are further exacerbated when collaboration requires intellectual property agreements. Lack of an accepted standard practice for research agreements, coupled with an outdated patent system, creates barriers to collaboration that are rarely overcome. Many ideas for collaboration either take years to initiate or never come to fruition, due to challenges with developing de novo legal research agreements.\nWhile industry and researchers are not philosophically opposed to sharing data to ensure reproducibility, protections from the aforementioned concerns are critical. Can the data be shared without the risk of lost intellectual property? If not, the incentives for innovation may be minimized. Who will bear the costs for ensuring that the replicating team fully understands the nuances of the data? Who will prevent competitors or others with malicious intent from inappropriately labeling valid research as \u201cjunk science\u201d? Such underhanded interventions could introduce confusion around valid earlier findings and unfairly distract and denigrate the primary researchers.\nUltimately, there is a growing sense that data will become less of a commodity over time if governments continue to support the development and maintenance of open access research networks. As the scale and quality of these surpass those of privately owned databases, society will benefit as the obstacles to collaboration and the value of retaining private ownership diminish.\n\nThe unreasonable effectiveness of data \nPeter Szolovits, PhD from the MIT Computer Science and Artificial Intelligence Lab, highlighted how big data can often trump good, but smaller, data. Researchers at Google have been making a similar argument from a decade-long experience with natural language processing, showing that for some important tasks an order of magnitude growth in the size of a dataset leads to improvements in performance that can overshadow improvements in modeling technique.[15] They have also argued that discarding rare events is a bad idea because although these may be individually rare, they could prove to be significant later when examined on a much larger scale.\nIn the clinical world, patient state depends on complex pathophysiology dictated by genetic predispositions, environmental exposures, treatments, and numerous other factors. While, there are many potential ways to formulate clinical outcomes into complex statistical models, it is often the simple models that give the best, and most interpretable, results. Some clinicians and epidemiologists have already used large sources of observational data to improve clinical practice, especially in identifying drug side-effects, for example, for rosiglitazone[16], and rofecoxib.[17] Cox proportional hazard, na\u00efve Bayes, linear and logistic regression, and similar models can use aggregated variables to summarize dynamic variation without adding additional complexity.\n\nThe story of MIMIC: Open-access critical care data \nSince researchers who seek to create new clinical knowledge and tools are dependent upon the availability of relevant data, restricting access to data introduces barriers that stifle research progress. This simple principle has been at the heart of the research of Roger Mark, MD, PhD since the 1980s, a time when his work was focused on developing real-time arrhythmia analysis tools for use in patient monitoring.\nLike today, the norm for researchers in the 1980s was to privately maintain closed databases for their own benefit. So when Mark\u2019s team needed data, they began the painstaking work of creating their own resource, collecting electrocardiograms from patients at Boston\u2019s BIDMC and in the process, adding over 100,000 annotations. Breaking from tradition, they openly shared the dataset, reasoning that the more people who analyzed it, the better the overall understanding of arrhythmias would become. This dataset become known as the MIT-Beth Israel Hospital (MIT-BIH) Arrhythmia Database.[18]\nThe consequences were far-reaching. Not only did the MIT-BIH Database stimulate research interest, it generated beneficial competition and became a shared resource for evaluating algorithms. Researchers competed to see whose work performed best on the standard data, eventually leading to the database becoming part of a federal requirement for evaluation of commercial algorithms. This success led the team to develop further resources unique in their openness, including PhysioNet, a platform for open physiologic data, and MIMIC II, a rich database of critical care data.\nPhysioNet has over 50,000 registered users in over 120 countries and international recognition for accelerating the pace of discovery.[19] Mark attributes much of PhysioNet\u2019s success to the progressive mindset of the participating collaborators. Success has required not only funding, but also a collaborative approach among partnering clinicians, researchers, hospital technologists, and local ethics committees. Participation of commercial partners was also required, and obtained, in order to decrypt the proprietary data formats output by their monitoring systems.\nReproducibility of research and open data are increasingly getting the attention they deserve, but changing practice requires support at all levels.[8] For open technology to be embraced, funders must recognize the added value from a robust database infrastructure and allocate funds accordingly. Researchers too must embrace open approaches that perhaps challenge some of the underlying career reward systems. With changing attitudes, and by engaging the creative energy of the worldwide research community, Mark\u2019s hope is that MIMIC will become a multinational resource leading to the generation of new knowledge and new tools.\n\nOpportunities and challenges in wearable sensor datasets \nThe ability to create and capture data is exploding and offers huge potential for health organizations around the world to save both lives and scarce resources. Yadid Ayzenberg, PhD discussed the \u201cOpportunities and Challenges in Wearable Sensor Data\u201d in his talk, focusing on how the combination of wearable technology and the near ubiquitous access to mobile phones have the potential to address some of the challenges in health care. Examples include the works of Poh et al.[20] and Sano and Picard[21], which used a wrist-worn electrodermal activity and accelerometry biosensor for detection of convulsive seizures and sleep stages.\nWearable technologies provide a way to transition from a traditional aperiodic \u201csnapshot\u201d monitoring approach to a continuous and longitudinal monitoring paradigm, increase patients\u2019 engagement in their care, and facilitate doctor-patient interactions. Already massive amounts of personal health data are being generated through consumer devices such as mobile phones and wristbands that monitor sleeping patterns, exercise, stress, calorie consumption and more. In most instances, however, the data are stored on a per-device basis, and there are unsolved issues concerning data management, ownership, privacy, and misuse. The noise and artifacts in the data measured by wearable sensors also present an important challenge. New analytic methods that transform \u201cdirty data\u201d into good quality data are needed.\n\nBig data, genomics, and public health \nAccording to Winston Hide, PhD, the promise of a new economy based on data-driven discovery and decision-making is also motivating his own field of genomics. Advances in genome sequencing technology will allow the cost of sequencing a genome to be less than $100 in the near future. Consequently, it is estimated that by 2015, one million genomes will be digitally available with high expectations for public health benefits. However, Hide cautions that there is still a \u201cgenome variants\u201d problem to be solved. This was exemplified by the case of Kira Peikoff who was predicted to have a 20% above average risk of developing psoriasis by one commercial sequencing product, while predicted to have a 2% below average risk for the same condition by another.[22]\nVariations in the reporting of genomic characteristics have two potential root causes. First, there is a sampling problem caused by the use of different sequencing technologies, which introduces errors in evaluating genome assembly. Second, there is an interpretation problem in defining the role of genes in disease which compromises the prediction of clinical outcomes as determined by the single-nucleotide polymorphism analysis tools.\nThe availability of millions of genomes will allow completion of the catalogue of genes associated with a particular disease in genome-wide association studies. However, Hide maintains that finding clear drug targets requires the creation of an evolving catalogue of functions which would interpret complex gene pathways[23], and the selection of cohorts that would not only depend on ethnicity (the classic phenotype), but also on physiological and even molecular differences.\nThe application of genomic tests to public health will contribute to the transformation of physicians into data-centric specialists and pave the way for \u201cprecision medicine\u201d.[24] This challenging way of delivering health care calls for new strategies and tactics for translating research into clinical practice. These will likely include the creation of open-access genomic and clinical databases, use of a common scientific language[25] and (open) data access tools. Regulatory bodies, such as the Food and Drug Administration, will have a role in guaranteeing the standardization and reliability of diagnostics based on genetic tests. These tools must guarantee the reproducibility[8] of discovered genome signals and contribute to the improvement of online platforms that map genetic features to diseases and their treatment (eg, Cancer Genome Atlas; PharmGKB).\n\nThe pitfalls and potential of big data in health care \nProponents of big data have made grandiose claims of expanding human knowledge by orders of magnitude through empirical analysis and data mining, but as Stanford professor John Ioannidis, MD, PhD says, \u201cwith big data comes big problems\u201d. Ioannidis discussed the darker side of data analysis, in which bias has led a large proportion of published medical science to come to the wrong conclusion. Author of the most downloaded paper in PLOS Medicine, \u201cWhy Most Published Research Findings Are False\u201d[26], Ioannidis argued that most statistically significant results are likely to be false positives. For example, using the national drug and cancer registry database of Sweden, Ioannidis and colleagues found that almost one third of the 560 medications evaluated in isolation were associated with a higher cancer risk.\nAs Ioannidis highlights, the issue is not in the quantity of data we have. Increasing sample sizes is a huge boon to the medical field. The issue resides in a lack of transparency. When he reviews a paper published in a journal on a new dataset, his thoughts immediately drift to those studies that were not published. This is quantified in the so-called \u201cvibration of effects\u201d, where depending on the confounding variables for which adjustments are made, completely opposite conclusions can be drawn. For vitamin E, for example, adjusting for a certain subset of confounders led to the conclusion that it increases the relative risk of mortality, whereas adjusting for another equally plausible set of confounders gave the opposite result, that is, a reduction in mortality risk. This may explain why 90% of effects in RCTs were lower in subsequent published trials.[27]\n\nComparative effectiveness using big data \nLimitations aside for now, ROS do provide an opportunity to conduct comparative effectiveness studies on research questions that would be unlikely to be examined by an RCT, or would be inherently biased if an RCT were conducted. The illustrative example presented by Una-May O\u2019Reilly, PhD was the question of the potential benefit of diuretic use to accelerate removal of fluids given during resuscitation in ICU patients who have recovered from sepsis. Retrospective analysis would be easily marred by \u201cselection bias\u201d. In fact, if patients are allocated (not randomized) to two groups, treatment and non-treatment, it is very likely that the allocation would be done on the basis of patient condition and biased by clinical severity resulting in unreliable results.\nIn a ROS, the data consists of a series of days during which the treatment was administered (D+) or not (D-). Because these decisions were being made on a daily basis, it is even harder to capture the covariance structure. O\u2019Reilly refers to this as the \u201cNon-Decision Day Dilemma\u201d. In order to deal with this, the covariance structure has to account for time-varying information with respect to a specific day. It is easy to take the treatment day as a reference and align all patients who received treatments with respect to this event (D+). For non-treated patients, aligning time-series is more complicated as every day is essentially a \u201cnon-decision day\u201d. Considering every single day would result in a widely unbalanced dataset where the length-of-stay influences the individual contribution of each patient. To account for this, it is possible to randomly sample N negative days (D*-) and pair them up with the positive instances (D+) based on a statistical similarity criterion with respect to the time from admission. This is achieved by defining a propensity score for each patient for every day during their ICU stay. The propensity score thus enables appropriate cohort matching such that comparative effectiveness can be appropriately assessed.\nThis modest example illustrates the sort of robust and reliable statistical technique that evidence-based medicine requires. It can reduce sample noise and improve the reliability of conclusions, and it leads toward methodology standardization across studies. Beyond these local improvements, meta-studies will also be a requirement to validate any local finding. These are only possible with data sharing and open data initiatives such as the MIMIC-II initiative. One strength of this database is the dual culture and scientific activity it generates because data science can only fully benefit from collaboration between data scientists and domain experts (in this case, intensivist physicians).\n\nConclusions \nAlthough the future of \u201cbig data\u201d in health care remains unclear, its role will be undeniably important. This conference was effective in collating the broad range of perspectives on the many challenges facing EBM in the 21st Century. As several speakers suggested, one possible opportunity is to adopt a pragmatic approach to EBM, combining RCT and ROS. This combination may employ ROSs to fill the gaps where it is impractical, unlikely or impossible to conduct a RCT or to drive hypothesis generation for further RCT analysis.\nIt is also crucial to acknowledge that any ROS requires a multidisciplinary approach, integrating clinical knowledge with a broad range of data analytic skills ranging from biostatistics, machine learning, and signal processing to data mining. Encouraging a change in physician culture can likely be accomplished through updating education programs as well as by creating centers for excellence that can showcase the impact of ROS to the broader medical fraternity. These centers for excellence should host open, transparent, easily accessible data warehouses, which will facilitate study reproducibility and allow for a new wave of collaborative learning. Only by understanding the potential biases of any analysis, and fostering a system of normative data sharing, will the medical community be able to gain reliable knowledge from data, and produce research findings that do not turn out to be false.\n\nAcknowledgements \nThe Organizing Committee would like to thank the speakers and panelists, the sponsors and all the attendees of the MIT Critical Data Conference and Marathon held in January 2014. The MIT Critical Data Marathon and Conference 2014 was sponsored by SAP, Philips Healthcare, Quanttus, Goodwin Procter and WilmerHale.\n\nConflict of interests \nOmar Badawi is senior clinical scientist at Philips Healthcare. Christopher Moses is founder and CEO of Smart Scheduling, Inc. Louis Mayaud is research director of Mensia Technologies SA. Thomas Brennan is data scientist at AthenaHealth Research.\n\nReferences \n\n\u2191 Celi, L.A.; Mark, R.G.; Stone, D.J.; Montgomery, R.A. (June 2013). \"\"Big data\" in the intensive care unit: Closing the data loop\". American Journal of Respiratory and Critical Care Medicine 187 (11): 1157\u20131160. doi:10.1164\/rccm.201212-2311ED. PMC PMC3734609. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3734609 .   \n\n\u2191 2.0 2.1 Grande, D.; Mitra, N.; Shah, A.; Wan, F.; Asch, D.A. (October 2013). \"Public preferences about secondary uses of electronic health information\". JAMA Internal Medicine 173 (19): 1798\u20131806. doi:10.1001\/jamainternmed.2013.9166. PMC PMC4083587. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4083587 .   \n\n\u2191 McAfee, A.; Brynjolfsson, E. (October 2012). \"Big data: the management revolution\". Harvard Business Review 90 (10): 60\u20136, 68, 128. PMID 23074865.   \n\n\u2191 Bourne, P.E. (2014). \"What Big Data means to me\". Journal of the American Medical Informatics Association 21 (2): 194. doi:10.1136\/amiajnl-2014-002651. PMC PMC3932474. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3932474 .   \n\n\u2191 \"The MIMIC II project\". Massachusetts Institute of Technology. Archived from the original on 11 August 2014. http:\/\/www.webcitation.org\/6RkH0iGr1 . Retrieved 11 August 2014 .   \n\n\u2191 \"Critical data: Empowering big data in critical care\". Massachusetts Institute of Technology. Archived from the original on 20 August 2014. http:\/\/www.webcitation.org\/6Rxk41BBR . Retrieved 20 August 2014 .   \n\n\u2191 Freedman, D.H. (04 October 2010). \"Lies, damned lies, and medical science\". The Atlantic. Archived from the original on 11 August 2014. http:\/\/www.webcitation.org\/6RkHK88k4 . Retrieved 11 August 2014 .   \n\n\u2191 8.0 8.1 8.2 8.3 \"Trouble at the lab\". The Economist. 19 October 2013. Archived from the original on 11 August 2014. http:\/\/www.webcitation.org\/6RkHeZm6r . Retrieved 11 August 2014 .   \n\n\u2191 Moses, C.; Celi, L.A.; Marshall, J. (June 2013). \"Pharmacovigilance: an active surveillance system to proactively identify risks for adverse events\". Population Health Management 16 (3): 147\u2013149. doi:10.1089\/pop.2012.0100. PMID 23530466.   \n\n\u2191 Safran, C.; Bloomrosen, M.; Hammond, W.E.; Labkoff, S.; Markel-Fox, S.; Tang, P.C. (January 2007). \"Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper\". Journal of the American Medical Informatics Association 14 (1): 1\u20139. doi:10.1197\/jamia.M2273. PMC PMC2329823. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823 .   \n\n\u2191 Geissbuhler, A.; Safran, C.; Buchan, I.; Bellazzi, R.; Labkoff, S.; Eilenberg, K.; et al. (January 2013). \"Trustworthy reuse of health data: a transnational perspective\". International Journal of Medical Informatics 82 (1): 1\u20139. doi:10.1016\/j.ijmedinf.2012.11.003. PMID 23182430.   \n\n\u2191 Cobb, J.P.; Cairns, C.B.; Bulger, E.; Wong, H.R.; Parsons, P.E.; Angus, D.C.; et al. (August 2009). \"The United States critical illness and injury trials group: an introduction\". The Journal of Trauma 67 (2 Suppl): S159\u2013S160. doi:10.1097\/TA.0b013e3181ad3473. PMID 19667851.   \n\n\u2191 Reinhart, C.M.; Rogoff, K.S. (2011). \"A Decade of Debt - Working Paper 16827\" (PDF). National Bureau of Economic Research. Archived from the original on 11 August 2014. http:\/\/www.webcitation.org\/6RkHujrwl . Retrieved 11 August 2014 .   \n\n\u2191 Herndon, T.; Ash, M.; Pollin, R. (2013). \"Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff - Working Paper Series No. 322\" (PDF). Political Economy Research Institute. Archived from the original on 11 August 2014. http:\/\/www.webcitation.org\/6RkI0b5Rf . Retrieved 11 August 2014 .   \n\n\u2191 Halevy, A.; Norvig, P.; Pereira, F. (2009). \"The unreasonable effectiveness of data\". IEEE Intelligent Systems 24 (2). doi:10.1109\/MIS.2009.36.   \n\n\u2191 Brownstein, J.S.; Murphy, S.N.; Goldfine, A.B.; Grant, R.W.; Sordo, M.; Gainer, V.; et al. (March 2010). \"Rapid identification of myocardial infarction risk associated with diabetes medications using electronic medical records\". Diabetes Care 33 (3): 526\u2013531. doi:10.2337\/dc09-1506. PMC PMC2827502. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827502 .   \n\n\u2191 Brownstein, J.S.; Sordo, M.; Kohane, I.S.; Mandl, K.D. (2007). \"The tell-tale heart: population-based surveillance reveals an association of rofecoxib and celecoxib with myocardial infarction\". PLoS One 2 (9): e840. doi:10.1371\/journal.pone.0000840. PMC PMC1950690. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1950690 .   \n\n\u2191 Moody, G.B.; Mark, R.G. (May 2001). \"The impact of the MIT-BIH arrhythmia database\". IEEE Engineering in Medicine and Biology Magazine 20 (3): 45\u201350. PMID 11446209.   \n\n\u2191 Kalil, T.; Green, E. (23 April 2013). \"Big Data is a Big Deal for Biomedical Research Internet\". Whitehouse Office of Science and Technology Policy. Archived from the original on 11 August 2014. http:\/\/www.webcitation.org\/6RkISsDjO . Retrieved 11 August 2014 .   \n\n\u2191 Poh, M.Z.; Loddenkemper, T.; Reinsberger, C.; Swenson, N.C.; Goyal, S.; Sabtala, M.C.; et al. (May 2012). \"Convulsive seizure detection using a wrist-worn electrodermal activity and accelerometry biosensor\". Epilepsia 53 (5): e93\u2013e97. doi:10.1111\/j.1528-1167.2012.03444.x. PMID 22432935.   \n\n\u2191 Sano, A.; Picard, R.W. (2011). \"Toward a taxonomy of autonomic sleep patterns with electrodermal activity\". Conference proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society 2011: 777\u2013780. doi:10.1109\/IEMBS.2011.6090178. PMID 22254426.   \n\n\u2191 Peikoff, K. (31 December 2013). \"I Had My DNA Picture Taken, With Varying Results\". The New York Times. Archived from the original on 11 August 2014. http:\/\/www.webcitation.org\/6RkIrPHId . Retrieved 11 August 2014 .   \n\n\u2191 Lander, A.D. (2010). \"The edges of understanding\". BMC Biology 8: 40. doi:10.1186\/1741-7007-8-40. PMC PMC2864098. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2864098 .   \n\n\u2191 Peterson, T.A.; Doughty, E.; Kann, M.G. (November 2013). \"Towards precision medicine: advances in computational approaches for the analysis of human variants\". Journal of Molecular Biology 425: 4047\u20134063. doi:10.1016\/j.jmb.2013.08.008. PMC PMC3807015. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3807015 .   \n\n\u2191 Sansone, S.A.; Rocca-Serra, P.; Field, D.. Maguire, E.; Taylor, C.; Hofmann, O.; et al. (February 2012). \"Toward interoperable bioscience data\". Nature Genetics 44 (2): 121\u2013126. doi:10.1038\/ng.1054. PMC PMC3428019. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019 .   \n\n\u2191 Ioannidis, J.P. (August 2005). \"Why most published research findings are false\". PLoS Medicine 2 (8): e124. doi:10.1371\/journal.pmed.0020124. PMC PMC1182327. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1182327 .   \n\n\u2191 Pereira, T.V.; Horwitz, R.I.; Ioannidis, J.P. (October 2012). \"Empirical evaluation of very large treatment effects of medical interventions\". JAMA 308 (16): 1676\u20131684. doi:10.1001\/jama.2012.13444. PMC 23093165. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=23093165 .   \n\n\nAbbreviations \nACO: accountable care organization\nBIDMC: Beth Israel Deaconess Medical Center\nCMS: Center for Medicare and Medicaid\nEBM: evidence-based medicine\nEMR: electronic medical record\nICU: intensive care unit\nMIMIC: Multiparameter Intelligent Monitoring in Intensive Care\nMIT: Massachusetts Institute of Technology\nMIT-BIH: Massachusetts Institute of Technology \u2013 Beth Israel Hospital\nRCT: randomized controlled trial\nROS: retrospective observational study\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. \nPer the distribution agreement, the following copyright information is also being added:\n\u00a9Omar Badawi, Thomas Brennan, Leo Anthony Celi, Mengling Feng, Marzyeh Ghassemi, Andrea Ippolito, Alistair Johnson, Roger G Mark, Louis Mayaud, George Moody, Christopher Moses, Tristan Naumann, Vipan Nikore, Marco Pimentel, Tom J Pollard, Mauro Santos, David J Stone, Andrew Zimolzak, MIT Critical Data Conference 2014 Organizing Committee. Originally published in JMIR Medical Informatics (http:\/\/medinform.jmir.org), 22.08.2014.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\">https:\/\/www.limswiki.org\/index.php\/Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:57.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,838 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","bcdf1ad9d7bcb8661bc008ff0ca9fb02_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Making_big_data_useful_for_health_care_A_summary_of_the_inaugural_MIT_Critical_Data_Conference skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Making big data useful for health care: A summary of the inaugural MIT Critical Data Conference<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>With growing concerns that big data will only augment the problem of unreliable research, the Laboratory of Computational Physiology at the Massachusetts Institute of Technology organized the Critical Data Conference in January 2014. Thought leaders from academia, government, and industry across disciplines \u2014 including clinical medicine, computer science, public health, <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">informatics<\/a>, biomedical research, health technology, statistics, and epidemiology \u2014 gathered and discussed the pitfalls and challenges of big data in health care. The key message from the conference is that the value of large amounts of data hinges on the ability of researchers to share data, methodologies, and findings in an open setting. If empirical value is to be from the analysis of retrospective data, groups must continuously work together on similar problems to create more effective peer review. This will lead to improvement in methodology and quality, with each iteration of analysis resulting in more reliability.\n<\/p><p><b>Keywords<\/b>: big data; open data; unreliable research; machine learning; knowledge creation\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Failure to store, analyze, and utilize the vast amount of data generated during clinical care has restricted both quality of care and advances in the practice of medicine. Other industries, such as finance and energy, have already embraced data analytics for the purpose of learning. While such innovations remain relatively limited in the clinical domain, interest in \u201cbig data in clinical care\u201d has dramatically increased. This is due partly to the widespread adoption of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical record<\/a> (EMR) systems and partly to the growing awareness that better data analytics are required to manage the complex enterprise of the health care system. For the most part, however, the clinical enterprise has not had to address the problems particular to \u201cbig data\u201d because it has not yet satisfactorily addressed more fundamental data management issues. It is now becoming apparent that we are on the cusp of a great transformation that will incorporate data and data science integrally within the health care domain. In addition to the necessary major digital enhancements of the retrospective analyses that have variably been in place, real time and predictive analytics will also become ubiquitous core functionalities in the more firmly data-based environment of the (near) future. The initial Massachusetts Institute of Technology (MIT) Critical Data Conference was conceived and conducted to address the many data issues involved in this important transformation.<sup id=\"rdp-ebb-cite_ref-CeliBig13_1-0\" class=\"reference\"><a href=\"#cite_note-CeliBig13-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GrandePublic13_2-0\" class=\"reference\"><a href=\"#cite_note-GrandePublic13-2\" rel=\"external_link\">[2]<\/a><\/sup>\n<\/p><p>Increasing interest in creating the clinical analog of \u201cbusiness intelligence\u201d has made evident the necessity of developing and nurturing a clinical culture that can manage and translate data-based findings, including those from \u201cbig data\u201d studies. Combining this improved secondary use of clinical data with a data-driven approach to learning will enable this new culture to close the clinical data feedback loop facilitating better and more personalized care. Authors have noted several hallmarks of \u201cbig data\u201d: very large datasets, a large number of unrelated and\/or unstructured datasets, or high speed or low latency of data creation.<sup id=\"rdp-ebb-cite_ref-McAfeeBig12_3-0\" class=\"reference\"><a href=\"#cite_note-McAfeeBig12-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BourneWhat14_4-0\" class=\"reference\"><a href=\"#cite_note-BourneWhat14-4\" rel=\"external_link\">[4]<\/a><\/sup> The intensive care unit (ICU) provides a potent example of a particularly data rich clinical domain with the potential for both clinical and financial benefits if these large amounts of data can be harnessed and systematically leveraged into guiding practice. Thus, we use the term \u201cCritical Data\u201d to refer to big data in the setting of the ICU.\n<\/p><p>This paper summarizes the lectures and group discussions that took place during the recent Critical Data Conference at MIT, Cambridge MA, on January 7, 2014. The conference was the second part of a two-part event that brought together clinicians, data scientists, statisticians and epidemiologists.\n<\/p><p>The event opened with a \u201cdata marathon\u201d on January 3-5, 2014 (Figure 1), which brought together teams of data scientists and clinicians to mine the Multiparameter Intelligent Monitoring in Intensive Care (MIMIC) database (version II). MIMIC II is an open-access database consisting of over 60,000 recorded ICU stays from the adult intensive care units at the Beth Israel Deaconess Medical Center (BIDMC) in Boston, MA.<sup id=\"rdp-ebb-cite_ref-MIMICII14_5-0\" class=\"reference\"><a href=\"#cite_note-MIMICII14-5\" rel=\"external_link\">[5]<\/a><\/sup> Over 100 people participated in the two-day data marathon, and posters of the projects were displayed at the Critical Data Conference.\n<\/p><p>The Critical Data Conference on January 7 was an approximately ten-hour program comprising two keynote addresses (Jeffrey Drazen, MD and John Ioannidis, MD, PhD), seven individual lectures, three panel discussions, and two poster sessions (Figure 2). The overall conference theme was meaningful secondary use of big data from critical care settings. Materials from the conference (program, slides, and videos) are available online at the MIT Critical Data conference site.<sup id=\"rdp-ebb-cite_ref-MITCrit14_6-0\" class=\"reference\"><a href=\"#cite_note-MITCrit14-6\" rel=\"external_link\">[6]<\/a><\/sup>\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Celi_JMIRMedInformatics2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"81c64e62e089d0a5e47f5c1fef6ed4d7\"><img alt=\"Fig1 Celi JMIRMedInformatics2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/8\/83\/Fig1_Celi_JMIRMedInformatics2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Presentation at the Critical Data Marathon. Photo credit: Andrew Zimolzak.<\/blockquote>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Celi_JMIRMedInformatics2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"26391f220d5a1525f5e8259b1c87f59a\"><img alt=\"Fig2 Celi JMIRMedInformatics2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f5\/Fig2_Celi_JMIRMedInformatics2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> Critical Data poster session. Photo credit: Andrew Zimolzak.<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"The_problem\">The problem<\/span><\/h2>\n<p>In his keynote address, Jeffrey Drazen, MD, Editor-in-Chief of the New England Journal of Medicine, noted that the number of evidence-based recommendations built on randomized controlled trials (RCTs), the current gold standard for data quality, is insufficient to address the majority of clinical decisions. Subsequently, clinicians are often left to practice medicine \u201cblindly.\u201d Without the knowledge generation required to capture the decisional factors involved in realistic clinical scenarios, clinical decision making is often less data-driven than determined by the \u201cplay of chance\u201d buttressed by past experience. Historically, a doctor took a history, performed a physical examination and made a diagnosis based on what he or she observed. As technology and medical theory progressed, knowledge such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> and imaging modalities helped mitigate chance in the diagnosis of disease. Rote application of existing knowledge is not enough, as physicians want to establish causality. Until now this has been done with theories, but moving forward, theories will be inadequate unless they are confirmed, translated to practice, and systematically disseminated in clinical practice.\n<\/p><p>This trial-and-error process continues today because data generated from routine care is most often not captured and is rarely disseminated for the purpose of improving population health. Even in <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a>-rich care settings like the ICU, the knowledge necessary to mitigate the play of chance is lacking.<sup id=\"rdp-ebb-cite_ref-FreedmanLies10_7-0\" class=\"reference\"><a href=\"#cite_note-FreedmanLies10-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-EconTrouble13_8-0\" class=\"reference\"><a href=\"#cite_note-EconTrouble13-8\" rel=\"external_link\">[8]<\/a><\/sup> As such, the ICU provides a fertile ground for potential improvement. Specifically, Drazen suggested a potential role for clinical data mining to answer questions that cannot be answered using RCTs.<sup id=\"rdp-ebb-cite_ref-MosesPharm13_9-0\" class=\"reference\"><a href=\"#cite_note-MosesPharm13-9\" rel=\"external_link\">[9]<\/a><\/sup> This approach would likely yield benefits both more quickly and with fewer resources.\n<\/p><p>Drazen concluded with the question \u201cAt what point is data good enough?\u201d Documented associations may be strong but not sufficiently \u201cproven\u201d to establish causality. Drazen drew a comparison with experimental physicists who hone future studies on the work of theorists as well as on prior experimental results: <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">biomedical informaticians<\/a> can identify meaningful associations that can then guide design of new RCTs where data quality can be increased by controlling for potential confounders. This will require cross-disciplinary collaboration of frontline clinicians, medical staff, database engineers and biomedical informaticians, in addition to strong partnerships with health information system vendors in order to close the loop from knowledge discovery during routine care to the real time application of best care for populations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Secondary_usage_of_clinical_data\">Secondary usage of clinical data<\/span><\/h2>\n<p>Charles Safran, MD, MS, Chief of the Division of Clinical Computing at the BIDMC and Harvard Medical School, spoke next, sharing the dream of evidence-based medicine (EBM): the ideal situation in which quality evidence would exist to guide clinicians through all the conundrums faced on a near-daily basis (eg, which test to order, how to interpret the test results, and what therapy to institute). For the last half-century, prospective RCTs have been the gold standard in EBM. Safran noted, as Drazen had, that such trials suffer from a number of limitations including economic burdens and design limitations. An RCT can only address a severely limited bundle of particularly well-posed clinical questions. For many clinical situations, it is either unethical or even impossible to proceed with an RCT. Furthermore, the inclusion and exclusion criteria often limit the generalizability of an RCT study and, given the time it normally takes to run an RCT, it is very difficult for these studies to remain current with the rapidly evolving practice of medicine.\n<\/p><p>Can another approach avoid at least some of the limitations of RCTs? Safran suggested that retrospective observational studies (ROS) utilizing EMR data are a promising avenue for generating EBM. Digital records contain extensive clinical information including medical history, diagnoses, medications, immunization dates, allergies, radiology images, and laboratory and test results. Consequently, routinely collected EMR data contains the rich, continuous and time-sensitive information needed to support clinical decision making and evidence generation.<sup id=\"rdp-ebb-cite_ref-SafranExpert07_10-0\" class=\"reference\"><a href=\"#cite_note-SafranExpert07-10\" rel=\"external_link\">[10]<\/a><\/sup> However, despite the many potential benefits, Safran pointed out that secondary use of EMR data is still subject to limitations: EMR data were not collected primarily for the purpose of evidence generation and data analytics but for real-time and longitudinal patient care.<sup id=\"rdp-ebb-cite_ref-GeissbuhlerTrust13_11-0\" class=\"reference\"><a href=\"#cite_note-GeissbuhlerTrust13-11\" rel=\"external_link\">[11]<\/a><\/sup> As a result, EMR data are often poorly structured, disorganized, unstandardized, and contaminated with errors, artifacts, and missing values.\n<\/p><p>Safran echoed one of Drazen\u2019s points by proposing that we should combine the usage of prospective RCTs and ROS in such a way that each complements the limitations of the other. Furthermore, he suggested the possibility of incorporating additional novel sources of data such as social media data, health data from portable sensors, and genetic data. While there are many barriers to establishing such a comprehensive framework, a big data picture of clinical, genetic, and treatment variables holds promise in revolutionizing diagnosis and treatment.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Connecting_patients.2C_providers.2C_and_payers\">Connecting patients, providers, and payers<\/span><\/h2>\n<p>For John Halamka, MD, MS, the Chief Information Officer of BIDMC, working with big data in <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" target=\"_blank\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospital<\/a> systems is hugely challenging but at the same time holds tremendous promise in providing more meaningful information to help clinicians treat patients across the continuum of care. In his position, Halamka has been tasked to aggregate data in novel ways in order to provide better care for BIDMC\u2019s patient population. One opportunity for furthering \u201cbig data in health care\u201d is to normalize the data collected via their EMR system and store it in large, centralized databases. In turn, analytic tools can then be applied to identify and isolate the quality data reporting measures required to participate as an Accountable Care Organization (ACO) under the Affordable Care Act.\n<\/p><p>Halamka emphasized that building these large datasets does not intrinsically provide value from the start, stating that \u201cworkflow is disparate, the vocabulary is disparate, and the people are disparate.\u201d Therefore, the normalization of data and its distillation into standard schemas are difficult due to discrepancies across longitudinal data. Further, since each vendor models concepts differently, there must be an emphasis on developing a \u201cleast common denominator\u201d concept map across vendors\u2019 offerings.\n<\/p><p>Nevertheless, through this normalization effort, doctors can utilize \u201cscorecards\u201d to evaluate their own patient population within and across the different payment models, such as Blue Cross Blue Shield\u2019s Alternative Quality Contact measures, the (CMS) Physician Quality Reporting System measures, and the CMS ACO measures. In addition, physicians can query this dataset to identify the most effective treatment regimes. However, such queries do pose privacy and security issues in the hospital setting, and these risks are further complicated by hospital staff utilizing personal mobile devices such as cell phones, laptops, and tablets.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Creating_a_data-driven_learning_system\">Creating a data-driven learning system<\/span><\/h2>\n<p>The problem posed to the first panel (Figure 3), comprising Gari Clifford, PhD, Perren Cobb, MD, and Joseph Frassica, MD, and moderated by Leo Anthony Celi, MD, MS, MPH, was how to create a data-driven learning system in clinical practice.<sup id=\"rdp-ebb-cite_ref-EconTrouble13_8-1\" class=\"reference\"><a href=\"#cite_note-EconTrouble13-8\" rel=\"external_link\">[8]<\/a><\/sup> Privacy concerns were cited as the central barrier, as there is a tradeoff between re-identification risk and the value of sharing. Furthermore, recent work shows patients are reluctant to share for certain purposes such as marketing, pharmaceutical, and quality improvement measures, indicating a need for public education about the benefits of data sharing and that shared data can be utilized without being used for marketing and other unwanted purposes.<sup id=\"rdp-ebb-cite_ref-GrandePublic13_2-1\" class=\"reference\"><a href=\"#cite_note-GrandePublic13-2\" rel=\"external_link\">[2]<\/a><\/sup>\n<\/p><p>There is also tension between intellectual property rights and transparency. Resolution of this may require collaboration between government, industry and academic institutions, as seen with the US Critical Illness and Injury Trials Group.<sup id=\"rdp-ebb-cite_ref-CobbUS09_12-0\" class=\"reference\"><a href=\"#cite_note-CobbUS09-12\" rel=\"external_link\">[12]<\/a><\/sup> There is also a risk that data sharing will make authors reluctant to write audacious or unconventional papers (as did Reinhart and Rogoff<sup id=\"rdp-ebb-cite_ref-ReinhartDecade11_13-0\" class=\"reference\"><a href=\"#cite_note-ReinhartDecade11-13\" rel=\"external_link\">[13]<\/a><\/sup>), if data sharing puts such papers at perceived higher risk of refutation (such as the refutation of Herndon et al<sup id=\"rdp-ebb-cite_ref-HerndonDoes11_14-0\" class=\"reference\"><a href=\"#cite_note-HerndonDoes11-14\" rel=\"external_link\">[14]<\/a><\/sup>).\n<\/p><p>Finally, the panel raised concerns about the high quantity but perceived low quality of the data that is actually captured. While there is hope that automatically captured data may be more accurate than manually entered data, there is also some risk that doing so will introduce additional noise, furthering the problem of quantity over quality. This concern poses the challenge of capturing more and higher quality data in order to promote reproducibility. Panelists observed that multidisciplinary conferences like the Critical Data Conference are especially beneficial in this regard, as they provide an opportunity for clinicians and data scientists to better understand the relation between real-world activity and the data that such activity generates.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Celi_JMIRMedInformatics2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2966279f0484a94ac0e3055f3472bc3e\"><img alt=\"Fig3 Celi JMIRMedInformatics2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/ff\/Fig3_Celi_JMIRMedInformatics2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3.<\/b> Data-driven learning system panel. Photo credit: Andrew Zimolzak.<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Physician_culture_as_a_barrier_to_spread_of_innovation\">Physician culture as a barrier to spread of innovation<\/span><\/h2>\n<p>In the following panel moderated by critical care physician Leo Anthony Celi, MD, MS, MPH, fellow intensivists Djillali Annane, MD, PhD, Peter Clardy, MD and Taylor Thompson, MD reflected on the barriers presented by the current clinician culture toward the goal of data-driven innovation in medicine (Figure 4). The panelists observed that historically, EBM was perceived to be incompatible with well-established observational trials and experience, perhaps instilling a residual degree of resistance. Consequently, echoing Safran\u2019s sentiments, it will be increasingly important that \u201cbig data\u201d is understood as a complement to RCTs and (patho)physiologic studies. Furthermore, condensing and filtering the vast quantity of data to make it applicable at the bedside will be key to adoption. The specific inclusion of clinicians during the design process will help to deter the creation of tools that inundate staff with extraneous information and burdensome extra tasks. Likewise the incorporation of \u201cbig data\u201d into medical education, in a way that students and resident trainees will be able to understand its importance in both everyday care and expediting research, is vital.\n<\/p><p>While the panel agreed that more evidence is required to determine whether big data can facilitate comparative effectiveness research, it was acknowledged that it is necessary to investigate this alternative since RCTs do not, and will not, provide answers to an important fraction of the decisions required on a daily basis. Scaling up RCTs to account for the thousands of decisions each day is not feasible, so big data approaches may provide the most effective way to fill these gaps. For example, three groups currently leading clinical trials research in the analysis of fluid resuscitation in critically ill patients have collaborated to create a common database architecture to allow for individual patient meta-analysis and for these trials to be evaluated in aggregate by an external monitoring committee.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Celi_JMIRMedInformatics2014_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5c6fdbd8c174f243d70e71324a826e6f\"><img alt=\"Fig4 Celi JMIRMedInformatics2014 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/36\/Fig4_Celi_JMIRMedInformatics2014_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4.<\/b> Physician culture panel. Photo credit: Andrew Zimolzak.<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"The_role_of_industry_in_the_data_revolution_in_health_care\">The role of industry in the data revolution in health care<\/span><\/h2>\n<p>There is concern that industry continues to view data as a potential source of revenue and would therefore be opposed to providing open access to what they consider to be proprietary data with business value. In the last panel discussion of the day, moderated by Ambar Bhattacharyya, MBA of Bessem Venture Partners, industry panelists Josh Gray, MBA of AthenaResearch, Enakshi Singh, MS of SAP, and Omar Badawi, PharmD, MPH of Philips Healthcare provided their insights on the topic.\n<\/p><p>They first addressed the issue of sharing databases freely, noting that concerns associated with data ownership are not restricted to industry. Similar problems and conflicts are observed among most stakeholders in health care data ownership: patients, hospitals, providers, payers, vendors, and academia. Generally speaking, industrial data owners want to protect their data from those who may use it competitively against them, share or sell the data to derive direct clinical value, or profit from possible insights. They also wish to avoid the overhead costs associated with sharing. They are interested in allowing society to leverage their data in order to make gains if, and only if, these other interests remain unaffected.\n<\/p><p>The costs for responsibly sharing secondary clinical data are not trivial. Although understanding the complexity of the data presents a significant challenge, understanding the workflow for entering data in the primary system is often even more complicated, requiring extensive support. Therefore, sharing secondary clinical data can be a costly initiative for industry, lowering its priority as a business objective. These challenges are further exacerbated when collaboration requires intellectual property agreements. Lack of an accepted standard practice for research agreements, coupled with an outdated patent system, creates barriers to collaboration that are rarely overcome. Many ideas for collaboration either take years to initiate or never come to fruition, due to challenges with developing <i>de novo<\/i> legal research agreements.\n<\/p><p>While industry and researchers are not philosophically opposed to sharing data to ensure reproducibility, protections from the aforementioned concerns are critical. Can the data be shared without the risk of lost intellectual property? If not, the incentives for innovation may be minimized. Who will bear the costs for ensuring that the replicating team fully understands the nuances of the data? Who will prevent competitors or others with malicious intent from inappropriately labeling valid research as \u201cjunk science\u201d? Such underhanded interventions could introduce confusion around valid earlier findings and unfairly distract and denigrate the primary researchers.\n<\/p><p>Ultimately, there is a growing sense that data will become less of a commodity over time if governments continue to support the development and maintenance of open access research networks. As the scale and quality of these surpass those of privately owned databases, society will benefit as the obstacles to collaboration and the value of retaining private ownership diminish.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_unreasonable_effectiveness_of_data\">The unreasonable effectiveness of data<\/span><\/h2>\n<p>Peter Szolovits, PhD from the MIT Computer Science and Artificial Intelligence Lab, highlighted how big data can often trump good, but smaller, data. Researchers at Google have been making a similar argument from a decade-long experience with natural language processing, showing that for some important tasks an order of magnitude growth in the size of a dataset leads to improvements in performance that can overshadow improvements in modeling technique.<sup id=\"rdp-ebb-cite_ref-HalevyThe09_15-0\" class=\"reference\"><a href=\"#cite_note-HalevyThe09-15\" rel=\"external_link\">[15]<\/a><\/sup> They have also argued that discarding rare events is a bad idea because although these may be individually rare, they could prove to be significant later when examined on a much larger scale.\n<\/p><p>In the clinical world, patient state depends on complex pathophysiology dictated by genetic predispositions, environmental exposures, treatments, and numerous other factors. While, there are many potential ways to formulate clinical outcomes into complex statistical models, it is often the simple models that give the best, and most interpretable, results. Some clinicians and epidemiologists have already used large sources of observational data to improve clinical practice, especially in identifying drug side-effects, for example, for rosiglitazone<sup id=\"rdp-ebb-cite_ref-BrownsteinRapid10_16-0\" class=\"reference\"><a href=\"#cite_note-BrownsteinRapid10-16\" rel=\"external_link\">[16]<\/a><\/sup>, and rofecoxib.<sup id=\"rdp-ebb-cite_ref-BrownsteinThe07_17-0\" class=\"reference\"><a href=\"#cite_note-BrownsteinThe07-17\" rel=\"external_link\">[17]<\/a><\/sup> Cox proportional hazard, na\u00efve Bayes, linear and logistic regression, and similar models can use aggregated variables to summarize dynamic variation without adding additional complexity.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_story_of_MIMIC:_Open-access_critical_care_data\">The story of MIMIC: Open-access critical care data<\/span><\/h2>\n<p>Since researchers who seek to create new clinical knowledge and tools are dependent upon the availability of relevant data, restricting access to data introduces barriers that stifle research progress. This simple principle has been at the heart of the research of Roger Mark, MD, PhD since the 1980s, a time when his work was focused on developing real-time arrhythmia analysis tools for use in patient monitoring.\n<\/p><p>Like today, the norm for researchers in the 1980s was to privately maintain closed databases for their own benefit. So when Mark\u2019s team needed data, they began the painstaking work of creating their own resource, collecting electrocardiograms from patients at Boston\u2019s BIDMC and in the process, adding over 100,000 annotations. Breaking from tradition, they openly shared the dataset, reasoning that the more people who analyzed it, the better the overall understanding of arrhythmias would become. This dataset become known as the MIT-Beth Israel Hospital (MIT-BIH) Arrhythmia Database.<sup id=\"rdp-ebb-cite_ref-MoodyThe01_18-0\" class=\"reference\"><a href=\"#cite_note-MoodyThe01-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/p><p>The consequences were far-reaching. Not only did the MIT-BIH Database stimulate research interest, it generated beneficial competition and became a shared resource for evaluating algorithms. Researchers competed to see whose work performed best on the standard data, eventually leading to the database becoming part of a federal requirement for evaluation of commercial algorithms. This success led the team to develop further resources unique in their openness, including PhysioNet, a platform for open physiologic data, and MIMIC II, a rich database of critical care data.\n<\/p><p>PhysioNet has over 50,000 registered users in over 120 countries and international recognition for accelerating the pace of discovery.<sup id=\"rdp-ebb-cite_ref-KalilBig13_19-0\" class=\"reference\"><a href=\"#cite_note-KalilBig13-19\" rel=\"external_link\">[19]<\/a><\/sup> Mark attributes much of PhysioNet\u2019s success to the progressive mindset of the participating collaborators. Success has required not only funding, but also a collaborative approach among partnering clinicians, researchers, hospital technologists, and local ethics committees. Participation of commercial partners was also required, and obtained, in order to decrypt the proprietary data formats output by their monitoring systems.\n<\/p><p>Reproducibility of research and open data are increasingly getting the attention they deserve, but changing practice requires support at all levels.<sup id=\"rdp-ebb-cite_ref-EconTrouble13_8-2\" class=\"reference\"><a href=\"#cite_note-EconTrouble13-8\" rel=\"external_link\">[8]<\/a><\/sup> For open technology to be embraced, funders must recognize the added value from a robust database infrastructure and allocate funds accordingly. Researchers too must embrace open approaches that perhaps challenge some of the underlying career reward systems. With changing attitudes, and by engaging the creative energy of the worldwide research community, Mark\u2019s hope is that MIMIC will become a multinational resource leading to the generation of new knowledge and new tools.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Opportunities_and_challenges_in_wearable_sensor_datasets\">Opportunities and challenges in wearable sensor datasets<\/span><\/h2>\n<p>The ability to create and capture data is exploding and offers huge potential for health organizations around the world to save both lives and scarce resources. Yadid Ayzenberg, PhD discussed the \u201cOpportunities and Challenges in Wearable Sensor Data\u201d in his talk, focusing on how the combination of wearable technology and the near ubiquitous access to mobile phones have the potential to address some of the challenges in health care. Examples include the works of Poh et al.<sup id=\"rdp-ebb-cite_ref-PohCon12_20-0\" class=\"reference\"><a href=\"#cite_note-PohCon12-20\" rel=\"external_link\">[20]<\/a><\/sup> and Sano and Picard<sup id=\"rdp-ebb-cite_ref-SanoToward11_21-0\" class=\"reference\"><a href=\"#cite_note-SanoToward11-21\" rel=\"external_link\">[21]<\/a><\/sup>, which used a wrist-worn electrodermal activity and accelerometry biosensor for detection of convulsive seizures and sleep stages.\n<\/p><p>Wearable technologies provide a way to transition from a traditional aperiodic \u201csnapshot\u201d monitoring approach to a continuous and longitudinal monitoring paradigm, increase patients\u2019 engagement in their care, and facilitate doctor-patient interactions. Already massive amounts of personal health data are being generated through consumer devices such as mobile phones and wristbands that monitor sleeping patterns, exercise, stress, calorie consumption and more. In most instances, however, the data are stored on a per-device basis, and there are unsolved issues concerning data management, ownership, privacy, and misuse. The noise and artifacts in the data measured by wearable sensors also present an important challenge. New analytic methods that transform \u201cdirty data\u201d into good quality data are needed.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Big_data.2C_genomics.2C_and_public_health\">Big data, genomics, and public health<\/span><\/h2>\n<p>According to Winston Hide, PhD, the promise of a new economy based on data-driven discovery and decision-making is also motivating his own field of genomics. Advances in genome sequencing technology will allow the cost of sequencing a genome to be less than $100 in the near future. Consequently, it is estimated that by 2015, one million genomes will be digitally available with high expectations for public health benefits. However, Hide cautions that there is still a \u201cgenome variants\u201d problem to be solved. This was exemplified by the case of Kira Peikoff who was predicted to have a 20% above average risk of developing psoriasis by one commercial sequencing product, while predicted to have a 2% below average risk for the same condition by another.<sup id=\"rdp-ebb-cite_ref-PeikoffIHad13_22-0\" class=\"reference\"><a href=\"#cite_note-PeikoffIHad13-22\" rel=\"external_link\">[22]<\/a><\/sup>\n<\/p><p>Variations in the reporting of genomic characteristics have two potential root causes. First, there is a sampling problem caused by the use of different sequencing technologies, which introduces errors in evaluating genome assembly. Second, there is an interpretation problem in defining the role of genes in disease which compromises the prediction of clinical outcomes as determined by the single-nucleotide polymorphism analysis tools.\n<\/p><p>The availability of millions of genomes will allow completion of the catalogue of genes associated with a particular disease in genome-wide association studies. However, Hide maintains that finding clear drug targets requires the creation of an evolving catalogue of functions which would interpret complex gene pathways<sup id=\"rdp-ebb-cite_ref-LanderThe10_23-0\" class=\"reference\"><a href=\"#cite_note-LanderThe10-23\" rel=\"external_link\">[23]<\/a><\/sup>, and the selection of cohorts that would not only depend on ethnicity (the classic phenotype), but also on physiological and even molecular differences.\n<\/p><p>The application of genomic tests to public health will contribute to the transformation of physicians into data-centric specialists and pave the way for \u201cprecision medicine\u201d.<sup id=\"rdp-ebb-cite_ref-PetersonTowards13_24-0\" class=\"reference\"><a href=\"#cite_note-PetersonTowards13-24\" rel=\"external_link\">[24]<\/a><\/sup> This challenging way of delivering health care calls for new strategies and tactics for translating research into clinical practice. These will likely include the creation of open-access genomic and clinical databases, use of a common scientific language<sup id=\"rdp-ebb-cite_ref-SansoneToward12_25-0\" class=\"reference\"><a href=\"#cite_note-SansoneToward12-25\" rel=\"external_link\">[25]<\/a><\/sup> and (open) data access tools. Regulatory bodies, such as the Food and Drug Administration, will have a role in guaranteeing the standardization and reliability of diagnostics based on genetic tests. These tools must guarantee the reproducibility<sup id=\"rdp-ebb-cite_ref-EconTrouble13_8-3\" class=\"reference\"><a href=\"#cite_note-EconTrouble13-8\" rel=\"external_link\">[8]<\/a><\/sup> of discovered genome signals and contribute to the improvement of online platforms that map genetic features to diseases and their treatment (eg, Cancer Genome Atlas; PharmGKB).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"The_pitfalls_and_potential_of_big_data_in_health_care\">The pitfalls and potential of big data in health care<\/span><\/h2>\n<p>Proponents of big data have made grandiose claims of expanding human knowledge by orders of magnitude through empirical analysis and data mining, but as Stanford professor John Ioannidis, MD, PhD says, \u201cwith big data comes big problems\u201d. Ioannidis discussed the darker side of data analysis, in which bias has led a large proportion of published medical science to come to the wrong conclusion. Author of the most downloaded paper in PLOS Medicine, \u201cWhy Most Published Research Findings Are False\u201d<sup id=\"rdp-ebb-cite_ref-IoannidisWhy05_26-0\" class=\"reference\"><a href=\"#cite_note-IoannidisWhy05-26\" rel=\"external_link\">[26]<\/a><\/sup>, Ioannidis argued that most statistically significant results are likely to be false positives. For example, using the national drug and cancer registry database of Sweden, Ioannidis and colleagues found that almost one third of the 560 medications evaluated in isolation were associated with a higher cancer risk.\n<\/p><p>As Ioannidis highlights, the issue is not in the quantity of data we have. Increasing sample sizes is a huge boon to the medical field. The issue resides in a lack of transparency. When he reviews a paper published in a journal on a new dataset, his thoughts immediately drift to those studies that were not published. This is quantified in the so-called \u201cvibration of effects\u201d, where depending on the confounding variables for which adjustments are made, completely opposite conclusions can be drawn. For vitamin E, for example, adjusting for a certain subset of confounders led to the conclusion that it increases the relative risk of mortality, whereas adjusting for another equally plausible set of confounders gave the opposite result, that is, a reduction in mortality risk. This may explain why 90% of effects in RCTs were lower in subsequent published trials.<sup id=\"rdp-ebb-cite_ref-PereiraEmp12_27-0\" class=\"reference\"><a href=\"#cite_note-PereiraEmp12-27\" rel=\"external_link\">[27]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Comparative_effectiveness_using_big_data\">Comparative effectiveness using big data<\/span><\/h2>\n<p>Limitations aside for now, ROS do provide an opportunity to conduct comparative effectiveness studies on research questions that would be unlikely to be examined by an RCT, or would be inherently biased if an RCT were conducted. The illustrative example presented by Una-May O\u2019Reilly, PhD was the question of the potential benefit of diuretic use to accelerate removal of fluids given during resuscitation in ICU patients who have recovered from sepsis. Retrospective analysis would be easily marred by \u201cselection bias\u201d. In fact, if patients are allocated (not randomized) to two groups, treatment and non-treatment, it is very likely that the allocation would be done on the basis of patient condition and biased by clinical severity resulting in unreliable results.\n<\/p><p>In a ROS, the data consists of a series of days during which the treatment was administered (D+) or not (D-). Because these decisions were being made on a daily basis, it is even harder to capture the covariance structure. O\u2019Reilly refers to this as the \u201cNon-Decision Day Dilemma\u201d. In order to deal with this, the covariance structure has to account for time-varying information with respect to a specific day. It is easy to take the treatment day as a reference and align all patients who received treatments with respect to this event (D+). For non-treated patients, aligning time-series is more complicated as every day is essentially a \u201cnon-decision day\u201d. Considering every single day would result in a widely unbalanced dataset where the length-of-stay influences the individual contribution of each patient. To account for this, it is possible to randomly sample N negative days (D*-) and pair them up with the positive instances (D+) based on a statistical similarity criterion with respect to the time from admission. This is achieved by defining a propensity score for each patient for every day during their ICU stay. The propensity score thus enables appropriate cohort matching such that comparative effectiveness can be appropriately assessed.\n<\/p><p>This modest example illustrates the sort of robust and reliable statistical technique that evidence-based medicine requires. It can reduce sample noise and improve the reliability of conclusions, and it leads toward methodology standardization across studies. Beyond these local improvements, meta-studies will also be a requirement to validate any local finding. These are only possible with data sharing and open data initiatives such as the MIMIC-II initiative. One strength of this database is the dual culture and scientific activity it generates because data science can only fully benefit from collaboration between data scientists and domain experts (in this case, intensivist physicians).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Although the future of \u201cbig data\u201d in health care remains unclear, its role will be undeniably important. This conference was effective in collating the broad range of perspectives on the many challenges facing EBM in the 21st Century. As several speakers suggested, one possible opportunity is to adopt a pragmatic approach to EBM, combining RCT and ROS. This combination may employ ROSs to fill the gaps where it is impractical, unlikely or impossible to conduct a RCT or to drive hypothesis generation for further RCT analysis.\n<\/p><p>It is also crucial to acknowledge that any ROS requires a multidisciplinary approach, integrating clinical knowledge with a broad range of data analytic skills ranging from biostatistics, machine learning, and signal processing to data mining. Encouraging a change in physician culture can likely be accomplished through updating education programs as well as by creating centers for excellence that can showcase the impact of ROS to the broader medical fraternity. These centers for excellence should host open, transparent, easily accessible data warehouses, which will facilitate study reproducibility and allow for a new wave of collaborative learning. Only by understanding the potential biases of any analysis, and fostering a system of normative data sharing, will the medical community be able to gain reliable knowledge from data, and produce research findings that do not turn out to be false.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The Organizing Committee would like to thank the speakers and panelists, the sponsors and all the attendees of the MIT Critical Data Conference and Marathon held in January 2014. The MIT Critical Data Marathon and Conference 2014 was sponsored by SAP, Philips Healthcare, Quanttus, Goodwin Procter and WilmerHale.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflict_of_interests\">Conflict of interests<\/span><\/h2>\n<p>Omar Badawi is senior clinical scientist at Philips Healthcare. Christopher Moses is founder and CEO of Smart Scheduling, Inc. Louis Mayaud is research director of Mensia Technologies SA. Thomas Brennan is data scientist at AthenaHealth Research.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<ol class=\"references\">\n<li id=\"cite_note-CeliBig13-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CeliBig13_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Celi, L.A.; Mark, R.G.; Stone, D.J.; Montgomery, R.A. (June 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3734609\" target=\"_blank\">\"\"Big data\" in the intensive care unit: Closing the data loop\"<\/a>. <i>American Journal of Respiratory and Critical Care Medicine<\/i> <b>187<\/b> (11): 1157\u20131160. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1164%2Frccm.201212-2311ED\" target=\"_blank\">10.1164\/rccm.201212-2311ED<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3734609\/\" target=\"_blank\">PMC3734609<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3734609\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3734609<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%22Big+data%22+in+the+intensive+care+unit%3A+Closing+the+data+loop&rft.jtitle=American+Journal+of+Respiratory+and+Critical+Care+Medicine&rft.aulast=Celi%2C+L.A.%3B+Mark%2C+R.G.%3B+Stone%2C+D.J.%3B+Montgomery%2C+R.A.&rft.au=Celi%2C+L.A.%3B+Mark%2C+R.G.%3B+Stone%2C+D.J.%3B+Montgomery%2C+R.A.&rft.date=June+2013&rft.volume=187&rft.issue=11&rft.pages=1157%E2%80%931160&rft_id=info:doi\/10.1164%2Frccm.201212-2311ED&rft_id=info:pmc\/PMC3734609&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3734609&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GrandePublic13-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GrandePublic13_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-GrandePublic13_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Grande, D.; Mitra, N.; Shah, A.; Wan, F.; Asch, D.A. (October 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4083587\" target=\"_blank\">\"Public preferences about secondary uses of electronic health information\"<\/a>. <i>JAMA Internal Medicine<\/i> <b>173<\/b> (19): 1798\u20131806. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjamainternmed.2013.9166\" target=\"_blank\">10.1001\/jamainternmed.2013.9166<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4083587\/\" target=\"_blank\">PMC4083587<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4083587\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4083587<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Public+preferences+about+secondary+uses+of+electronic+health+information&rft.jtitle=JAMA+Internal+Medicine&rft.aulast=Grande%2C+D.%3B+Mitra%2C+N.%3B+Shah%2C+A.%3B+Wan%2C+F.%3B+Asch%2C+D.A.&rft.au=Grande%2C+D.%3B+Mitra%2C+N.%3B+Shah%2C+A.%3B+Wan%2C+F.%3B+Asch%2C+D.A.&rft.date=October+2013&rft.volume=173&rft.issue=19&rft.pages=1798%E2%80%931806&rft_id=info:doi\/10.1001%2Fjamainternmed.2013.9166&rft_id=info:pmc\/PMC4083587&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4083587&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McAfeeBig12-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McAfeeBig12_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McAfee, A.; Brynjolfsson, E. (October 2012). \"Big data: the management revolution\". <i>Harvard Business Review<\/i> <b>90<\/b> (10): 60\u20136, 68, 128. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23074865\" target=\"_blank\">23074865<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Big+data%3A+the+management+revolution&rft.jtitle=Harvard+Business+Review&rft.aulast=McAfee%2C+A.%3B+Brynjolfsson%2C+E.&rft.au=McAfee%2C+A.%3B+Brynjolfsson%2C+E.&rft.date=October+2012&rft.volume=90&rft.issue=10&rft.pages=60%E2%80%936%2C+68%2C+128&rft_id=info:pmid\/23074865&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BourneWhat14-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BourneWhat14_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bourne, P.E. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3932474\" target=\"_blank\">\"What Big Data means to me\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>21<\/b> (2): 194. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2014-002651\" target=\"_blank\">10.1136\/amiajnl-2014-002651<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3932474\/\" target=\"_blank\">PMC3932474<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3932474\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3932474<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=What+Big+Data+means+to+me&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Bourne%2C+P.E.&rft.au=Bourne%2C+P.E.&rft.date=2014&rft.volume=21&rft.issue=2&rft.pages=194&rft_id=info:doi\/10.1136%2Famiajnl-2014-002651&rft_id=info:pmc\/PMC3932474&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3932474&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MIMICII14-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MIMICII14_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6RkH0iGr1\" target=\"_blank\">\"The MIMIC II project\"<\/a>. Massachusetts Institute of Technology. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/mimic.physionet.org\" target=\"_blank\">the original<\/a> on 11 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6RkH0iGr1\" target=\"_blank\">http:\/\/www.webcitation.org\/6RkH0iGr1<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+MIMIC+II+project&rft.atitle=&rft.pub=Massachusetts+Institute+of+Technology&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6RkH0iGr1&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MITCrit14-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MITCrit14_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6Rxk41BBR\" target=\"_blank\">\"Critical data: Empowering big data in critical care\"<\/a>. Massachusetts Institute of Technology. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/criticaldata.mit.edu\/past-events\/\" target=\"_blank\">the original<\/a> on 20 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6Rxk41BBR\" target=\"_blank\">http:\/\/www.webcitation.org\/6Rxk41BBR<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Critical+data%3A+Empowering+big+data+in+critical+care&rft.atitle=&rft.pub=Massachusetts+Institute+of+Technology&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6Rxk41BBR&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FreedmanLies10-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FreedmanLies10_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Freedman, D.H. (04 October 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6RkHK88k4\" target=\"_blank\">\"Lies, damned lies, and medical science\"<\/a>. <i>The Atlantic<\/i>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.theatlantic.com\/magazine\/archive\/2010\/11\/lies-damned-lies-and-medical-science\/308269\/\" target=\"_blank\">the original<\/a> on 11 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6RkHK88k4\" target=\"_blank\">http:\/\/www.webcitation.org\/6RkHK88k4<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Lies%2C+damned+lies%2C+and+medical+science&rft.atitle=The+Atlantic&rft.aulast=Freedman%2C+D.H.&rft.au=Freedman%2C+D.H.&rft.date=04+October+2010&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6RkHK88k4&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EconTrouble13-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-EconTrouble13_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-EconTrouble13_8-1\" rel=\"external_link\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-EconTrouble13_8-2\" rel=\"external_link\">8.2<\/a><\/sup> <sup><a href=\"#cite_ref-EconTrouble13_8-3\" rel=\"external_link\">8.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6RkHeZm6r\" target=\"_blank\">\"Trouble at the lab\"<\/a>. <i>The Economist<\/i>. 19 October 2013. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.economist.com\/news\/briefing\/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble\" target=\"_blank\">the original<\/a> on 11 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6RkHeZm6r\" target=\"_blank\">http:\/\/www.webcitation.org\/6RkHeZm6r<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Trouble+at+the+lab&rft.atitle=The+Economist&rft.date=19+October+2013&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6RkHeZm6r&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MosesPharm13-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MosesPharm13_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moses, C.; Celi, L.A.; Marshall, J. (June 2013). \"Pharmacovigilance: an active surveillance system to proactively identify risks for adverse events\". <i>Population Health Management<\/i> <b>16<\/b> (3): 147\u2013149. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fpop.2012.0100\" target=\"_blank\">10.1089\/pop.2012.0100<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23530466\" target=\"_blank\">23530466<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pharmacovigilance%3A+an+active+surveillance+system+to+proactively+identify+risks+for+adverse+events&rft.jtitle=Population+Health+Management&rft.aulast=Moses%2C+C.%3B+Celi%2C+L.A.%3B+Marshall%2C+J.&rft.au=Moses%2C+C.%3B+Celi%2C+L.A.%3B+Marshall%2C+J.&rft.date=June+2013&rft.volume=16&rft.issue=3&rft.pages=147%E2%80%93149&rft_id=info:doi\/10.1089%2Fpop.2012.0100&rft_id=info:pmid\/23530466&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SafranExpert07-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SafranExpert07_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Safran, C.; Bloomrosen, M.; Hammond, W.E.; Labkoff, S.; Markel-Fox, S.; Tang, P.C. (January 2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823\" target=\"_blank\">\"Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>14<\/b> (1): 1\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2273\" target=\"_blank\">10.1197\/jamia.M2273<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2329823\/\" target=\"_blank\">PMC2329823<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+national+framework+for+the+secondary+use+of+health+data%3A+an+American+Medical+Informatics+Association+White+Paper&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Safran%2C+C.%3B+Bloomrosen%2C+M.%3B+Hammond%2C+W.E.%3B+Labkoff%2C+S.%3B+Markel-Fox%2C+S.%3B+Tang%2C+P.C.&rft.au=Safran%2C+C.%3B+Bloomrosen%2C+M.%3B+Hammond%2C+W.E.%3B+Labkoff%2C+S.%3B+Markel-Fox%2C+S.%3B+Tang%2C+P.C.&rft.date=January+2007&rft.volume=14&rft.issue=1&rft.pages=1%E2%80%939&rft_id=info:doi\/10.1197%2Fjamia.M2273&rft_id=info:pmc\/PMC2329823&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2329823&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GeissbuhlerTrust13-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GeissbuhlerTrust13_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Geissbuhler, A.; Safran, C.; Buchan, I.; Bellazzi, R.; Labkoff, S.; Eilenberg, K.; et al. (January 2013). \"Trustworthy reuse of health data: a transnational perspective\". <i>International Journal of Medical Informatics<\/i> <b>82<\/b> (1): 1\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2012.11.003\" target=\"_blank\">10.1016\/j.ijmedinf.2012.11.003<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23182430\" target=\"_blank\">23182430<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Trustworthy+reuse+of+health+data%3A+a+transnational+perspective&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Geissbuhler%2C+A.%3B+Safran%2C+C.%3B+Buchan%2C+I.%3B+Bellazzi%2C+R.%3B+Labkoff%2C+S.%3B+Eilenberg%2C+K.%3B+et+al.&rft.au=Geissbuhler%2C+A.%3B+Safran%2C+C.%3B+Buchan%2C+I.%3B+Bellazzi%2C+R.%3B+Labkoff%2C+S.%3B+Eilenberg%2C+K.%3B+et+al.&rft.date=January+2013&rft.volume=82&rft.issue=1&rft.pages=1%E2%80%939&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2012.11.003&rft_id=info:pmid\/23182430&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CobbUS09-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CobbUS09_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cobb, J.P.; Cairns, C.B.; Bulger, E.; Wong, H.R.; Parsons, P.E.; Angus, D.C.; et al. (August 2009). \"The United States critical illness and injury trials group: an introduction\". <i>The Journal of Trauma<\/i> <b>67<\/b> (2 Suppl): S159\u2013S160. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FTA.0b013e3181ad3473\" target=\"_blank\">10.1097\/TA.0b013e3181ad3473<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19667851\" target=\"_blank\">19667851<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+United+States+critical+illness+and+injury+trials+group%3A+an+introduction&rft.jtitle=The+Journal+of+Trauma&rft.aulast=Cobb%2C+J.P.%3B+Cairns%2C+C.B.%3B+Bulger%2C+E.%3B+Wong%2C+H.R.%3B+Parsons%2C+P.E.%3B+Angus%2C+D.C.%3B+et+al.&rft.au=Cobb%2C+J.P.%3B+Cairns%2C+C.B.%3B+Bulger%2C+E.%3B+Wong%2C+H.R.%3B+Parsons%2C+P.E.%3B+Angus%2C+D.C.%3B+et+al.&rft.date=August+2009&rft.volume=67&rft.issue=2+Suppl&rft.pages=S159%E2%80%93S160&rft_id=info:doi\/10.1097%2FTA.0b013e3181ad3473&rft_id=info:pmid\/19667851&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReinhartDecade11-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReinhartDecade11_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Reinhart, C.M.; Rogoff, K.S. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6RkHujrwl\" target=\"_blank\">\"A Decade of Debt - Working Paper 16827\"<\/a> (PDF). National Bureau of Economic Research. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.nber.org\/papers\/w16827.pdf\" target=\"_blank\">the original<\/a> on 11 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6RkHujrwl\" target=\"_blank\">http:\/\/www.webcitation.org\/6RkHujrwl<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+Decade+of+Debt+-+Working+Paper+16827&rft.atitle=&rft.aulast=Reinhart%2C+C.M.%3B+Rogoff%2C+K.S.&rft.au=Reinhart%2C+C.M.%3B+Rogoff%2C+K.S.&rft.date=2011&rft.pub=National+Bureau+of+Economic+Research&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6RkHujrwl&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HerndonDoes11-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HerndonDoes11_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Herndon, T.; Ash, M.; Pollin, R. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6RkI0b5Rf\" target=\"_blank\">\"Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff - Working Paper Series No. 322\"<\/a> (PDF). Political Economy Research Institute. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.peri.umass.edu\/fileadmin\/pdf\/working_papers\/working_papers_301-350\/WP322.pdf\" target=\"_blank\">the original<\/a> on 11 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6RkI0b5Rf\" target=\"_blank\">http:\/\/www.webcitation.org\/6RkI0b5Rf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Does+High+Public+Debt+Consistently+Stifle+Economic+Growth%3F+A+Critique+of+Reinhart+and+Rogoff+-+Working+Paper+Series+No.+322&rft.atitle=&rft.aulast=Herndon%2C+T.%3B+Ash%2C+M.%3B+Pollin%2C+R.&rft.au=Herndon%2C+T.%3B+Ash%2C+M.%3B+Pollin%2C+R.&rft.date=2013&rft.pub=Political+Economy+Research+Institute&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6RkI0b5Rf&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HalevyThe09-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HalevyThe09_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Halevy, A.; Norvig, P.; Pereira, F. (2009). \"The unreasonable effectiveness of data\". <i>IEEE Intelligent Systems<\/i> <b>24<\/b> (2). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMIS.2009.36\" target=\"_blank\">10.1109\/MIS.2009.36<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+unreasonable+effectiveness+of+data&rft.jtitle=IEEE+Intelligent+Systems&rft.aulast=Halevy%2C+A.%3B+Norvig%2C+P.%3B+Pereira%2C+F.&rft.au=Halevy%2C+A.%3B+Norvig%2C+P.%3B+Pereira%2C+F.&rft.date=2009&rft.volume=24&rft.issue=2&rft_id=info:doi\/10.1109%2FMIS.2009.36&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrownsteinRapid10-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrownsteinRapid10_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brownstein, J.S.; Murphy, S.N.; Goldfine, A.B.; Grant, R.W.; Sordo, M.; Gainer, V.; et al. (March 2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827502\" target=\"_blank\">\"Rapid identification of myocardial infarction risk associated with diabetes medications using electronic medical records\"<\/a>. <i>Diabetes Care<\/i> <b>33<\/b> (3): 526\u2013531. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2337%2Fdc09-1506\" target=\"_blank\">10.2337\/dc09-1506<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2827502\/\" target=\"_blank\">PMC2827502<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827502\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827502<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Rapid+identification+of+myocardial+infarction+risk+associated+with+diabetes+medications+using+electronic+medical+records&rft.jtitle=Diabetes+Care&rft.aulast=Brownstein%2C+J.S.%3B+Murphy%2C+S.N.%3B+Goldfine%2C+A.B.%3B+Grant%2C+R.W.%3B+Sordo%2C+M.%3B+Gainer%2C+V.%3B+et+al.&rft.au=Brownstein%2C+J.S.%3B+Murphy%2C+S.N.%3B+Goldfine%2C+A.B.%3B+Grant%2C+R.W.%3B+Sordo%2C+M.%3B+Gainer%2C+V.%3B+et+al.&rft.date=March+2010&rft.volume=33&rft.issue=3&rft.pages=526%E2%80%93531&rft_id=info:doi\/10.2337%2Fdc09-1506&rft_id=info:pmc\/PMC2827502&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2827502&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrownsteinThe07-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrownsteinThe07_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brownstein, J.S.; Sordo, M.; Kohane, I.S.; Mandl, K.D. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1950690\" target=\"_blank\">\"The tell-tale heart: population-based surveillance reveals an association of rofecoxib and celecoxib with myocardial infarction\"<\/a>. <i>PLoS One<\/i> <b>2<\/b> (9): e840. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pone.0000840\" target=\"_blank\">10.1371\/journal.pone.0000840<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1950690\/\" target=\"_blank\">PMC1950690<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1950690\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1950690<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+tell-tale+heart%3A+population-based+surveillance+reveals+an+association+of+rofecoxib+and+celecoxib+with+myocardial+infarction&rft.jtitle=PLoS+One&rft.aulast=Brownstein%2C+J.S.%3B+Sordo%2C+M.%3B+Kohane%2C+I.S.%3B+Mandl%2C+K.D.&rft.au=Brownstein%2C+J.S.%3B+Sordo%2C+M.%3B+Kohane%2C+I.S.%3B+Mandl%2C+K.D.&rft.date=2007&rft.volume=2&rft.issue=9&rft.pages=e840&rft_id=info:doi\/10.1371%2Fjournal.pone.0000840&rft_id=info:pmc\/PMC1950690&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1950690&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoodyThe01-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MoodyThe01_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moody, G.B.; Mark, R.G. (May 2001). \"The impact of the MIT-BIH arrhythmia database\". <i>IEEE Engineering in Medicine and Biology Magazine<\/i> <b>20<\/b> (3): 45\u201350. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11446209\" target=\"_blank\">11446209<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+the+MIT-BIH+arrhythmia+database&rft.jtitle=IEEE+Engineering+in+Medicine+and+Biology+Magazine&rft.aulast=Moody%2C+G.B.%3B+Mark%2C+R.G.&rft.au=Moody%2C+G.B.%3B+Mark%2C+R.G.&rft.date=May+2001&rft.volume=20&rft.issue=3&rft.pages=45%E2%80%9350&rft_id=info:pmid\/11446209&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KalilBig13-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KalilBig13_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Kalil, T.; Green, E. (23 April 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6RkISsDjO\" target=\"_blank\">\"Big Data is a Big Deal for Biomedical Research Internet\"<\/a>. Whitehouse Office of Science and Technology Policy. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.whitehouse.gov\/blog\/2013\/04\/23\/big-data-big-deal-biomedical-research\" target=\"_blank\">the original<\/a> on 11 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6RkISsDjO\" target=\"_blank\">http:\/\/www.webcitation.org\/6RkISsDjO<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Big+Data+is+a+Big+Deal+for+Biomedical+Research+Internet&rft.atitle=&rft.aulast=Kalil%2C+T.%3B+Green%2C+E.&rft.au=Kalil%2C+T.%3B+Green%2C+E.&rft.date=23+April+2013&rft.pub=Whitehouse+Office+of+Science+and+Technology+Policy&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6RkISsDjO&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PohCon12-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PohCon12_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Poh, M.Z.; Loddenkemper, T.; Reinsberger, C.; Swenson, N.C.; Goyal, S.; Sabtala, M.C.; et al. (May 2012). \"Convulsive seizure detection using a wrist-worn electrodermal activity and accelerometry biosensor\". <i>Epilepsia<\/i> <b>53<\/b> (5): e93\u2013e97. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1528-1167.2012.03444.x\" target=\"_blank\">10.1111\/j.1528-1167.2012.03444.x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22432935\" target=\"_blank\">22432935<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Convulsive+seizure+detection+using+a+wrist-worn+electrodermal+activity+and+accelerometry+biosensor&rft.jtitle=Epilepsia&rft.aulast=Poh%2C+M.Z.%3B+Loddenkemper%2C+T.%3B+Reinsberger%2C+C.%3B+Swenson%2C+N.C.%3B+Goyal%2C+S.%3B+Sabtala%2C+M.C.%3B+et+al.&rft.au=Poh%2C+M.Z.%3B+Loddenkemper%2C+T.%3B+Reinsberger%2C+C.%3B+Swenson%2C+N.C.%3B+Goyal%2C+S.%3B+Sabtala%2C+M.C.%3B+et+al.&rft.date=May+2012&rft.volume=53&rft.issue=5&rft.pages=e93%E2%80%93e97&rft_id=info:doi\/10.1111%2Fj.1528-1167.2012.03444.x&rft_id=info:pmid\/22432935&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SanoToward11-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SanoToward11_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sano, A.; Picard, R.W. (2011). \"Toward a taxonomy of autonomic sleep patterns with electrodermal activity\". <i>Conference proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society<\/i> <b>2011<\/b>: 777\u2013780. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FIEMBS.2011.6090178\" target=\"_blank\">10.1109\/IEMBS.2011.6090178<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22254426\" target=\"_blank\">22254426<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+taxonomy+of+autonomic+sleep+patterns+with+electrodermal+activity&rft.jtitle=Conference+proceedings%3A+Annual+International+Conference+of+the+IEEE+Engineering+in+Medicine+and+Biology+Society&rft.aulast=Sano%2C+A.%3B+Picard%2C+R.W.&rft.au=Sano%2C+A.%3B+Picard%2C+R.W.&rft.date=2011&rft.volume=2011&rft.pages=777%E2%80%93780&rft_id=info:doi\/10.1109%2FIEMBS.2011.6090178&rft_id=info:pmid\/22254426&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PeikoffIHad13-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PeikoffIHad13_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Peikoff, K. (31 December 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/6RkIrPHId\" target=\"_blank\">\"I Had My DNA Picture Taken, With Varying Results\"<\/a>. <i>The New York Times<\/i>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.nytimes.com\/2013\/12\/31\/science\/i-had-my-dna-picture-taken-with-varying-results.html?_r=0\" target=\"_blank\">the original<\/a> on 11 August 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/6RkIrPHId\" target=\"_blank\">http:\/\/www.webcitation.org\/6RkIrPHId<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 August 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=I+Had+My+DNA+Picture+Taken%2C+With+Varying+Results&rft.atitle=The+New+York+Times&rft.aulast=Peikoff%2C+K.&rft.au=Peikoff%2C+K.&rft.date=31+December+2013&rft_id=http%3A%2F%2Fwww.webcitation.org%2F6RkIrPHId&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LanderThe10-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LanderThe10_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lander, A.D. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2864098\" target=\"_blank\">\"The edges of understanding\"<\/a>. <i>BMC Biology<\/i> <b>8<\/b>: 40. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1741-7007-8-40\" target=\"_blank\">10.1186\/1741-7007-8-40<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2864098\/\" target=\"_blank\">PMC2864098<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2864098\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2864098<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+edges+of+understanding&rft.jtitle=BMC+Biology&rft.aulast=Lander%2C+A.D.&rft.au=Lander%2C+A.D.&rft.date=2010&rft.volume=8&rft.pages=40&rft_id=info:doi\/10.1186%2F1741-7007-8-40&rft_id=info:pmc\/PMC2864098&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2864098&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PetersonTowards13-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PetersonTowards13_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Peterson, T.A.; Doughty, E.; Kann, M.G. (November 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3807015\" target=\"_blank\">\"Towards precision medicine: advances in computational approaches for the analysis of human variants\"<\/a>. <i>Journal of Molecular Biology<\/i> <b>425<\/b>: 4047\u20134063. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jmb.2013.08.008\" target=\"_blank\">10.1016\/j.jmb.2013.08.008<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3807015\/\" target=\"_blank\">PMC3807015<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3807015\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3807015<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+precision+medicine%3A+advances+in+computational+approaches+for+the+analysis+of+human+variants&rft.jtitle=Journal+of+Molecular+Biology&rft.aulast=Peterson%2C+T.A.%3B+Doughty%2C+E.%3B+Kann%2C+M.G.&rft.au=Peterson%2C+T.A.%3B+Doughty%2C+E.%3B+Kann%2C+M.G.&rft.date=November+2013&rft.volume=425&rft.pages=4047%E2%80%934063&rft_id=info:doi\/10.1016%2Fj.jmb.2013.08.008&rft_id=info:pmc\/PMC3807015&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3807015&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SansoneToward12-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SansoneToward12_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sansone, S.A.; Rocca-Serra, P.; Field, D.. Maguire, E.; Taylor, C.; Hofmann, O.; et al. (February 2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019\" target=\"_blank\">\"Toward interoperable bioscience data\"<\/a>. <i>Nature Genetics<\/i> <b>44<\/b> (2): 121\u2013126. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fng.1054\" target=\"_blank\">10.1038\/ng.1054<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3428019\/\" target=\"_blank\">PMC3428019<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3428019<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+interoperable+bioscience+data&rft.jtitle=Nature+Genetics&rft.aulast=Sansone%2C+S.A.%3B+Rocca-Serra%2C+P.%3B+Field%2C+D..+Maguire%2C+E.%3B+Taylor%2C+C.%3B+Hofmann%2C+O.%3B+et+al.&rft.au=Sansone%2C+S.A.%3B+Rocca-Serra%2C+P.%3B+Field%2C+D..+Maguire%2C+E.%3B+Taylor%2C+C.%3B+Hofmann%2C+O.%3B+et+al.&rft.date=February+2012&rft.volume=44&rft.issue=2&rft.pages=121%E2%80%93126&rft_id=info:doi\/10.1038%2Fng.1054&rft_id=info:pmc\/PMC3428019&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3428019&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IoannidisWhy05-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IoannidisWhy05_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ioannidis, J.P. (August 2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1182327\" target=\"_blank\">\"Why most published research findings are false\"<\/a>. <i>PLoS Medicine<\/i> <b>2<\/b> (8): e124. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.0020124\" target=\"_blank\">10.1371\/journal.pmed.0020124<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1182327\/\" target=\"_blank\">PMC1182327<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1182327\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1182327<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Why+most+published+research+findings+are+false&rft.jtitle=PLoS+Medicine&rft.aulast=Ioannidis%2C+J.P.&rft.au=Ioannidis%2C+J.P.&rft.date=August+2005&rft.volume=2&rft.issue=8&rft.pages=e124&rft_id=info:doi\/10.1371%2Fjournal.pmed.0020124&rft_id=info:pmc\/PMC1182327&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1182327&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PereiraEmp12-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PereiraEmp12_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pereira, T.V.; Horwitz, R.I.; Ioannidis, J.P. (October 2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=23093165\" target=\"_blank\">\"Empirical evaluation of very large treatment effects of medical interventions\"<\/a>. <i>JAMA<\/i> <b>308<\/b> (16): 1676\u20131684. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2012.13444\" target=\"_blank\">10.1001\/jama.2012.13444<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/23093165\/\" target=\"_blank\">23093165<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=23093165\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=23093165<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Empirical+evaluation+of+very+large+treatment+effects+of+medical+interventions&rft.jtitle=JAMA&rft.aulast=Pereira%2C+T.V.%3B+Horwitz%2C+R.I.%3B+Ioannidis%2C+J.P.&rft.au=Pereira%2C+T.V.%3B+Horwitz%2C+R.I.%3B+Ioannidis%2C+J.P.&rft.date=October+2012&rft.volume=308&rft.issue=16&rft.pages=1676%E2%80%931684&rft_id=info:doi\/10.1001%2Fjama.2012.13444&rft_id=info:pmc\/23093165&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3D23093165&rfr_id=info:sid\/en.wikipedia.org:Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<dl><dd><b>ACO<\/b>: accountable care organization<\/dd>\n<dd><b>BIDMC<\/b>: Beth Israel Deaconess Medical Center<\/dd>\n<dd><b>CMS<\/b>: Center for Medicare and Medicaid<\/dd>\n<dd><b>EBM<\/b>: evidence-based medicine<\/dd>\n<dd><b>EMR<\/b>: electronic medical record<\/dd>\n<dd><b>ICU<\/b>: intensive care unit<\/dd>\n<dd><b>MIMIC<\/b>: Multiparameter Intelligent Monitoring in Intensive Care<\/dd>\n<dd><b>MIT<\/b>: Massachusetts Institute of Technology<\/dd>\n<dd><b>MIT-BIH<\/b>: Massachusetts Institute of Technology \u2013 Beth Israel Hospital<\/dd>\n<dd><b>RCT<\/b>: randomized controlled trial<\/dd>\n<dd><b>ROS<\/b>: retrospective observational study<\/dd><\/dl>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. \n<\/p><p>Per the distribution agreement, the following copyright information is also being added:\n<\/p><p>\u00a9Omar Badawi, Thomas Brennan, Leo Anthony Celi, Mengling Feng, Marzyeh Ghassemi, Andrea Ippolito, Alistair Johnson, Roger G Mark, Louis Mayaud, George Moody, Christopher Moses, Tristan Naumann, Vipan Nikore, Marco Pimentel, Tom J Pollard, Mauro Santos, David J Stone, Andrew Zimolzak, MIT Critical Data Conference 2014 Organizing Committee. Originally published in JMIR Medical Informatics (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/medinform.jmir.org\" target=\"_blank\">http:\/\/medinform.jmir.org<\/a>), 22.08.2014.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210229\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.684 seconds\nReal time usage: 0.722 seconds\nPreprocessor visited node count: 21664\/1000000\nPreprocessor generated node count: 33292\/1000000\nPost\u2010expand include size: 181418\/2097152 bytes\nTemplate argument size: 63844\/2097152 bytes\nHighest expansion depth: 17\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 654.524 1 - -total\n 77.39% 506.557 27 - Template:Citation\/core\n 61.67% 403.648 19 - Template:Cite_journal\n 20.17% 132.013 8 - Template:Cite_web\n 10.52% 68.886 1 - Template:Infobox_journal_article\n 10.13% 66.323 1 - Template:Infobox\n 7.75% 50.734 35 - Template:Citation\/identifier\n 6.10% 39.901 80 - Template:Infobox\/row\n 5.09% 33.313 38 - Template:Citation\/make_link\n 2.96% 19.367 81 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7635-0!*!0!!en!5!* and timestamp 20181213210228 and revision id 23414\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference\">https:\/\/www.limswiki.org\/index.php\/Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","bcdf1ad9d7bcb8661bc008ff0ca9fb02_images":["https:\/\/www.limswiki.org\/images\/8\/83\/Fig1_Celi_JMIRMedInformatics2014_2-2.jpg","https:\/\/www.limswiki.org\/images\/f\/f5\/Fig2_Celi_JMIRMedInformatics2014_2-2.jpg","https:\/\/www.limswiki.org\/images\/f\/ff\/Fig3_Celi_JMIRMedInformatics2014_2-2.jpg","https:\/\/www.limswiki.org\/images\/3\/36\/Fig4_Celi_JMIRMedInformatics2014_2-2.jpg"],"bcdf1ad9d7bcb8661bc008ff0ca9fb02_timestamp":1544734948,"fee95fa41e5e6f66dc7579d2ee634b76_type":"article","fee95fa41e5e6f66dc7579d2ee634b76_title":"Incorporating domain knowledge in chemical and biomedical named entity recognition with word representations (Munkhdalai et al. 2015)","fee95fa41e5e6f66dc7579d2ee634b76_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations","fee95fa41e5e6f66dc7579d2ee634b76_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Incorporating domain knowledge in chemical and biomedical named entity recognition with word representations\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nIncorporating domain knowledge in chemical and biomedical named entity recognition with word representationsJournal\n \nJournal of CheminformaticsAuthor(s)\n \nMunkhdalai, Tsendsuren; Li, Meijing; Batsuren, Khuyagbaatar; Park, Hyeon Ah; Choi, Nak Hyeon; Ryu, Keun HoAuthor affiliation(s)\n \nChungbuk National UniversityPrimary contact\n \nEmail: khryu@dblab.chungbuk.ac.krYear published\n \n2015Volume and issue\n \n7 (Suppl 1)Page(s)\n \nS9DOI\n \n10.1186\/1758-2946-7-S1-S9ISSN\n \n1758-2946Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.jcheminf.com\/content\/7\/S1\/S9Download\n \nhttp:\/\/www.jcheminf.com\/content\/pdf\/1758-2946-7-S1-S9.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Methods \n\n3.1 Preprocessing \n3.2 Feature processing \n3.3 Supervised learning \n\n\n4 Results \n\n4.1 Dataset \n4.2 Evaluation measure \n4.3 Chemical and drug named entity recognition \n4.4 Gene and protein mention recognition \n\n\n5 Discussion \n6 Conclusions \n7 Competing interests \n8 Authors' contributions \n9 Acknowledgements \n10 Declarations \n\n10.1 Funding \n\n\n11 References \n12 Notes \n\n\n\nAbstract \nBackground: Chemical and biomedical Named Entity Recognition (NER) is an essential prerequisite task before effective text mining can begin for biochemical-text data. Exploiting unlabeled text data to leverage system performance has been an active and challenging research topic in text mining due to the recent growth in the amount of biomedical literature.\nWe present a semi-supervised learning method that efficiently exploits unlabeled data in order to incorporate domain knowledge into a named entity recognition model and to leverage system performance. The proposed method includes Natural Language Processing (NLP) tasks for text preprocessing, learning word representation features from a large amount of text data for feature extraction, and conditional random fields for token classification. Other than the free text in the domain, the proposed method does not rely on any lexicon nor any dictionary in order to keep the system applicable to other NER tasks in bio-text data.\nResults: We extended BANNER, a biomedical NER system, with the proposed method. This yields an integrated system that can be applied to chemical and drug NER or biomedical NER. We call our branch of the BANNER system BANNER-CHEMDNER, which is scalable over millions of documents, processing about 530 documents per minute, is configurable via XML, and can be plugged into other systems by using the BANNER Unstructured Information Management Architecture (UIMA) interface.\nBANNER-CHEMDNER achieved an 85.68% and an 86.47% F-measure on the testing sets of CHEMDNER Chemical Entity Mention (CEM) and Chemical Document Indexing (CDI) subtasks, respectively, and achieved an 87.04% F-measure on the official testing set of the BioCreative II gene mention task, showing remarkable performance in both chemical and biomedical NER. BANNER-CHEMDNER system is available at: https:\/\/bitbucket.org\/tsendeemts\/banner-chemdner.\nKeywords: Feature Representation Learning; Semi-Supervised Learning; Named Entity Recognition; Conditional Random Fields\n\nBackground \nAs biomedical literature on servers grows exponentially in the form of semi-structured documents, biomedical text mining has been intensively investigated to find information in a more accurate and efficient manner. One essential task in developing such an information extraction system is the Named Entity Recognition (NER) process, which basically defines the boundaries between typical words and biomedical terminology in a particular text, and assigns the terminology to specific categories based on domain knowledge.\nNER performance in the newswire domain is indistinguishable from human performance, because it has an accuracy that is above 90%. However, performance has not been the same in the biomedical and chemical domain. It has been hampered by problems such as the number of new terms being created on a regular basis, the lack of standardization of technical terms between authors, and often by the fact that technical terms, such as gene names, often occur with other terminologies.[1]\nProposed solutions include rule-based, dictionary-based, and Machine Learning (ML) approaches. In the dictionary-based approach, a prepared terminology list is matched through a given text to retrieve chunks containing the location of the terminology words.[2][3] However, medical and chemical text can contain new terminology that has yet to be included in the dictionary.\nThe rule-based approach defines particular rules by observing the general features of the entities in a text.[4] In order to identify any named entity in text data, a rule-generation process has to process a huge amount of text to collect accurate rules. In addition, the rules are usually collected by domain experts, requiring a lot of effort.\nSince the machine learning approach was adopted, significant progress in biomedical and chemical NER has been achieved with methods like the Markov Model[5], the Support Vector Machine (SVM)[6][7][8], the Maximum Entropy Markov Model[9][10], and Conditional Random Fields (CRF).[2][11][12][13] However, most of the studies rely on supervised machine learning, and thus, system performance is limited by the training set that is usually built by a domain expert. Studies have shown that the word, the word n-gram and the character n-gram, and the traditional orthographic features are the base for NER, but are poor at representing domain background. \nIn order to incorporate domain knowledge into an ML model, Semi-Supervised Learning (SSL) techniques have been applied to NER. SSL is an ML approach that typically uses a large amount of unlabeled and a small amount of labeled data to build a more accurate classification model than would be built using only labeled data. SSL has received significant attention for two reasons. First, preparing a large amount of data for training requires a lot of time and effort. Second, since SSL exploits unlabeled data, the accuracy of classifiers is generally improved. There have been two different directions in SSL methods: 1) semi-supervised model induction approaches, which are the traditional methods and which incorporate domain knowledge from unlabeled data into the classification model during the training phase[14][15], and 2) supervised model induction with unsupervised, possibly semi-supervised, feature learning. The approaches in the second research direction induce better feature representation by learning from a large unlabeled corpus. Recently, the studies that apply the word representation features induced on the large text corpus have reported improvement over baseline systems in many Natural Language Processing tasks.[16][17][18]\nSeveral BioCreative shared tasks have been organized in order to evaluate researcher advancements in chemical and biomedical text mining.[19][20] BioCreative IV CHEMDNER Track consists of two subtasks: the Chemical Document Indexing subtask, where participants are asked to provide a ranked list of chemical entities found in each of the PubMed documents, and the Chemical Entity Mention recognition subtask, where they are asked to submit the start and end indices corresponding to all the chemical entities mentioned in a particular document.[21] This study extends our previous work participated in CHEMDNER task in the following ways. First, the unlabeled data used to build unsupervised models is now enriched with a large collection of PMC articles. Second, we induce word vector class models from word vectors as word representations. Third, we explore several variations of the unsupervised models built with a larger text collection than the one used before. Finally, we take a step towards a unified NER system in biomedical, chemical and medical domain by training and evaluating a biomedical NER model. These changes lead an improvement outperforming our official entries for CHEMDNER CEM and CDI subtasks by a 0.93% and a 0.73% F-measure, respectively. \nIn order to incorporate domain knowledge into the machine learning model to leverage overall system performance, we propose a semi-supervised learning method that efficiently exploits unlabeled data. The proposed method includes NLP tasks for text preprocessing and learning word representation features from a large amount of raw text data, in addition to the word, the word n-gram, the character n-gram, and the traditional orthographic information (baseline features) for feature extraction, and applies CRF for token labeling. Our method does not rely on any lexicon, nor any dictionary other than the free text in the domain, in order to keep the system applicable to the other NER tasks in bio-text data, even though the usage of such resources is reported to considerably boost system performance.\nDuring the development, we extended the BANNER system[22] with the proposed method, since that system is used in many biomedical text mining systems[23][24][25][26], showing state-of-the-art performance in biomedical Named Entity Recognition. BANNER extracts the most fundamental features for NER such as orthographic, letter n-gram and word prefix features, builds on top of a CRF model, and includes two types of postprocessing rule, namely parenthesis matching and abbreviations resolving. Our extension yields an integrated system that can be applied to chemical and drug NER or biomedical NER. We call our branch of the BANNER system BANNER-CHEMDNER, which is scalable and configurable, and can easily be plugged into other systems. BANNER-CHEMDNER shows an 85.68% and an 86.47% F-measure on the testing sets of CHEMDNER CEM and CDI subtasks, respectively, and an 87.04% F-measure on the official testing set of the BioCreative II gene mention task. \n\nMethods \nOur chemical and drug NER system design is shown in Figure 1. First, we perform preprocessing on MEDLINE and PMC document collection and then extract two different feature sets, a base feature set and a word representation feature set, in the feature processing phase. The unlabeled set of the collection is fed to unsupervised learning of the feature processing phase to build word classes. Finally, we apply the CRF sequence-labeling method to the extracted feature vectors to train the NER model. These steps will be described in subsequent sections. \n\n\n\nFigure 1. System design for chemical and drug Named Entity Recognition. The solid lines represent the flow of labeled data, and the dotted lines represent the flow of unlabeled data.\nPreprocessing \nPreprocessing is where text data is cleaned and processed via NLP tasks and is a preparatory task for feature processing.\nFirst, the text data is cleansed by removing non-informative characters and replacing special characters with corresponding spellings. The text is then tokenized with a tokenization tool. We evaluated two different tokenization strategies: a simple white space tokenizer and the BANNER simple tokenizer. The white space tokenizer splits the text simply, based on blanks within it, whereas the BANNER tokenizer breaks tokens into either a contiguous block of letters and\/or digits or a single punctuation mark. Finally, the lemma and the part-of-speech (POS) information were obtained for a further usage in the feature extraction phase. In BANNER-CHEMDNER, BioLemmatizer[27] was used for lemma extraction, which resulted in a significant improvement in overall system performance.\nIn addition to these preprocessing steps, special care is taken to parse the PMC XML documents to get the full text for the unlabeled data collection. \n\nFeature processing \nWe extract features from the preprocessed text to represent each token as a feature vector, and then an ML algorithm is employed to build a model for NER.\nThe proposed method includes extraction of the baseline and the word representation feature sets. The baseline feature set is essential in NER, but is poor at representing the domain background because it only carries some morphological and shallow-syntax information of words. On the other hand, the word representation features can be extracted by learning on a large amount of text and may be capable of introducing domain background to the NER model.\nThe entire feature set for a token is expanded to include features for the surroundings with a two-length sliding window. The word, the word n-gram, the character n-gram, lemma and the traditional orthographic information are extracted as the baseline feature set. The regular expressions that reveal orthographic information are matched to the tokens to give orthographic information. These baseline features are summarized in Table 1. \n\r\n\n\nTable 1: The baseline features\n\n\n\n\n\n\n Feature description\n\n Note\/Regular expression\n\n\nRoman number\n\n[ivxdlcm]+|[IVXDLCM]+\n\n\nPunctuation\n\n[,\\\\.;:?!]\n\n\nStart with dash\n\n\"-.*\n\n\nNucleotide sequence\n\n[atgcu]+\n\n\nNumber\n\n[0-9]+\n\n\nCapitalized\n\n[A-Z] [a-z]*\n\n\nQuote\n\n[\\\"`']\n\n\nThe lemma for the current token\n\nProvided by BioLemmatizer[23]\n\n\n2, 3 and 4-character prefixes and suffixes\n\n\n\n\n2 and 3 character n-grams\n\nToken start or end indicators are included\n\n\n2 and 3 word n-grams\n\n\n\n\n\nFor word representation features, we train Brown clustering models[28] and Word Vector (WV) models[17] on a large PubMed and PMC document collection. Brown clustering is a hierarchical word clustering method, grouping words in an input corpus to maximize the mutual information of bigrams. Therefore, the quality of a partition can be computed as a sum of mutual information weights between clusters. It runs in time O(V \u00d7 K2), where V is the size of the vocabulary and K is the number of clusters. \nThe VW model is induced via a Recurrent Neural Network (RNN) and can be seen as a language model that consists of n-dimensional continuous valued vectors, each of which represents a word in the training corpus. The RNN instance is trained to predict either the middle word of a token sequence captured in a window (CBOW) or surrounding words given the middle word of the sequence (skip-gram) depending on the model architecture.[17] The RNN becomes a log-linear classifier, once its non-linear hidden layer is removed, so the training process speeds up allowing millions of documents to process within an hour. We used a tool implemented by Mikolov et al.[17] to build our WV model from the PubMed collection.\nFurther, the word vectors are clustered using a K-means algorithm to drive a Word Vector Class (WVC) model. Since Brown clustering is a bigram model, this model may not be able to carry wide context information of a word, whereas the WVC model is an n-gram model (usually n = 5) and learns broad context information from the domain corpus. We drive the cluster label prefixes with 4, 6, 10 and 20 lengths in the Brown model by following the experiment of Turian et al.[16], and the WVC models induced from 250-dimension WVs as word representation features.\nFor feature extraction, we do not rely on any lexicon nor any dictionary other than the free text in the domain in order to keep the system applicable to other NER tasks in bio-text data, even though the usage of such resources is reported to considerably boost system performance. Most of the top performing systems participated in CHEMDNER task use the domain lexicon and observed a considerable performance boost.[29]\n\nSupervised learning \nCRF - a probabilistic undirected graphical model has been used successfully in a large number of studies on NER, because it takes advantage of sequence labelling by treating each sentence as a sequence of tokens. We apply a second-order CRF model, where the current label is conditioned on the previous two using a Begin, Inside, Outside (BIO) tagging format of the tokens. In the BIO tagging format, each token is classified either at the beginning, inside or outside of a named entity, and a postprocessing task forms the named entity mentions by merging the tagged tokens.\nWe use a Machine Learning for Language Toolkit (MALLET) library[30] for training the CRF model, because the BANNER system provides a convenient interface to work with it. The BANNER system also includes two types of general postprocessing that could be useful for any NER tasks in bio-text data. The first type is based on the symmetry of parenthesis, brackets or double quotation marks. Since these punctuation marks are always paired, BANNER drops any named entity mention containing mismatched parentheses, brackets or double quotation marks. The second type of postprocessing is dedicated to resolving abbreviations of named entities. \n\nResults \nDataset \nBecause the proposed method is a semi-supervised learning method exploiting unlabeled data during feature extraction, we prepared a large document collection of domain text in addition to annotated datasets.\nWe evaluated the system for chemical and drug NER with a CHEMDNER dataset provided by the BioCreative IV CHEMDNER task organizers. The dataset consists of 10,000 annotated documents subdivided into training and development sets of 3,500 documents each, and a testing set of 3,000 documents. The BioCreative II Gene Mention (BC2GM) dataset was used to compare the system against existing systems of the biomedical NER. The BC2GM dataset originally splits into subsets of 15,000 and 5,000 sentences for training and testing, respectively.\nFor the unlabeled data, we collected around 1.4 million PubMed abstracts and full text articles from the whole PMC database available at the time (over 2 million documents). After preprocessing, we derived two different text corpora: a PubMed abstract corpus consisting of a vocabulary of 1,136,085 entries for induction of Brown clustering models, and a merged corpus of both resources with a vocabulary of 4,359,932 entries for training WV models. Given the limited resources and time, we were able to induce the Brown clustering models only with the PubMed abstract corpus. We would like to build a Brown model with the full corpus. However, this would take several months, making it impossible to test on the CHEMDNER testing set in the given period. \n\nEvaluation measure \nFor both CDI and CEM subtasks, an exact matching criteria was used to examine three different result types. False negative (FN) and False positives (FP) are incorrect negative and positive predictions. True positives (TP) results corresponded to correct positive predictions, which are actual correct predictions. The evaluation is based on the performance measures p (precision), r (recall), and F.\nRecall denotes the percentage of correctly labeled positive results over all positive cases and is calculated as: r = TP\/(TP+FN).\nPrecision is the percentage of correctly labeled positive results over all positive labeled results and is calculated as: p = TP\/(TP+FP).\nThe F-measure is the harmonic average of precision and recall, and a balanced F-measure is expressed as: F1 = 2pr\/(p+r). \n\nChemical and drug named entity recognition \nWe trained the second-order CRF models with different features on the training set and evaluated the models on the development set. Consequently, after noticing the best settings for the hyperparameters, we trained the models on a merged set of the training and the development sets and reported the performance on the testing set. Table 2 and Table 3 show the performance comparison of the different runs with varied feature settings for CDI and CEM subtasks. Both tasks are interconnected, and their performance measures are interchangeable. \n\r\n\n\nTable 2: CDI subtask evaluation results of different runs with varied features\n\n\n\n\n\n\n\n\nDevelopment set\n\nTesting set\n\n\n Features\n\n Pre\n\n Rec\n\n F-scr\n\n Pre\n\n Rec\n\n F-scr\n\n\nBANNER setup\n\n82.83\n\n78.71\n\n80.72\n\n85.36\n\n85.29\n\n85.32\n\n\nBaseline\n\n81.71\n\n82.30\n\n82.00\n\n75.87\n\n70.55\n\n73.11\n\n\nBaseline + Brown 300\n\n81.20\n\n82.96\n\n82.58\n\n86.03\n\n85.45\n\n85.74\n\n\nBaseline + Brown 1000\n\n81.96\n\n83.24\n\n82.59\n\n86.04\n\n85.60\n\n85.82\n\n\nBaseline + Brown 1000 + WVC 1000\n\n82.73\n\n83.89\n\n83.31\n\n86.23\n\n85.37\n\n85.80\n\n\nBaseline + Brown 1000 + Brown 300\n\n82.10\n\n83.42\n\n82.76\n\n86.46\n\n85.63\n\n86.04\n\n\nBaseline + Brown 1000 + WVC 300\n\n82.43\n\n83.82\n\n83.12\n\n86.06\n\n86.06\n\n86.06\n\n\nBaseline + Brown 1000 + WVC 500\n\n82.78\n\n83.56\n\n83.17\n\n86.12\n\n86.20\n\n86.16\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 300\n\n83.10\n\n83.78\n\n83.44\n\n86.10\n\n86.31\n\n86.20\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 1000\n\n82.78\n\n83.76\n\n83.27\n\n86.19\n\n86.40\n\n86.28\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 300 + WVC 1000\n\n82.30\n\n84.05\n\n83.16\n\n86.47\n\n86.47\n\n86.47\n\n\nFeature groups are separated by (+). The parameters followed Brown and WVC are the number of classes induced in each model. Pre: Precision, Rec: Recall, F-scr: F-score.\n\n\n\r\n\n\nTable 3: CEM subtask evaluation results of different runs with varied features\n\n\n\n\n\n\n\n\nDevelopment set\n\nTesting set\n\n\n Features\n\n Pre\n\n Rec\n\n F-scr\n\n Pre\n\n Rec\n\n F-scr\n\n\nBANNER setup\n\n85.59\n\n72.74\n\n78.64\n\n88.2\n\n80.74\n\n84.31\n\n\nBaseline\n\n84.40\n\n77.34\n\n80.71\n\n79.81\n\n63.16\n\n70.51\n\n\nBaseline + Brown 300\n\n84.60\n\n78.47\n\n81.42\n\n88.67\n\n81.17\n\n84.75\n\n\nBaseline + Brown 1000\n\n84.60\n\n79.34\n\n81.89\n\n88.71\n\n81.39\n\n84.89\n\n\nBaseline + Brown 1000 + WVC 1000\n\n85.25\n\n80.30\n\n82.70\n\n88.79\n\n81.45\n\n84.96\n\n\nBaseline + Brown 1000 + Brown 300\n\n84.76\n\n79.46\n\n82.03\n\n89.10\n\n81.54\n\n85.20\n\n\nBaseline + Brown 1000 + WVC 300\n\n84.98\n\n80.07\n\n82.45\n\n88.65\n\n82.13\n\n85.26\n\n\nBaseline + Brown 1000 + WVC 500\n\n85.32\n\n79.92\n\n82.53\n\n88.77\n\n82.42\n\n85.48\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 300\n\n85.58\n\n80.10\n\n82.75\n\n88.57\n\n82.60\n\n85.48\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 1000\n\n85.28\n\n80.28\n\n82.70\n\n88.80\n\n82.60\n\n85.59\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 300 + WVC 1000\n\n84.89\n\n80.35\n\n82.56\n\n88.9\n\n82.68\n\n85.68\n\n\nFeature groups are separated by (+). The parameters followed Brown and WVC are the number of classes induced in each model. Pre: Precision, Rec: Recall, F-scr: F-score.\n\n\nWe started conducting a run with a basic feature setting and a BANNER setup, and gradually increased the complexity of the feature space for further runs. The inclusion of lemma by BioLemmatizer as a feature (baseline feature setup) in addition to BANNER feature set yielded a significant improvement on the development set. Surprisingly, the model based on the baseline features converged to an optimum, possibly a local optimum in the first 30 iterations of the training, and reported a worthless performance when evaluating the testing set. A Brown model with a larger number of clusters tended to obtain a higher F-measure. Unlike Brown clustering, a large or a lower number of WVCs degraded the performance. We found the WVC model with 500 different classes the best performing one on this task. During the induction of the WVC models, we set the WV dimension to 250 which is a trade-off value between computational cost and WV quality. Further, the combination of the different WVC models significantly improved the F-measure. We achieved the best performance, an 85.68% F-measure for CEM and an 86.47% F-measure for CDI subtasks, with the model based on the baseline feature set, the 1000-Brown clustering, and 300, 500 and 1000 WVCs (the baseline + Brown 1000 + WVC 300 + WVC 500 + WVC 1000 setup). This result is 0.93% and 0.73% higher than our best entries for CHEMDNER CEM and CDI subtasks.[31] The word representation features extracted from the unlabeled data boosted performance by a 1.37% and a 1.14% F-measure for CEM and CDI subtasks, respectively. \n\nGene and protein mention recognition \nWe trained BANNER-CHEMDNER on the BC2GM training set and evaluated it on the testing set, following the convention of the BioCreative II gene mention task. Table 4 lists the different runs with varied features included in the NER model. We expected similar results to be obtained in chemical and drug NER. However, the combination of the features did not show the expected improvement, and the best result (an 87.04% F-measure) that we obtained was a run with a model based on the baseline feature set, the 1000-Brown clustering and 500 WVCs (baseline + Brown 1000 + WVC 500). \n\r\n\n\nTable 4: BioCreative II gene mention evaluation results of different runs with varied features\n\n\n\n\n\n\n\n\nTesting set\n\n\n Features\n\n Pre\n\n Rec\n\n F-scr\n\n\nBaseline\n\n86.88\n\n84.09\n\n85.47\n\n\nBaseline + Brown 300\n\n86.49\n\n83.79\n\n85.12\n\n\nBaseline + Brown 1000\n\n86.82\n\n84.27\n\n85.53\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 300\n\n87.09\n\n84.98\n\n86.02\n\n\nBaseline + Brown 1000 + WVC 300\n\n87.95\n\n84.27\n\n86.07\n\n\nBaseline + Brown 1000 + WVC 500 + WVC 1000\n\n87.92\n\n85.49\n\n86.69\n\n\nBaseline + Brown 1000 + WVC 1000\n\n88.13\n\n85.58\n\n86.84\n\n\nBaseline + Brown 1000 + WVC 500\n\n88.02\n\n86.09\n\n87.04\n\n\nFeature groups are separated by (+). The parameters followed Brown and WVC are the number of classes induced in each model. Pre: Precision, Rec: Recall, F-scr: F-score.\n\n\nWe finally compared our best result to the task entries and the BANNER system result (Table 5). \n\r\n\n\nTable 5: Comparison of different systems on the BioCreative II testing set\n\n\n\n\n\n\n System or author\n\n BioCreative II rank\n\n Pre\n\n Rec\n\n F-scr\n\n\nAndo[32]\n\n1\n\n88.48\n\n85.97\n\n87.21\n\n\nBANNER-CHEMDNER\n\n-\n\n88.02\n\n86.08\n\n87.04\n\n\nKuo et al.[33]\n\n2\n\n89.30\n\n84.49\n\n86.83\n\n\nHuang et al.[34]\n\n3\n\n84.93\n\n88.28\n\n86.57\n\n\nBANNER\n\n-\n\n88.66\n\n84.32\n\n86.43\n\n\n\nAndo focused on a semi-supervised approach, alternating structure optimization. The system learns better feature representations from a large collection of PubMed text and uses a regularized linear classifier during the supervised training on annotated data. Kuo et al.[33] utilized domain lexicons and bi-direction parsing models of CRFs. The results from left-right and right-left parsing models were combined to produce a set of higher recall mention answers. Huang et al.[34] explored an ensemble of SVM and CRFs models. They applied intersection to the tagging results of the two SVM models and then union with the tagging results of the CRF model in their ensemble approach. \n\nDiscussion \nSeveral different runs other than the ones reported in the results section were carried out. We observed that the whitespace tokenizer performs better than the BANNER simple tokenizer for extraction of word representation features. A model with an additional feature set of raw WVs was trained and evaluated. However, we found that the word WVs did not always improve performance, and in some cases, degraded system performance with the CRF model. That is, the continuous valued WV features add some level of complexity to the model, overfitting the CRF model. Even though the WV features with the CRF model did not achieve improvement, those continuous valued features could be useful in conjunction with other classifiers, such as perceptron and support vector machines.[8] Another finding is that the Brown cluster features always improve F-measure, and the improvement is significant when the model is built on the domain text corpus.\n\nConclusions \nWe proposed a semi-supervised learning method that exploits unlabeled data efficiently in order to incorporate domain knowledge into a Named Entity Recognition model and to leverage overall system performance. The key feature of the method is learning word representations from a large amount of text data for feature extraction. The generally applicable word representation features were reported to boost system performance significantly for both chemical and biomedical NER.\nWe extended BANNER, a biomedical NER system, with the proposed method. This yields an integrated system that can be applied for chemical and drug NER or biomedical NER. We call our branch of the BANNER system BANNER-CHEMDNER.\nBANNER-CHEMDNER achieves an 85.68% and an 86.47% F-measure on the testing set for CHEMDNER CEM and CDI subtasks, respectively, and an 87.04% F-measure on the official testing set of the BioCreative II gene mention task, showing remarkable performance for both chemical and biomedical NER.\nOur future work should be towards a unified NER system in biomedical, chemical and medical domain, based on the generally applicable word representations.\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors' contributions \nTM conceived the study, participated in its design, developed the extension program, and drafted the manuscript. ML carried out calculations and helped draft the manuscript. KB participated in data analysis and helped draft the manuscript. HAP participated in study design. NHC helped draft the manuscript. KHR provided system design, valuable guidance, editing, and research grant. All authors read and approved the final manuscript.\n\nAcknowledgements \nThis research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (No-2013R1A2A2A01068923) and by a National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2008-0062611). We would like to thank the BANNER system authors for developing BANNER extendable.\n\nDeclarations \nFunding \nThis research and publication was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (No-2013R1A2A2A01068923) and by a National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2008-0062611).\nThis article has been published as part of Journal of Cheminformatics Volume 7 Supplement 1, 2015: Text mining for chemistry and the CHEMDNER track. The full contents of the supplement are available online at http:\/\/www.jcheminf.com\/supplements\/7\/S1.\n\nReferences \n\n\n\u2191 Dai, H.J.; Chang, Y.C.; Tsai, R.T.H.; Hsu, W.L. (2010). \"New Challenges for Biological Text-Mining in the Next Decade\". Journal of Computer Science and Technology 25 (1): 169-179. doi:10.1007\/s11390-010-9313-5.   \n\n\u2191 2.0 2.1 Rockt\u00e4schel, T.; Weidlich, M.; Leser, U. (2012). \"ChemSpot: A hybrid system for chemical named entity recognition\". Bioinformatics 28 (12): 1633-1640. doi:10.1093\/bioinformatics\/bts183. PMID 22500000.   \n\n\u2191 Rockt\u00e4schel, T.; Weidlich, M.; Leser, U. (2009). \"A dictionary to identify small molecules and drugs in free text\". Bioinformatics 25 (22): 2983-2991. doi:10.1093\/bioinformatics\/btp535. PMID 19759196.   \n\n\u2191 Segura-Bedmar, I.; Mart\u00ednez, P.; Segura-Bedmar, M. (2008). \"Drug name recognition and classification in biomedical texts: A case study outlining approaches underpinning automated systems\". Drug Discovery Today 13 (17\u201318): 816\u2013823. doi:10.1016\/j.drudis.2008.06.001. PMID 18602492.   \n\n\u2191 Zhao, S. (2004). \"Named Entity Recognition in Biomedical Texts using an HMM Model\". Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications 2004: 84-87. http:\/\/dl.acm.org\/citation.cfm?id=1567594.1567613 .   \n\n\u2191 Zhao, G.; Su, J. (2004). \"Exploring deep knowledge resources in biomedical name recognition\". Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications 2004: 96-99. http:\/\/dl.acm.org\/citation.cfm?id=1567616 .   \n\n\u2191 Mitsumori, T.; Fation, S.; Murata, M.; Doi, K.; Doi, H. (2005). \"Gene\/protein name recognition based on support vector machine using dictionary as features\". BMC Bioinformatics 6 (Suppl 1): S8. doi:10.1186\/1471-2105-6-S1-S8. PMC PMC1869022. PMID 15960842. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1869022 .   \n\n\u2191 8.0 8.1 Collier, N; Takeuchi, K. (2004). \"Comparison of character-level and part of speech features for name recognition in biomedical texts\". Journal of Biomedical Informatics 37 (6): 423-35. doi:10.1016\/j.jbi.2004.08.008. PMID 15542016.   \n\n\u2191 Finkel, J.; Dingare, S.; Nguyen, H.; Nissim, M.; Manning, C.; Sinclair, G. (2004). \"Exploiting context for biomedical entity recognition: from syntax to the web\". Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications 2004: 88\u201391. http:\/\/dl.acm.org\/citation.cfm?id=1567614 .   \n\n\u2191 Corbett, P.; Copestake, A. (2008). \"Cascaded classifiers for confidence-based chemical named entity recognition\". BMC Bioinformatics 9 (Suppl 11): S4. doi:10.1186\/1471-2105-9-S11-S4. PMC PMC2586753. PMID 19025690. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2586753 .   \n\n\u2191 Settles, B. (2004). \"Biomedical named entity recognition using conditional random fields and rich feature sets\". Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications 2004: 104-107. http:\/\/dl.acm.org\/citation.cfm?id=1567618 .   \n\n\u2191 Hsu, C.; Chang, Y.; Kuo, C.; Lin, Y.; Huang, H.; Chung, I. (2008). \"Integrating high dimensional bi-directional parsing models for gene mention tagging\". Bioinformatics 24 (13): i286-94. doi:10.1093\/bioinformatics\/btn183. PMC PMC2718659. PMID 18586726. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2718659 .   \n\n\u2191 Li, Y.; Lin, H.; Yang, Z. (2009). \"Incorporating rich background knowledge for gene named entity classification and recognition\". BMC Bioinformatics 10: 223. doi:10.1186\/1471-2105-10-223. PMC PMC2725142. PMID 19615051. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2725142 .   \n\n\u2191 Munkhdalai, T.; Li, M.; Kim, T.; Namsrai, O.; Jeong, S.; Shin, J.; Ryu, K.H. (2012). \"Bio Named Entity Recognition based on Co-training Algorithm\". Proceedings 26th International Conference on Advanced Information Networking and Applications Workshops 2012: 857-862. doi:10.1109\/WAINA.2012.75.   \n\n\u2191 Munkhdalai, T.; Li, M.; Yun, U.; Namsrai, O.; Ryu, K.H. (2012). \"An Active Co-Training Algorithm for Biomedical Named-Entity Recognition\". Journal of Information Processing Systems 8 (4): 575-588. doi:10.3745\/JIPS.2012.8.4.575.   \n\n\u2191 16.0 16.1 Turian, J.; Ratinov, L.; Bengio, Y. (2010). \"Word representations: A simple and general method for semi-supervised learning\". Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics 2010: 384-394. http:\/\/cogcomp.cs.illinois.edu\/page\/publication_view\/653 .   \n\n\u2191 17.0 17.1 17.2 17.3 Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. (2013). \"Efficient Estimation of Word Representations in Vector Space\". Proceedings of Workshop at ICLR. http:\/\/arxiv.org\/abs\/1301.3781 .   \n\n\u2191 Socher, R.; Lin, C.C.; Ng, A.Y.; Manning, C.D. (2011). \"Parsing Natural Scenes and Natural Language with Recursive Neural Networks\". Proceedings of the 28th International Conference on Machine Learning 2011: 129\u2013136. http:\/\/machinelearning.wustl.edu\/mlpapers\/papers\/ICML2011Socher_125 .   \n\n\u2191 Arighi, C.N.; Lu, Z.; Krallinger, M.; Cohen, K.B.; Wilbur, J.W.; Valencia, A.; Hirschman, L.; Wu, C.H. (2011). \"Overview of the BioCreative III Workshop\". Bioinformatics 12 (Suppl 8): S1. doi:10.1186\/1471-2105-12-S8-S1. PMC PMC3269932. PMID 22151647. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3269932 .   \n\n\u2191 Arighi, C.N.; Wu, C.H.; Cohen, K.B.; Hirschman, L.; Krallinger, M.; Valencia, A.; Lu, Z.; Wilbur, J.W.; Wiegers, T.C. (2014). \"BioCreative-IV virtual issue\". Database 2014: bau039. doi:10.1093\/database\/bau039. PMC PMC4030502. PMID 24852177. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030502 .   \n\n\u2191 Krallinger, M.; Rabal, O.; Leitner, F.; et al. (2015). \"The CHEMDNER corpus of chemicals and drugs and its annotation principles\". Journal of Cheminformatics 7 (Suppl 1): S2. doi:10.1186\/1758-2946-7-S1-S2. PMC PMC4331692. PMID 25810773. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331692 .   \n\n\u2191 Leaman, R.; Gonzalez, G. (2008). \"BANNER: an executable survey of advances in biomedical named entity recognition\". Proceedings of Pacific Symposium on Biocomputing 2008: 652-63. PMID 18229723.   \n\n\u2191 23.0 23.1 Leaman, R.; Islamaj, D.R.; Lu, Z. (2013). \"DNorm: disease name normalization with pairwise learning to rank\". Bioinformatics 29 (1): 22-28. doi:10.1093\/bioinformatics\/bts639. PMID 23104890.   \n\n\u2191 Wei, C.H.; Harris, B.R.; Kao, H.Y.; Lu, Z. (2013). \"tmVar: a text mining approach for extracting sequence variants in biomedical literature\". Bioinformatics 29 (11): 1433-1439. doi:10.1093\/bioinformatics\/btt156. PMC PMC3661051. PMID 23564842. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3661051 .   \n\n\u2191 Bj\u00f6rne, J.; Ginter, F.; Pyysalo, S.; Tsujii, J.; Salakoski, T. (2010). \"Complex event extraction at PubMed scale\". Bioinformatics 26 (12): i382-i390. doi:10.1093\/bioinformatics\/btq180. PMC PMC2881365. PMID 20529932. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2881365 .   \n\n\u2191 Gerner, M.; Sarafraz, F.; Bergman, C.M.; Nenadic, G. (2012). \"BioContext: an integrated text mining system for large-scale extraction and contextualization of biomolecular events\". Bioinformatics 28 (16): 2154-2161. doi:10.1093\/bioinformatics\/bts332. PMC PMC3413385. PMID 22711795. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3413385 .   \n\n\u2191 Liu, H.; Christiansen, T.; Baumgartner, W.A.; Verspoor, K. (2012). \"BioLemmatizer: a lemmatization tool for morphological processing of biomedical text\". Journal of Biomedical Semantic 3: 3. doi:10.1186\/2041-1480-3-3. PMC PMC3359276. PMID 22464129. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3359276 .   \n\n\u2191 Brown, P.F.; deSouza, P.V.; Mercer, R.L.; Pietra, V.J.D.; Lai, J.C. (1992). \"Class-Based n-gram Models of Natural Language\". Computational Linguistics 18 (4): 467-479. http:\/\/dl.acm.org\/citation.cfm?id=176316 .   \n\n\u2191 Krallinger, M.; Leitner, F.; Rabal, O.; Vazquez, M.; Oyarzabal, J.; Valencia, A. (1995). \"CHEMDNER: The drugs and chemical names extraction challenge\". Journal of Cheminformatics 7 (Suppl 1): S1. doi:10.1186\/1758-2946-7-S1-S1. PMC PMC4331685. PMID 25810766. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331685 .   \n\n\u2191 McCallum, Andrew Kachites (2013). \"MALLET: A Machine Learning for Language Toolkit\". University of Massachusetts. http:\/\/mallet.cs.umass.edu\/ .   \n\n\u2191 Munkhdalai, T.; Li, M.; Batsuren, K.; Ryu, K.H. (2013). \"BANNER-CHEMDNER: Incorporating Domain Knowledge in Chemical and Drug Named Entity Recognition\". Proceedings of the Fourth BioCreative Challenge Evaluation Workshop 2: 135-139. http:\/\/www.biocreative.org\/resources\/publications\/chemdner-proceed-publications\/ .   \n\n\u2191 Ando, R.K. (2007). \"BioCreative II Gene Mention Tagging System at IBM Watson\". Proceedings of the Second BioCreative Challenge Evaluation Workshop: 101-103. http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.121.8576 .   \n\n\u2191 33.0 33.1 Kuo, C.J.; Chang, Y.M.; Huang, H.S.; Lin, K.T.; Yang, B.H.; Lin, Y.S.; Hsu, C.N.; Chung, I.F. (2007). \"Rich feature set, unification of bidirectional parsing and dictionary filtering for high F-score gene mention tagging\" (PDF). Proceedings of the Second BioCreative Challenge Evaluation Workshop: 105-107. http:\/\/www.iis.sinica.edu.tw\/~chunnan\/DOWNLOADS\/T111_v16.pdf .   \n\n\u2191 34.0 34.1 Kuo, C.J.; Chang, Y.M.; Huang, H.S.; Lin, K.T.; Yang, B.H.; Lin, Y.S.; Hsu, C.N.; Chung, I.F. (2007). \"High-recall gene mention recognition by unification of multiple backward parsing models\". Proceedings of the Second BioCreative Challenge Evaluation Workshop: 109-111.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases citation information was missing and was added to make the reference more useful. Numbers in Tables 2\u20135 were all converted to two decimal places for consistency.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\">https:\/\/www.limswiki.org\/index.php\/Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on reporting\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:56.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,521 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","fee95fa41e5e6f66dc7579d2ee634b76_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Incorporating domain knowledge in chemical and biomedical named entity recognition with word representations<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Chemical and biomedical Named Entity Recognition (NER) is an essential prerequisite task before effective text mining can begin for biochemical-text data. Exploiting unlabeled text data to leverage system performance has been an active and challenging research topic in text mining due to the recent growth in the amount of biomedical literature.\n<\/p><p>We present a semi-supervised learning method that efficiently exploits unlabeled data in order to incorporate domain knowledge into a named entity recognition model and to leverage system performance. The proposed method includes Natural Language Processing (NLP) tasks for text preprocessing, learning word representation features from a large amount of text data for feature extraction, and conditional random fields for token classification. Other than the free text in the domain, the proposed method does not rely on any lexicon nor any dictionary in order to keep the system applicable to other NER tasks in bio-text data.\n<\/p><p><b>Results:<\/b> We extended BANNER, a biomedical NER system, with the proposed method. This yields an integrated system that can be applied to chemical and drug NER or biomedical NER. We call our branch of the BANNER system BANNER-CHEMDNER, which is scalable over millions of documents, processing about 530 documents per minute, is configurable via XML, and can be plugged into other systems by using the BANNER Unstructured Information Management Architecture (UIMA) interface.\n<\/p><p>BANNER-CHEMDNER achieved an 85.68% and an 86.47% F-measure on the testing sets of CHEMDNER Chemical Entity Mention (CEM) and Chemical Document Indexing (CDI) subtasks, respectively, and achieved an 87.04% F-measure on the official testing set of the BioCreative II gene mention task, showing remarkable performance in both chemical and biomedical NER. BANNER-CHEMDNER system is available at: <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/bitbucket.org\/tsendeemts\/banner-chemdner\" target=\"_blank\">https:\/\/bitbucket.org\/tsendeemts\/banner-chemdner<\/a>.\n<\/p><p><b>Keywords:<\/b> Feature Representation Learning; Semi-Supervised Learning; Named Entity Recognition; Conditional Random Fields\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>As biomedical literature on servers grows exponentially in the form of semi-structured documents, biomedical text mining has been intensively investigated to find <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> in a more accurate and efficient manner. One essential task in developing such an information extraction system is the Named Entity Recognition (NER) process, which basically defines the boundaries between typical words and biomedical terminology in a particular text, and assigns the terminology to specific categories based on domain knowledge.\n<\/p><p>NER performance in the newswire domain is indistinguishable from human performance, because it has an accuracy that is above 90%. However, performance has not been the same in the biomedical and chemical domain. It has been hampered by problems such as the number of new terms being created on a regular basis, the lack of standardization of technical terms between authors, and often by the fact that technical terms, such as gene names, often occur with other terminologies.<sup id=\"rdp-ebb-cite_ref-DaiNew10_1-0\" class=\"reference\"><a href=\"#cite_note-DaiNew10-1\" rel=\"external_link\">[1]<\/a><\/sup>\n<\/p><p>Proposed solutions include rule-based, dictionary-based, and Machine Learning (ML) approaches. In the dictionary-based approach, a prepared terminology list is matched through a given text to retrieve chunks containing the location of the terminology words.<sup id=\"rdp-ebb-cite_ref-RocktaschelChem12_2-0\" class=\"reference\"><a href=\"#cite_note-RocktaschelChem12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HettneADict09_3-0\" class=\"reference\"><a href=\"#cite_note-HettneADict09-3\" rel=\"external_link\">[3]<\/a><\/sup> However, medical and chemical text can contain new terminology that has yet to be included in the dictionary.\n<\/p><p>The rule-based approach defines particular rules by observing the general features of the entities in a text.<sup id=\"rdp-ebb-cite_ref-SeguraDrug08_4-0\" class=\"reference\"><a href=\"#cite_note-SeguraDrug08-4\" rel=\"external_link\">[4]<\/a><\/sup> In order to identify any named entity in text data, a rule-generation process has to process a huge amount of text to collect accurate rules. In addition, the rules are usually collected by domain experts, requiring a lot of effort.\n<\/p><p>Since the machine learning approach was adopted, significant progress in biomedical and chemical NER has been achieved with methods like the Markov Model<sup id=\"rdp-ebb-cite_ref-ZhaoNamed04_5-0\" class=\"reference\"><a href=\"#cite_note-ZhaoNamed04-5\" rel=\"external_link\">[5]<\/a><\/sup>, the Support Vector Machine (SVM)<sup id=\"rdp-ebb-cite_ref-ZhaoExp04_6-0\" class=\"reference\"><a href=\"#cite_note-ZhaoExp04-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MitsumoriGene05_7-0\" class=\"reference\"><a href=\"#cite_note-MitsumoriGene05-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CollierComp04_8-0\" class=\"reference\"><a href=\"#cite_note-CollierComp04-8\" rel=\"external_link\">[8]<\/a><\/sup>, the Maximum Entropy Markov Model<sup id=\"rdp-ebb-cite_ref-FinkelExp04_9-0\" class=\"reference\"><a href=\"#cite_note-FinkelExp04-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CorbettCasc08_10-0\" class=\"reference\"><a href=\"#cite_note-CorbettCasc08-10\" rel=\"external_link\">[10]<\/a><\/sup>, and Conditional Random Fields (CRF).<sup id=\"rdp-ebb-cite_ref-RocktaschelChem12_2-1\" class=\"reference\"><a href=\"#cite_note-RocktaschelChem12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SettlesBio04_11-0\" class=\"reference\"><a href=\"#cite_note-SettlesBio04-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HsuInt08_12-0\" class=\"reference\"><a href=\"#cite_note-HsuInt08-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiInc09_13-0\" class=\"reference\"><a href=\"#cite_note-LiInc09-13\" rel=\"external_link\">[13]<\/a><\/sup> However, most of the studies rely on supervised machine learning, and thus, system performance is limited by the training set that is usually built by a domain expert. Studies have shown that the word, the word <i>n<\/i>-gram and the character <i>n<\/i>-gram, and the traditional orthographic features are the base for NER, but are poor at representing domain background. \n<\/p><p>In order to incorporate domain knowledge into an ML model, Semi-Supervised Learning (SSL) techniques have been applied to NER. SSL is an ML approach that typically uses a large amount of unlabeled and a small amount of labeled data to build a more accurate classification model than would be built using only labeled data. SSL has received significant attention for two reasons. First, preparing a large amount of data for training requires a lot of time and effort. Second, since SSL exploits unlabeled data, the accuracy of classifiers is generally improved. There have been two different directions in SSL methods: 1) semi-supervised model induction approaches, which are the traditional methods and which incorporate domain knowledge from unlabeled data into the classification model during the training phase<sup id=\"rdp-ebb-cite_ref-MunkhdalaiBio12_14-0\" class=\"reference\"><a href=\"#cite_note-MunkhdalaiBio12-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MunkhdalaiAn12_15-0\" class=\"reference\"><a href=\"#cite_note-MunkhdalaiAn12-15\" rel=\"external_link\">[15]<\/a><\/sup>, and 2) supervised model induction with unsupervised, possibly semi-supervised, feature learning. The approaches in the second research direction induce better feature representation by learning from a large unlabeled corpus. Recently, the studies that apply the word representation features induced on the large text corpus have reported improvement over baseline systems in many Natural Language Processing tasks.<sup id=\"rdp-ebb-cite_ref-TurianWord10_16-0\" class=\"reference\"><a href=\"#cite_note-TurianWord10-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MikolovEff13_17-0\" class=\"reference\"><a href=\"#cite_note-MikolovEff13-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SocherPars11_18-0\" class=\"reference\"><a href=\"#cite_note-SocherPars11-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/p><p>Several BioCreative shared tasks have been organized in order to evaluate researcher advancements in chemical and biomedical text mining.<sup id=\"rdp-ebb-cite_ref-ArighiOver11_19-0\" class=\"reference\"><a href=\"#cite_note-ArighiOver11-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ArighiBio14_20-0\" class=\"reference\"><a href=\"#cite_note-ArighiBio14-20\" rel=\"external_link\">[20]<\/a><\/sup> BioCreative IV CHEMDNER Track consists of two subtasks: the Chemical Document Indexing subtask, where participants are asked to provide a ranked list of chemical entities found in each of the PubMed documents, and the Chemical Entity Mention recognition subtask, where they are asked to submit the start and end indices corresponding to all the chemical entities mentioned in a particular document.<sup id=\"rdp-ebb-cite_ref-KrallingerThe15_21-0\" class=\"reference\"><a href=\"#cite_note-KrallingerThe15-21\" rel=\"external_link\">[21]<\/a><\/sup> This study extends our previous work participated in CHEMDNER task in the following ways. First, the unlabeled data used to build unsupervised models is now enriched with a large collection of PMC articles. Second, we induce word vector class models from word vectors as word representations. Third, we explore several variations of the unsupervised models built with a larger text collection than the one used before. Finally, we take a step towards a unified NER system in biomedical, chemical and medical domain by training and evaluating a biomedical NER model. These changes lead an improvement outperforming our official entries for CHEMDNER CEM and CDI subtasks by a 0.93% and a 0.73% F-measure, respectively. \n<\/p><p>In order to incorporate domain knowledge into the machine learning model to leverage overall system performance, we propose a semi-supervised learning method that efficiently exploits unlabeled data. The proposed method includes NLP tasks for text preprocessing and learning word representation features from a large amount of raw text data, in addition to the word, the word n-gram, the character n-gram, and the traditional orthographic information (baseline features) for feature extraction, and applies CRF for token labeling. Our method does not rely on any lexicon, nor any dictionary other than the free text in the domain, in order to keep the system applicable to the other NER tasks in bio-text data, even though the usage of such resources is reported to considerably boost system performance.\n<\/p><p>During the development, we extended the BANNER system<sup id=\"rdp-ebb-cite_ref-LeamanBAN08_22-0\" class=\"reference\"><a href=\"#cite_note-LeamanBAN08-22\" rel=\"external_link\">[22]<\/a><\/sup> with the proposed method, since that system is used in many biomedical text mining systems<sup id=\"rdp-ebb-cite_ref-LeamanDNorm13_23-0\" class=\"reference\"><a href=\"#cite_note-LeamanDNorm13-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-Wei_tmVar13_24-0\" class=\"reference\"><a href=\"#cite_note-Wei_tmVar13-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BjornComp10_25-0\" class=\"reference\"><a href=\"#cite_note-BjornComp10-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GernerBio12_26-0\" class=\"reference\"><a href=\"#cite_note-GernerBio12-26\" rel=\"external_link\">[26]<\/a><\/sup>, showing state-of-the-art performance in biomedical Named Entity Recognition. BANNER extracts the most fundamental features for NER such as orthographic, letter n-gram and word prefix features, builds on top of a CRF model, and includes two types of postprocessing rule, namely parenthesis matching and abbreviations resolving. Our extension yields an integrated system that can be applied to chemical and drug NER or biomedical NER. We call our branch of the BANNER system BANNER-CHEMDNER, which is scalable and configurable, and can easily be plugged into other systems. BANNER-CHEMDNER shows an 85.68% and an 86.47% F-measure on the testing sets of CHEMDNER CEM and CDI subtasks, respectively, and an 87.04% F-measure on the official testing set of the BioCreative II gene mention task. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>Our chemical and drug NER system design is shown in Figure 1. First, we perform preprocessing on MEDLINE and PMC document collection and then extract two different feature sets, a base feature set and a word representation feature set, in the feature processing phase. The unlabeled set of the collection is fed to unsupervised learning of the feature processing phase to build word classes. Finally, we apply the CRF sequence-labeling method to the extracted feature vectors to train the NER model. These steps will be described in subsequent sections. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Munkhdalai_JCheminformatics2015_7-1.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"ddb3d9da2f1bfe573a2e2808ee75de59\"><img alt=\"Fig1 Munkhdalai JCheminformatics2015 7-1.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/c4\/Fig1_Munkhdalai_JCheminformatics2015_7-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1. System design for chemical and drug Named Entity Recognition.<\/b> The solid lines represent the flow of labeled data, and the dotted lines represent the flow of unlabeled data.<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Preprocessing\">Preprocessing<\/span><\/h3>\n<p>Preprocessing is where text data is cleaned and processed via NLP tasks and is a preparatory task for feature processing.\n<\/p><p>First, the text data is cleansed by removing non-informative characters and replacing special characters with corresponding spellings. The text is then tokenized with a tokenization tool. We evaluated two different tokenization strategies: a simple white space tokenizer and the BANNER simple tokenizer. The white space tokenizer splits the text simply, based on blanks within it, whereas the BANNER tokenizer breaks tokens into either a contiguous block of letters and\/or digits or a single punctuation mark. Finally, the lemma and the part-of-speech (POS) information were obtained for a further usage in the feature extraction phase. In BANNER-CHEMDNER, BioLemmatizer<sup id=\"rdp-ebb-cite_ref-LiuBioL12_27-0\" class=\"reference\"><a href=\"#cite_note-LiuBioL12-27\" rel=\"external_link\">[27]<\/a><\/sup> was used for lemma extraction, which resulted in a significant improvement in overall system performance.\n<\/p><p>In addition to these preprocessing steps, special care is taken to parse the PMC XML documents to get the full text for the unlabeled data collection. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Feature_processing\">Feature processing<\/span><\/h3>\n<p>We extract features from the preprocessed text to represent each token as a feature vector, and then an ML algorithm is employed to build a model for NER.\n<\/p><p>The proposed method includes extraction of the baseline and the word representation feature sets. The baseline feature set is essential in NER, but is poor at representing the domain background because it only carries some morphological and shallow-syntax information of words. On the other hand, the word representation features can be extracted by learning on a large amount of text and may be capable of introducing domain background to the NER model.\n<\/p><p>The entire feature set for a token is expanded to include features for the surroundings with a two-length sliding window. The word, the word <i>n<\/i>-gram, the character n-gram, lemma and the traditional orthographic information are extracted as the baseline feature set. The regular expressions that reveal orthographic information are matched to the tokens to give orthographic information. These baseline features are summarized in Table 1. \n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 1<\/b>: The baseline features<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Feature description\n<\/th>\n<th> Note\/Regular expression\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Roman number\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">[ivxdlcm]+|[IVXDLCM]+\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Punctuation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">[,\\\\.;:?!]\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Start with dash\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\"-.*\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nucleotide sequence\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">[atgcu]+\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Number\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">[0-9]+\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Capitalized\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">[A-Z] [a-z]*\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Quote\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">[\\\"`']\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The lemma for the current token\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Provided by BioLemmatizer<sup id=\"rdp-ebb-cite_ref-LeamanDNorm13_23-1\" class=\"reference\"><a href=\"#cite_note-LeamanDNorm13-23\" rel=\"external_link\">[23]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2, 3 and 4-character prefixes and suffixes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 and 3 character n-grams\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Token start or end indicators are included\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2 and 3 word n-grams\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>For word representation features, we train Brown clustering models<sup id=\"rdp-ebb-cite_ref-BrownClass92_28-0\" class=\"reference\"><a href=\"#cite_note-BrownClass92-28\" rel=\"external_link\">[28]<\/a><\/sup> and Word Vector (WV) models<sup id=\"rdp-ebb-cite_ref-MikolovEff13_17-1\" class=\"reference\"><a href=\"#cite_note-MikolovEff13-17\" rel=\"external_link\">[17]<\/a><\/sup> on a large PubMed and PMC document collection. Brown clustering is a hierarchical word clustering method, grouping words in an input corpus to maximize the mutual information of bigrams. Therefore, the quality of a partition can be computed as a sum of mutual information weights between clusters. It runs in time O(V \u00d7 K<sup>2<\/sup>), where V is the size of the vocabulary and K is the number of clusters. \n<\/p><p>The VW model is induced via a Recurrent Neural Network (RNN) and can be seen as a language model that consists of <i>n<\/i>-dimensional continuous valued vectors, each of which represents a word in the training corpus. The RNN instance is trained to predict either the middle word of a token sequence captured in a window (CBOW) or surrounding words given the middle word of the sequence (skip-gram) depending on the model architecture.<sup id=\"rdp-ebb-cite_ref-MikolovEff13_17-2\" class=\"reference\"><a href=\"#cite_note-MikolovEff13-17\" rel=\"external_link\">[17]<\/a><\/sup> The RNN becomes a log-linear classifier, once its non-linear hidden layer is removed, so the training process speeds up allowing millions of documents to process within an hour. We used a tool implemented by Mikolov et al.<sup id=\"rdp-ebb-cite_ref-MikolovEff13_17-3\" class=\"reference\"><a href=\"#cite_note-MikolovEff13-17\" rel=\"external_link\">[17]<\/a><\/sup> to build our WV model from the PubMed collection.\n<\/p><p>Further, the word vectors are clustered using a K-means algorithm to drive a Word Vector Class (WVC) model. Since Brown clustering is a bigram model, this model may not be able to carry wide context information of a word, whereas the WVC model is an <i>n<\/i>-gram model (usually <i>n<\/i> = 5) and learns broad context information from the domain corpus. We drive the cluster label prefixes with 4, 6, 10 and 20 lengths in the Brown model by following the experiment of Turian et al.<sup id=\"rdp-ebb-cite_ref-TurianWord10_16-1\" class=\"reference\"><a href=\"#cite_note-TurianWord10-16\" rel=\"external_link\">[16]<\/a><\/sup>, and the WVC models induced from 250-dimension WVs as word representation features.\n<\/p><p>For feature extraction, we do not rely on any lexicon nor any dictionary other than the free text in the domain in order to keep the system applicable to other NER tasks in bio-text data, even though the usage of such resources is reported to considerably boost system performance. Most of the top performing systems participated in CHEMDNER task use the domain lexicon and observed a considerable performance boost.<sup id=\"rdp-ebb-cite_ref-KrallingerCHEM15_29-0\" class=\"reference\"><a href=\"#cite_note-KrallingerCHEM15-29\" rel=\"external_link\">[29]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Supervised_learning\">Supervised learning<\/span><\/h3>\n<p>CRF - a probabilistic undirected graphical model has been used successfully in a large number of studies on NER, because it takes advantage of sequence labelling by treating each sentence as a sequence of tokens. We apply a second-order CRF model, where the current label is conditioned on the previous two using a Begin, Inside, Outside (BIO) tagging format of the tokens. In the BIO tagging format, each token is classified either at the beginning, inside or outside of a named entity, and a postprocessing task forms the named entity mentions by merging the tagged tokens.\n<\/p><p>We use a Machine Learning for Language Toolkit (MALLET) library<sup id=\"rdp-ebb-cite_ref-McCallumMallet02_30-0\" class=\"reference\"><a href=\"#cite_note-McCallumMallet02-30\" rel=\"external_link\">[30]<\/a><\/sup> for training the CRF model, because the BANNER system provides a convenient interface to work with it. The BANNER system also includes two types of general postprocessing that could be useful for any NER tasks in bio-text data. The first type is based on the symmetry of parenthesis, brackets or double quotation marks. Since these punctuation marks are always paired, BANNER drops any named entity mention containing mismatched parentheses, brackets or double quotation marks. The second type of postprocessing is dedicated to resolving abbreviations of named entities. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Dataset\">Dataset<\/span><\/h3>\n<p>Because the proposed method is a semi-supervised learning method exploiting unlabeled data during feature extraction, we prepared a large document collection of domain text in addition to annotated datasets.\n<\/p><p>We evaluated the system for chemical and drug NER with a CHEMDNER dataset provided by the BioCreative IV CHEMDNER task organizers. The dataset consists of 10,000 annotated documents subdivided into training and development sets of 3,500 documents each, and a testing set of 3,000 documents. The BioCreative II Gene Mention (BC2GM) dataset was used to compare the system against existing systems of the biomedical NER. The BC2GM dataset originally splits into subsets of 15,000 and 5,000 sentences for training and testing, respectively.\n<\/p><p>For the unlabeled data, we collected around 1.4 million PubMed abstracts and full text articles from the whole PMC database available at the time (over 2 million documents). After preprocessing, we derived two different text corpora: a PubMed abstract corpus consisting of a vocabulary of 1,136,085 entries for induction of Brown clustering models, and a merged corpus of both resources with a vocabulary of 4,359,932 entries for training WV models. Given the limited resources and time, we were able to induce the Brown clustering models only with the PubMed abstract corpus. We would like to build a Brown model with the full corpus. However, this would take several months, making it impossible to test on the CHEMDNER testing set in the given period. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Evaluation_measure\">Evaluation measure<\/span><\/h3>\n<p>For both CDI and CEM subtasks, an exact matching criteria was used to examine three different result types. False negative (<i>FN<\/i>) and False positives (<i>FP<\/i>) are incorrect negative and positive predictions. True positives (<i>TP<\/i>) results corresponded to correct positive predictions, which are actual correct predictions. The evaluation is based on the performance measures <i>p<\/i> (precision), <i>r<\/i> (recall), and <i>F<\/i>.\n<\/p><p>Recall denotes the percentage of correctly labeled positive results over all positive cases and is calculated as: <i>r = TP\/(TP+FN)<\/i>.\n<\/p><p>Precision is the percentage of correctly labeled positive results over all positive labeled results and is calculated as: <i>p = TP\/(TP+FP)<\/i>.\n<\/p><p>The F-measure is the harmonic average of precision and recall, and a balanced F-measure is expressed as: <i>F<sub>1<\/sub> = 2pr\/(p+r)<\/i>. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Chemical_and_drug_named_entity_recognition\">Chemical and drug named entity recognition<\/span><\/h3>\n<p>We trained the second-order CRF models with different features on the training set and evaluated the models on the development set. Consequently, after noticing the best settings for the hyperparameters, we trained the models on a merged set of the training and the development sets and reported the performance on the testing set. Table 2 and Table 3 show the performance comparison of the different runs with varied feature settings for CDI and CEM subtasks. Both tasks are interconnected, and their performance measures are interchangeable. \n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 2<\/b>: CDI subtask evaluation results of different runs with varied features<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th>\n<\/th>\n<th colspan=\"3\">Development set\n<\/th>\n<th colspan=\"3\">Testing set\n<\/th><\/tr>\n<tr>\n<th> Features\n<\/th>\n<th> Pre\n<\/th>\n<th> Rec\n<\/th>\n<th> F-scr\n<\/th>\n<th> Pre\n<\/th>\n<th> Rec\n<\/th>\n<th> F-scr\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BANNER setup\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.83\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">78.71\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.72\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.36\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.29\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.32\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.71\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">75.87\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">70.55\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">73.11\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.96\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.58\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.45\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.74\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.96\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.24\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.59\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.82\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.73\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.89\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.23\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.37\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.80\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + Brown 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.42\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.76\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.63\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.04\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.43\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.82\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.06\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.06\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.06\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.78\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.56\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.12\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.20\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.16\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>83.10<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.78\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>83.44<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.31\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.20\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.78\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.76\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.28\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 300 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>84.05<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>86.47<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>86.47<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>86.47<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\">Feature groups are separated by (+). The parameters followed Brown and WVC are the number of classes induced in each model. Pre: Precision, Rec: Recall, F-scr: F-score.\n<\/td><\/tr><\/table>\n<\/td><\/tr><\/table>\n<p><br \/>\n<\/p>\n<dl><dd><b>Table 3<\/b>: CEM subtask evaluation results of different runs with varied features<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th>\n<\/th>\n<th colspan=\"3\">Development set\n<\/th>\n<th colspan=\"3\">Testing set\n<\/th><\/tr>\n<tr>\n<th> Features\n<\/th>\n<th> Pre\n<\/th>\n<th> Rec\n<\/th>\n<th> F-scr\n<\/th>\n<th> Pre\n<\/th>\n<th> Rec\n<\/th>\n<th> F-scr\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BANNER setup\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.59\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">72.74\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">78.64\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.74\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.31\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">77.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.71\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">79.81\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">63.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">70.51\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">78.47\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.42\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.67\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.75\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">79.34\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.89\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.71\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.89\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.70\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.79\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.45\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.96\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + Brown 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.76\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">79.46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.03\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">89.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">81.54\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.20\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.98\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.45\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.65\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.13\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.26\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">79.92\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.53\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.77\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.42\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.48\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>85.58<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>82.75<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.57\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.48\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">80.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.70\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.80\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.60\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.59\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 300 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.89\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>80.35<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">82.56\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>88.9<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>82.68<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>85.68<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\">Feature groups are separated by (+). The parameters followed Brown and WVC are the number of classes induced in each model. Pre: Precision, Rec: Recall, F-scr: F-score.\n<\/td><\/tr><\/table>\n<\/td><\/tr><\/table>\n<p>We started conducting a run with a basic feature setting and a BANNER setup, and gradually increased the complexity of the feature space for further runs. The inclusion of lemma by BioLemmatizer as a feature (baseline feature setup) in addition to BANNER feature set yielded a significant improvement on the development set. Surprisingly, the model based on the baseline features converged to an optimum, possibly a local optimum in the first 30 iterations of the training, and reported a worthless performance when evaluating the testing set. A Brown model with a larger number of clusters tended to obtain a higher F-measure. Unlike Brown clustering, a large or a lower number of WVCs degraded the performance. We found the WVC model with 500 different classes the best performing one on this task. During the induction of the WVC models, we set the WV dimension to 250 which is a trade-off value between computational cost and WV quality. Further, the combination of the different WVC models significantly improved the F-measure. We achieved the best performance, an 85.68% F-measure for CEM and an 86.47% F-measure for CDI subtasks, with the model based on the baseline feature set, the 1000-Brown clustering, and 300, 500 and 1000 WVCs (the baseline + Brown 1000 + WVC 300 + WVC 500 + WVC 1000 setup). This result is 0.93% and 0.73% higher than our best entries for CHEMDNER CEM and CDI subtasks.<sup id=\"rdp-ebb-cite_ref-MunkhdalaiBANNER13_31-0\" class=\"reference\"><a href=\"#cite_note-MunkhdalaiBANNER13-31\" rel=\"external_link\">[31]<\/a><\/sup> The word representation features extracted from the unlabeled data boosted performance by a 1.37% and a 1.14% F-measure for CEM and CDI subtasks, respectively. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Gene_and_protein_mention_recognition\">Gene and protein mention recognition<\/span><\/h3>\n<p>We trained BANNER-CHEMDNER on the BC2GM training set and evaluated it on the testing set, following the convention of the BioCreative II gene mention task. Table 4 lists the different runs with varied features included in the NER model. We expected similar results to be obtained in chemical and drug NER. However, the combination of the features did not show the expected improvement, and the best result (an 87.04% F-measure) that we obtained was a run with a model based on the baseline feature set, the 1000-Brown clustering and 500 WVCs (baseline + Brown 1000 + WVC 500). \n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 4<\/b>: BioCreative II gene mention evaluation results of different runs with varied features<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th>\n<\/th>\n<th colspan=\"3\">Testing set\n<\/th><\/tr>\n<tr>\n<th> Features\n<\/th>\n<th> Pre\n<\/th>\n<th> Rec\n<\/th>\n<th> F-scr\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.88\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.47\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.49\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">83.79\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.12\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.82\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.53\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">87.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.98\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.02\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 300\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">87.95\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.27\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.07\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">87.92\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.49\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.69\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 1000\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>88.13<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.58\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.84\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Baseline + Brown 1000 + WVC 500\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>86.09<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>87.04<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\">Feature groups are separated by (+). The parameters followed Brown and WVC are the number of classes induced in each model. Pre: Precision, Rec: Recall, F-scr: F-score.\n<\/td><\/tr><\/table>\n<\/td><\/tr><\/table>\n<p>We finally compared our best result to the task entries and the BANNER system result (Table 5). \n<\/p><p><br \/>\n<\/p>\n<dl><dd><b>Table 5<\/b>: Comparison of different systems on the BioCreative II testing set<\/dd><\/dl>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> System or author\n<\/th>\n<th> BioCreative II rank\n<\/th>\n<th> Pre\n<\/th>\n<th> Rec\n<\/th>\n<th> F-scr\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ando<sup id=\"rdp-ebb-cite_ref-Andobio7_32-0\" class=\"reference\"><a href=\"#cite_note-Andobio7-32\" rel=\"external_link\">[32]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.48\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">85.97\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">87.21\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BANNER-CHEMDNER\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.02\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.08\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">87.04\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Kuo et al.<sup id=\"rdp-ebb-cite_ref-KuoRich07_33-0\" class=\"reference\"><a href=\"#cite_note-KuoRich07-33\" rel=\"external_link\">[33]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">89.30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.49\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.83\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Huang et al.<sup id=\"rdp-ebb-cite_ref-HuangHigh07_34-0\" class=\"reference\"><a href=\"#cite_note-HuangHigh07-34\" rel=\"external_link\">[34]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.93\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.57\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BANNER\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">88.66\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">84.32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">86.43\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Ando focused on a semi-supervised approach, alternating structure optimization. The system learns better feature representations from a large collection of PubMed text and uses a regularized linear classifier during the supervised training on annotated data. Kuo et al.<sup id=\"rdp-ebb-cite_ref-KuoRich07_33-1\" class=\"reference\"><a href=\"#cite_note-KuoRich07-33\" rel=\"external_link\">[33]<\/a><\/sup> utilized domain lexicons and bi-direction parsing models of CRFs. The results from left-right and right-left parsing models were combined to produce a set of higher recall mention answers. Huang et al.<sup id=\"rdp-ebb-cite_ref-HuangHigh07_34-1\" class=\"reference\"><a href=\"#cite_note-HuangHigh07-34\" rel=\"external_link\">[34]<\/a><\/sup> explored an ensemble of SVM and CRFs models. They applied intersection to the tagging results of the two SVM models and then union with the tagging results of the CRF model in their ensemble approach. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Several different runs other than the ones reported in the results section were carried out. We observed that the whitespace tokenizer performs better than the BANNER simple tokenizer for extraction of word representation features. A model with an additional feature set of raw WVs was trained and evaluated. However, we found that the word WVs did not always improve performance, and in some cases, degraded system performance with the CRF model. That is, the continuous valued WV features add some level of complexity to the model, overfitting the CRF model. Even though the WV features with the CRF model did not achieve improvement, those continuous valued features could be useful in conjunction with other classifiers, such as perceptron and support vector machines.<sup id=\"rdp-ebb-cite_ref-CollierComp04_8-1\" class=\"reference\"><a href=\"#cite_note-CollierComp04-8\" rel=\"external_link\">[8]<\/a><\/sup> Another finding is that the Brown cluster features always improve F-measure, and the improvement is significant when the model is built on the domain text corpus.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>We proposed a semi-supervised learning method that exploits unlabeled data efficiently in order to incorporate domain knowledge into a Named Entity Recognition model and to leverage overall system performance. The key feature of the method is learning word representations from a large amount of text data for feature extraction. The generally applicable word representation features were reported to boost system performance significantly for both chemical and biomedical NER.\n<\/p><p>We extended BANNER, a biomedical NER system, with the proposed method. This yields an integrated system that can be applied for chemical and drug NER or biomedical NER. We call our branch of the BANNER system BANNER-CHEMDNER.\n<\/p><p>BANNER-CHEMDNER achieves an 85.68% and an 86.47% F-measure on the testing set for CHEMDNER CEM and CDI subtasks, respectively, and an 87.04% F-measure on the official testing set of the BioCreative II gene mention task, showing remarkable performance for both chemical and biomedical NER.\n<\/p><p>Our future work should be towards a unified NER system in biomedical, chemical and medical domain, based on the generally applicable word representations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.27_contributions\">Authors' contributions<\/span><\/h2>\n<p>TM conceived the study, participated in its design, developed the extension program, and drafted the manuscript. ML carried out calculations and helped draft the manuscript. KB participated in data analysis and helped draft the manuscript. HAP participated in study design. NHC helped draft the manuscript. KHR provided system design, valuable guidance, editing, and research grant. All authors read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (No-2013R1A2A2A01068923) and by a National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2008-0062611). We would like to thank the BANNER system authors for developing BANNER extendable.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This research and publication was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (No-2013R1A2A2A01068923) and by a National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2008-0062611).\n<\/p><p>This article has been published as part of <i>Journal of Cheminformatics<\/i> Volume 7 Supplement 1, 2015: Text mining for chemistry and the CHEMDNER track. The full contents of the supplement are available online at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.jcheminf.com\/supplements\/7\/S1\" target=\"_blank\">http:\/\/www.jcheminf.com\/supplements\/7\/S1<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-DaiNew10-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DaiNew10_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dai, H.J.; Chang, Y.C.; Tsai, R.T.H.; Hsu, W.L. (2010). \"New Challenges for Biological Text-Mining in the Next Decade\". <i>Journal of Computer Science and Technology<\/i> <b>25<\/b> (1): 169-179. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11390-010-9313-5\" target=\"_blank\">10.1007\/s11390-010-9313-5<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=New+Challenges+for+Biological+Text-Mining+in+the+Next+Decade&rft.jtitle=Journal+of+Computer+Science+and+Technology&rft.aulast=Dai%2C+H.J.%3B+Chang%2C+Y.C.%3B+Tsai%2C+R.T.H.%3B+Hsu%2C+W.L.&rft.au=Dai%2C+H.J.%3B+Chang%2C+Y.C.%3B+Tsai%2C+R.T.H.%3B+Hsu%2C+W.L.&rft.date=2010&rft.volume=25&rft.issue=1&rft.pages=169-179&rft_id=info:doi\/10.1007%2Fs11390-010-9313-5&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RocktaschelChem12-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RocktaschelChem12_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-RocktaschelChem12_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rockt\u00e4schel, T.; Weidlich, M.; Leser, U. (2012). \"ChemSpot: A hybrid system for chemical named entity recognition\". <i>Bioinformatics<\/i> <b>28<\/b> (12): 1633-1640. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbts183\" target=\"_blank\">10.1093\/bioinformatics\/bts183<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22500000\" target=\"_blank\">22500000<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ChemSpot%3A+A+hybrid+system+for+chemical+named+entity+recognition&rft.jtitle=Bioinformatics&rft.aulast=Rockt%C3%A4schel%2C+T.%3B+Weidlich%2C+M.%3B+Leser%2C+U.&rft.au=Rockt%C3%A4schel%2C+T.%3B+Weidlich%2C+M.%3B+Leser%2C+U.&rft.date=2012&rft.volume=28&rft.issue=12&rft.pages=1633-1640&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbts183&rft_id=info:pmid\/22500000&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HettneADict09-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HettneADict09_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rockt\u00e4schel, T.; Weidlich, M.; Leser, U. (2009). \"A dictionary to identify small molecules and drugs in free text\". <i>Bioinformatics<\/i> <b>25<\/b> (22): 2983-2991. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp535\" target=\"_blank\">10.1093\/bioinformatics\/btp535<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19759196\" target=\"_blank\">19759196<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+dictionary+to+identify+small+molecules+and+drugs+in+free+text&rft.jtitle=Bioinformatics&rft.aulast=Rockt%C3%A4schel%2C+T.%3B+Weidlich%2C+M.%3B+Leser%2C+U.&rft.au=Rockt%C3%A4schel%2C+T.%3B+Weidlich%2C+M.%3B+Leser%2C+U.&rft.date=2009&rft.volume=25&rft.issue=22&rft.pages=2983-2991&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp535&rft_id=info:pmid\/19759196&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SeguraDrug08-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SeguraDrug08_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Segura-Bedmar, I.; Mart\u00ednez, P.; Segura-Bedmar, M. (2008). \"Drug name recognition and classification in biomedical texts: A case study outlining approaches underpinning automated systems\". <i>Drug Discovery Today<\/i> <b>13<\/b> (17\u201318): 816\u2013823. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.drudis.2008.06.001\" target=\"_blank\">10.1016\/j.drudis.2008.06.001<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18602492\" target=\"_blank\">18602492<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Drug+name+recognition+and+classification+in+biomedical+texts%3A+A+case+study+outlining+approaches+underpinning+automated+systems&rft.jtitle=Drug+Discovery+Today&rft.aulast=Segura-Bedmar%2C+I.%3B+Mart%C3%ADnez%2C+P.%3B+Segura-Bedmar%2C+M.&rft.au=Segura-Bedmar%2C+I.%3B+Mart%C3%ADnez%2C+P.%3B+Segura-Bedmar%2C+M.&rft.date=2008&rft.volume=13&rft.issue=17%E2%80%9318&rft.pages=816%E2%80%93823&rft_id=info:doi\/10.1016%2Fj.drudis.2008.06.001&rft_id=info:pmid\/18602492&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhaoNamed04-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhaoNamed04_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhao, S. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567594.1567613\" target=\"_blank\">\"Named Entity Recognition in Biomedical Texts using an HMM Model\"<\/a>. <i>Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications<\/i> <b>2004<\/b>: 84-87<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567594.1567613\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?id=1567594.1567613<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Named+Entity+Recognition+in+Biomedical+Texts+using+an+HMM+Model&rft.jtitle=Proceedings+of+the+Joint+Workshop+on+Natural+Language+Processing+in+Biomedicine+and+its+Applications&rft.aulast=Zhao%2C+S.&rft.au=Zhao%2C+S.&rft.date=2004&rft.volume=2004&rft.pages=84-87&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D1567594.1567613&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZhaoExp04-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZhaoExp04_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Zhao, G.; Su, J. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567616\" target=\"_blank\">\"Exploring deep knowledge resources in biomedical name recognition\"<\/a>. <i>Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications<\/i> <b>2004<\/b>: 96-99<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567616\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?id=1567616<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+deep+knowledge+resources+in+biomedical+name+recognition&rft.jtitle=Proceedings+of+the+Joint+Workshop+on+Natural+Language+Processing+in+Biomedicine+and+its+Applications&rft.aulast=Zhao%2C+G.%3B+Su%2C+J.&rft.au=Zhao%2C+G.%3B+Su%2C+J.&rft.date=2004&rft.volume=2004&rft.pages=96-99&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D1567616&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MitsumoriGene05-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MitsumoriGene05_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mitsumori, T.; Fation, S.; Murata, M.; Doi, K.; Doi, H. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1869022\" target=\"_blank\">\"Gene\/protein name recognition based on support vector machine using dictionary as features\"<\/a>. <i>BMC Bioinformatics<\/i> <b>6<\/b> (Suppl 1): S8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-6-S1-S8\" target=\"_blank\">10.1186\/1471-2105-6-S1-S8<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1869022\/\" target=\"_blank\">PMC1869022<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15960842\" target=\"_blank\">15960842<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1869022\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1869022<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene%2Fprotein+name+recognition+based+on+support+vector+machine+using+dictionary+as+features&rft.jtitle=BMC+Bioinformatics&rft.aulast=Mitsumori%2C+T.%3B+Fation%2C+S.%3B+Murata%2C+M.%3B+Doi%2C+K.%3B+Doi%2C+H.&rft.au=Mitsumori%2C+T.%3B+Fation%2C+S.%3B+Murata%2C+M.%3B+Doi%2C+K.%3B+Doi%2C+H.&rft.date=2005&rft.volume=6&rft.issue=Suppl+1&rft.pages=S8&rft_id=info:doi\/10.1186%2F1471-2105-6-S1-S8&rft_id=info:pmc\/PMC1869022&rft_id=info:pmid\/15960842&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1869022&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CollierComp04-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CollierComp04_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-CollierComp04_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Collier, N; Takeuchi, K. (2004). \"Comparison of character-level and part of speech features for name recognition in biomedical texts\". <i>Journal of Biomedical Informatics<\/i> <b>37<\/b> (6): 423-35. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2004.08.008\" target=\"_blank\">10.1016\/j.jbi.2004.08.008<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15542016\" target=\"_blank\">15542016<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Comparison+of+character-level+and+part+of+speech+features+for+name+recognition+in+biomedical+texts&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Collier%2C+N%3B+Takeuchi%2C+K.&rft.au=Collier%2C+N%3B+Takeuchi%2C+K.&rft.date=2004&rft.volume=37&rft.issue=6&rft.pages=423-35&rft_id=info:doi\/10.1016%2Fj.jbi.2004.08.008&rft_id=info:pmid\/15542016&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FinkelExp04-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FinkelExp04_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Finkel, J.; Dingare, S.; Nguyen, H.; Nissim, M.; Manning, C.; Sinclair, G. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567614\" target=\"_blank\">\"Exploiting context for biomedical entity recognition: from syntax to the web\"<\/a>. <i>Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications<\/i> <b>2004<\/b>: 88\u201391<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567614\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?id=1567614<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploiting+context+for+biomedical+entity+recognition%3A+from+syntax+to+the+web&rft.jtitle=Proceedings+of+the+Joint+Workshop+on+Natural+Language+Processing+in+Biomedicine+and+its+Applications&rft.aulast=Finkel%2C+J.%3B+Dingare%2C+S.%3B+Nguyen%2C+H.%3B+Nissim%2C+M.%3B+Manning%2C+C.%3B+Sinclair%2C+G.&rft.au=Finkel%2C+J.%3B+Dingare%2C+S.%3B+Nguyen%2C+H.%3B+Nissim%2C+M.%3B+Manning%2C+C.%3B+Sinclair%2C+G.&rft.date=2004&rft.volume=2004&rft.pages=88%E2%80%9391&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D1567614&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CorbettCasc08-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CorbettCasc08_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Corbett, P.; Copestake, A. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2586753\" target=\"_blank\">\"Cascaded classifiers for confidence-based chemical named entity recognition\"<\/a>. <i>BMC Bioinformatics<\/i> <b>9<\/b> (Suppl 11): S4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-9-S11-S4\" target=\"_blank\">10.1186\/1471-2105-9-S11-S4<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2586753\/\" target=\"_blank\">PMC2586753<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19025690\" target=\"_blank\">19025690<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2586753\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2586753<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cascaded+classifiers+for+confidence-based+chemical+named+entity+recognition&rft.jtitle=BMC+Bioinformatics&rft.aulast=Corbett%2C+P.%3B+Copestake%2C+A.&rft.au=Corbett%2C+P.%3B+Copestake%2C+A.&rft.date=2008&rft.volume=9&rft.issue=Suppl+11&rft.pages=S4&rft_id=info:doi\/10.1186%2F1471-2105-9-S11-S4&rft_id=info:pmc\/PMC2586753&rft_id=info:pmid\/19025690&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2586753&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SettlesBio04-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SettlesBio04_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Settles, B. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567618\" target=\"_blank\">\"Biomedical named entity recognition using conditional random fields and rich feature sets\"<\/a>. <i>Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications<\/i> <b>2004<\/b>: 104-107<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1567618\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?id=1567618<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Biomedical+named+entity+recognition+using+conditional+random+fields+and+rich+feature+sets&rft.jtitle=Proceedings+of+the+Joint+Workshop+on+Natural+Language+Processing+in+Biomedicine+and+its+Applications&rft.aulast=Settles%2C+B.&rft.au=Settles%2C+B.&rft.date=2004&rft.volume=2004&rft.pages=104-107&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D1567618&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HsuInt08-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HsuInt08_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hsu, C.; Chang, Y.; Kuo, C.; Lin, Y.; Huang, H.; Chung, I. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2718659\" target=\"_blank\">\"Integrating high dimensional bi-directional parsing models for gene mention tagging\"<\/a>. <i>Bioinformatics<\/i> <b>24<\/b> (13): i286-94. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtn183\" target=\"_blank\">10.1093\/bioinformatics\/btn183<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2718659\/\" target=\"_blank\">PMC2718659<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18586726\" target=\"_blank\">18586726<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2718659\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2718659<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrating+high+dimensional+bi-directional+parsing+models+for+gene+mention+tagging&rft.jtitle=Bioinformatics&rft.aulast=Hsu%2C+C.%3B+Chang%2C+Y.%3B+Kuo%2C+C.%3B+Lin%2C+Y.%3B+Huang%2C+H.%3B+Chung%2C+I.&rft.au=Hsu%2C+C.%3B+Chang%2C+Y.%3B+Kuo%2C+C.%3B+Lin%2C+Y.%3B+Huang%2C+H.%3B+Chung%2C+I.&rft.date=2008&rft.volume=24&rft.issue=13&rft.pages=i286-94&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtn183&rft_id=info:pmc\/PMC2718659&rft_id=info:pmid\/18586726&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2718659&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiInc09-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiInc09_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, Y.; Lin, H.; Yang, Z. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2725142\" target=\"_blank\">\"Incorporating rich background knowledge for gene named entity classification and recognition\"<\/a>. <i>BMC Bioinformatics<\/i> <b>10<\/b>: 223. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-10-223\" target=\"_blank\">10.1186\/1471-2105-10-223<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2725142\/\" target=\"_blank\">PMC2725142<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19615051\" target=\"_blank\">19615051<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2725142\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2725142<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Incorporating+rich+background+knowledge+for+gene+named+entity+classification+and+recognition&rft.jtitle=BMC+Bioinformatics&rft.aulast=Li%2C+Y.%3B+Lin%2C+H.%3B+Yang%2C+Z.&rft.au=Li%2C+Y.%3B+Lin%2C+H.%3B+Yang%2C+Z.&rft.date=2009&rft.volume=10&rft.pages=223&rft_id=info:doi\/10.1186%2F1471-2105-10-223&rft_id=info:pmc\/PMC2725142&rft_id=info:pmid\/19615051&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2725142&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MunkhdalaiBio12-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MunkhdalaiBio12_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Munkhdalai, T.; Li, M.; Kim, T.; Namsrai, O.; Jeong, S.; Shin, J.; Ryu, K.H. (2012). \"Bio Named Entity Recognition based on Co-training Algorithm\". <i>Proceedings 26th International Conference on Advanced Information Networking and Applications Workshops<\/i> <b>2012<\/b>: 857-862. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FWAINA.2012.75\" target=\"_blank\">10.1109\/WAINA.2012.75<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bio+Named+Entity+Recognition+based+on+Co-training+Algorithm&rft.jtitle=Proceedings+26th+International+Conference+on+Advanced+Information+Networking+and+Applications+Workshops&rft.aulast=Munkhdalai%2C+T.%3B+Li%2C+M.%3B+Kim%2C+T.%3B+Namsrai%2C+O.%3B+Jeong%2C+S.%3B+Shin%2C+J.%3B+Ryu%2C+K.H.&rft.au=Munkhdalai%2C+T.%3B+Li%2C+M.%3B+Kim%2C+T.%3B+Namsrai%2C+O.%3B+Jeong%2C+S.%3B+Shin%2C+J.%3B+Ryu%2C+K.H.&rft.date=2012&rft.volume=2012&rft.pages=857-862&rft_id=info:doi\/10.1109%2FWAINA.2012.75&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MunkhdalaiAn12-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MunkhdalaiAn12_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Munkhdalai, T.; Li, M.; Yun, U.; Namsrai, O.; Ryu, K.H. (2012). \"An Active Co-Training Algorithm for Biomedical Named-Entity Recognition\". <i>Journal of Information Processing Systems<\/i> <b>8<\/b> (4): 575-588. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3745%2FJIPS.2012.8.4.575\" target=\"_blank\">10.3745\/JIPS.2012.8.4.575<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+Active+Co-Training+Algorithm+for+Biomedical+Named-Entity+Recognition&rft.jtitle=Journal+of+Information+Processing+Systems&rft.aulast=Munkhdalai%2C+T.%3B+Li%2C+M.%3B+Yun%2C+U.%3B+Namsrai%2C+O.%3B+Ryu%2C+K.H.&rft.au=Munkhdalai%2C+T.%3B+Li%2C+M.%3B+Yun%2C+U.%3B+Namsrai%2C+O.%3B+Ryu%2C+K.H.&rft.date=2012&rft.volume=8&rft.issue=4&rft.pages=575-588&rft_id=info:doi\/10.3745%2FJIPS.2012.8.4.575&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TurianWord10-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-TurianWord10_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-TurianWord10_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Turian, J.; Ratinov, L.; Bengio, Y. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/cogcomp.cs.illinois.edu\/page\/publication_view\/653\" target=\"_blank\">\"Word representations: A simple and general method for semi-supervised learning\"<\/a>. <i>Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics<\/i> <b>2010<\/b>: 384-394<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/cogcomp.cs.illinois.edu\/page\/publication_view\/653\" target=\"_blank\">http:\/\/cogcomp.cs.illinois.edu\/page\/publication_view\/653<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Word+representations%3A+A+simple+and+general+method+for+semi-supervised+learning&rft.jtitle=Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics&rft.aulast=Turian%2C+J.%3B+Ratinov%2C+L.%3B+Bengio%2C+Y.&rft.au=Turian%2C+J.%3B+Ratinov%2C+L.%3B+Bengio%2C+Y.&rft.date=2010&rft.volume=2010&rft.pages=384-394&rft_id=http%3A%2F%2Fcogcomp.cs.illinois.edu%2Fpage%2Fpublication_view%2F653&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MikolovEff13-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MikolovEff13_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-MikolovEff13_17-1\" rel=\"external_link\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-MikolovEff13_17-2\" rel=\"external_link\">17.2<\/a><\/sup> <sup><a href=\"#cite_ref-MikolovEff13_17-3\" rel=\"external_link\">17.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/arxiv.org\/abs\/1301.3781\" target=\"_blank\">\"Efficient Estimation of Word Representations in Vector Space\"<\/a>. <i>Proceedings of Workshop at ICLR<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/arxiv.org\/abs\/1301.3781\" target=\"_blank\">http:\/\/arxiv.org\/abs\/1301.3781<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Efficient+Estimation+of+Word+Representations+in+Vector+Space&rft.jtitle=Proceedings+of+Workshop+at+ICLR&rft.aulast=Mikolov%2C+T.%3B+Chen%2C+K.%3B+Corrado%2C+G.%3B+Dean%2C+J.&rft.au=Mikolov%2C+T.%3B+Chen%2C+K.%3B+Corrado%2C+G.%3B+Dean%2C+J.&rft.date=2013&rft_id=http%3A%2F%2Farxiv.org%2Fabs%2F1301.3781&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SocherPars11-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SocherPars11_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Socher, R.; Lin, C.C.; Ng, A.Y.; Manning, C.D. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/machinelearning.wustl.edu\/mlpapers\/papers\/ICML2011Socher_125\" target=\"_blank\">\"Parsing Natural Scenes and Natural Language with Recursive Neural Networks\"<\/a>. <i>Proceedings of the 28th International Conference on Machine Learning<\/i> <b>2011<\/b>: 129\u2013136<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/machinelearning.wustl.edu\/mlpapers\/papers\/ICML2011Socher_125\" target=\"_blank\">http:\/\/machinelearning.wustl.edu\/mlpapers\/papers\/ICML2011Socher_125<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Parsing+Natural+Scenes+and+Natural+Language+with+Recursive+Neural+Networks&rft.jtitle=Proceedings+of+the+28th+International+Conference+on+Machine+Learning&rft.aulast=Socher%2C+R.%3B+Lin%2C+C.C.%3B+Ng%2C+A.Y.%3B+Manning%2C+C.D.&rft.au=Socher%2C+R.%3B+Lin%2C+C.C.%3B+Ng%2C+A.Y.%3B+Manning%2C+C.D.&rft.date=2011&rft.volume=2011&rft.pages=129%E2%80%93136&rft_id=http%3A%2F%2Fmachinelearning.wustl.edu%2Fmlpapers%2Fpapers%2FICML2011Socher_125&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArighiOver11-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArighiOver11_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Arighi, C.N.; Lu, Z.; Krallinger, M.; Cohen, K.B.; Wilbur, J.W.; Valencia, A.; Hirschman, L.; Wu, C.H. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3269932\" target=\"_blank\">\"Overview of the BioCreative III Workshop\"<\/a>. <i>Bioinformatics<\/i> <b>12<\/b> (Suppl 8): S1. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-S8-S1\" target=\"_blank\">10.1186\/1471-2105-12-S8-S1<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3269932\/\" target=\"_blank\">PMC3269932<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22151647\" target=\"_blank\">22151647<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3269932\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3269932<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Overview+of+the+BioCreative+III+Workshop&rft.jtitle=Bioinformatics&rft.aulast=Arighi%2C+C.N.%3B+Lu%2C+Z.%3B+Krallinger%2C+M.%3B+Cohen%2C+K.B.%3B+Wilbur%2C+J.W.%3B+Valencia%2C+A.%3B+Hirschman%2C+L.%3B+Wu%2C+C.H.&rft.au=Arighi%2C+C.N.%3B+Lu%2C+Z.%3B+Krallinger%2C+M.%3B+Cohen%2C+K.B.%3B+Wilbur%2C+J.W.%3B+Valencia%2C+A.%3B+Hirschman%2C+L.%3B+Wu%2C+C.H.&rft.date=2011&rft.volume=12&rft.issue=Suppl+8&rft.pages=S1&rft_id=info:doi\/10.1186%2F1471-2105-12-S8-S1&rft_id=info:pmc\/PMC3269932&rft_id=info:pmid\/22151647&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3269932&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ArighiBio14-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ArighiBio14_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Arighi, C.N.; Wu, C.H.; Cohen, K.B.; Hirschman, L.; Krallinger, M.; Valencia, A.; Lu, Z.; Wilbur, J.W.; Wiegers, T.C. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030502\" target=\"_blank\">\"BioCreative-IV virtual issue\"<\/a>. <i>Database<\/i> <b>2014<\/b>: bau039. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fdatabase%2Fbau039\" target=\"_blank\">10.1093\/database\/bau039<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4030502\/\" target=\"_blank\">PMC4030502<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24852177\" target=\"_blank\">24852177<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030502\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030502<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioCreative-IV+virtual+issue&rft.jtitle=Database&rft.aulast=Arighi%2C+C.N.%3B+Wu%2C+C.H.%3B+Cohen%2C+K.B.%3B+Hirschman%2C+L.%3B+Krallinger%2C+M.%3B+Valencia%2C+A.%3B+Lu%2C+Z.%3B+Wilbur%2C+J.W.%3B+Wiegers%2C+T.C.&rft.au=Arighi%2C+C.N.%3B+Wu%2C+C.H.%3B+Cohen%2C+K.B.%3B+Hirschman%2C+L.%3B+Krallinger%2C+M.%3B+Valencia%2C+A.%3B+Lu%2C+Z.%3B+Wilbur%2C+J.W.%3B+Wiegers%2C+T.C.&rft.date=2014&rft.volume=2014&rft.pages=bau039&rft_id=info:doi\/10.1093%2Fdatabase%2Fbau039&rft_id=info:pmc\/PMC4030502&rft_id=info:pmid\/24852177&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4030502&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrallingerThe15-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrallingerThe15_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krallinger, M.; Rabal, O.; Leitner, F.; et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331692\" target=\"_blank\">\"The CHEMDNER corpus of chemicals and drugs and its annotation principles\"<\/a>. <i>Journal of Cheminformatics<\/i> <b>7<\/b> (Suppl 1): S2. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1758-2946-7-S1-S2\" target=\"_blank\">10.1186\/1758-2946-7-S1-S2<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4331692\/\" target=\"_blank\">PMC4331692<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25810773\" target=\"_blank\">25810773<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331692\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331692<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+CHEMDNER+corpus+of+chemicals+and+drugs+and+its+annotation+principles&rft.jtitle=Journal+of+Cheminformatics&rft.aulast=Krallinger%2C+M.%3B+Rabal%2C+O.%3B+Leitner%2C+F.%3B+et+al.&rft.au=Krallinger%2C+M.%3B+Rabal%2C+O.%3B+Leitner%2C+F.%3B+et+al.&rft.date=2015&rft.volume=7&rft.issue=Suppl+1&rft.pages=S2&rft_id=info:doi\/10.1186%2F1758-2946-7-S1-S2&rft_id=info:pmc\/PMC4331692&rft_id=info:pmid\/25810773&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4331692&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeamanBAN08-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LeamanBAN08_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leaman, R.; Gonzalez, G. (2008). \"BANNER: an executable survey of advances in biomedical named entity recognition\". <i>Proceedings of Pacific Symposium on Biocomputing<\/i> <b>2008<\/b>: 652-63. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18229723\" target=\"_blank\">18229723<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BANNER%3A+an+executable+survey+of+advances+in+biomedical+named+entity+recognition&rft.jtitle=Proceedings+of+Pacific+Symposium+on+Biocomputing&rft.aulast=Leaman%2C+R.%3B+Gonzalez%2C+G.&rft.au=Leaman%2C+R.%3B+Gonzalez%2C+G.&rft.date=2008&rft.volume=2008&rft.pages=652-63&rft_id=info:pmid\/18229723&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeamanDNorm13-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LeamanDNorm13_23-0\" rel=\"external_link\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-LeamanDNorm13_23-1\" rel=\"external_link\">23.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Leaman, R.; Islamaj, D.R.; Lu, Z. (2013). \"DNorm: disease name normalization with pairwise learning to rank\". <i>Bioinformatics<\/i> <b>29<\/b> (1): 22-28. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbts639\" target=\"_blank\">10.1093\/bioinformatics\/bts639<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23104890\" target=\"_blank\">23104890<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DNorm%3A+disease+name+normalization+with+pairwise+learning+to+rank&rft.jtitle=Bioinformatics&rft.aulast=Leaman%2C+R.%3B+Islamaj%2C+D.R.%3B+Lu%2C+Z.&rft.au=Leaman%2C+R.%3B+Islamaj%2C+D.R.%3B+Lu%2C+Z.&rft.date=2013&rft.volume=29&rft.issue=1&rft.pages=22-28&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbts639&rft_id=info:pmid\/23104890&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Wei_tmVar13-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Wei_tmVar13_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wei, C.H.; Harris, B.R.; Kao, H.Y.; Lu, Z. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3661051\" target=\"_blank\">\"tmVar: a text mining approach for extracting sequence variants in biomedical literature\"<\/a>. <i>Bioinformatics<\/i> <b>29<\/b> (11): 1433-1439. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt156\" target=\"_blank\">10.1093\/bioinformatics\/btt156<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3661051\/\" target=\"_blank\">PMC3661051<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23564842\" target=\"_blank\">23564842<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3661051\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3661051<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=tmVar%3A+a+text+mining+approach+for+extracting+sequence+variants+in+biomedical+literature&rft.jtitle=Bioinformatics&rft.aulast=Wei%2C+C.H.%3B+Harris%2C+B.R.%3B+Kao%2C+H.Y.%3B+Lu%2C+Z.&rft.au=Wei%2C+C.H.%3B+Harris%2C+B.R.%3B+Kao%2C+H.Y.%3B+Lu%2C+Z.&rft.date=2013&rft.volume=29&rft.issue=11&rft.pages=1433-1439&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt156&rft_id=info:pmc\/PMC3661051&rft_id=info:pmid\/23564842&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3661051&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BjornComp10-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BjornComp10_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bj\u00f6rne, J.; Ginter, F.; Pyysalo, S.; Tsujii, J.; Salakoski, T. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2881365\" target=\"_blank\">\"Complex event extraction at PubMed scale\"<\/a>. <i>Bioinformatics<\/i> <b>26<\/b> (12): i382-i390. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtq180\" target=\"_blank\">10.1093\/bioinformatics\/btq180<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2881365\/\" target=\"_blank\">PMC2881365<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20529932\" target=\"_blank\">20529932<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2881365\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2881365<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Complex+event+extraction+at+PubMed+scale&rft.jtitle=Bioinformatics&rft.aulast=Bj%C3%B6rne%2C+J.%3B+Ginter%2C+F.%3B+Pyysalo%2C+S.%3B+Tsujii%2C+J.%3B+Salakoski%2C+T.&rft.au=Bj%C3%B6rne%2C+J.%3B+Ginter%2C+F.%3B+Pyysalo%2C+S.%3B+Tsujii%2C+J.%3B+Salakoski%2C+T.&rft.date=2010&rft.volume=26&rft.issue=12&rft.pages=i382-i390&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtq180&rft_id=info:pmc\/PMC2881365&rft_id=info:pmid\/20529932&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2881365&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GernerBio12-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GernerBio12_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gerner, M.; Sarafraz, F.; Bergman, C.M.; Nenadic, G. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3413385\" target=\"_blank\">\"BioContext: an integrated text mining system for large-scale extraction and contextualization of biomolecular events\"<\/a>. <i>Bioinformatics<\/i> <b>28<\/b> (16): 2154-2161. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbts332\" target=\"_blank\">10.1093\/bioinformatics\/bts332<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3413385\/\" target=\"_blank\">PMC3413385<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22711795\" target=\"_blank\">22711795<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3413385\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3413385<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioContext%3A+an+integrated+text+mining+system+for+large-scale+extraction+and+contextualization+of+biomolecular+events&rft.jtitle=Bioinformatics&rft.aulast=Gerner%2C+M.%3B+Sarafraz%2C+F.%3B+Bergman%2C+C.M.%3B+Nenadic%2C+G.&rft.au=Gerner%2C+M.%3B+Sarafraz%2C+F.%3B+Bergman%2C+C.M.%3B+Nenadic%2C+G.&rft.date=2012&rft.volume=28&rft.issue=16&rft.pages=2154-2161&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbts332&rft_id=info:pmc\/PMC3413385&rft_id=info:pmid\/22711795&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3413385&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiuBioL12-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiuBioL12_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liu, H.; Christiansen, T.; Baumgartner, W.A.; Verspoor, K. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3359276\" target=\"_blank\">\"BioLemmatizer: a lemmatization tool for morphological processing of biomedical text\"<\/a>. <i>Journal of Biomedical Semantic<\/i> <b>3<\/b>: 3. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F2041-1480-3-3\" target=\"_blank\">10.1186\/2041-1480-3-3<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3359276\/\" target=\"_blank\">PMC3359276<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22464129\" target=\"_blank\">22464129<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3359276\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3359276<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioLemmatizer%3A+a+lemmatization+tool+for+morphological+processing+of+biomedical+text&rft.jtitle=Journal+of+Biomedical+Semantic&rft.aulast=Liu%2C+H.%3B+Christiansen%2C+T.%3B+Baumgartner%2C+W.A.%3B+Verspoor%2C+K.&rft.au=Liu%2C+H.%3B+Christiansen%2C+T.%3B+Baumgartner%2C+W.A.%3B+Verspoor%2C+K.&rft.date=2012&rft.volume=3&rft.pages=3&rft_id=info:doi\/10.1186%2F2041-1480-3-3&rft_id=info:pmc\/PMC3359276&rft_id=info:pmid\/22464129&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3359276&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrownClass92-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrownClass92_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brown, P.F.; deSouza, P.V.; Mercer, R.L.; Pietra, V.J.D.; Lai, J.C. (1992). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=176316\" target=\"_blank\">\"Class-Based n-gram Models of Natural Language\"<\/a>. <i>Computational Linguistics<\/i> <b>18<\/b> (4): 467-479<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=176316\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?id=176316<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Class-Based+n-gram+Models+of+Natural+Language&rft.jtitle=Computational+Linguistics&rft.aulast=Brown%2C+P.F.%3B+deSouza%2C+P.V.%3B+Mercer%2C+R.L.%3B+Pietra%2C+V.J.D.%3B+Lai%2C+J.C.&rft.au=Brown%2C+P.F.%3B+deSouza%2C+P.V.%3B+Mercer%2C+R.L.%3B+Pietra%2C+V.J.D.%3B+Lai%2C+J.C.&rft.date=1992&rft.volume=18&rft.issue=4&rft.pages=467-479&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D176316&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrallingerCHEM15-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrallingerCHEM15_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Krallinger, M.; Leitner, F.; Rabal, O.; Vazquez, M.; Oyarzabal, J.; Valencia, A. (1995). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331685\" target=\"_blank\">\"CHEMDNER: The drugs and chemical names extraction challenge\"<\/a>. <i>Journal of Cheminformatics<\/i> <b>7<\/b> (Suppl 1): S1. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1758-2946-7-S1-S1\" target=\"_blank\">10.1186\/1758-2946-7-S1-S1<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4331685\/\" target=\"_blank\">PMC4331685<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25810766\" target=\"_blank\">25810766<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331685\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4331685<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=CHEMDNER%3A+The+drugs+and+chemical+names+extraction+challenge&rft.jtitle=Journal+of+Cheminformatics&rft.aulast=Krallinger%2C+M.%3B+Leitner%2C+F.%3B+Rabal%2C+O.%3B+Vazquez%2C+M.%3B+Oyarzabal%2C+J.%3B+Valencia%2C+A.&rft.au=Krallinger%2C+M.%3B+Leitner%2C+F.%3B+Rabal%2C+O.%3B+Vazquez%2C+M.%3B+Oyarzabal%2C+J.%3B+Valencia%2C+A.&rft.date=1995&rft.volume=7&rft.issue=Suppl+1&rft.pages=S1&rft_id=info:doi\/10.1186%2F1758-2946-7-S1-S1&rft_id=info:pmc\/PMC4331685&rft_id=info:pmid\/25810766&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4331685&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McCallumMallet02-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McCallumMallet02_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">McCallum, Andrew Kachites (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/mallet.cs.umass.edu\/\" target=\"_blank\">\"MALLET: A Machine Learning for Language Toolkit\"<\/a>. University of Massachusetts<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mallet.cs.umass.edu\/\" target=\"_blank\">http:\/\/mallet.cs.umass.edu\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MALLET%3A+A+Machine+Learning+for+Language+Toolkit&rft.atitle=&rft.aulast=McCallum%2C+Andrew+Kachites&rft.au=McCallum%2C+Andrew+Kachites&rft.date=2013&rft.pub=University+of+Massachusetts&rft_id=http%3A%2F%2Fmallet.cs.umass.edu%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MunkhdalaiBANNER13-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MunkhdalaiBANNER13_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Munkhdalai, T.; Li, M.; Batsuren, K.; Ryu, K.H. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biocreative.org\/resources\/publications\/chemdner-proceed-publications\/\" target=\"_blank\">\"BANNER-CHEMDNER: Incorporating Domain Knowledge in Chemical and Drug Named Entity Recognition\"<\/a>. <i>Proceedings of the Fourth BioCreative Challenge Evaluation Workshop<\/i> <b>2<\/b>: 135-139<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.biocreative.org\/resources\/publications\/chemdner-proceed-publications\/\" target=\"_blank\">http:\/\/www.biocreative.org\/resources\/publications\/chemdner-proceed-publications\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BANNER-CHEMDNER%3A+Incorporating+Domain+Knowledge+in+Chemical+and+Drug+Named+Entity+Recognition&rft.jtitle=Proceedings+of+the+Fourth+BioCreative+Challenge+Evaluation+Workshop&rft.aulast=Munkhdalai%2C+T.%3B+Li%2C+M.%3B+Batsuren%2C+K.%3B+Ryu%2C+K.H.&rft.au=Munkhdalai%2C+T.%3B+Li%2C+M.%3B+Batsuren%2C+K.%3B+Ryu%2C+K.H.&rft.date=2013&rft.volume=2&rft.pages=135-139&rft_id=http%3A%2F%2Fwww.biocreative.org%2Fresources%2Fpublications%2Fchemdner-proceed-publications%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Andobio7-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Andobio7_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ando, R.K. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.121.8576\" target=\"_blank\">\"BioCreative II Gene Mention Tagging System at IBM Watson\"<\/a>. <i>Proceedings of the Second BioCreative Challenge Evaluation Workshop<\/i>: 101-103<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.121.8576\" target=\"_blank\">http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.121.8576<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioCreative+II+Gene+Mention+Tagging+System+at+IBM+Watson&rft.jtitle=Proceedings+of+the+Second+BioCreative+Challenge+Evaluation+Workshop&rft.aulast=Ando%2C+R.K.&rft.au=Ando%2C+R.K.&rft.date=2007&rft.pages=101-103&rft_id=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fsummary%3Fdoi%3D10.1.1.121.8576&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KuoRich07-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KuoRich07_33-0\" rel=\"external_link\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-KuoRich07_33-1\" rel=\"external_link\">33.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kuo, C.J.; Chang, Y.M.; Huang, H.S.; Lin, K.T.; Yang, B.H.; Lin, Y.S.; Hsu, C.N.; Chung, I.F. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.iis.sinica.edu.tw\/~chunnan\/DOWNLOADS\/T111_v16.pdf\" target=\"_blank\">\"Rich feature set, unification of bidirectional parsing and dictionary filtering for high F-score gene mention tagging\"<\/a> (PDF). <i>Proceedings of the Second BioCreative Challenge Evaluation Workshop<\/i>: 105-107<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.iis.sinica.edu.tw\/~chunnan\/DOWNLOADS\/T111_v16.pdf\" target=\"_blank\">http:\/\/www.iis.sinica.edu.tw\/~chunnan\/DOWNLOADS\/T111_v16.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Rich+feature+set%2C+unification+of+bidirectional+parsing+and+dictionary+filtering+for+high+F-score+gene+mention+tagging&rft.jtitle=Proceedings+of+the+Second+BioCreative+Challenge+Evaluation+Workshop&rft.aulast=Kuo%2C+C.J.%3B+Chang%2C+Y.M.%3B+Huang%2C+H.S.%3B+Lin%2C+K.T.%3B+Yang%2C+B.H.%3B+Lin%2C+Y.S.%3B+Hsu%2C+C.N.%3B+Chung%2C+I.F.&rft.au=Kuo%2C+C.J.%3B+Chang%2C+Y.M.%3B+Huang%2C+H.S.%3B+Lin%2C+K.T.%3B+Yang%2C+B.H.%3B+Lin%2C+Y.S.%3B+Hsu%2C+C.N.%3B+Chung%2C+I.F.&rft.date=2007&rft.pages=105-107&rft_id=http%3A%2F%2Fwww.iis.sinica.edu.tw%2F%7Echunnan%2FDOWNLOADS%2FT111_v16.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HuangHigh07-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HuangHigh07_34-0\" rel=\"external_link\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-HuangHigh07_34-1\" rel=\"external_link\">34.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kuo, C.J.; Chang, Y.M.; Huang, H.S.; Lin, K.T.; Yang, B.H.; Lin, Y.S.; Hsu, C.N.; Chung, I.F. (2007). \"High-recall gene mention recognition by unification of multiple backward parsing models\". <i>Proceedings of the Second BioCreative Challenge Evaluation Workshop<\/i>: 109-111.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=High-recall+gene+mention+recognition+by+unification+of+multiple+backward+parsing+models&rft.jtitle=Proceedings+of+the+Second+BioCreative+Challenge+Evaluation+Workshop&rft.aulast=Kuo%2C+C.J.%3B+Chang%2C+Y.M.%3B+Huang%2C+H.S.%3B+Lin%2C+K.T.%3B+Yang%2C+B.H.%3B+Lin%2C+Y.S.%3B+Hsu%2C+C.N.%3B+Chung%2C+I.F.&rft.au=Kuo%2C+C.J.%3B+Chang%2C+Y.M.%3B+Huang%2C+H.S.%3B+Lin%2C+K.T.%3B+Yang%2C+B.H.%3B+Lin%2C+Y.S.%3B+Hsu%2C+C.N.%3B+Chung%2C+I.F.&rft.date=2007&rft.pages=109-111&rfr_id=info:sid\/en.wikipedia.org:Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases citation information was missing and was added to make the reference more useful. Numbers in Tables 2\u20135 were all converted to two decimal places for consistency.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210228\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.808 seconds\nReal time usage: 0.833 seconds\nPreprocessor visited node count: 27210\/1000000\nPreprocessor generated node count: 35099\/1000000\nPost\u2010expand include size: 232039\/2097152 bytes\nTemplate argument size: 80893\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 793.248 1 - -total\n 87.21% 691.790 1 - Template:Reflist\n 78.90% 625.880 33 - Template:Cite_journal\n 77.13% 611.837 34 - Template:Citation\/core\n 8.56% 67.882 50 - Template:Citation\/identifier\n 7.91% 62.735 1 - Template:Infobox_journal_article\n 7.58% 60.139 1 - Template:Infobox\n 4.38% 34.735 80 - Template:Infobox\/row\n 4.17% 33.050 34 - Template:Citation\/make_link\n 3.08% 24.428 112 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7854-0!*!0!!en!5!* and timestamp 20181213210227 and revision id 23412\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations\">https:\/\/www.limswiki.org\/index.php\/Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","fee95fa41e5e6f66dc7579d2ee634b76_images":["https:\/\/www.limswiki.org\/images\/c\/c4\/Fig1_Munkhdalai_JCheminformatics2015_7-1.jpg"],"fee95fa41e5e6f66dc7579d2ee634b76_timestamp":1544734947,"3eb9d25818bfd5a80515d585fb6a6712_type":"article","3eb9d25818bfd5a80515d585fb6a6712_title":"Factors associated with adoption of health information technology: A conceptual model based on a systematic review (Kruse et al. 2014)","3eb9d25818bfd5a80515d585fb6a6712_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review","3eb9d25818bfd5a80515d585fb6a6712_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Factors associated with adoption of health information technology: A conceptual model based on a systematic review\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nFactors associated with adoption of health information technology: A conceptual model based on a systematic reviewJournal\n \nJMIR Medical InformaticsAuthor(s)\n \nKruse, Clemens Scott; DeShazo, Jonathan; Forest, Kim; Fulton, LawrenceAuthor affiliation(s)\n \nTexas State University, Virginia Commonwealth University, Baylor UniversityPrimary contact\n \nEmail: scottkruse@txstate.edu; Phone: 1.210.355.4742Editors\n \nEysenbach, G.Year published\n \n2014Volume and issue\n \n2 (1)Page(s)\n \ne9DOI\n \n10.2196\/medinform.3106ISSN\n \n2291-9694Distribution license\n \nCreative Commons Attribution 2.0Website\n \nhttp:\/\/medinform.jmir.org\/2014\/1\/e9\/Download\n \nhttp:\/\/medinform.jmir.org\/article\/download\/medinform_v2i1e9\/2 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n\n2.1 Background \n2.2 Organizational theories \n2.3 Strategy and decision making \n\n\n3 Methods \n\n3.1 Literature review process \n3.2 EHR adoption and internal organizational influence \n3.3 EHR adoption and external environmental influence \n3.4 Overview of the conceptual model \n\n\n4 Results \n\n4.1 Chosen articles \n4.2 EHR adoption and internal organizational influence \n4.3 EHR adoption and external environmental influence \n4.4 Overview of the conceptual model \n\n\n5 Discussion \n\n5.1 Principal findings \n5.2 Limitations \n5.3 Conclusions \n\n\n6 Authors' contributions \n7 Conflicts of interest \n8 References \n9 Abbreviations \n10 Notes \n\n\n\nAbstract \nBackground: The Health Information Technology for Economic and Clinical Health Act (HITECH) allocated $19.2 billion to incentivize adoption of the electronic health record (EHR). Since 2009, Meaningful Use Criteria have dominated information technology (IT) strategy. Health care organizations have struggled to meet expectations and avoid penalties to reimbursements from the Center for Medicare and Medicaid Services (CMS). Organizational theories attempt to explain factors that influence organizational change, and many theories address changes in organizational strategy. However, due to the complexities of the health care industry, existing organizational theories fall short of demonstrating association with significant health care IT implementations. There is no organizational theory for health care that identifies, groups, and analyzes both internal and external factors of influence for large health care IT implementations like adoption of the EHR.\nObjective: The purpose of this systematic review is to identify a full-spectrum of both internal organizational and external environmental factors associated with the adoption of health information technology (HIT), specifically the EHR. The result is a conceptual model that is commensurate with the complexity of with the health care sector.\nMethods: We performed a systematic literature search in PubMed (restricted to English), EBSCO Host, and Google Scholar for both empirical studies and theory-based writing from 1993-2013 that demonstrated association between influential factors and three modes of HIT: EHR, electronic medical record (EMR), and computerized provider order entry (CPOE). We also looked at published books on organizational theories. We made notes and noted trends on adoption factors. These factors were grouped as adoption factors associated with various versions of EHR adoption.\nResults: The resulting conceptual model summarizes the diversity of independent variables (IVs) and dependent variables (DVs) used in articles, editorials, books, as well as quantitative and qualitative studies (n=83). As of 2009, only 16.30% (815\/4999) of nonfederal, acute-care hospitals had adopted a fully interoperable EHR. From the 83 articles reviewed in this study, 16\/83 (19%) identified internal organizational factors and 9\/83 (11%) identified external environmental factors associated with adoption of the EHR, EMR, or CPOE. The conceptual model for EHR adoption associates each variable with the work that identified it.\nConclusions: Commonalities exist in the literature for internal organizational and external environmental factors associated with the adoption of the EHR and\/or CPOE. The conceptual model for EHR adoption associates internal and external factors, specific to the health care industry, associated with adoption of the EHR. It becomes apparent that these factors have some level of association, but the association is not consistently calculated individually or in combination. To better understand effective adoption strategies, empirical studies should be performed from this conceptual model to quantify the positive or negative effect of each factor.\nKeywords: electronic health record (EHR); electronic medical record (EMR); health information technology (HIT); medical information systems; computerized provider order entry (CPOE); adoption \n\nIntroduction \nBackground \nThe US Government passed the Health Information Technology for Economic and Clinical Health (HITECH) act[1] to incentivize adoption of the electronic health record (EHR) and to assuage the short run (SR) effects of cost to the health care organization in the adoption process. The three phases of Meaningful Use consume information technology (IT) strategies in the SR because of the HITECH act\u2019s timeline for health care organizations to qualify for monetary incentives.[2][3]\nAdoption of the EHR is a significant goal. International vernacular for the EHR varies; for example, electronic patient record, computerized patient records, electronic medical records (EMRs), and digital medical record. The defining difference, as defined by the Institute of Medicine, the health arm of the US National Academy of Sciences, focuses on the longitudinal and interoperable nature of the electronic patient record.[4] Without these capabilities, the patient record is greatly limited in scope. The longitudinal and interoperable nuances of the EHR are not the only significant advantages; there are eventual cost savings as well.\nStudies estimate that adoption of the EHR could eventually save more than $813 billion annually, prevent 200,000 adverse drug events, and enhance the doctor-patient relationship through increased communication.[5] Unfortunately, these benefits are realized in the long run (LR), while the investment to adopt the EHR is expended in the SR. A large deficit in the SR could inhibit a health care organization\u2019s ability to compete or survive in heavily competitive environment.\nThe environment of health care is unique in a competitive environment. The health care organization develops an organizational strategy based on the local environment. To increase an organization\u2019s ability to compete, its strategy might also include cost reduction, and EHR adoption runs counter to this goal in the SR. The health care environment faces many sources of influence, including a reluctance to accept technology.\nThere has been a tremendous amount of research dedicated to the study of acceptance of technology, specifically the Technology Acceptance Model (TAM).[6] More recent work has suggested modifications to the TAM that explain a perception of usefulness and intentions from the aspect of social influence and the cognitive instrumental process.[7][8] Several organizational theories have been developed. These focus on the sources of influence and the reason for their existence.\n\nOrganizational theories \nOrganizational theories address influence, but none adequately addresses the complexity of the health care organization. Payers, providers, and patients all control resources that exert influence. The nature of the competitive environment will also exert influence on decisions. External influence from those who control resources can be explained through resource dependence theory.[9][10] Internal and external influences can be explained by the Diffusion of Innovation Theory through its introduction of compatibility, complexity, trialability, observability, and relative advantage.[11][12][13][14]\nAccording to resource dependence theory, health care organizations with the greatest level of dependence on other organizations that control the resources will feel the greatest level of environmental influence on its decisions.[14] The Resource Dependence Theory describes an external interdependence of organizations. External Control of Organizations,[14] which is an adaptation of Resource Dependence Theory, provides good insight for this study. The authors\u2019 premise is that the external environment creates a social context and plays an important role in how organizational decisions are made. The lack of absolute independence requires some degree of interorganizational exchange of goods or services.[14] As organizations build and negotiate relationships with each other in the exchange of resources, positions of power are established. No one organization can provide all of its own resources, so each organization becomes dependent on the other organizations that control the resources.\nSimilar to Resource Dependence, the Diffusion of Innovation Theory describes a social system that influences through communication channels.[11][12][13] Diffusion of Innovation attempts to explain how \u201can innovation, is communicated through channels over time among members of a social system\u201d.[13] This theory accounts for 49-97% of variance in the rate of adoption of innovation through five factors: compatibility, complexity, trialability, observability, and relative advantage. These factors are sorted into three categories of a predictive model for EHR adoption: innovation determinants, organizational determinants, and environmental determinants.[8] The next several paragraphs exercise the five factors to this study.\nThe concept of compatibility[13] goes beyond answering the question, \u201cis a product\/service right for a market?\u201d It also asks, \u201cis the market ready for the product\/service?\u201d For instance, the Chevy Nova failed in Spanish-speaking markets because in Spanish the word \u201cNova\u201d means \u201cdoes not go.\u201d Promotion of conservation techniques to farmers in the United States initially failed because farmers associated conservation with lower crop yield. Boiling water to sanitize it makes perfect sense to a market that is familiar with germ theory, but primitive tribes in Peru only heated water for sicker, weaker members; as a result, the concept failed when initially introduced and dysentery continued to flourish. In relation to this study, the concept of compatibility might ask, \u201cis the market ready for the EHR?\u201d\nThe concept of complexity[13] is appropriate to this study because innovation can be a double-edged sword. On one hand, it is new and may offer some improvement to a product or service. However, it might also be perceived as too complex; and perception can be a powerful force. If the Baby Boomer generation perceives computers to be too complex, and this perception causes computer anxiety, its users may reject its adoption and use.[15] The older physicians in a hospital have greater seniority, and are therefore, more influential in the hospital\u2019s decision to adopt the EHR. Would this same generation of providers influence the health care organization considering EHR adoption?\nThe concept of trialability[13] applies more to the early-adopter group than the other groups: innovators, early-majority, late-majority, and laggards. In the early phase of promotion for a new product or service, the vendor might lower the risk of adoption by offering free trials or samples to potential users. Once the user is confident of the new item\u2019s efficacy, then he\/she is more likely to pay full price for its use. When a new producer of an EHR enters the marketplace, it must incentivize the use of its product because it is not known in the industry. The user accepts a risk by trying the new EHR, but the risk is overcome by the incentive. Once the new EHR gains momentum in the industry, adoption enters the early-majority phase. The new EHR has already gained momentum in the industry, and the producer does not need to incentivize its use.\nThe concept of observability[13] is also highly applicable to this study. Decision makers in a hospital that has not yet adopted an EHR will observe the experiences of other hospitals that have adopted it. Vendors will promote or advertise specifically to the nonadopters and help them observe how the EHR can benefit its organization. External players in the health care organization\u2019s competitive environment will provide some level of observability.\nRelative advantage is a multifaceted concept for this study. In health care, the most important factor is provision of health, as well as the treatment and prevention of disease. If adoption of the EHR speaks directly to the health care organization\u2019s primary purpose, then it might provide relative advantage over competitors that have not adopted it. Another concept is that of social prestige.[13] Unless a health care organization can serve as an example to other health care organizations (observability), there may not be a sufficient level of relative advantage to be considered.\n\nStrategy and decision making \nStrategy can be a multifaceted concept, and organizations around the world hire strategy experts to help identify and focus on a market forces. An operational definition of strategy is borrowed from education[16] and is adapted to health care: strategy is defined as instruments by which health care organizations manage their organizational processes and deal with their environments in order to select a portfolio of activities and find appropriate position in the health care industry (italics indicate a change in wording from the authors\u2019 definition). It follows that adoption of an EHR would alter how a health care organization manages its organizational processes, so this definition of strategy is a good fit for the health care industry. However, two significant considerations in the health care environment are the level of local competiveness, and how health care organizations compete.[17]\nStudies have shown that decision making in the health care industry is often based on how the organization competes, whether in a single-market or multimarket environment.[18] In either environment, decision-making varies on competition, and the health care industry competes in clusters.[18] The way health care organizations compete will also affect its organizational structure. A four-cluster solution was identified as a reliable, internally valid, and stable model for health networks and a five-cluster solution for health systems.[19] Differentiation and centralization are particularly important in distinguishing unique clusters of organizations. High differentiation typically occurs with low centralization, which suggests that a broader scope of activity is more difficult to centrally coordinate. Integration is also important, but the authors find that health networks and systems typically engage in both ownership-based and contractual-based integration or they are not integrated at all.\nAsh and Bates[20] studied the EHR adoption rates and the factors and forces affecting system adoption through surveys (85\/650, 13.1%). Only 106 of the 650 (16.3%) of hospitals surveyed had adopted some form of EHR, 63\/106 (59.4%) had implemented a full Computerized Provider Order Entry (CPOE) solution, and the other 43\/106 (40.6%) implemented a partial CPOE solution. A full one-third of adopters were either Veterans Affairs or military hospitals. Additionally, 481\/650 (73.8%) of those who planned to implement a full solution intended to do so within 5 years. Ash and Bates[20] also found that the size of hospital is positively-associated with component adoption; specifically CPOE adoption. The authors inferred from their results that the primary reasons to adopt the EHR is to gain the quality-of-care advantages of CPOE. This inference reinforced our inclusion of CPOE as a dependent variable.\nFactors that influence health information system (HIS) adoption in US hospitals have been studied by others as well (n=1441).[21] Results showed that HIS adoption is influenced by the hospital market, organizational, and financial factors. Larger, system-affiliated, and for-profit hospitals with more preferred provider organization contracts are more likely to adopt managerial information systems than other hospitals. Operating revenue is positively associated with HIS adoption. The study also identified hostility as an aspect of environmental uncertainty, and that organizations often turn to technological adoption to regain competitive advantage.\nA knowledge-based taxonomy of critical factors for adopting an EHR was developed from a systematic literature review.[22] The researchers selected 68 of 3400 (2.00%) articles to identify six factors of adoption, listed in order of importance: user attitude toward information systems, workflow impact, interoperability, technical support, communication among users, and expert support.\nAlternative measures of EHR adoption among hospitals have been studied.[23] Authors analyzed a 2009 information technology supplement survey distributed by the American Hospital Association (AHA). The survey focused on 24 EHR functionalities in various areas: electronic clinical documentation, results viewing, CPOE, and clinical decision support. They found that 142 of 3937 (3.60%) acute-care hospitals in the United States of responding hospitals have implemented all 24 functions, 386\/3937 (9.80%) of hospitals have implemented at least 20 functions, and 1437\/3937 (36.50%) have implemented at least one-half of the functions. The researchers added that EHR adoption is a complex process.\nOthers have studied the relationship between hospital financial position and the adoption of the EHR.[24] Through a cross-sectional study of secondary data from several sources, including the AHA (2442\/5752, 42.51% acute-care hospitals in the United States), researchers identified five independent and one dependent variable. Of the five independent variables (IVs), only liquidity was positively-associated with EHR. Asset turnover was negatively-associated with EHR adoption. Bed size, a control variable, was positively-associated with EHR adoption. The authors concluded that hospitals adopt EHRs as a strategic move to better align themselves with their environment.\nBecause commonly used elements of organizational strategy are difficult to change, several of the variables were categorized as internal organizational factors. Research has assessed variables of hospital influence in five categories: (1) capacity as measured by number of beds in groupings by intervals of 100, (2) management, or ownership, (3) organizational focus, or teaching status, (4) competitive location and alternatives, and (5) state regulatory pressures.[25]\nAlthough resources have been consumed to study factors associated with adoption of HIT, there is a gap in the literature that provides a conceptual model to guide the design of empirical models. It may seem backward to design a conceptual model after so many studies have already been conducted, but the gap remains. The aim of this study was to develop a conceptual model from a systematic literature review that associates both internal and external factors associated with adoption of the EHR. The intent of the conceptual model is to enable future empirical models.\n\nMethods \nLiterature review process \nSearch terms were selected based on the experience of the authors in the field of health care administration. The time frame of 1993-2013 was selected as convenience. It was assumed that 2 decades would be sufficient to capture trends.\nFigure 1 illustrates the literature review process that identified 83 sources consisting of empirical studies, articles, editorials, commentaries, opinion papers, organizational theories, and text books. The intent of no limits to the type of papers was to mitigate the risk of missing something significant from a study that was not catalogued properly within a key word catalogue like the Dublin Core.\nThe 83 records were reviewed for content and evidence. After discarding 58 articles for lack of evidence, three additional references were added because they were key concepts upon which other studies were based. Of the remaining 25 articles, a list of factors was identified as IVs. Some factors were grouped under a similar category for the purposes of simplification of the conceptual model. The dependent variable (DV) started as adoption of the EHR, but the studies from those chosen were not as specific. From personal experience, many studies seem to discuss the EHR, but call it something else: most commonly the EMR. That is why EMR was included in the search. Because so few ERHs exist without some form of CPOE, the latter term was included in the search criteria.\nOur study combines the influences highlighted by previous work and examines determinants of EHR adoption. Examining EHR adoption at the health care organization level will demonstrate validity between this study and others that have used the hospital as the unit of analysis.\n\n\n\nFigure 1. Literature review process\nEHR adoption and internal organizational influence \nSeveral influences in the environment exert pressure on the health care organization to adopt EHR. Influences range from incentives from the federal government to the nature of local competitive community. US federal incentives provide a heavy influence for EHR implementation, under specific conditions, and imposes penalties for a lack of EHR implementation.\nThe internal politics of one organization serve as one source of influence. A hospital is part of a community, which serves as an external influence. Further, if a hospital is also part of a larger multihospital system (MHS), then the politics of the broad MHS will also exert influence on local decisions.\n\nEHR adoption and external environmental influence \nThe patient is external to the organization, and for our study, the patient primarily serves as an external influence. Although some employees of the health care organization might also be patients, and this relationship could create a small internal influence, this study considers those few stake holders in the internal organizational factor of users. The providers are internal to the organization, and for our study, providers serve as an internal organizational influence. The payer is a significant influence[14], and the Center for Medicare and Medicaid Services (CMS) serves as a good example of this significant influence.[26] The HITECH act provides monetary incentives for EHR adoption. Those who do not implement all aspects specified in the stages of adoption are not eligible for the incentives. In this way, the CMS disincentivizes those organizations that do not adopt the EHR. If payments from the CMS were of little consequence to the health care organization\u2019s revenue, then the health care organization might decide differently about EHR adoption. A competing health care organization is an external market force in the environment. Third-party payers might compare health care organizations based on maturity of automation because mature clinical components like CPOE will result in more accurate billing. Such forces incentivize a health care organization to adopt the EHR.\n\nOverview of the conceptual model \nThe premise for an EHR adoption conceptual model is that that environmental influences affect organizational strategy of the health care organizations that adopt the EHR.[13][14][20][22][24][25] Diffusion of Innovation theory provides three categories of a predictive model for EHR adoption: innovation determinants, organizational determinants, and environmental determinants.[13] Resource Dependence Theory provides a category of a predictive model for EHR adoption, the competitive environment. In construction of the EHR adoption conceptual model, several constructs emerged.[14]\nElements of organizational strategy are not variables that can be easily changed[19]; therefore, elements typically ascribed to strategy, such as size, ownership, and fiscal stability, will be absorbed into the IVs of influence. This research proposes a model, whereby environmental factors are associated with an organization\u2019s decision to adopt the EHR.\nResource Dependence Theory explains environmental influences and the external interdependence of organizations.[14] The authors\u2019 premise is that the external environment creates a social context and plays an important role in how organizational decisions are made. The interdependence of organizations widens the field of stakeholders, and this relationship effect should be defined.\nDisparate stakeholders have different interests with reference to different components of the EHR. These interests may be different in the SR interests versus the LR interests. SR interests are those that are immediate, such as current year expenditures. LR interests are further out when all inputs are variable. The SR interests of cost can often compete with the LR potential of cost savings and greater safety. Both the SR and LR interests are affected by the external environment.[17]\nIn a highly competitive environment, SR cost implications could often win over any long-term savings. The number of patients in a market is fixed in the SR, and a highly competitive market will affect each competitor\u2019s share of that market. The SR costs of EHR implementation might be insurmountable by an organization in this market because it could not afford to lose ground without significant capital reserves or the ability to borrow cheaply.[17] However, in a less competitive market, the LR interests of potential cost savings have a better chance of influencing the decision to implement an EHR because the costs incurred in the SR are justified by the long-term benefits.[17]\nExternal stakeholders that control resources important to the health care organization can exert significant influence. For instance, a health care organization that receives a significant amount of revenue from the CMS will be influenced more by incentives provided by the CMS than an organization that receives a significant cash flow from private third parties. The relative influence of various external stakeholders may be captured by an analysis of the structure of the market in which a health care organization operates.\nStakeholders have varying interests with regard to the capabilities and effects of EHR components depending upon their relationship with the health care organization. Private payers have both SR and LR interests in the EHR. In the SR, their focus is on minimizing expenditures. Because the health care organization would pass on the implementation costs through higher contract costs, payers would not be equal in the SR. In addition, the disruption of EHR implementation could potentially affect care processes and therefore increase claims. Payers would be interested in the LR benefits of the EHR: potential cost savings, better disease management, and increased safety. However, the SR interests of the private payers might overshadow the LR benefits of the EHR. Public payers enable care of the indigent and elderly. As part of the United States Department of Health and Human Services (HHS), the CMS is highly interested in disease management, public health, safety, and research, and it may value these LR capabilities of the EHR more than the SR costs. The CMS, as part of HHS, would also favor the EHR because it supports the Presidential directive to promote the establishment of the Nationwide Health Information Network that links electronic patient records through health information exchanges.\nProviders and patients value face time with each other. During EHR implementation, providers might spend less time in communication with patients. Providers must adapt their processes and clinic-to-administrative schedules. Any disruption or action that is perceived as deleterious to this relationship could result in a negative reaction to EHR implementation. As a result, physicians might oppose EHR adoption, or they might simply support the EHR solution with the shortest implementation time or least administrative burden. Patients might not like the reduced face time with the provider, but they might be attracted to EHR components such as e-prescribing, e-results, personal health records, and email access to the provider. These desirable features are available to the patient when the health care organization chooses to adopt various portions of the CPOE component to the EHR.\n\nResults \nChosen articles \nThe articles chosen for final inclusion were read once more to make a list of variables. The variables from the studies were listed as internal and external. There were significant commonalities in the variables used, so they were combined in the model.\n\nEHR adoption and internal organizational influence \nAs depicted in Figure 1, 16 references identified internal factors.[7][8][12][14][15][19][21][22][23][24][27][28][29][30][31][32][33] Six identified size of the health care organization, and six identified strategic alliances. Five identified ownership and five identified complexity of care. Four identified capital expenditures. Three identified users, and three identified teaching status. Two identified user attitude toward HIS, and two identified communication among users. Workflow impact, interoperability, technical support, expert support, physician arrangements, unity of effort, and user computer anxiety were all identified by one study, independently.\nThe dependent variable was not consistent: seven references used EHR adoption[23][24][25][28][30][31][33], two used \u201celectronic capture of clinical data,\u201d[23][25] one used a generic DV of technology adoption[15], and six used CPOE[20][23][25][28][29][31].\n\nEHR adoption and external environmental influence \nAs depicted in Figure 1, nine studies identified external environmental factors[12][14][15][19][22][24][28][30][31]. Five studies identified buyers, four studies identified patients, three studies identified competitiveness, two studies identified location, and one identified interdependence factors external to the organization that are associated with adoption of the EHR.\n\nOverview of the conceptual model \nAs previously stated, there was overlap between the sources\/theories. There were four internal forces and seven external forces identified through multiple works by three authors[11][12][13][14][22]. However, it was unclear in existing literature the degree to which these forces can influence a health care organization\u2019s decision to adopt the EHR. A complex conceptual model should provide insight into the strength and direction of the influence on the complex health care organization. The resulting conceptual model, depicted in Figure 2, posits a complex relationship between environmental influences, organizational strategy, and EHR adoption.\nThis framework captures both internal and external factors that influence the adoption of the EHR. The positive (+) and negative (\u2212) signs in the model describe the relationship identified by the associated authors. For instance, Gin et al.[24] identified a positive relationship between the external environmental factors of public payer (IV) and competitiveness (IV) and an association with the adoption of an EHR (DV). That is to say, the greater the percentages of an organization\u2019s reimbursements that come from a public source like CMS, the stronger the association of the organization\u2019s adoption of the EHR. Likewise, the greater the Herfindahl Index of the local competitive environment, the stronger the association of the organization\u2019s adoption of the EHR. Age is another interesting factor because of its negative relationship with adoption. The older the patient population (external environmental IV)[15] and provider population (internal organizational IV)[30], the lower association with the adoption of the EHR (DV). The + and \u2212 signs above the arrows between the IVs and DVs indicates the variety of positive and negative associations with the adoption of an EHR.\n\n\n\nFigure 2. Conceptual model of factors associated with adoption of the EHR\nDiscussion \nPrincipal findings \nThe main findings of this study were that nine studies identified external factors and 16 studies identified internal factors associated with the adoption of EHR. These factors were depicted in a conceptual model to describe relationships to EHR adoption.\nThe conceptual model for EHR adoption illustrates a framework within which both administrators and policy makers can work to understand the levers that exert significant influence in the adoption of EHR. The extensive literature review conducted by this study builds a unique model from which empirical studies can be designed.\nIdentifying relationships between the adoption factors and adoption of the EHR becomes significant because it identifies levers that will produce a desired action. For instance, if a hospital has a majority of senior providers, perhaps from the Baby Boom generation, the administrators become aware of the additional effort that needs to go into user acceptance. A hospital that has a majority of new providers will not need to expend the resources on user acceptance, because studies already show a penchant for technology in younger generations. Similar inferences on other factors of adoption could be made, and some would require additional study.\nFor instance, the literature on workflow impact is split. There seems to be evidence that the presence of the EHR both enhances and complicates the providers\u2019 workflow. This observation clearly begs additional questions. Were subjects for the data in different phases of adoption of the EHR? Was the hospital that responded negatively in the middle of an EHR implementation? Logically, a large implementation of any technology will become disruptive to the organization. Several studies could emerge from this relationship alone.\nEmpirical models could easily be designed to further investigate specific relationships between the IVs and DVs. The set of studies on CPOE was interesting. Although there were some overlaps with adoption of the EHR, there were also studies that only looked at CPOE. There does not seem to be an abundance of evidence in the literature about CPOE, and yet the AHA regularly collects data on six different versions of CPOE: laboratory, radiology, pharmacy, nursing, physician notes, and consults. There was no data to be found about the use or efficacy on CPOE consults. It might be interesting to determine the reason for this paucity of data.\n\nLimitations \nThe EHR adoption conceptual model associates internal and external factors with the adoption of the EHR, but it is primarily based on an extensive literature review. So far, it is not empirically tested. However, data are available to test the theory. Because the findings of our study are descriptive in nature, we do not opine on appropriate medical use of the information.\nCaution should be identified with the interaction of variables. Some variables will most likely confound or mask the effects of others. For instance, is there a direct relationship between the number of beds of a hospital and the number of full-time equivalents? There are staffing models that would most likely answer that question. If there is a similar relationship, then one of these variables should be eliminated in favor of the stronger one. Otherwise, the effects of the weaker variable will be masked by the other. A false conclusion could easily be identified concerning the masked variable.\nA majority of references for this study were from the United States, with one exception from Hong Kong. The internal validity of this study is strongest within the US health care sector. The conceptual model might be limited outside the United States because of the nature of competition between hospitals. The analysis of 83 articles identified studies that used similar methods: survey or secondary data analysis. Many authors analyzed data from the AHA, a well-established data set in the United States. These data are self-reported, which comes with limited bias.\n\nConclusions \nThis study also identified overlap between studies in terms of variables. One interpretation of this overlap could be that the variables and associated studies are highly reliable. The key word\/phrase searches described in the Methods section identifies the databases queried and results given. Other researchers should be able to duplicate or update this conceptual model going forward.\n\nAuthors' contributions \nThe topic of research and conceptual model in this original article are from a chapter of the PhD dissertation of Kruse C DeShazo J, and Kim F sat on the dissertation committee, which helped direct the research and develop the model. Fulton L helped Kruse C extract portions of the dissertation worthy of publication.\n\nConflicts of interest \nNone declared.\n\nReferences \n\n\n\u2191 \"Title XIII: Health Information Technology for Economic and Clinical Health Act\" (PDF). American Recovery and Reinvestment Act of 2009. ONC. 19 February 2009. https:\/\/www.healthit.gov\/sites\/default\/files\/hitech_act_excerpt_from_arra_with_index.pdf . Retrieved 18 March 2014 .   \n\n\u2191 Elnahal, S.M.; Joynt, K.E.; Bristol, S.J.; Jha, A.K. (2011). \"Electronic health record functions differ between best and worst hospitals\". American Journal of Managed Care 17 (4): e121\u2013e147. PMC PMC3335431. PMID 21774097. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3335431 .   \n\n\u2191 Song, P.H.; McAlearney, A.S.; Robbins, J.; McCullough, J.S. (2011). \"Exploring the business case for ambulatory electronic health record system adoption\". Journal of Healthcare Management 56 (3): 169-80; discussion 181-2. PMID 21714372.   \n\n\u2191 \"Key Capabilities of an Electronic Health Record System\". National Academies Institute of Medicine. 31 July 2003. http:\/\/iom.nationalacademies.org\/Reports\/2003\/Key-Capabilities-of-an-Electronic-Health-Record-System.aspx . Retrieved 20 March 2014 .   \n\n\u2191 Hillestad, Richard; Bigelow, James; Bower, Anthony; Girosi, Federico; Meili, Robin; Scoville, Richard; Taylor, Roger (2005). \"Can electronic medical record systems transform health care? Potential health benefits, savings, and costs\". Health Affairs 24 (5): 1103\u20131117. doi:10.1377\/hlthaff.24.5.1103. PMID 16162551.   \n\n\u2191 Davis, F. (1989). \"Perceived usefulness, perceived ease of use, and user acceptance of information technology\". MIS Quarterly 13 (3): 319\u2013340. doi:10.2307\/249008.   \n\n\u2191 7.0 7.1 Venkatesh, V; Davis, F.D. (2000). \"A theoretical extension of the Technology Acceptance Model: four longitudinal field studies\". Management Science 46 (2): 186\u2013204. doi:10.1287\/mnsc.46.2.186.11926.   \n\n\u2191 8.0 8.1 8.2 Venkatesh, V; Morris, M.; Davis, G.; Davis, F. (2003). \"User acceptance of information technology: Toward a unified view\". MIS Quarterly 27 (3): 425\u2013478. doi:10.2307\/30036540.   \n\n\u2191 Aldrich, H.E.; Pfeffer, J. (1976). \"Environments of organizations\". Annual Review of Sociology 2 (1): 79\u2013105. doi:10.1146\/annurev.so.02.080176.000455.   \n\n\u2191 Pfeffer, J.; Salancik, G. (1978). The External Control of Organizations: A Resource Dependence Perspective. New York, NY: Harper & Row.   \n\n\u2191 11.0 11.1 11.2 Scholz, Carolyn Elaine (01 May 2001). \"Applying Rogers' theory of diffusion of innovations to examine older females' perceptions of size labels for apparel\". University of Manitoba. http:\/\/mspace.lib.umanitoba.ca\/xmlui\/handle\/1993\/2732 .   \n\n\u2191 12.0 12.1 12.2 12.3 12.4 Rogers, E.M. (1995). Diffusion of Innovations (4th ed.). New York, NY: Free Press. ISBN 9780029266717.   \n\n\u2191 13.00 13.01 13.02 13.03 13.04 13.05 13.06 13.07 13.08 13.09 13.10 Rogers, E.M. (2003). Diffusion of Innovations (5th ed.). New York, NY: Free Press. ISBN 9780743222099.   \n\n\u2191 14.00 14.01 14.02 14.03 14.04 14.05 14.06 14.07 14.08 14.09 14.10 Pfeffer, J.; Salancik, G. (2003). The External Control of Organizations: A Resource Dependence Perspective. Stanford, CA: Stanford Business Books. ISBN 9780804747899.   \n\n\u2191 15.0 15.1 15.2 15.3 15.4 Czaja, S.J.; Charness, N.; Fisk, A.D.; Hertzog, C.; Nair, S.N.; Rogers, W.A.; Sharit, J. (2006). \"Factors predicting the use of technology: findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE)\". Psychology and Aging 21 (2): 333\u2013352. doi:10.1037\/0882-7974.21.2.333. PMC PMC1524856. PMID 16768579. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524856 .   \n\n\u2191 Fumasoli, T.; Lepori, B. (2011). \"Patterns of strategies in Swiss higher education institutions\". Higher Education 61 (2): 157\u2013178. doi:10.1007\/s10734-010-9330-x.   \n\n\u2191 17.0 17.1 17.2 17.3 Henderson, J.. Health Economics and Policy (with Economic Applications) (2nd ed.). Mason, OH: South-Western College Pub. ISBN 9780538481175.   \n\n\u2191 18.0 18.1 Sikka, V.; Luke, R.D.; Ozcan, Y.A. (2009). \"The efficiency of hospital-based clusters: evaluating system performance using data envelopment analysis\". Health Care Management Review 34 (3): 251\u2013261. doi:10.1097\/HMR.0b013e3181a16ba7. PMID 19625830.   \n\n\u2191 19.0 19.1 19.2 19.3 Bazzoli, G.J.; Shortell, S.M.; Dubbs, N.; Chan, C.; Kralovec, P. (1999). \"A taxonomy of health networks and systems: Bringing order out of chaos\". Health Services Research Journal 33 (6): 1683\u20131717. PMC PMC1070343. PMID 10029504. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1070343 .   \n\n\u2191 20.0 20.1 20.2 20.3 Ash, J.S.; Bates, D.W. (2005). \"Factors and forces affecting EHR system adoption: Report of a 2004 ACMI discussion\". Journal of the American Medical Informatics Association 12 (1): 8\u201312. doi:10.1197\/jamia.M1684. PMC PMC543830. PMID 15492027. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC543830 .   \n\n\u2191 21.0 21.1 Wang, B.B.; Wan, T.T.; Burke, D.E.; Bazzoli, G.J.; Lin, B.Y. (2005). \"Factors influencing health information system adoption in American hospitals\". Health Care Management Review 30 (1): 44-51. PMID 15773253.   \n\n\u2191 22.0 22.1 22.2 22.3 22.4 Castillo, V.H.; Mart\u00ednez-Garc\u00eda, A.I.; Pulido, J.R. (2010). \"A knowledge-based taxonomy of critical factors for adopting electronic health record systems by physicians: a systematic literature review\". BMC Medical Informatics & Decision Making 10: 60. doi:10.1186\/1472-6947-10-60. PMC PMC2970582. PMID 20950458. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2970582 .   \n\n\u2191 23.0 23.1 23.2 23.3 23.4 Blavin, F.E.; Buntin, M.J.; Friedman, C.P. (2010). \"Alternative measures of electronic health record adoption among hospitals\". American Journal of Managed Care 16 (12 Suppl HIT): e293-e301. PMID 21322296.   \n\n\u2191 24.0 24.1 24.2 24.3 24.4 24.5 Ginn, G.O.; Shen, J.J.; Moseley, C.B. (2011). \"Hospital financial position and the adoption of electronic health records\". Journal of Healthcare Management 56 (5): 337\u201350; discussion 351\u20132. PMID 21991681.   \n\n\u2191 25.0 25.1 25.2 25.3 25.4 Farley, D.E.; Hogan, C. (1990). \"Case-mix specialization in the market for hospital services\". Health Services Research Journal 25 (5): 757\u2013783. PMC PMC1065663. PMID 2123838. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1065663 .   \n\n\u2191 \"NHE Fact Sheet\". CMS.gov. Centers for Medicare and Medicaid Services. 2014. https:\/\/www.cms.gov\/Research-Statistics-Data-and-Systems\/Statistics-Trends-and-Reports\/NationalHealthExpendData\/NHE-Fact-Sheet.html . Retrieved 18 March 2014 .   \n\n\u2191 Bazzoli, G.J.; Chan, B.; Shortell, S.M.; D'Aunno, T. (2000). \"The financial performance of hospitals belonging to health networks and systems\". Inquiry 37 (3): 234\u2013252. PMID 11111282.   \n\n\u2191 28.0 28.1 28.2 28.3 DesRoches, C.M.; Campbell, E.G.; Vogeli, C.; Zheng, J.; Rao, S.R.; Shields, A.E. Donelan, K.; Rosenbaum, S.; Bristol, S.J.; Jha, A.K. (2010). \"Electronic health records' limited successes suggest more targeted uses\". Health Affairs 29 (4): 639\u2013646. doi:10.1377\/hlthaff.2009.1086. PMID 20368593.   \n\n\u2191 29.0 29.1 Harshberger, C.A.; Harper, A.J.; Carro, G.W.; Spath, W.E.; Hui, W.C.; Lawton, J.M.; Brockstein, B.E. (2011). \"Outcomes of computerized physician order entry in an electronic health record after implementation in an outpatient oncology setting\". Journal of Oncology Practice 7 (4): 233\u2013237. doi:10.1200\/JOP.2011.000261. PMC PMC3140445. PMID 22043187. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3140445 .   \n\n\u2191 30.0 30.1 30.2 30.3 Decker, S.L.; Jamoom, E.W.; Sisk, J.E. (2012). \"Physicians in nonprimary care and small practices and those age 55 and older lag in adopting electronic health record systems\". Health Affairs 31 (5): 1108\u20131114. doi:10.1377\/hlthaff.2011.1121. PMID 22535502.   \n\n\u2191 31.0 31.1 31.2 31.3 Wolf, L.; Harvell, J.; Jha, A.K. (2012). \"Hospitals ineligible for federal meaningful-use incentives have dismally low rates of adoption of electronic health records\". Health Affairs 31 (3): 505\u2013513. doi:10.1377\/hlthaff.2011.0351. PMID 22392661.   \n\n\u2191 Lluch, M. (2011). \"Healthcare professionals' organisational barriers to health information technologies-a literature review\". International Journal of Medical Informatics 80 (12): 849\u2013862. doi:10.1016\/j.ijmedinf.2011.09.005. PMID 22000677.   \n\n\u2191 33.0 33.1 Cheung, C.S.; Tong, E.L.; Cheung, N.T.; Chan, W.M.; Wang, H.H.; Kwan, M.W.; Fan, C.K.; Liu, K.Q.; Wong, M.C. (2013). \"Factors associated with adoption of the electronic health record system among primary care physicians\". JMIR Medical Informatics 1 (1): e1. doi:10.2196\/medinform.2766. PMC PMC4288082. PMID 25599989. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4288082 .   \n\n\nAbbreviations \nAHA: American Hospital Association\nARRA: American Recovery and Reinvestment Act\nCMS: Center for Medicare and Medicaid Services\nCPOE: computerized provider order entry\nDV: dependent variables\nEHR: electronic health record\nEMR: electronic medical record\nHHS: the Department of Health and Human Services\nHIMSS: Health Information Management Systems Society\nHIS: health information system\nHIT: health information technology\nHITECH: Health Information Technology for Economic and Clinical Health\nIT: information technology\nIV: independent variables\nLR: long run\nMHS: multihospital system\nSR: short run\nTAM: Technology Acceptance Model\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful.\nPer the distribution agreement, the following copyright information is also being added: \n\u00a9Clemens Scott Kruse, Jonathan DeShazo, Forest Kim, Lawrence Fulton. Originally published in JMIR Medical Informatics (http:\/\/medinform.jmir.org), 23.05.2014.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\">https:\/\/www.limswiki.org\/index.php\/Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:53.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,355 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","3eb9d25818bfd5a80515d585fb6a6712_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Factors_associated_with_adoption_of_health_information_technology_A_conceptual_model_based_on_a_systematic_review skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Factors associated with adoption of health information technology: A conceptual model based on a systematic review<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> The <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Information_Technology_for_Economic_and_Clinical_Health_Act\" title=\"Health Information Technology for Economic and Clinical Health Act\" target=\"_blank\" class=\"wiki-link\" data-key=\"89c9e20984bbda2e628b6d1d28ec3ad5\">Health Information Technology for Economic and Clinical Health Act<\/a> (HITECH) allocated $19.2 billion to incentivize adoption of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a> (EHR). Since 2009, Meaningful Use Criteria have dominated information technology (IT) strategy. Health care organizations have struggled to meet expectations and avoid penalties to reimbursements from the <a href=\"https:\/\/www.limswiki.org\/index.php\/Centers_for_Medicare_and_Medicaid_Services\" title=\"Centers for Medicare and Medicaid Services\" target=\"_blank\" class=\"wiki-link\" data-key=\"654b4449e4816e190325b420c264df1a\">Center for Medicare and Medicaid Services<\/a> (CMS). Organizational theories attempt to explain factors that influence organizational change, and many theories address changes in organizational strategy. However, due to the complexities of the health care industry, existing organizational theories fall short of demonstrating association with significant health care IT implementations. There is no organizational theory for health care that identifies, groups, and analyzes both internal and external factors of influence for large health care IT implementations like adoption of the EHR.\n<\/p><p><b>Objective:<\/b> The purpose of this systematic review is to identify a full-spectrum of both internal organizational and external environmental factors associated with the adoption of <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_information_technology\" title=\"Health information technology\" target=\"_blank\" class=\"wiki-link\" data-key=\"9c8ef822470559f757db89f3fa234cc0\">health information technology<\/a> (HIT), specifically the EHR. The result is a conceptual model that is commensurate with the complexity of with the health care sector.\n<\/p><p><b>Methods:<\/b> We performed a systematic literature search in PubMed (restricted to English), EBSCO Host, and Google Scholar for both empirical studies and theory-based writing from 1993-2013 that demonstrated association between influential factors and three modes of HIT: EHR, <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_medical_record\" title=\"Electronic medical record\" target=\"_blank\" class=\"wiki-link\" data-key=\"99a695d2af23397807da0537d29d0be7\">electronic medical record<\/a> (EMR), and computerized provider order entry (CPOE). We also looked at published books on organizational theories. We made notes and noted trends on adoption factors. These factors were grouped as adoption factors associated with various versions of EHR adoption.\n<\/p><p><b>Results:<\/b> The resulting conceptual model summarizes the diversity of independent variables (IVs) and dependent variables (DVs) used in articles, editorials, books, as well as quantitative and qualitative studies (n=83). As of 2009, only 16.30% (815\/4999) of nonfederal, acute-care <a href=\"https:\/\/www.limswiki.org\/index.php\/Hospital\" title=\"Hospital\" target=\"_blank\" class=\"wiki-link\" data-key=\"b8f070c66d8123fe91063594befebdff\">hospitals<\/a> had adopted a fully interoperable EHR. From the 83 articles reviewed in this study, 16\/83 (19%) identified internal organizational factors and 9\/83 (11%) identified external environmental factors associated with adoption of the EHR, EMR, or CPOE. The conceptual model for EHR adoption associates each variable with the work that identified it.\n<\/p><p><b>Conclusions:<\/b> Commonalities exist in the literature for internal organizational and external environmental factors associated with the adoption of the EHR and\/or CPOE. The conceptual model for EHR adoption associates internal and external factors, specific to the health care industry, associated with adoption of the EHR. It becomes apparent that these factors have some level of association, but the association is not consistently calculated individually or in combination. To better understand effective adoption strategies, empirical studies should be performed from this conceptual model to quantify the positive or negative effect of each factor.\n<\/p><p><b>Keywords:<\/b> electronic health record (EHR); electronic medical record (EMR); health information technology (HIT); medical information systems; computerized provider order entry (CPOE); adoption \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h3>\n<p>The US Government passed the Health Information Technology for Economic and Clinical Health (HITECH) act<sup id=\"rdp-ebb-cite_ref-HITECH_1-0\" class=\"reference\"><a href=\"#cite_note-HITECH-1\" rel=\"external_link\">[1]<\/a><\/sup> to incentivize adoption of the electronic health record (EHR) and to assuage the short run (SR) effects of cost to the health care organization in the adoption process. The three phases of Meaningful Use consume information technology (IT) strategies in the SR because of the HITECH act\u2019s timeline for health care organizations to qualify for monetary incentives.<sup id=\"rdp-ebb-cite_ref-ElnahalElec_2-0\" class=\"reference\"><a href=\"#cite_note-ElnahalElec-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SongExp_3-0\" class=\"reference\"><a href=\"#cite_note-SongExp-3\" rel=\"external_link\">[3]<\/a><\/sup>\n<\/p><p>Adoption of the EHR is a significant goal. International vernacular for the EHR varies; for example, electronic patient record, computerized patient records, electronic medical records (EMRs), and digital medical record. The defining difference, as defined by the Institute of Medicine, the health arm of the US National Academy of Sciences, focuses on the longitudinal and interoperable nature of the electronic patient record.<sup id=\"rdp-ebb-cite_ref-NatAcadKey_4-0\" class=\"reference\"><a href=\"#cite_note-NatAcadKey-4\" rel=\"external_link\">[4]<\/a><\/sup> Without these capabilities, the patient record is greatly limited in scope. The longitudinal and interoperable nuances of the EHR are not the only significant advantages; there are eventual cost savings as well.\n<\/p><p>Studies estimate that adoption of the EHR could eventually save more than $813 billion annually, prevent 200,000 adverse drug events, and enhance the doctor-patient relationship through increased communication.<sup id=\"rdp-ebb-cite_ref-HillestadCan_5-0\" class=\"reference\"><a href=\"#cite_note-HillestadCan-5\" rel=\"external_link\">[5]<\/a><\/sup> Unfortunately, these benefits are realized in the long run (LR), while the investment to adopt the EHR is expended in the SR. A large deficit in the SR could inhibit a health care organization\u2019s ability to compete or survive in heavily competitive environment.\n<\/p><p>The environment of health care is unique in a competitive environment. The health care organization develops an organizational strategy based on the local environment. To increase an organization\u2019s ability to compete, its strategy might also include cost reduction, and EHR adoption runs counter to this goal in the SR. The health care environment faces many sources of influence, including a reluctance to accept technology.\n<\/p><p>There has been a tremendous amount of research dedicated to the study of acceptance of technology, specifically the Technology Acceptance Model (TAM).<sup id=\"rdp-ebb-cite_ref-DavisPerc_6-0\" class=\"reference\"><a href=\"#cite_note-DavisPerc-6\" rel=\"external_link\">[6]<\/a><\/sup> More recent work has suggested modifications to the TAM that explain a perception of usefulness and intentions from the aspect of social influence and the cognitive instrumental process.<sup id=\"rdp-ebb-cite_ref-VenkateshasATheo_7-0\" class=\"reference\"><a href=\"#cite_note-VenkateshasATheo-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VenkateshasUser_8-0\" class=\"reference\"><a href=\"#cite_note-VenkateshasUser-8\" rel=\"external_link\">[8]<\/a><\/sup> Several organizational theories have been developed. These focus on the sources of influence and the reason for their existence.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Organizational_theories\">Organizational theories<\/span><\/h3>\n<p>Organizational theories address influence, but none adequately addresses the complexity of the health care organization. Payers, providers, and patients all control resources that exert influence. The nature of the competitive environment will also exert influence on decisions. External influence from those who control resources can be explained through resource dependence theory.<sup id=\"rdp-ebb-cite_ref-AldrichEnv_9-0\" class=\"reference\"><a href=\"#cite_note-AldrichEnv-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PfefferTheEx1_10-0\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx1-10\" rel=\"external_link\">[10]<\/a><\/sup> Internal and external influences can be explained by the Diffusion of Innovation Theory through its introduction of compatibility, complexity, trialability, observability, and relative advantage.<sup id=\"rdp-ebb-cite_ref-ScholzApp_11-0\" class=\"reference\"><a href=\"#cite_note-ScholzApp-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RogersDiff4_12-0\" class=\"reference\"><a href=\"#cite_note-RogersDiff4-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-0\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-0\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p><p>According to resource dependence theory, health care organizations with the greatest level of dependence on other organizations that control the resources will feel the greatest level of environmental influence on its decisions.<sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-1\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup> The Resource Dependence Theory describes an external interdependence of organizations. External Control of Organizations,<sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-2\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup> which is an adaptation of Resource Dependence Theory, provides good insight for this study. The authors\u2019 premise is that the external environment creates a social context and plays an important role in how organizational decisions are made. The lack of absolute independence requires some degree of interorganizational exchange of goods or services.<sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-3\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup> As organizations build and negotiate relationships with each other in the exchange of resources, positions of power are established. No one organization can provide all of its own resources, so each organization becomes dependent on the other organizations that control the resources.\n<\/p><p>Similar to Resource Dependence, the Diffusion of Innovation Theory describes a social system that influences through communication channels.<sup id=\"rdp-ebb-cite_ref-ScholzApp_11-1\" class=\"reference\"><a href=\"#cite_note-ScholzApp-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RogersDiff4_12-1\" class=\"reference\"><a href=\"#cite_note-RogersDiff4-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-1\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> Diffusion of Innovation attempts to explain how \u201can innovation, is communicated through channels over time among members of a social system\u201d.<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-2\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> This theory accounts for 49-97% of variance in the rate of adoption of innovation through five factors: compatibility, complexity, trialability, observability, and relative advantage. These factors are sorted into three categories of a predictive model for EHR adoption: innovation determinants, organizational determinants, and environmental determinants.<sup id=\"rdp-ebb-cite_ref-VenkateshasUser_8-1\" class=\"reference\"><a href=\"#cite_note-VenkateshasUser-8\" rel=\"external_link\">[8]<\/a><\/sup> The next several paragraphs exercise the five factors to this study.\n<\/p><p>The concept of compatibility<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-3\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> goes beyond answering the question, \u201cis a product\/service right for a market?\u201d It also asks, \u201cis the market ready for the product\/service?\u201d For instance, the Chevy Nova failed in Spanish-speaking markets because in Spanish the word \u201cNova\u201d means \u201cdoes not go.\u201d Promotion of conservation techniques to farmers in the United States initially failed because farmers associated conservation with lower crop yield. Boiling water to sanitize it makes perfect sense to a market that is familiar with germ theory, but primitive tribes in Peru only heated water for sicker, weaker members; as a result, the concept failed when initially introduced and dysentery continued to flourish. In relation to this study, the concept of compatibility might ask, \u201cis the market ready for the EHR?\u201d\n<\/p><p>The concept of complexity<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-4\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> is appropriate to this study because innovation can be a double-edged sword. On one hand, it is new and may offer some improvement to a product or service. However, it might also be perceived as too complex; and perception can be a powerful force. If the Baby Boomer generation perceives computers to be too complex, and this perception causes computer anxiety, its users may reject its adoption and use.<sup id=\"rdp-ebb-cite_ref-CzajaFact_15-0\" class=\"reference\"><a href=\"#cite_note-CzajaFact-15\" rel=\"external_link\">[15]<\/a><\/sup> The older physicians in a hospital have greater seniority, and are therefore, more influential in the hospital\u2019s decision to adopt the EHR. Would this same generation of providers influence the health care organization considering EHR adoption?\n<\/p><p>The concept of trialability<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-5\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> applies more to the early-adopter group than the other groups: innovators, early-majority, late-majority, and laggards. In the early phase of promotion for a new product or service, the vendor might lower the risk of adoption by offering free trials or samples to potential users. Once the user is confident of the new item\u2019s efficacy, then he\/she is more likely to pay full price for its use. When a new producer of an EHR enters the marketplace, it must incentivize the use of its product because it is not known in the industry. The user accepts a risk by trying the new EHR, but the risk is overcome by the incentive. Once the new EHR gains momentum in the industry, adoption enters the early-majority phase. The new EHR has already gained momentum in the industry, and the producer does not need to incentivize its use.\n<\/p><p>The concept of observability<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-6\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> is also highly applicable to this study. Decision makers in a hospital that has not yet adopted an EHR will observe the experiences of other hospitals that have adopted it. Vendors will promote or advertise specifically to the nonadopters and help them observe how the EHR can benefit its organization. External players in the health care organization\u2019s competitive environment will provide some level of observability.\n<\/p><p>Relative advantage is a multifaceted concept for this study. In health care, the most important factor is provision of health, as well as the treatment and prevention of disease. If adoption of the EHR speaks directly to the health care organization\u2019s primary purpose, then it might provide relative advantage over competitors that have not adopted it. Another concept is that of social prestige.<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-7\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> Unless a health care organization can serve as an example to other health care organizations (observability), there may not be a sufficient level of relative advantage to be considered.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Strategy_and_decision_making\">Strategy and decision making<\/span><\/h3>\n<p>Strategy can be a multifaceted concept, and organizations around the world hire strategy experts to help identify and focus on a market forces. An operational definition of strategy is borrowed from education<sup id=\"rdp-ebb-cite_ref-FumasoliPatt_16-0\" class=\"reference\"><a href=\"#cite_note-FumasoliPatt-16\" rel=\"external_link\">[16]<\/a><\/sup> and is adapted to health care: strategy is defined as instruments by which health care organizations manage their organizational processes and deal with their environments in order to select a portfolio of activities and find appropriate position in the health care industry (italics indicate a change in wording from the authors\u2019 definition). It follows that adoption of an EHR would alter how a health care organization manages its organizational processes, so this definition of strategy is a good fit for the health care industry. However, two significant considerations in the health care environment are the level of local competiveness, and how health care organizations compete.<sup id=\"rdp-ebb-cite_ref-HendersonHealth_17-0\" class=\"reference\"><a href=\"#cite_note-HendersonHealth-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/p><p>Studies have shown that decision making in the health care industry is often based on how the organization competes, whether in a single-market or multimarket environment.<sup id=\"rdp-ebb-cite_ref-SikkaTheEff_18-0\" class=\"reference\"><a href=\"#cite_note-SikkaTheEff-18\" rel=\"external_link\">[18]<\/a><\/sup> In either environment, decision-making varies on competition, and the health care industry competes in clusters.<sup id=\"rdp-ebb-cite_ref-SikkaTheEff_18-1\" class=\"reference\"><a href=\"#cite_note-SikkaTheEff-18\" rel=\"external_link\">[18]<\/a><\/sup> The way health care organizations compete will also affect its organizational structure. A four-cluster solution was identified as a reliable, internally valid, and stable model for health networks and a five-cluster solution for health systems.<sup id=\"rdp-ebb-cite_ref-BazzoliATax_19-0\" class=\"reference\"><a href=\"#cite_note-BazzoliATax-19\" rel=\"external_link\">[19]<\/a><\/sup> Differentiation and centralization are particularly important in distinguishing unique clusters of organizations. High differentiation typically occurs with low centralization, which suggests that a broader scope of activity is more difficult to centrally coordinate. Integration is also important, but the authors find that health networks and systems typically engage in both ownership-based and contractual-based integration or they are not integrated at all.\n<\/p><p>Ash and Bates<sup id=\"rdp-ebb-cite_ref-AshFact_20-0\" class=\"reference\"><a href=\"#cite_note-AshFact-20\" rel=\"external_link\">[20]<\/a><\/sup> studied the EHR adoption rates and the factors and forces affecting system adoption through surveys (85\/650, 13.1%). Only 106 of the 650 (16.3%) of hospitals surveyed had adopted some form of EHR, 63\/106 (59.4%) had implemented a full Computerized Provider Order Entry (CPOE) solution, and the other 43\/106 (40.6%) implemented a partial CPOE solution. A full one-third of adopters were either Veterans Affairs or military hospitals. Additionally, 481\/650 (73.8%) of those who planned to implement a full solution intended to do so within 5 years. Ash and Bates<sup id=\"rdp-ebb-cite_ref-AshFact_20-1\" class=\"reference\"><a href=\"#cite_note-AshFact-20\" rel=\"external_link\">[20]<\/a><\/sup> also found that the size of hospital is positively-associated with component adoption; specifically CPOE adoption. The authors inferred from their results that the primary reasons to adopt the EHR is to gain the quality-of-care advantages of CPOE. This inference reinforced our inclusion of CPOE as a dependent variable.\n<\/p><p>Factors that influence health information system (HIS) adoption in US hospitals have been studied by others as well (n=1441).<sup id=\"rdp-ebb-cite_ref-WangFact_21-0\" class=\"reference\"><a href=\"#cite_note-WangFact-21\" rel=\"external_link\">[21]<\/a><\/sup> Results showed that HIS adoption is influenced by the hospital market, organizational, and financial factors. Larger, system-affiliated, and for-profit hospitals with more preferred provider organization contracts are more likely to adopt managerial information systems than other hospitals. Operating revenue is positively associated with HIS adoption. The study also identified hostility as an aspect of environmental uncertainty, and that organizations often turn to technological adoption to regain competitive advantage.\n<\/p><p>A knowledge-based taxonomy of critical factors for adopting an EHR was developed from a systematic literature review.<sup id=\"rdp-ebb-cite_ref-CastilloAKnow_22-0\" class=\"reference\"><a href=\"#cite_note-CastilloAKnow-22\" rel=\"external_link\">[22]<\/a><\/sup> The researchers selected 68 of 3400 (2.00%) articles to identify six factors of adoption, listed in order of importance: user attitude toward information systems, workflow impact, interoperability, technical support, communication among users, and expert support.\n<\/p><p>Alternative measures of EHR adoption among hospitals have been studied.<sup id=\"rdp-ebb-cite_ref-BlavinAlt_23-0\" class=\"reference\"><a href=\"#cite_note-BlavinAlt-23\" rel=\"external_link\">[23]<\/a><\/sup> Authors analyzed a 2009 information technology supplement survey distributed by the American Hospital Association (AHA). The survey focused on 24 EHR functionalities in various areas: electronic clinical documentation, results viewing, CPOE, and clinical decision support. They found that 142 of 3937 (3.60%) acute-care hospitals in the United States of responding hospitals have implemented all 24 functions, 386\/3937 (9.80%) of hospitals have implemented at least 20 functions, and 1437\/3937 (36.50%) have implemented at least one-half of the functions. The researchers added that EHR adoption is a complex process.\n<\/p><p>Others have studied the relationship between hospital financial position and the adoption of the EHR.<sup id=\"rdp-ebb-cite_ref-GinnHos_24-0\" class=\"reference\"><a href=\"#cite_note-GinnHos-24\" rel=\"external_link\">[24]<\/a><\/sup> Through a cross-sectional study of secondary data from several sources, including the AHA (2442\/5752, 42.51% acute-care hospitals in the United States), researchers identified five independent and one dependent variable. Of the five independent variables (IVs), only liquidity was positively-associated with EHR. Asset turnover was negatively-associated with EHR adoption. Bed size, a control variable, was positively-associated with EHR adoption. The authors concluded that hospitals adopt EHRs as a strategic move to better align themselves with their environment.\n<\/p><p>Because commonly used elements of organizational strategy are difficult to change, several of the variables were categorized as internal organizational factors. Research has assessed variables of hospital influence in five categories: (1) capacity as measured by number of beds in groupings by intervals of 100, (2) management, or ownership, (3) organizational focus, or teaching status, (4) competitive location and alternatives, and (5) state regulatory pressures.<sup id=\"rdp-ebb-cite_ref-FarleyCase_25-0\" class=\"reference\"><a href=\"#cite_note-FarleyCase-25\" rel=\"external_link\">[25]<\/a><\/sup>\n<\/p><p>Although resources have been consumed to study factors associated with adoption of HIT, there is a gap in the literature that provides a conceptual model to guide the design of empirical models. It may seem backward to design a conceptual model after so many studies have already been conducted, but the gap remains. The aim of this study was to develop a conceptual model from a systematic literature review that associates both internal and external factors associated with adoption of the EHR. The intent of the conceptual model is to enable future empirical models.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Literature_review_process\">Literature review process<\/span><\/h3>\n<p>Search terms were selected based on the experience of the authors in the field of health care administration. The time frame of 1993-2013 was selected as convenience. It was assumed that 2 decades would be sufficient to capture trends.\n<\/p><p>Figure 1 illustrates the literature review process that identified 83 sources consisting of empirical studies, articles, editorials, commentaries, opinion papers, organizational theories, and text books. The intent of no limits to the type of papers was to mitigate the risk of missing something significant from a study that was not catalogued properly within a key word catalogue like the Dublin Core.\n<\/p><p>The 83 records were reviewed for content and evidence. After discarding 58 articles for lack of evidence, three additional references were added because they were key concepts upon which other studies were based. Of the remaining 25 articles, a list of factors was identified as IVs. Some factors were grouped under a similar category for the purposes of simplification of the conceptual model. The dependent variable (DV) started as adoption of the EHR, but the studies from those chosen were not as specific. From personal experience, many studies seem to discuss the EHR, but call it something else: most commonly the EMR. That is why EMR was included in the search. Because so few ERHs exist without some form of CPOE, the latter term was included in the search criteria.\n<\/p><p>Our study combines the influences highlighted by previous work and examines determinants of EHR adoption. Examining EHR adoption at the health care organization level will demonstrate validity between this study and others that have used the hospital as the unit of analysis.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Kruse_JMIRMedInfo2014_2-1.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"ec89a7d639788231e73abe3665ce33ab\"><img alt=\"Fig1 Kruse JMIRMedInfo2014 2-1.jpg\" src=\"https:\/\/www.limswiki.org\/images\/6\/65\/Fig1_Kruse_JMIRMedInfo2014_2-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Literature review process<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"EHR_adoption_and_internal_organizational_influence\">EHR adoption and internal organizational influence<\/span><\/h3>\n<p>Several influences in the environment exert pressure on the health care organization to adopt EHR. Influences range from incentives from the federal government to the nature of local competitive community. US federal incentives provide a heavy influence for EHR implementation, under specific conditions, and imposes penalties for a lack of EHR implementation.\n<\/p><p>The internal politics of one organization serve as one source of influence. A hospital is part of a community, which serves as an external influence. Further, if a hospital is also part of a larger multihospital system (MHS), then the politics of the broad MHS will also exert influence on local decisions.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"EHR_adoption_and_external_environmental_influence\">EHR adoption and external environmental influence<\/span><\/h3>\n<p>The patient is external to the organization, and for our study, the patient primarily serves as an external influence. Although some employees of the health care organization might also be patients, and this relationship could create a small internal influence, this study considers those few stake holders in the internal organizational factor of users. The providers are internal to the organization, and for our study, providers serve as an internal organizational influence. The payer is a significant influence<sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-4\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup>, and the Center for Medicare and Medicaid Services (CMS) serves as a good example of this significant influence.<sup id=\"rdp-ebb-cite_ref-NHEFact_26-0\" class=\"reference\"><a href=\"#cite_note-NHEFact-26\" rel=\"external_link\">[26]<\/a><\/sup> The HITECH act provides monetary incentives for EHR adoption. Those who do not implement all aspects specified in the stages of adoption are not eligible for the incentives. In this way, the CMS disincentivizes those organizations that do not adopt the EHR. If payments from the CMS were of little consequence to the health care organization\u2019s revenue, then the health care organization might decide differently about EHR adoption. A competing health care organization is an external market force in the environment. Third-party payers might compare health care organizations based on maturity of automation because mature clinical components like CPOE will result in more accurate billing. Such forces incentivize a health care organization to adopt the EHR.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Overview_of_the_conceptual_model\">Overview of the conceptual model<\/span><\/h3>\n<p>The premise for an EHR adoption conceptual model is that that environmental influences affect organizational strategy of the health care organizations that adopt the EHR.<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-8\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-5\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AshFact_20-2\" class=\"reference\"><a href=\"#cite_note-AshFact-20\" rel=\"external_link\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CastilloAKnow_22-1\" class=\"reference\"><a href=\"#cite_note-CastilloAKnow-22\" rel=\"external_link\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GinnHos_24-1\" class=\"reference\"><a href=\"#cite_note-GinnHos-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FarleyCase_25-1\" class=\"reference\"><a href=\"#cite_note-FarleyCase-25\" rel=\"external_link\">[25]<\/a><\/sup> Diffusion of Innovation theory provides three categories of a predictive model for EHR adoption: innovation determinants, organizational determinants, and environmental determinants.<sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-9\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup> Resource Dependence Theory provides a category of a predictive model for EHR adoption, the competitive environment. In construction of the EHR adoption conceptual model, several constructs emerged.<sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-6\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p><p>Elements of organizational strategy are not variables that can be easily changed<sup id=\"rdp-ebb-cite_ref-BazzoliATax_19-1\" class=\"reference\"><a href=\"#cite_note-BazzoliATax-19\" rel=\"external_link\">[19]<\/a><\/sup>; therefore, elements typically ascribed to strategy, such as size, ownership, and fiscal stability, will be absorbed into the IVs of influence. This research proposes a model, whereby environmental factors are associated with an organization\u2019s decision to adopt the EHR.\n<\/p><p>Resource Dependence Theory explains environmental influences and the external interdependence of organizations.<sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-7\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup> The authors\u2019 premise is that the external environment creates a social context and plays an important role in how organizational decisions are made. The interdependence of organizations widens the field of stakeholders, and this relationship effect should be defined.\n<\/p><p>Disparate stakeholders have different interests with reference to different components of the EHR. These interests may be different in the SR interests versus the LR interests. SR interests are those that are immediate, such as current year expenditures. LR interests are further out when all inputs are variable. The SR interests of cost can often compete with the LR potential of cost savings and greater safety. Both the SR and LR interests are affected by the external environment.<sup id=\"rdp-ebb-cite_ref-HendersonHealth_17-1\" class=\"reference\"><a href=\"#cite_note-HendersonHealth-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/p><p>In a highly competitive environment, SR cost implications could often win over any long-term savings. The number of patients in a market is fixed in the SR, and a highly competitive market will affect each competitor\u2019s share of that market. The SR costs of EHR implementation might be insurmountable by an organization in this market because it could not afford to lose ground without significant capital reserves or the ability to borrow cheaply.<sup id=\"rdp-ebb-cite_ref-HendersonHealth_17-2\" class=\"reference\"><a href=\"#cite_note-HendersonHealth-17\" rel=\"external_link\">[17]<\/a><\/sup> However, in a less competitive market, the LR interests of potential cost savings have a better chance of influencing the decision to implement an EHR because the costs incurred in the SR are justified by the long-term benefits.<sup id=\"rdp-ebb-cite_ref-HendersonHealth_17-3\" class=\"reference\"><a href=\"#cite_note-HendersonHealth-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/p><p>External stakeholders that control resources important to the health care organization can exert significant influence. For instance, a health care organization that receives a significant amount of revenue from the CMS will be influenced more by incentives provided by the CMS than an organization that receives a significant cash flow from private third parties. The relative influence of various external stakeholders may be captured by an analysis of the structure of the market in which a health care organization operates.\n<\/p><p>Stakeholders have varying interests with regard to the capabilities and effects of EHR components depending upon their relationship with the health care organization. Private payers have both SR and LR interests in the EHR. In the SR, their focus is on minimizing expenditures. Because the health care organization would pass on the implementation costs through higher contract costs, payers would not be equal in the SR. In addition, the disruption of EHR implementation could potentially affect care processes and therefore increase claims. Payers would be interested in the LR benefits of the EHR: potential cost savings, better disease management, and increased safety. However, the SR interests of the private payers might overshadow the LR benefits of the EHR. Public payers enable care of the indigent and elderly. As part of the <a href=\"https:\/\/www.limswiki.org\/index.php\/United_States_Department_of_Health_and_Human_Services\" title=\"United States Department of Health and Human Services\" target=\"_blank\" class=\"wiki-link\" data-key=\"efa106bcbb93039b1a6c3c596daedec3\">United States Department of Health and Human Services<\/a> (HHS), the CMS is highly interested in disease management, public health, safety, and research, and it may value these LR capabilities of the EHR more than the SR costs. The CMS, as part of HHS, would also favor the EHR because it supports the Presidential directive to promote the establishment of the Nationwide Health Information Network that links electronic patient records through health information exchanges.\n<\/p><p>Providers and patients value face time with each other. During EHR implementation, providers might spend less time in communication with patients. Providers must adapt their processes and clinic-to-administrative schedules. Any disruption or action that is perceived as deleterious to this relationship could result in a negative reaction to EHR implementation. As a result, physicians might oppose EHR adoption, or they might simply support the EHR solution with the shortest implementation time or least administrative burden. Patients might not like the reduced face time with the provider, but they might be attracted to EHR components such as e-prescribing, e-results, personal health records, and email access to the provider. These desirable features are available to the patient when the health care organization chooses to adopt various portions of the CPOE component to the EHR.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Chosen_articles\">Chosen articles<\/span><\/h3>\n<p>The articles chosen for final inclusion were read once more to make a list of variables. The variables from the studies were listed as internal and external. There were significant commonalities in the variables used, so they were combined in the model.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"EHR_adoption_and_internal_organizational_influence_2\">EHR adoption and internal organizational influence<\/span><\/h3>\n<p>As depicted in Figure 1, 16 references identified internal factors.<sup id=\"rdp-ebb-cite_ref-VenkateshasATheo_7-1\" class=\"reference\"><a href=\"#cite_note-VenkateshasATheo-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-VenkateshasUser_8-2\" class=\"reference\"><a href=\"#cite_note-VenkateshasUser-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RogersDiff4_12-2\" class=\"reference\"><a href=\"#cite_note-RogersDiff4-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-8\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CzajaFact_15-1\" class=\"reference\"><a href=\"#cite_note-CzajaFact-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BazzoliATax_19-2\" class=\"reference\"><a href=\"#cite_note-BazzoliATax-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WangFact_21-1\" class=\"reference\"><a href=\"#cite_note-WangFact-21\" rel=\"external_link\">[21]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CastilloAKnow_22-2\" class=\"reference\"><a href=\"#cite_note-CastilloAKnow-22\" rel=\"external_link\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BlavinAlt_23-1\" class=\"reference\"><a href=\"#cite_note-BlavinAlt-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GinnHos_24-2\" class=\"reference\"><a href=\"#cite_note-GinnHos-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BazzoliTheFin_27-0\" class=\"reference\"><a href=\"#cite_note-BazzoliTheFin-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DesRochesElec_28-0\" class=\"reference\"><a href=\"#cite_note-DesRochesElec-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HarshbergerOut_29-0\" class=\"reference\"><a href=\"#cite_note-HarshbergerOut-29\" rel=\"external_link\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeckerPhys_30-0\" class=\"reference\"><a href=\"#cite_note-DeckerPhys-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WolfHos_31-0\" class=\"reference\"><a href=\"#cite_note-WolfHos-31\" rel=\"external_link\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LluchHealth_32-0\" class=\"reference\"><a href=\"#cite_note-LluchHealth-32\" rel=\"external_link\">[32]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CheungFact_33-0\" class=\"reference\"><a href=\"#cite_note-CheungFact-33\" rel=\"external_link\">[33]<\/a><\/sup> Six identified size of the health care organization, and six identified strategic alliances. Five identified ownership and five identified complexity of care. Four identified capital expenditures. Three identified users, and three identified teaching status. Two identified user attitude toward HIS, and two identified communication among users. Workflow impact, interoperability, technical support, expert support, physician arrangements, unity of effort, and user computer anxiety were all identified by one study, independently.\n<\/p><p>The dependent variable was not consistent: seven references used EHR adoption<sup id=\"rdp-ebb-cite_ref-BlavinAlt_23-2\" class=\"reference\"><a href=\"#cite_note-BlavinAlt-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GinnHos_24-3\" class=\"reference\"><a href=\"#cite_note-GinnHos-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FarleyCase_25-2\" class=\"reference\"><a href=\"#cite_note-FarleyCase-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DesRochesElec_28-1\" class=\"reference\"><a href=\"#cite_note-DesRochesElec-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeckerPhys_30-1\" class=\"reference\"><a href=\"#cite_note-DeckerPhys-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WolfHos_31-1\" class=\"reference\"><a href=\"#cite_note-WolfHos-31\" rel=\"external_link\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CheungFact_33-1\" class=\"reference\"><a href=\"#cite_note-CheungFact-33\" rel=\"external_link\">[33]<\/a><\/sup>, two used \u201celectronic capture of clinical data,\u201d<sup id=\"rdp-ebb-cite_ref-BlavinAlt_23-3\" class=\"reference\"><a href=\"#cite_note-BlavinAlt-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FarleyCase_25-3\" class=\"reference\"><a href=\"#cite_note-FarleyCase-25\" rel=\"external_link\">[25]<\/a><\/sup> one used a generic DV of technology adoption<sup id=\"rdp-ebb-cite_ref-CzajaFact_15-2\" class=\"reference\"><a href=\"#cite_note-CzajaFact-15\" rel=\"external_link\">[15]<\/a><\/sup>, and six used CPOE<sup id=\"rdp-ebb-cite_ref-AshFact_20-3\" class=\"reference\"><a href=\"#cite_note-AshFact-20\" rel=\"external_link\">[20]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BlavinAlt_23-4\" class=\"reference\"><a href=\"#cite_note-BlavinAlt-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-FarleyCase_25-4\" class=\"reference\"><a href=\"#cite_note-FarleyCase-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DesRochesElec_28-2\" class=\"reference\"><a href=\"#cite_note-DesRochesElec-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HarshbergerOut_29-1\" class=\"reference\"><a href=\"#cite_note-HarshbergerOut-29\" rel=\"external_link\">[29]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WolfHos_31-2\" class=\"reference\"><a href=\"#cite_note-WolfHos-31\" rel=\"external_link\">[31]<\/a><\/sup>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"EHR_adoption_and_external_environmental_influence_2\">EHR adoption and external environmental influence<\/span><\/h3>\n<p>As depicted in Figure 1, nine studies identified external environmental factors<sup id=\"rdp-ebb-cite_ref-RogersDiff4_12-3\" class=\"reference\"><a href=\"#cite_note-RogersDiff4-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-9\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CzajaFact_15-3\" class=\"reference\"><a href=\"#cite_note-CzajaFact-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BazzoliATax_19-3\" class=\"reference\"><a href=\"#cite_note-BazzoliATax-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CastilloAKnow_22-3\" class=\"reference\"><a href=\"#cite_note-CastilloAKnow-22\" rel=\"external_link\">[22]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GinnHos_24-4\" class=\"reference\"><a href=\"#cite_note-GinnHos-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DesRochesElec_28-3\" class=\"reference\"><a href=\"#cite_note-DesRochesElec-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeckerPhys_30-2\" class=\"reference\"><a href=\"#cite_note-DeckerPhys-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WolfHos_31-3\" class=\"reference\"><a href=\"#cite_note-WolfHos-31\" rel=\"external_link\">[31]<\/a><\/sup>. Five studies identified buyers, four studies identified patients, three studies identified competitiveness, two studies identified location, and one identified interdependence factors external to the organization that are associated with adoption of the EHR.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Overview_of_the_conceptual_model_2\">Overview of the conceptual model<\/span><\/h3>\n<p>As previously stated, there was overlap between the sources\/theories. There were four internal forces and seven external forces identified through multiple works by three authors<sup id=\"rdp-ebb-cite_ref-ScholzApp_11-2\" class=\"reference\"><a href=\"#cite_note-ScholzApp-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RogersDiff4_12-4\" class=\"reference\"><a href=\"#cite_note-RogersDiff4-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RogersDiff5_13-10\" class=\"reference\"><a href=\"#cite_note-RogersDiff5-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PfefferTheEx2_14-10\" class=\"reference\"><a href=\"#cite_note-PfefferTheEx2-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CastilloAKnow_22-4\" class=\"reference\"><a href=\"#cite_note-CastilloAKnow-22\" rel=\"external_link\">[22]<\/a><\/sup>. However, it was unclear in existing literature the degree to which these forces can influence a health care organization\u2019s decision to adopt the EHR. A complex conceptual model should provide insight into the strength and direction of the influence on the complex health care organization. The resulting conceptual model, depicted in Figure 2, posits a complex relationship between environmental influences, organizational strategy, and EHR adoption.\n<\/p><p>This framework captures both internal and external factors that influence the adoption of the EHR. The positive (+) and negative (\u2212) signs in the model describe the relationship identified by the associated authors. For instance, Gin et al.<sup id=\"rdp-ebb-cite_ref-GinnHos_24-5\" class=\"reference\"><a href=\"#cite_note-GinnHos-24\" rel=\"external_link\">[24]<\/a><\/sup> identified a positive relationship between the external environmental factors of public payer (IV) and competitiveness (IV) and an association with the adoption of an EHR (DV). That is to say, the greater the percentages of an organization\u2019s reimbursements that come from a public source like CMS, the stronger the association of the organization\u2019s adoption of the EHR. Likewise, the greater the Herfindahl Index of the local competitive environment, the stronger the association of the organization\u2019s adoption of the EHR. Age is another interesting factor because of its negative relationship with adoption. The older the patient population (external environmental IV)<sup id=\"rdp-ebb-cite_ref-CzajaFact_15-4\" class=\"reference\"><a href=\"#cite_note-CzajaFact-15\" rel=\"external_link\">[15]<\/a><\/sup> and provider population (internal organizational IV)<sup id=\"rdp-ebb-cite_ref-DeckerPhys_30-3\" class=\"reference\"><a href=\"#cite_note-DeckerPhys-30\" rel=\"external_link\">[30]<\/a><\/sup>, the lower association with the adoption of the EHR (DV). The + and \u2212 signs above the arrows between the IVs and DVs indicates the variety of positive and negative associations with the adoption of an EHR.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Kruse_JMIRMedInfo2014_2-1.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"1300fea96984cfd4924e9a811929d694\"><img alt=\"Fig2 Kruse JMIRMedInfo2014 2-1.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/3a\/Fig2_Kruse_JMIRMedInfo2014_2-1.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> Conceptual model of factors associated with adoption of the EHR<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Principal_findings\">Principal findings<\/span><\/h3>\n<p>The main findings of this study were that nine studies identified external factors and 16 studies identified internal factors associated with the adoption of EHR. These factors were depicted in a conceptual model to describe relationships to EHR adoption.\n<\/p><p>The conceptual model for EHR adoption illustrates a framework within which both administrators and policy makers can work to understand the levers that exert significant influence in the adoption of EHR. The extensive literature review conducted by this study builds a unique model from which empirical studies can be designed.\n<\/p><p>Identifying relationships between the adoption factors and adoption of the EHR becomes significant because it identifies levers that will produce a desired action. For instance, if a hospital has a majority of senior providers, perhaps from the Baby Boom generation, the administrators become aware of the additional effort that needs to go into user acceptance. A hospital that has a majority of new providers will not need to expend the resources on user acceptance, because studies already show a penchant for technology in younger generations. Similar inferences on other factors of adoption could be made, and some would require additional study.\n<\/p><p>For instance, the literature on workflow impact is split. There seems to be evidence that the presence of the EHR both enhances and complicates the providers\u2019 workflow. This observation clearly begs additional questions. Were subjects for the data in different phases of adoption of the EHR? Was the hospital that responded negatively in the middle of an EHR implementation? Logically, a large implementation of any technology will become disruptive to the organization. Several studies could emerge from this relationship alone.\n<\/p><p>Empirical models could easily be designed to further investigate specific relationships between the IVs and DVs. The set of studies on CPOE was interesting. Although there were some overlaps with adoption of the EHR, there were also studies that only looked at CPOE. There does not seem to be an abundance of evidence in the literature about CPOE, and yet the AHA regularly collects data on six different versions of CPOE: laboratory, radiology, pharmacy, nursing, physician notes, and consults. There was no data to be found about the use or efficacy on CPOE consults. It might be interesting to determine the reason for this paucity of data.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limitations\">Limitations<\/span><\/h3>\n<p>The EHR adoption conceptual model associates internal and external factors with the adoption of the EHR, but it is primarily based on an extensive literature review. So far, it is not empirically tested. However, data are available to test the theory. Because the findings of our study are descriptive in nature, we do not opine on appropriate medical use of the information.\n<\/p><p>Caution should be identified with the interaction of variables. Some variables will most likely confound or mask the effects of others. For instance, is there a direct relationship between the number of beds of a hospital and the number of full-time equivalents? There are staffing models that would most likely answer that question. If there is a similar relationship, then one of these variables should be eliminated in favor of the stronger one. Otherwise, the effects of the weaker variable will be masked by the other. A false conclusion could easily be identified concerning the masked variable.\n<\/p><p>A majority of references for this study were from the United States, with one exception from Hong Kong. The internal validity of this study is strongest within the US health care sector. The conceptual model might be limited outside the United States because of the nature of competition between hospitals. The analysis of 83 articles identified studies that used similar methods: survey or secondary data analysis. Many authors analyzed data from the AHA, a well-established data set in the United States. These data are self-reported, which comes with limited bias.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h3>\n<p>This study also identified overlap between studies in terms of variables. One interpretation of this overlap could be that the variables and associated studies are highly reliable. The key word\/phrase searches described in the Methods section identifies the databases queried and results given. Other researchers should be able to duplicate or update this conceptual model going forward.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.27_contributions\">Authors' contributions<\/span><\/h2>\n<p>The topic of research and conceptual model in this original article are from a chapter of the PhD dissertation of Kruse C DeShazo J, and Kim F sat on the dissertation committee, which helped direct the research and develop the model. Fulton L helped Kruse C extract portions of the dissertation worthy of publication.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h2>\n<p>None declared.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HITECH-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HITECH_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.healthit.gov\/sites\/default\/files\/hitech_act_excerpt_from_arra_with_index.pdf\" target=\"_blank\">\"Title XIII: Health Information Technology for Economic and Clinical Health Act\"<\/a> (PDF). <i>American Recovery and Reinvestment Act of 2009<\/i>. ONC. 19 February 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.healthit.gov\/sites\/default\/files\/hitech_act_excerpt_from_arra_with_index.pdf\" target=\"_blank\">https:\/\/www.healthit.gov\/sites\/default\/files\/hitech_act_excerpt_from_arra_with_index.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 March 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Title+XIII%3A+Health+Information+Technology+for+Economic+and+Clinical+Health+Act&rft.atitle=American+Recovery+and+Reinvestment+Act+of+2009&rft.date=19+February+2009&rft.pub=ONC&rft_id=https%3A%2F%2Fwww.healthit.gov%2Fsites%2Fdefault%2Ffiles%2Fhitech_act_excerpt_from_arra_with_index.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ElnahalElec-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ElnahalElec_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Elnahal, S.M.; Joynt, K.E.; Bristol, S.J.; Jha, A.K. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3335431\" target=\"_blank\">\"Electronic health record functions differ between best and worst hospitals\"<\/a>. <i>American Journal of Managed Care<\/i> <b>17<\/b> (4): e121\u2013e147. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3335431\/\" target=\"_blank\">PMC3335431<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21774097\" target=\"_blank\">21774097<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3335431\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3335431<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+record+functions+differ+between+best+and+worst+hospitals&rft.jtitle=American+Journal+of+Managed+Care&rft.aulast=Elnahal%2C+S.M.%3B+Joynt%2C+K.E.%3B+Bristol%2C+S.J.%3B+Jha%2C+A.K.&rft.au=Elnahal%2C+S.M.%3B+Joynt%2C+K.E.%3B+Bristol%2C+S.J.%3B+Jha%2C+A.K.&rft.date=2011&rft.volume=17&rft.issue=4&rft.pages=e121%E2%80%93e147&rft_id=info:pmc\/PMC3335431&rft_id=info:pmid\/21774097&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3335431&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SongExp-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SongExp_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Song, P.H.; McAlearney, A.S.; Robbins, J.; McCullough, J.S. (2011). \"Exploring the business case for ambulatory electronic health record system adoption\". <i>Journal of Healthcare Management<\/i> <b>56<\/b> (3): 169-80; discussion 181-2. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21714372\" target=\"_blank\">21714372<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+the+business+case+for+ambulatory+electronic+health+record+system+adoption&rft.jtitle=Journal+of+Healthcare+Management&rft.aulast=Song%2C+P.H.%3B+McAlearney%2C+A.S.%3B+Robbins%2C+J.%3B+McCullough%2C+J.S.&rft.au=Song%2C+P.H.%3B+McAlearney%2C+A.S.%3B+Robbins%2C+J.%3B+McCullough%2C+J.S.&rft.date=2011&rft.volume=56&rft.issue=3&rft.pages=169-80%3B+discussion+181-2&rft_id=info:pmid\/21714372&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NatAcadKey-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NatAcadKey_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/iom.nationalacademies.org\/Reports\/2003\/Key-Capabilities-of-an-Electronic-Health-Record-System.aspx\" target=\"_blank\">\"Key Capabilities of an Electronic Health Record System\"<\/a>. National Academies Institute of Medicine. 31 July 2003<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/iom.nationalacademies.org\/Reports\/2003\/Key-Capabilities-of-an-Electronic-Health-Record-System.aspx\" target=\"_blank\">http:\/\/iom.nationalacademies.org\/Reports\/2003\/Key-Capabilities-of-an-Electronic-Health-Record-System.aspx<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 March 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Key+Capabilities+of+an+Electronic+Health+Record+System&rft.atitle=&rft.date=31+July+2003&rft.pub=National+Academies+Institute+of+Medicine&rft_id=http%3A%2F%2Fiom.nationalacademies.org%2FReports%2F2003%2FKey-Capabilities-of-an-Electronic-Health-Record-System.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HillestadCan-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HillestadCan_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hillestad, Richard; Bigelow, James; Bower, Anthony; Girosi, Federico; Meili, Robin; Scoville, Richard; Taylor, Roger (2005). \"Can electronic medical record systems transform health care? Potential health benefits, savings, and costs\". <i>Health Affairs<\/i> <b>24<\/b> (5): 1103\u20131117. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.24.5.1103\" target=\"_blank\">10.1377\/hlthaff.24.5.1103<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16162551\" target=\"_blank\">16162551<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Can+electronic+medical+record+systems+transform+health+care%3F+Potential+health+benefits%2C+savings%2C+and+costs&rft.jtitle=Health+Affairs&rft.aulast=Hillestad%2C+Richard%3B+Bigelow%2C+James%3B+Bower%2C+Anthony%3B+Girosi%2C+Federico%3B+Meili%2C+Robin%3B+Scoville%2C+Richard%3B+Taylor%2C+Roger&rft.au=Hillestad%2C+Richard%3B+Bigelow%2C+James%3B+Bower%2C+Anthony%3B+Girosi%2C+Federico%3B+Meili%2C+Robin%3B+Scoville%2C+Richard%3B+Taylor%2C+Roger&rft.date=2005&rft.volume=24&rft.issue=5&rft.pages=1103%E2%80%931117&rft_id=info:doi\/10.1377%2Fhlthaff.24.5.1103&rft_id=info:pmid\/16162551&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DavisPerc-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DavisPerc_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Davis, F. (1989). \"Perceived usefulness, perceived ease of use, and user acceptance of information technology\". <i>MIS Quarterly<\/i> <b>13<\/b> (3): 319\u2013340. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2307%2F249008\" target=\"_blank\">10.2307\/249008<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Perceived+usefulness%2C+perceived+ease+of+use%2C+and+user+acceptance+of+information+technology&rft.jtitle=MIS+Quarterly&rft.aulast=Davis%2C+F.&rft.au=Davis%2C+F.&rft.date=1989&rft.volume=13&rft.issue=3&rft.pages=319%E2%80%93340&rft_id=info:doi\/10.2307%2F249008&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VenkateshasATheo-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VenkateshasATheo_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-VenkateshasATheo_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Venkatesh, V; Davis, F.D. (2000). \"A theoretical extension of the Technology Acceptance Model: four longitudinal field studies\". <i>Management Science<\/i> <b>46<\/b> (2): 186\u2013204. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1287%2Fmnsc.46.2.186.11926\" target=\"_blank\">10.1287\/mnsc.46.2.186.11926<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+theoretical+extension+of+the+Technology+Acceptance+Model%3A+four+longitudinal+field+studies&rft.jtitle=Management+Science&rft.aulast=Venkatesh%2C+V%3B+Davis%2C+F.D.&rft.au=Venkatesh%2C+V%3B+Davis%2C+F.D.&rft.date=2000&rft.volume=46&rft.issue=2&rft.pages=186%E2%80%93204&rft_id=info:doi\/10.1287%2Fmnsc.46.2.186.11926&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VenkateshasUser-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-VenkateshasUser_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-VenkateshasUser_8-1\" rel=\"external_link\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-VenkateshasUser_8-2\" rel=\"external_link\">8.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Venkatesh, V; Morris, M.; Davis, G.; Davis, F. (2003). \"User acceptance of information technology: Toward a unified view\". <i>MIS Quarterly<\/i> <b>27<\/b> (3): 425\u2013478. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2307%2F30036540\" target=\"_blank\">10.2307\/30036540<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=User+acceptance+of+information+technology%3A+Toward+a+unified+view&rft.jtitle=MIS+Quarterly&rft.aulast=Venkatesh%2C+V%3B+Morris%2C+M.%3B+Davis%2C+G.%3B+Davis%2C+F.&rft.au=Venkatesh%2C+V%3B+Morris%2C+M.%3B+Davis%2C+G.%3B+Davis%2C+F.&rft.date=2003&rft.volume=27&rft.issue=3&rft.pages=425%E2%80%93478&rft_id=info:doi\/10.2307%2F30036540&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AldrichEnv-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AldrichEnv_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aldrich, H.E.; Pfeffer, J. (1976). \"Environments of organizations\". <i>Annual Review of Sociology<\/i> <b>2<\/b> (1): 79\u2013105. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1146%2Fannurev.so.02.080176.000455\" target=\"_blank\">10.1146\/annurev.so.02.080176.000455<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Environments+of+organizations&rft.jtitle=Annual+Review+of+Sociology&rft.aulast=Aldrich%2C+H.E.%3B+Pfeffer%2C+J.&rft.au=Aldrich%2C+H.E.%3B+Pfeffer%2C+J.&rft.date=1976&rft.volume=2&rft.issue=1&rft.pages=79%E2%80%93105&rft_id=info:doi\/10.1146%2Fannurev.so.02.080176.000455&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PfefferTheEx1-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PfefferTheEx1_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Pfeffer, J.; Salancik, G. (1978). <i>The External Control of Organizations: A Resource Dependence Perspective<\/i>. New York, NY: Harper & Row.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+External+Control+of+Organizations%3A+A+Resource+Dependence+Perspective&rft.aulast=Pfeffer%2C+J.%3B+Salancik%2C+G.&rft.au=Pfeffer%2C+J.%3B+Salancik%2C+G.&rft.date=1978&rft.place=New+York%2C+NY&rft.pub=Harper+%26+Row&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScholzApp-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ScholzApp_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-ScholzApp_11-1\" rel=\"external_link\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-ScholzApp_11-2\" rel=\"external_link\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Scholz, Carolyn Elaine (01 May 2001). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/mspace.lib.umanitoba.ca\/xmlui\/handle\/1993\/2732\" target=\"_blank\">\"Applying Rogers' theory of diffusion of innovations to examine older females' perceptions of size labels for apparel\"<\/a>. University of Manitoba<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mspace.lib.umanitoba.ca\/xmlui\/handle\/1993\/2732\" target=\"_blank\">http:\/\/mspace.lib.umanitoba.ca\/xmlui\/handle\/1993\/2732<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Applying+Rogers%27+theory+of+diffusion+of+innovations+to+examine+older+females%27+perceptions+of+size+labels+for+apparel&rft.atitle=&rft.aulast=Scholz%2C+Carolyn+Elaine&rft.au=Scholz%2C+Carolyn+Elaine&rft.date=01+May+2001&rft.pub=University+of+Manitoba&rft_id=http%3A%2F%2Fmspace.lib.umanitoba.ca%2Fxmlui%2Fhandle%2F1993%2F2732&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RogersDiff4-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RogersDiff4_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff4_12-1\" rel=\"external_link\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff4_12-2\" rel=\"external_link\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff4_12-3\" rel=\"external_link\">12.3<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff4_12-4\" rel=\"external_link\">12.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Rogers, E.M. (1995). <i>Diffusion of Innovations<\/i> (4th ed.). New York, NY: Free Press. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780029266717.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Diffusion+of+Innovations&rft.aulast=Rogers%2C+E.M.&rft.au=Rogers%2C+E.M.&rft.date=1995&rft.edition=4th&rft.place=New+York%2C+NY&rft.pub=Free+Press&rft.isbn=9780029266717&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RogersDiff5-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RogersDiff5_13-0\" rel=\"external_link\">13.00<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-1\" rel=\"external_link\">13.01<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-2\" rel=\"external_link\">13.02<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-3\" rel=\"external_link\">13.03<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-4\" rel=\"external_link\">13.04<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-5\" rel=\"external_link\">13.05<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-6\" rel=\"external_link\">13.06<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-7\" rel=\"external_link\">13.07<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-8\" rel=\"external_link\">13.08<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-9\" rel=\"external_link\">13.09<\/a><\/sup> <sup><a href=\"#cite_ref-RogersDiff5_13-10\" rel=\"external_link\">13.10<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Rogers, E.M. (2003). <i>Diffusion of Innovations<\/i> (5th ed.). New York, NY: Free Press. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780743222099.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Diffusion+of+Innovations&rft.aulast=Rogers%2C+E.M.&rft.au=Rogers%2C+E.M.&rft.date=2003&rft.edition=5th&rft.place=New+York%2C+NY&rft.pub=Free+Press&rft.isbn=9780743222099&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PfefferTheEx2-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PfefferTheEx2_14-0\" rel=\"external_link\">14.00<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-1\" rel=\"external_link\">14.01<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-2\" rel=\"external_link\">14.02<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-3\" rel=\"external_link\">14.03<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-4\" rel=\"external_link\">14.04<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-5\" rel=\"external_link\">14.05<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-6\" rel=\"external_link\">14.06<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-7\" rel=\"external_link\">14.07<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-8\" rel=\"external_link\">14.08<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-9\" rel=\"external_link\">14.09<\/a><\/sup> <sup><a href=\"#cite_ref-PfefferTheEx2_14-10\" rel=\"external_link\">14.10<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Pfeffer, J.; Salancik, G. (2003). <i>The External Control of Organizations: A Resource Dependence Perspective<\/i>. Stanford, CA: Stanford Business Books. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780804747899.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+External+Control+of+Organizations%3A+A+Resource+Dependence+Perspective&rft.aulast=Pfeffer%2C+J.%3B+Salancik%2C+G.&rft.au=Pfeffer%2C+J.%3B+Salancik%2C+G.&rft.date=2003&rft.place=Stanford%2C+CA&rft.pub=Stanford+Business+Books&rft.isbn=9780804747899&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CzajaFact-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CzajaFact_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-CzajaFact_15-1\" rel=\"external_link\">15.1<\/a><\/sup> <sup><a href=\"#cite_ref-CzajaFact_15-2\" rel=\"external_link\">15.2<\/a><\/sup> <sup><a href=\"#cite_ref-CzajaFact_15-3\" rel=\"external_link\">15.3<\/a><\/sup> <sup><a href=\"#cite_ref-CzajaFact_15-4\" rel=\"external_link\">15.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Czaja, S.J.; Charness, N.; Fisk, A.D.; Hertzog, C.; Nair, S.N.; Rogers, W.A.; Sharit, J. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524856\" target=\"_blank\">\"Factors predicting the use of technology: findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE)\"<\/a>. <i>Psychology and Aging<\/i> <b>21<\/b> (2): 333\u2013352. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1037%2F0882-7974.21.2.333\" target=\"_blank\">10.1037\/0882-7974.21.2.333<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1524856\/\" target=\"_blank\">PMC1524856<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16768579\" target=\"_blank\">16768579<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524856\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524856<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Factors+predicting+the+use+of+technology%3A+findings+from+the+Center+for+Research+and+Education+on+Aging+and+Technology+Enhancement+%28CREATE%29&rft.jtitle=Psychology+and+Aging&rft.aulast=Czaja%2C+S.J.%3B+Charness%2C+N.%3B+Fisk%2C+A.D.%3B+Hertzog%2C+C.%3B+Nair%2C+S.N.%3B+Rogers%2C+W.A.%3B+Sharit%2C+J.&rft.au=Czaja%2C+S.J.%3B+Charness%2C+N.%3B+Fisk%2C+A.D.%3B+Hertzog%2C+C.%3B+Nair%2C+S.N.%3B+Rogers%2C+W.A.%3B+Sharit%2C+J.&rft.date=2006&rft.volume=21&rft.issue=2&rft.pages=333%E2%80%93352&rft_id=info:doi\/10.1037%2F0882-7974.21.2.333&rft_id=info:pmc\/PMC1524856&rft_id=info:pmid\/16768579&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1524856&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FumasoliPatt-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FumasoliPatt_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fumasoli, T.; Lepori, B. (2011). \"Patterns of strategies in Swiss higher education institutions\". <i>Higher Education<\/i> <b>61<\/b> (2): 157\u2013178. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10734-010-9330-x\" target=\"_blank\">10.1007\/s10734-010-9330-x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Patterns+of+strategies+in+Swiss+higher+education+institutions&rft.jtitle=Higher+Education&rft.aulast=Fumasoli%2C+T.%3B+Lepori%2C+B.&rft.au=Fumasoli%2C+T.%3B+Lepori%2C+B.&rft.date=2011&rft.volume=61&rft.issue=2&rft.pages=157%E2%80%93178&rft_id=info:doi\/10.1007%2Fs10734-010-9330-x&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HendersonHealth-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HendersonHealth_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-HendersonHealth_17-1\" rel=\"external_link\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-HendersonHealth_17-2\" rel=\"external_link\">17.2<\/a><\/sup> <sup><a href=\"#cite_ref-HendersonHealth_17-3\" rel=\"external_link\">17.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Henderson, J.. <i>Health Economics and Policy (with Economic Applications)<\/i> (2nd ed.). Mason, OH: South-Western College Pub. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780538481175.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Health+Economics+and+Policy+%28with+Economic+Applications%29&rft.aulast=Henderson%2C+J.&rft.au=Henderson%2C+J.&rft.edition=2nd&rft.place=Mason%2C+OH&rft.pub=South-Western+College+Pub&rft.isbn=9780538481175&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SikkaTheEff-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SikkaTheEff_18-0\" rel=\"external_link\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-SikkaTheEff_18-1\" rel=\"external_link\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sikka, V.; Luke, R.D.; Ozcan, Y.A. (2009). \"The efficiency of hospital-based clusters: evaluating system performance using data envelopment analysis\". <i>Health Care Management Review<\/i> <b>34<\/b> (3): 251\u2013261. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FHMR.0b013e3181a16ba7\" target=\"_blank\">10.1097\/HMR.0b013e3181a16ba7<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19625830\" target=\"_blank\">19625830<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+efficiency+of+hospital-based+clusters%3A+evaluating+system+performance+using+data+envelopment+analysis&rft.jtitle=Health+Care+Management+Review&rft.aulast=Sikka%2C+V.%3B+Luke%2C+R.D.%3B+Ozcan%2C+Y.A.&rft.au=Sikka%2C+V.%3B+Luke%2C+R.D.%3B+Ozcan%2C+Y.A.&rft.date=2009&rft.volume=34&rft.issue=3&rft.pages=251%E2%80%93261&rft_id=info:doi\/10.1097%2FHMR.0b013e3181a16ba7&rft_id=info:pmid\/19625830&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BazzoliATax-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BazzoliATax_19-0\" rel=\"external_link\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-BazzoliATax_19-1\" rel=\"external_link\">19.1<\/a><\/sup> <sup><a href=\"#cite_ref-BazzoliATax_19-2\" rel=\"external_link\">19.2<\/a><\/sup> <sup><a href=\"#cite_ref-BazzoliATax_19-3\" rel=\"external_link\">19.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bazzoli, G.J.; Shortell, S.M.; Dubbs, N.; Chan, C.; Kralovec, P. (1999). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1070343\" target=\"_blank\">\"A taxonomy of health networks and systems: Bringing order out of chaos\"<\/a>. <i>Health Services Research Journal<\/i> <b>33<\/b> (6): 1683\u20131717. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1070343\/\" target=\"_blank\">PMC1070343<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/10029504\" target=\"_blank\">10029504<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1070343\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1070343<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+taxonomy+of+health+networks+and+systems%3A+Bringing+order+out+of+chaos&rft.jtitle=Health+Services+Research+Journal&rft.aulast=Bazzoli%2C+G.J.%3B+Shortell%2C+S.M.%3B+Dubbs%2C+N.%3B+Chan%2C+C.%3B+Kralovec%2C+P.&rft.au=Bazzoli%2C+G.J.%3B+Shortell%2C+S.M.%3B+Dubbs%2C+N.%3B+Chan%2C+C.%3B+Kralovec%2C+P.&rft.date=1999&rft.volume=33&rft.issue=6&rft.pages=1683%E2%80%931717&rft_id=info:pmc\/PMC1070343&rft_id=info:pmid\/10029504&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1070343&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AshFact-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AshFact_20-0\" rel=\"external_link\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-AshFact_20-1\" rel=\"external_link\">20.1<\/a><\/sup> <sup><a href=\"#cite_ref-AshFact_20-2\" rel=\"external_link\">20.2<\/a><\/sup> <sup><a href=\"#cite_ref-AshFact_20-3\" rel=\"external_link\">20.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ash, J.S.; Bates, D.W. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC543830\" target=\"_blank\">\"Factors and forces affecting EHR system adoption: Report of a 2004 ACMI discussion\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>12<\/b> (1): 8\u201312. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M1684\" target=\"_blank\">10.1197\/jamia.M1684<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC543830\/\" target=\"_blank\">PMC543830<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15492027\" target=\"_blank\">15492027<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC543830\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC543830<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Factors+and+forces+affecting+EHR+system+adoption%3A+Report+of+a+2004+ACMI+discussion&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Ash%2C+J.S.%3B+Bates%2C+D.W.&rft.au=Ash%2C+J.S.%3B+Bates%2C+D.W.&rft.date=2005&rft.volume=12&rft.issue=1&rft.pages=8%E2%80%9312&rft_id=info:doi\/10.1197%2Fjamia.M1684&rft_id=info:pmc\/PMC543830&rft_id=info:pmid\/15492027&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC543830&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangFact-21\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WangFact_21-0\" rel=\"external_link\">21.0<\/a><\/sup> <sup><a href=\"#cite_ref-WangFact_21-1\" rel=\"external_link\">21.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, B.B.; Wan, T.T.; Burke, D.E.; Bazzoli, G.J.; Lin, B.Y. (2005). \"Factors influencing health information system adoption in American hospitals\". <i>Health Care Management Review<\/i> <b>30<\/b> (1): 44-51. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15773253\" target=\"_blank\">15773253<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Factors+influencing+health+information+system+adoption+in+American+hospitals&rft.jtitle=Health+Care+Management+Review&rft.aulast=Wang%2C+B.B.%3B+Wan%2C+T.T.%3B+Burke%2C+D.E.%3B+Bazzoli%2C+G.J.%3B+Lin%2C+B.Y.&rft.au=Wang%2C+B.B.%3B+Wan%2C+T.T.%3B+Burke%2C+D.E.%3B+Bazzoli%2C+G.J.%3B+Lin%2C+B.Y.&rft.date=2005&rft.volume=30&rft.issue=1&rft.pages=44-51&rft_id=info:pmid\/15773253&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CastilloAKnow-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CastilloAKnow_22-0\" rel=\"external_link\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-CastilloAKnow_22-1\" rel=\"external_link\">22.1<\/a><\/sup> <sup><a href=\"#cite_ref-CastilloAKnow_22-2\" rel=\"external_link\">22.2<\/a><\/sup> <sup><a href=\"#cite_ref-CastilloAKnow_22-3\" rel=\"external_link\">22.3<\/a><\/sup> <sup><a href=\"#cite_ref-CastilloAKnow_22-4\" rel=\"external_link\">22.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Castillo, V.H.; Mart\u00ednez-Garc\u00eda, A.I.; Pulido, J.R. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2970582\" target=\"_blank\">\"A knowledge-based taxonomy of critical factors for adopting electronic health record systems by physicians: a systematic literature review\"<\/a>. <i>BMC Medical Informatics & Decision Making<\/i> <b>10<\/b>: 60. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-10-60\" target=\"_blank\">10.1186\/1472-6947-10-60<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2970582\/\" target=\"_blank\">PMC2970582<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20950458\" target=\"_blank\">20950458<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2970582\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2970582<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+knowledge-based+taxonomy+of+critical+factors+for+adopting+electronic+health+record+systems+by+physicians%3A+a+systematic+literature+review&rft.jtitle=BMC+Medical+Informatics+%26+Decision+Making&rft.aulast=Castillo%2C+V.H.%3B+Mart%C3%ADnez-Garc%C3%ADa%2C+A.I.%3B+Pulido%2C+J.R.&rft.au=Castillo%2C+V.H.%3B+Mart%C3%ADnez-Garc%C3%ADa%2C+A.I.%3B+Pulido%2C+J.R.&rft.date=2010&rft.volume=10&rft.pages=60&rft_id=info:doi\/10.1186%2F1472-6947-10-60&rft_id=info:pmc\/PMC2970582&rft_id=info:pmid\/20950458&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2970582&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlavinAlt-23\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BlavinAlt_23-0\" rel=\"external_link\">23.0<\/a><\/sup> <sup><a href=\"#cite_ref-BlavinAlt_23-1\" rel=\"external_link\">23.1<\/a><\/sup> <sup><a href=\"#cite_ref-BlavinAlt_23-2\" rel=\"external_link\">23.2<\/a><\/sup> <sup><a href=\"#cite_ref-BlavinAlt_23-3\" rel=\"external_link\">23.3<\/a><\/sup> <sup><a href=\"#cite_ref-BlavinAlt_23-4\" rel=\"external_link\">23.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blavin, F.E.; Buntin, M.J.; Friedman, C.P. (2010). \"Alternative measures of electronic health record adoption among hospitals\". <i>American Journal of Managed Care<\/i> <b>16<\/b> (12 Suppl HIT): e293-e301. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21322296\" target=\"_blank\">21322296<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Alternative+measures+of+electronic+health+record+adoption+among+hospitals&rft.jtitle=American+Journal+of+Managed+Care&rft.aulast=Blavin%2C+F.E.%3B+Buntin%2C+M.J.%3B+Friedman%2C+C.P.&rft.au=Blavin%2C+F.E.%3B+Buntin%2C+M.J.%3B+Friedman%2C+C.P.&rft.date=2010&rft.volume=16&rft.issue=12+Suppl+HIT&rft.pages=e293-e301&rft_id=info:pmid\/21322296&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GinnHos-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GinnHos_24-0\" rel=\"external_link\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-GinnHos_24-1\" rel=\"external_link\">24.1<\/a><\/sup> <sup><a href=\"#cite_ref-GinnHos_24-2\" rel=\"external_link\">24.2<\/a><\/sup> <sup><a href=\"#cite_ref-GinnHos_24-3\" rel=\"external_link\">24.3<\/a><\/sup> <sup><a href=\"#cite_ref-GinnHos_24-4\" rel=\"external_link\">24.4<\/a><\/sup> <sup><a href=\"#cite_ref-GinnHos_24-5\" rel=\"external_link\">24.5<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ginn, G.O.; Shen, J.J.; Moseley, C.B. (2011). \"Hospital financial position and the adoption of electronic health records\". <i>Journal of Healthcare Management<\/i> <b>56<\/b> (5): 337\u201350; discussion 351\u20132. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21991681\" target=\"_blank\">21991681<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hospital+financial+position+and+the+adoption+of+electronic+health+records&rft.jtitle=Journal+of+Healthcare+Management&rft.aulast=Ginn%2C+G.O.%3B+Shen%2C+J.J.%3B+Moseley%2C+C.B.&rft.au=Ginn%2C+G.O.%3B+Shen%2C+J.J.%3B+Moseley%2C+C.B.&rft.date=2011&rft.volume=56&rft.issue=5&rft.pages=337%E2%80%9350%3B+discussion+351%E2%80%932&rft_id=info:pmid\/21991681&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FarleyCase-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FarleyCase_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-FarleyCase_25-1\" rel=\"external_link\">25.1<\/a><\/sup> <sup><a href=\"#cite_ref-FarleyCase_25-2\" rel=\"external_link\">25.2<\/a><\/sup> <sup><a href=\"#cite_ref-FarleyCase_25-3\" rel=\"external_link\">25.3<\/a><\/sup> <sup><a href=\"#cite_ref-FarleyCase_25-4\" rel=\"external_link\">25.4<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Farley, D.E.; Hogan, C. (1990). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1065663\" target=\"_blank\">\"Case-mix specialization in the market for hospital services\"<\/a>. <i>Health Services Research Journal<\/i> <b>25<\/b> (5): 757\u2013783. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1065663\/\" target=\"_blank\">PMC1065663<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/2123838\" target=\"_blank\">2123838<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1065663\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1065663<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Case-mix+specialization+in+the+market+for+hospital+services&rft.jtitle=Health+Services+Research+Journal&rft.aulast=Farley%2C+D.E.%3B+Hogan%2C+C.&rft.au=Farley%2C+D.E.%3B+Hogan%2C+C.&rft.date=1990&rft.volume=25&rft.issue=5&rft.pages=757%E2%80%93783&rft_id=info:pmc\/PMC1065663&rft_id=info:pmid\/2123838&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1065663&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NHEFact-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NHEFact_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.cms.gov\/Research-Statistics-Data-and-Systems\/Statistics-Trends-and-Reports\/NationalHealthExpendData\/NHE-Fact-Sheet.html\" target=\"_blank\">\"NHE Fact Sheet\"<\/a>. <i>CMS.gov<\/i>. Centers for Medicare and Medicaid Services. 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.cms.gov\/Research-Statistics-Data-and-Systems\/Statistics-Trends-and-Reports\/NationalHealthExpendData\/NHE-Fact-Sheet.html\" target=\"_blank\">https:\/\/www.cms.gov\/Research-Statistics-Data-and-Systems\/Statistics-Trends-and-Reports\/NationalHealthExpendData\/NHE-Fact-Sheet.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 18 March 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=NHE+Fact+Sheet&rft.atitle=CMS.gov&rft.date=2014&rft.pub=Centers+for+Medicare+and+Medicaid+Services&rft_id=https%3A%2F%2Fwww.cms.gov%2FResearch-Statistics-Data-and-Systems%2FStatistics-Trends-and-Reports%2FNationalHealthExpendData%2FNHE-Fact-Sheet.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BazzoliTheFin-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BazzoliTheFin_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bazzoli, G.J.; Chan, B.; Shortell, S.M.; D'Aunno, T. (2000). \"The financial performance of hospitals belonging to health networks and systems\". <i>Inquiry<\/i> <b>37<\/b> (3): 234\u2013252. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11111282\" target=\"_blank\">11111282<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+financial+performance+of+hospitals+belonging+to+health+networks+and+systems&rft.jtitle=Inquiry&rft.aulast=Bazzoli%2C+G.J.%3B+Chan%2C+B.%3B+Shortell%2C+S.M.%3B+D%27Aunno%2C+T.&rft.au=Bazzoli%2C+G.J.%3B+Chan%2C+B.%3B+Shortell%2C+S.M.%3B+D%27Aunno%2C+T.&rft.date=2000&rft.volume=37&rft.issue=3&rft.pages=234%E2%80%93252&rft_id=info:pmid\/11111282&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DesRochesElec-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DesRochesElec_28-0\" rel=\"external_link\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-DesRochesElec_28-1\" rel=\"external_link\">28.1<\/a><\/sup> <sup><a href=\"#cite_ref-DesRochesElec_28-2\" rel=\"external_link\">28.2<\/a><\/sup> <sup><a href=\"#cite_ref-DesRochesElec_28-3\" rel=\"external_link\">28.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DesRoches, C.M.; Campbell, E.G.; Vogeli, C.; Zheng, J.; Rao, S.R.; Shields, A.E. Donelan, K.; Rosenbaum, S.; Bristol, S.J.; Jha, A.K. (2010). \"Electronic health records' limited successes suggest more targeted uses\". <i>Health Affairs<\/i> <b>29<\/b> (4): 639\u2013646. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.2009.1086\" target=\"_blank\">10.1377\/hlthaff.2009.1086<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20368593\" target=\"_blank\">20368593<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+health+records%27+limited+successes+suggest+more+targeted+uses&rft.jtitle=Health+Affairs&rft.aulast=DesRoches%2C+C.M.%3B+Campbell%2C+E.G.%3B+Vogeli%2C+C.%3B+Zheng%2C+J.%3B+Rao%2C+S.R.%3B+Shields%2C+A.E.+Donelan%2C+K.%3B+Rosenbaum%2C+S.%3B+Bristol%2C+S.J.%3B+Jha%2C+A.K.&rft.au=DesRoches%2C+C.M.%3B+Campbell%2C+E.G.%3B+Vogeli%2C+C.%3B+Zheng%2C+J.%3B+Rao%2C+S.R.%3B+Shields%2C+A.E.+Donelan%2C+K.%3B+Rosenbaum%2C+S.%3B+Bristol%2C+S.J.%3B+Jha%2C+A.K.&rft.date=2010&rft.volume=29&rft.issue=4&rft.pages=639%E2%80%93646&rft_id=info:doi\/10.1377%2Fhlthaff.2009.1086&rft_id=info:pmid\/20368593&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HarshbergerOut-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-HarshbergerOut_29-0\" rel=\"external_link\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-HarshbergerOut_29-1\" rel=\"external_link\">29.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Harshberger, C.A.; Harper, A.J.; Carro, G.W.; Spath, W.E.; Hui, W.C.; Lawton, J.M.; Brockstein, B.E. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3140445\" target=\"_blank\">\"Outcomes of computerized physician order entry in an electronic health record after implementation in an outpatient oncology setting\"<\/a>. <i>Journal of Oncology Practice<\/i> <b>7<\/b> (4): 233\u2013237. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1200%2FJOP.2011.000261\" target=\"_blank\">10.1200\/JOP.2011.000261<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3140445\/\" target=\"_blank\">PMC3140445<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22043187\" target=\"_blank\">22043187<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3140445\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3140445<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Outcomes+of+computerized+physician+order+entry+in+an+electronic+health+record+after+implementation+in+an+outpatient+oncology+setting&rft.jtitle=Journal+of+Oncology+Practice&rft.aulast=Harshberger%2C+C.A.%3B+Harper%2C+A.J.%3B+Carro%2C+G.W.%3B+Spath%2C+W.E.%3B+Hui%2C+W.C.%3B+Lawton%2C+J.M.%3B+Brockstein%2C+B.E.&rft.au=Harshberger%2C+C.A.%3B+Harper%2C+A.J.%3B+Carro%2C+G.W.%3B+Spath%2C+W.E.%3B+Hui%2C+W.C.%3B+Lawton%2C+J.M.%3B+Brockstein%2C+B.E.&rft.date=2011&rft.volume=7&rft.issue=4&rft.pages=233%E2%80%93237&rft_id=info:doi\/10.1200%2FJOP.2011.000261&rft_id=info:pmc\/PMC3140445&rft_id=info:pmid\/22043187&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3140445&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeckerPhys-30\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DeckerPhys_30-0\" rel=\"external_link\">30.0<\/a><\/sup> <sup><a href=\"#cite_ref-DeckerPhys_30-1\" rel=\"external_link\">30.1<\/a><\/sup> <sup><a href=\"#cite_ref-DeckerPhys_30-2\" rel=\"external_link\">30.2<\/a><\/sup> <sup><a href=\"#cite_ref-DeckerPhys_30-3\" rel=\"external_link\">30.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Decker, S.L.; Jamoom, E.W.; Sisk, J.E. (2012). \"Physicians in nonprimary care and small practices and those age 55 and older lag in adopting electronic health record systems\". <i>Health Affairs<\/i> <b>31<\/b> (5): 1108\u20131114. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.2011.1121\" target=\"_blank\">10.1377\/hlthaff.2011.1121<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22535502\" target=\"_blank\">22535502<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Physicians+in+nonprimary+care+and+small+practices+and+those+age+55+and+older+lag+in+adopting+electronic+health+record+systems&rft.jtitle=Health+Affairs&rft.aulast=Decker%2C+S.L.%3B+Jamoom%2C+E.W.%3B+Sisk%2C+J.E.&rft.au=Decker%2C+S.L.%3B+Jamoom%2C+E.W.%3B+Sisk%2C+J.E.&rft.date=2012&rft.volume=31&rft.issue=5&rft.pages=1108%E2%80%931114&rft_id=info:doi\/10.1377%2Fhlthaff.2011.1121&rft_id=info:pmid\/22535502&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WolfHos-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WolfHos_31-0\" rel=\"external_link\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-WolfHos_31-1\" rel=\"external_link\">31.1<\/a><\/sup> <sup><a href=\"#cite_ref-WolfHos_31-2\" rel=\"external_link\">31.2<\/a><\/sup> <sup><a href=\"#cite_ref-WolfHos_31-3\" rel=\"external_link\">31.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wolf, L.; Harvell, J.; Jha, A.K. (2012). \"Hospitals ineligible for federal meaningful-use incentives have dismally low rates of adoption of electronic health records\". <i>Health Affairs<\/i> <b>31<\/b> (3): 505\u2013513. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.2011.0351\" target=\"_blank\">10.1377\/hlthaff.2011.0351<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22392661\" target=\"_blank\">22392661<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hospitals+ineligible+for+federal+meaningful-use+incentives+have+dismally+low+rates+of+adoption+of+electronic+health+records&rft.jtitle=Health+Affairs&rft.aulast=Wolf%2C+L.%3B+Harvell%2C+J.%3B+Jha%2C+A.K.&rft.au=Wolf%2C+L.%3B+Harvell%2C+J.%3B+Jha%2C+A.K.&rft.date=2012&rft.volume=31&rft.issue=3&rft.pages=505%E2%80%93513&rft_id=info:doi\/10.1377%2Fhlthaff.2011.0351&rft_id=info:pmid\/22392661&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LluchHealth-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LluchHealth_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lluch, M. (2011). \"Healthcare professionals' organisational barriers to health information technologies-a literature review\". <i>International Journal of Medical Informatics<\/i> <b>80<\/b> (12): 849\u2013862. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ijmedinf.2011.09.005\" target=\"_blank\">10.1016\/j.ijmedinf.2011.09.005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22000677\" target=\"_blank\">22000677<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Healthcare+professionals%27+organisational+barriers+to+health+information+technologies-a+literature+review&rft.jtitle=International+Journal+of+Medical+Informatics&rft.aulast=Lluch%2C+M.&rft.au=Lluch%2C+M.&rft.date=2011&rft.volume=80&rft.issue=12&rft.pages=849%E2%80%93862&rft_id=info:doi\/10.1016%2Fj.ijmedinf.2011.09.005&rft_id=info:pmid\/22000677&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CheungFact-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-CheungFact_33-0\" rel=\"external_link\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-CheungFact_33-1\" rel=\"external_link\">33.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cheung, C.S.; Tong, E.L.; Cheung, N.T.; Chan, W.M.; Wang, H.H.; Kwan, M.W.; Fan, C.K.; Liu, K.Q.; Wong, M.C. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4288082\" target=\"_blank\">\"Factors associated with adoption of the electronic health record system among primary care physicians\"<\/a>. <i>JMIR Medical Informatics<\/i> <b>1<\/b> (1): e1. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2196%2Fmedinform.2766\" target=\"_blank\">10.2196\/medinform.2766<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4288082\/\" target=\"_blank\">PMC4288082<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25599989\" target=\"_blank\">25599989<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4288082\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4288082<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Factors+associated+with+adoption+of+the+electronic+health+record+system+among+primary+care+physicians&rft.jtitle=JMIR+Medical+Informatics&rft.aulast=Cheung%2C+C.S.%3B+Tong%2C+E.L.%3B+Cheung%2C+N.T.%3B+Chan%2C+W.M.%3B+Wang%2C+H.H.%3B+Kwan%2C+M.W.%3B+Fan%2C+C.K.%3B+Liu%2C+K.Q.%3B+Wong%2C+M.C.&rft.au=Cheung%2C+C.S.%3B+Tong%2C+E.L.%3B+Cheung%2C+N.T.%3B+Chan%2C+W.M.%3B+Wang%2C+H.H.%3B+Kwan%2C+M.W.%3B+Fan%2C+C.K.%3B+Liu%2C+K.Q.%3B+Wong%2C+M.C.&rft.date=2013&rft.volume=1&rft.issue=1&rft.pages=e1&rft_id=info:doi\/10.2196%2Fmedinform.2766&rft_id=info:pmc\/PMC4288082&rft_id=info:pmid\/25599989&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4288082&rfr_id=info:sid\/en.wikipedia.org:Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>AHA:<\/b> American Hospital Association\n<\/p><p><b>ARRA:<\/b> American Recovery and Reinvestment Act\n<\/p><p><b>CMS:<\/b> Center for Medicare and Medicaid Services\n<\/p><p><b>CPOE:<\/b> computerized provider order entry\n<\/p><p><b>DV:<\/b> dependent variables\n<\/p><p><b>EHR:<\/b> electronic health record\n<\/p><p><b>EMR:<\/b> electronic medical record\n<\/p><p><b>HHS:<\/b> the Department of Health and Human Services\n<\/p><p><b>HIMSS:<\/b> Health Information Management Systems Society\n<\/p><p><b>HIS:<\/b> health information system\n<\/p><p><b>HIT:<\/b> health information technology\n<\/p><p><b>HITECH:<\/b> Health Information Technology for Economic and Clinical Health\n<\/p><p><b>IT:<\/b> information technology\n<\/p><p><b>IV:<\/b> independent variables\n<\/p><p><b>LR:<\/b> long run\n<\/p><p><b>MHS:<\/b> multihospital system\n<\/p><p><b>SR:<\/b> short run\n<\/p><p><b>TAM:<\/b> Technology Acceptance Model\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases the PubMed ID was missing and was added to make the reference more useful.\n<\/p><p>Per the distribution agreement, the following copyright information is also being added: \n<\/p><p>\u00a9Clemens Scott Kruse, Jonathan DeShazo, Forest Kim, Lawrence Fulton. Originally published in JMIR Medical Informatics (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/medinform.jmir.org\" target=\"_blank\">http:\/\/medinform.jmir.org<\/a>), 23.05.2014.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210227\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.848 seconds\nReal time usage: 0.892 seconds\nPreprocessor visited node count: 26769\/1000000\nPreprocessor generated node count: 41354\/1000000\nPost\u2010expand include size: 206352\/2097152 bytes\nTemplate argument size: 66672\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 848.471 1 - -total\n 79.42% 673.872 1 - Template:Reflist\n 68.09% 577.694 33 - Template:Citation\/core\n 52.72% 447.278 24 - Template:Cite_journal\n 10.04% 85.220 1 - Template:Infobox_journal_article\n 9.82% 83.312 5 - Template:Cite_book\n 9.70% 82.262 4 - Template:Cite_web\n 9.62% 81.650 1 - Template:Infobox\n 7.07% 60.017 47 - Template:Citation\/identifier\n 5.42% 45.945 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7822-0!*!0!!en!5!* and timestamp 20181213210226 and revision id 23408\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review\">https:\/\/www.limswiki.org\/index.php\/Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","3eb9d25818bfd5a80515d585fb6a6712_images":["https:\/\/www.limswiki.org\/images\/6\/65\/Fig1_Kruse_JMIRMedInfo2014_2-1.jpg","https:\/\/www.limswiki.org\/images\/3\/3a\/Fig2_Kruse_JMIRMedInfo2014_2-1.jpg"],"3eb9d25818bfd5a80515d585fb6a6712_timestamp":1544734946,"43a50f3587a6a8e1ea355835ec07b4a1_type":"article","43a50f3587a6a8e1ea355835ec07b4a1_title":"Beyond information retrieval and electronic health record use: Competencies in clinical informatics for medical education (Hersh et al. 2014)","43a50f3587a6a8e1ea355835ec07b4a1_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education","43a50f3587a6a8e1ea355835ec07b4a1_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Beyond information retrieval and electronic health record use: Competencies in clinical informatics for medical education\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nBeyond information retrieval and electronic health record use: Competencies in clinical informatics for medical educationJournal\n \nAdvances in Medical Education and PracticeAuthor(s)\n \nHersh, William R.; Gorman, Paul N.; Biagioli, Frances E.; Mohan, Vishnu; Gold, Jeffrey A.; Mejicano, George C.Author affiliation(s)\n \nOregon Health & Science University, Portland, OR, USAYear published\n \n2014Volume and issue\n \n2014 (5)Page(s)\n \n205\u2014212DOI\n \n10.2147\/AMEP.S63903ISSN\n \n1179-7258Distribution license\n \nCreative Commons Attribution-NonCommercial-ShareAlike 3.0 UnportedWebsite\n \nhttps:\/\/www.dovepress.com\/Download\n \nhttps:\/\/www.dovepress.com\/getfile.php?fileID=20654 (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Previous work developing competencies in informatics \n4 Methods \n5 Results \n\n5.1 Developing competencies and learning objectives \n5.2 Implementing competencies and objectives \n\n\n6 Conclusion and future directions \n7 Acknowledgments \n8 Disclosures \n9 References \n10 Notes \n\n\n\nAbstract \nPhysicians in the 21st century will increasingly interact in diverse ways with information systems, requiring competence in many aspects of clinical informatics. In recent years, many medical school curricula have added content in information retrieval (search) and basic use of the electronic health record. However, this omits the growing number of other ways that physicians are interacting with information that includes activities such as clinical decision support, quality measurement and improvement, personal health records, telemedicine, and personalized medicine. We describe a process whereby six faculty members representing different perspectives came together to define competencies in clinical informatics for a curriculum transformation process occurring at Oregon Health & Science University. From the broad competencies, we also developed specific learning objectives and milestones, an implementation schedule, and mapping to general competency domains. We present our work to encourage debate and refinement as well as facilitate evaluation in this area.\nKeywords: curriculum transformation, clinical decision support, patient safety, health care quality, patient engagement\n\nIntroduction \nPhysicians and medical students have been using health information technology (HIT) for decades. During this time, the role of HIT has changed dramatically from a useful tool for data access and occasional information retrieval to a ubiquitous presence that permeates health care and medical practice in myriad ways. As the use of HIT has accelerated and the underlying science of biomedical and health informatics has advanced, medical education has lagged behind, leaving physicians and students alike to learn largely on their own how to make use of these tools. While some medical schools have introduced limited aspects of biomedical informatics into their curricula, most of this is focused on training to perform basic tasks such as accessing knowledge resources or basic use of an electronic health record (EHR).\nTwenty-first century clinicians face a much more digital world than their predecessors. The quantity of biomedical knowledge continues to expand, with an attendant increase in the primary scientific literature.[1] Secondary sources that summarize this information proliferate as well, not only for use by clinicians but also by patients and consumers. Programs such as the \u201cmeaningful use\u201d financial incentives of the Health Information for Technology and Clinical Health Act require comprehensive adoption and use of the EHR.[2][3] Patients, especially the aging \u201cbaby boomer\u201d generation, want to interact with the health care system the same way they interact with airlines, banks, and retailers, ie, through digital means using technologies such as the personal health record.[4] Patients, payers, and purchasers demand more accountability in health care quality, safety, and cost,[5] leading to an expectation of measurement and reporting of quality of care as a routine part of participation in new delivery mechanisms such as primary care medical homes and accountable care organizations. At the same time, telemedicine applications extend the reach of health care systems and clinicians in both rural and urban settings. Ideally, for all of these applications, patient data must move readily across organizational boundaries via health information exchange, while privacy and confidentiality are protected.[6] The growing quantity of clinical and administrative data in these systems also affords an opportunity for advanced analysis that can enable better deployment of resources and coordination of care, facilitation of personalized and precision medicine, and advancement of clinical and translational research.[7] Together, these advances are moving health care toward the global vision of the learning health system put forth by the Institute of Medicine (IOM).[8][9]\nThe importance of these developments is further evidenced by the recent establishment of a new medical subspecialty of clinical informatics.[10] Practicing physicians are now beginning to become board-certified in this new subspecialty, with fellowship programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) to soon be established. This underscores the need for medical students to be introduced to the concepts and competencies of this new medical subspecialty as part of the undergraduate curriculum.\nIn this paper, we describe our work to expand the undergraduate medical curriculum to include a comprehensive set of competencies in clinical informatics. We go beyond the usual focus on information retrieval from knowledge sources and data access through the EHR, aiming to address the expanded and diverse roles of HIT in contemporary health care delivery, personal health, public health, and clinical and translational research. We describe the development of a novel biomedical informatics curriculum that is being implemented within a comprehensive transformation of the undergraduate curriculum at our institution. Our future plans include evaluation of the impact of this work, and we encourage others to participate in this evaluation and refinement process.\n\nPrevious work developing competencies in informatics \nThe need to include biomedical informatics in medical education has been recognized in previous publications of competencies, curricula, and other learning materials for informatics. One early, high-profile resource was part of the Medical Student Objectives Project of the Association of American Medical Colleges, which framed proposed objectives in terms of five key roles of physicians: clinicians, life-long learners, educators, researchers, and managers.[11] Although current medical students believe these objectives are important to their education[12], uptake of these objectives has been modest.[13] More recently, the Alliance for Clinical Education issued a policy statement calling for competencies related to the EHR[14], although also noting that many challenges to implementing such competencies exist.[15] Others have also noted the challenges to education about the EHR, citing problems analogous to those of clinicians incorporating EHRs into practice, such as integration into workflows, and concerns about unanticipated and undesirable impacts.[16] It is clear, however, that competencies in the use of the EHR are aligned with more general physician educational competencies[17][18], and that informatics curricula can be incorporated into undergraduate medical training[19][20] and evaluated.[21]\n\nMethods \nIn 2012, the Oregon Health & Science University, School of Medicine began a comprehensive transformation of the undergraduate medical curriculum for the 21st century. From the outset, the Dean and school leadership recognized the importance of informatics to physicians and health care in the future. With participation and support from the Senior Associate Dean for Education (GM), a team of five physician-faculty was convened to lead this effort, including the following:\n\n An informatician involved nationally in informatics education leadership and development of the new clinical informatics subspecialty (WH)\n An internist-informatician who is chair of the medical school curriculum committee and actively engaged in the new curriculum development team (PG)\n A family physician and innovator in using simulation for teaching and assessing EHR use by medical students (FB)\n A critical care physician investigator using EHR simulation to assess residents\u2019 ability to detect important patterns in critical care data (JG)\n An informatician and former internal medicine residency associate program director also involved in the previous two above activities (VM)\nUsing an exploratory qualitative methodology based on action research and working with the curriculum transformation leadership, we developed a broad set of competencies in informatics for medical education. With each competency, we developed a set of learning objectives, a designation of when it would be most appropriate to introduce the concept during the medical school curriculum, and how each mapped to the broader ACGME competencies.\n\nResults \nDeveloping competencies and learning objectives \nBefore beginning to identify the competencies for student learning, we needed to develop the rationale for including these topics in the curriculum. This led us to map the competencies to the six ACGME core competency domains[22] and develop explicit learning objectives within each competency, followed by their implementation in the emerging curriculum. Discerning the rationale for our competencies was a narrative process, with their implementation enumerating the specific areas deemed important to incorporate into the medical school curriculum, as it was being concurrently designed.\nOur first step was to provide the rationale to include various topic areas in the curriculum. A first critical concept we agreed upon was that informatics is not the same as computer literacy. Computer literacy is one of many requirements to use informatics successfully, but knowing how to use a computing device (personal computer, tablet, or smartphone) is not the same as having skills in informatics, ie, using that device to improve health, health care delivery, public health, or research.\nWe concurred that one fundamental skill currently being taught in many medical schools for 21st century clinicians was something taught in many medical schools since the late 20th century, namely, how to find information to apply to patient care. The needed informatics skills start, and do not stop, with the knowledge of how to enter simple queries into common general (eg, Google\u2122, http:\/\/www.google.com) or medical (eg, PubMed, http:\/\/www.pubmed.gov) search engines. Clinicians must also know how to formulate a clinical question as an answerable one, and then be able to select the appropriate resource and make optimal use of it. This includes knowing what content is in different search systems. He or she must also know about specialized resources such as the Agency for Healthcare Research and Quality Guidelines Clearinghouse (http:\/\/www.guideline.gov) and the Centers for Disease Control and Prevention travel site (http:\/\/wwwnc.cdc.gov\/travel). Clinicians should also have an understanding of the major commercial publishers as well as what professional societies offer. Once clinicians know how to use a search site, they must be able to phrase an appropriate query and know how to maximize the search using advanced query features. To take maximum advantage of available knowledge resources, it is necessary to understand the advanced search features that are available, whether in general web resources such as Google or dedicated biomedical knowledge resources such as PubMed.\nOnce information is retrieved, clinicians must know how to critically appraise the information retrieved and decide, if warranted, to apply it to the patient or population. As with searching, the type of appraisal varies with the search engine used. With output from general search engines like Google, the trustworthiness of the information must be assessed. Clinicians must also know how to make optimal use of patient data and information. They must know how to use informatics to achieve the triple aim of better health, better care, and lower cost.[5] An appropriate framework for applying this emanates from the IOM concept of the \u201clearning health system\u201d[8][9], which presents a compelling vision for a health care system that is patient-centered, population-based, and promotes learning from data.\nBeing able to use data and information also means understanding that the patient health record is more than \u201ccharting\u201d, and that its value goes beyond being able to read it. Clinicians must be facile with all aspects of the EHR, able to easily move from one vendor system to another and to understand the critical role of health information exchange in making any single record as complete as possible.[6] This issue is magnified by the increasing amount of data the clinicians must view and manage.[23] Further, as the health care focus moves from one of quantity to one of value, clinicians must pay attention to quality, patient safety, and cost. This requires coordination of care, in not just providing medical procedures, nursing interventions, therapies, etc in isolation, but also teamwork and communication.[24]\nFrom this vantage, the health record is no longer a passive collection of information used mainly to justify billing. Rather, it is a source of data, organized into coherent information that allows the health care team to deliver the best, safest, and most cost-effective care. This means that the 21st century clinician must have a basic understanding of informatics issues, such as capturing data that is correct and complete as well as consistent in its expression.[25] He or she must be able to work in partnership with informatics professionals to achieve what we know is so critical in the application of informatics: adhering to standards, achieving system interoperability, appropriately and optimally implementing clinical decision support, and maintaining security to assure privacy and confidentiality.[26]\nClinicians also require understanding of areas of quality measurement and improvement, especially as it applies to their direct practice.[5] The clinician must understand concepts of health care quality and how the EHR and other sources of data are increasingly used to measure it. They must also know how to work with clinical leaders to select, implement, and improve quality measures. This should be introduced as part of the larger learning health system.[8][9] This discussion should also include the related issues of patient safety and medical errors[27], including safety of HIT.[28]\nPatient engagement is another component for safe, effective, and coordinated care.[29] Not only does patient engagement improve health care, but patients, including Internet-savvy baby boomers, will increasingly demand it. Patients want health care that adapts the online features of other modern industries, allowing them the ability to view their own data and interact with their clinicians and health care system (eg, online scheduling of appointments, prescription refills, and even consultations that are appropriate via the Internet or other electronic media).[29] These will likely take the form of a personal health record, accessible from a patient portal that allows access to all information, not just that from the system of the provider organization.[4] Because of this natural evolution of health care, clinicians must also have knowledge and understanding of the appropriate use of telemedicine and telehealth (such as e-visits), both for remote locations and as a convenient option locally in patients\u2019 homes and other settings.\nComplementary to the patient-centric view, clinicians must also understand population-based care[30] and the informatics underlying it.[31] The clinician and their team care for populations of patients; they must be able to view their care needs and results across their patient population. When a new test or treatment is determined to be highly effective, the clinician must be able to quickly identify patients who are candidates for it. They must also be able to identify outliers in their populations who require intervention, such as those with excessively high blood pressure or blood sugar, missed appointments or screen tests, or those at risk for hospital (re)admission.\nAnother area where 21st century clinicians must understand the key issues is in bioinformatics, especially as it relates to personalized medicine.[32] Modern clinicians need not understand complicated gene sequencing algorithms, but they should have an understanding of how genomics and related areas are transforming our understanding of maintaining health and diagnosing and treating disease.[33] Now that the vision of personalized medicine is coming to fruition, clinicians will increasingly rely on the help of decision support and other tools for assistance in applying it to individual patients. He or she should have a basic understanding of genome-wide association studies and their ramifications.[34]\nClinicians must also understand the strengths and limitations of clinical and translational research. They must understand the differences between and the value of experimental and observational studies. Ideally, students will have participated in research while in their training. But even if not, they should understand issues such as data quality, study design, and the limitations that come from the sharp focus perspective of a clinical study. Students should participate in the learning health system laid out in the vision of groups like the IOM.\n\nImplementing competencies and objectives \nBased on the above narrative, our group of diverse clinician-educators developed through an iterative process a set of 13 competencies in clinical informatics (Table 1). Each competency was mapped to one or more of the six ACGME general competency domains. We then developed more detailed learning objectives and milestones within each competency. We also categorized each learning objective for its presence in the early (at the beginning), middle (during the preclinical portion), or late (during clinical experiences) portion of the curriculum.\t\n\n\n\nTable 1 Competencies in clinical informatics and specific learning objective\/milestone within each\r\n\nNotes: All learning objectives\/milestones are designated for time to start (early: start of medical school; mid: later in preclinical years; and late: during clinical experiences); Each is also mapped into one or more of the ACGME core competencies: patient care and procedural skills (PCPS), medical knowledge (MK), practice-based learning and improvement (PBLI), interpersonal and communication skills (ICS), professionalism (Prof), and systems-based practice (SBP).\r\n\n\nAbbreviations: ACGME, Accreditation Council for Graduate Medical Education; CDS, clinical decision support; HIE, health information exchange; HIPAA, Health Insurance Portability and Accountability Act.\nThese learning objectives are now being woven into Oregon Health & Science University\u2019s new curriculum (Table 2). As seen in Table 2, we have designated where the specific learning activities will be placed in the major portions of the curriculum from orientation through preclinical sciences, clinical experiences, and intersession in the fourth year.\n\n\n\nTable 2 Informatics in various portions of curriculum in the preclinical (orientation, fundamentals, and preclinical sciences) and clinical (clinical experiences and intersession) years\r\n\nAbbreviations: CDS, clinical decision support; DDI, drug\u2013drug interaction; EBM, evidence-based medicine; EHR, electronic health record; EKG, electrocardiogram; HIE, health information exchange; HIPAA, Health Insurance Portability and Accountability Act; IT, information technology; OHSU, Oregon Health & Science University; PHR, personal health record; VA, Veterans Affairs. \nConclusion and future directions \nOur analysis has shown there are a substantial number of informatics competencies and a large body of associated knowledge that the 21st century clinician needs to learn and apply. From a pedagogical standpoint, there are also issues in how to organize, deliver, and assess this content. Certainly, one approach is to provide this content as a separate course, isolated from the rest of the curriculum. However, a better approach would be to tightly and comprehensively integrate informatics concepts longitudinally into the learning curriculum since clinical informatics is emerging as a core competency of medical practice, applicable in all basic science disciplines and clinical specialties.\nA next major step for this work will be to develop evaluation activities for the competencies and learning activities. These will vary based on institutional factors (class size, whether students are asynchronous, etc), faculty preference, and funding limitations (desire to use simulation for many things, but cost may be prohibitive). As such, different learning activities will require different evaluation methods. For example, the competency \u201cProvide care using population health management approaches\u201d might be best measured by having students describe or select effective system change options based on immunizations rates of the clinic to which they are assigned, while the competency \u201cProvide clinical care via telemedicine and refer those for whom it is necessary\u201d might best be measured with an objective structured clinical examination simulation.\nThe presence of these competencies also indicates a need for educators who are specialists in informatics to (collaboratively with clinical educators) design the learning and deliver learning experiences that are appropriate for lecture, group discussions, self-paced and self-directed methods, and other settings. But informatics is one of those topics that is best infused throughout the curriculum, especially in clinical settings where it is being used.\nThere are a number of future steps for this work. First, we recognize that this set of competencies represents the views of one medical school\u2019s informatics faculty and must be validated with informatics experts from other institutions. Second, we must evaluate our own implementation of this curriculum to determine how these competencies are delivered to a medical student audience. Finally, we must evaluate this entire process with students from our institution and others to determine which competencies and learning experiences are most valuable for them in their future clinical practice.\n\nAcknowledgments \nDr Biagioli was supported in part by NIH Grant 1R25CA158571. Drs Hersh and Mejicano were supported in part by the Accelerating Change in Medical Education grant of the American Medical Association.\n\nDisclosures \nThe authors report no conflicts of interest in this work.\n\nReferences \n\n\n\u2191 Bastian, H.; Glasziou, P.; Chalmers, I. (2010). \"Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?\". PLoS Medicine 7 (9): e1000326. doi:10.1371\/journal.pmed.1000326. PMC PMC2943439. PMID 20877712. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2943439 .   \n\n\u2191 Blumenthal, D. (2011). \"Wiring the health system \u2013 origins and provisions of a new federal program\". The New England Journal of Medicine 365 (24): 2323\u20132329. doi:10.1056\/NEJMsr1110507. PMID 22168647.   \n\n\u2191 Blumenthal, D. (2011). \"Implementation of the federal health information technology initiative\". The New England Journal of Medicine 365 (25): 2426\u20132431. doi:10.1056\/NEJMsr1112158. PMID 22187990.   \n\n\u2191 4.0 4.1 Miller, H.; Yasnoff, W.; Burde, H. (2009). Personal Health Records: The Essential Missing Element in 21st Century Healthcare. Chicago, IL: Healthcare Information and Management Systems Society. pp. 178. ISBN 9780980069761.   \n\n\u2191 5.0 5.1 5.2 Berwick, D.M.; Nolan, T.W.; Whittington, J. (2008). \"The triple aim: care, health, and cost\". Health Affairs 27 (3): 759\u2013769. doi:10.1377\/hlthaff.27.3.759. PMID 18474969.   \n\n\u2191 6.0 6.1 Kuperman, G.J. (2011). \"Health-information exchange: why are we doing it, and what are we doing?\". Journal of the American Medical Informatics Association 18 (5): 678\u2013682. doi:10.1136\/amiajnl-2010-000021. PMC PMC3168299. PMID 21676940. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168299 .   \n\n\u2191 Safran, C.; Bloomrosen, M.; Hammond, W.E.; et al. (2007). \"Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper\". Journal of the American Medical Informatics Association 14 (1): 1\u20139. doi:10.1197\/jamia.M2273. PMC PMC2329823. PMID 17077452. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823 .   \n\n\u2191 8.0 8.1 8.2 Friedman, C.P.; Wong, A.K.; Blumenthal, D. (2010). \"Achieving a nationwide learning health system\". Science Translation Medicine 2 (57): 57cm29. doi:10.1126\/scitranslmed.3001456.   \n\n\u2191 9.0 9.1 9.2 Smith, M.; Saunders, R.; Stuckhardt, L.; McGinnis, J.M., ed. (2012). Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: National Academies Press. pp. 416. ISBN 9780309260732.   \n\n\u2191 Lehmann, C.U.; Shorte, V.; Gundlapalli, A.V. (2013). \"Clinical informatics sub-specialty board certification\". Pediatrics in Review 34 (11): 525\u2013530. doi:10.1542\/pir.34-11-525. PMID 24187144.   \n\n\u2191 The Informatics Panel and the Population Health Perspective Panel (1999). \"Contemporary issues in medicine \u2013 medical informatics and population health: report II of the Medical School Objectives Project\". Academic Medicine 74 (2): 130\u2013141. http:\/\/journals.lww.com\/academicmedicine\/Abstract\/1999\/02000\/Contemporary_issues_in_medicine_medical.12.aspx .   \n\n\u2191 Beaudoin, D.; Richardson, S.J.; Sheng, X.; Mitchell, J. (2013). \"Medical students\u2019 perspectives on biomedical informatics learning objectives\". International Journal of Medical Education (4): 1\u20138. doi:10.5116\/ijme.50ce.316b. PMC PMC4205536. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4205536 .   \n\n\u2191 McGowan, J.; Passiment, M.; Hoffman, H. (2007). \"Educating medical students as competent users of health information technologies: the MSOP data\". Studies in Health Technology and Informatics 129 (Pt. 2): 1414-8. PMID 17911947.   \n\n\u2191 Hammoud, M.M.; Dalymple, J.L.; Christner, J.G.; et al. (2012). \"Medical student documentation in electronic health records: a collaborative statement from the Alliance for Clinical Education\". Teaching and Learning in Medicine 24 (3): 257\u2013266. doi:10.1080\/10401334.2012.692284. PMID 22775791.   \n\n\u2191 Hammoud, M.M.; Margo, K.; Christner, J.G.;, Fisher, J.; Fischer, S.H.; Pangaro, L.N. (2012). \"Opportunities and challenges in integrating electronic health records into undergraduate medical education: a national survey of clerkship directors\". Teaching and Learning in Medicine 24 (3): 219\u2013224. doi:10.1080\/10401334.2012.692267. PMID 22775785.   \n\n\u2191 Ellaway, R.H.; Graves, L.; Greene, P.S. (2013). \"Medical education in an electronic health record-mediated world\". Medical Teacher 35 (4): 282\u2013286. doi:10.3109\/0142159X.2013.773396. PMID 23464893.   \n\n\u2191 Tierney, M.J.; Pageler, N.M.; Kahana, M.; Pantaleoni, J.L.; Longhurst, C.A. (2013). \"Medical education in the electronic medical record (EMR) era: benefits, challenges, and future directions\". Academic Medicine 88 (6): 748\u2013752. doi:10.1097\/ACM.0b013e3182905ceb. PMID 23619078.   \n\n\u2191 Pageler, N.M.; Friedman, C.P.; Longhurst, C.A. (2013). \"Refocusing medical education in the EMR era\". Journal of the American Medical Association 310 (21): 2249\u20132250. doi:10.1001\/jama.2013.282326. PMID 24302083.   \n\n\u2191 Silverman, H.; Cohen, T.; Fridsma, D. (2012). \"The evolution of a novel biomedical informatics curriculum for medical students\". Academic Medicine 87 (1): 84\u201390. doi:10.1097\/ACM.0b013e31823a599e. PMID 22104049.   \n\n\u2191 Milano, C.E.; Hardman, J.A.; Plesiu, A.; Rdesinski, R.E.; Biagioli, F.E. (2014). \"Simulated electronic health record (Sim-EHR) curriculum: teaching EHR skills and use of the EHR for disease management and prevention\". Academic Medicine 89 (3): 399\u2013403. doi:10.1097\/ACM.0000000000000149. PMC PMC4035239. PMID 24448035. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4035239 .   \n\n\u2191 Reis, S.; Sagi, D.; Eisenberg, O.; et al. (2013). \"The impact of residents\u2019 training in Electronic Medical Record (EMR) use on their competence: report of a pragmatic trial\". Patient Education and Counseling 93 (3): 515\u2013521. doi:10.1016\/j.pec.2013.08.007. PMID 24011647.   \n\n\u2191 Englander, R.; Cameron, T.; Ballard, A.J.; Dodge, J.; Bull, J.; Aschenbrener, C.A. (2013). \"Toward a common taxonomy of competency domains for the health professions and competencies for physicians\". Academic Medicine 88 (8): 1088\u20131094. doi:10.1097\/ACM.0b013e31829a3b2b. PMID 23807109.   \n\n\u2191 Manor-Shulman, O.; Beyene, J.; Frndova, H.; Parshuram, C.S. (2008). \"Quantifying the volume of documented clinical information in critical illness\". Journal of Critical Care 23 (2): 245\u2013250. doi:10.1016\/j.jcrc.2007.06.003. PMID 18538218.   \n\n\u2191 Doherty, R.B.; Crowley, R.A.; Health and Public Policy Committee of the American College of Physicians (2013). \"Principles supporting dynamic clinical care teams: an American College of Physicians position paper\". Annals of Internal Medicine 159 (9): 620\u2013626. doi:10.7326\/0003-4819-159-9-201311050-00710. PMID 24042251.   \n\n\u2191 Hersh, W.R.; Weiner, M.G.; Embi, P.J.; et al. (2013). \"Caveats for the use of operational electronic health record data in comparative effectiveness research\". Medical Care 51 (8 Suppl 3): S30\u2013S37. doi:10.1097\/MLR.0b013e31829b1dbd. PMC PMC3748381. PMID 23774517. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3748381 .   \n\n\u2191 Hersh, W.R. (2009). \"A stimulus to define informatics and health information technology\". BMC Medical Informatics & Decision Making (9): 24. doi:10.1186\/1472-6947-9-24.   \n\n\u2191 Shekelle, P.G.; Pronovost, P.J.; Wachter, R.M.; et al. (2013). \"The top patient safety strategies that can be encouraged for adoption now\". Annals of Internal Medicine 158 (5 Pt 2): 365\u2013368. doi:10.7326\/0003-4819-158-5-201303051-00001. PMID 23460091.   \n\n\u2191 Committee on Patient Safety and Health Information Technology; Board on Health Care Services; Institute of Medicine (2012). Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: National Academies Press. ISBN 9780309221122.   \n\n\u2191 29.0 29.1 Ricciardi, L.; Mostashari, F.; Murphy, J.; Daniel, J.G.; Siminerio, E.P. (2013). \"A national action plan to support consumer engagement via e-health\". Health Affairs 32 (2): 376\u2013384. doi:10.1377\/hlthaff.2012.1216. PMID 23381531.   \n\n\u2191 Institute for Health Technology Transformation (2012) (PDF). Population Health Management \u2013 A Roadmap for Provider-Based Automation in a New Era of Healthcare. New York, NY: Institute for Health Technology Transformation. http:\/\/ihealthtran.com\/pdf\/PHMReport.pdf .   \n\n\u2191 Diamond, C.C.; Mostashari, F.; Shirky, C. (2009). \"A national action plan to support consumer engagement via e-health\". Health Affairs 28 (2): 454\u2013466. doi:10.1377\/hlthaff.28.2.454. PMID 19276005.   \n\n\u2191 Hamburg, M.A.; Collins, F.S. (2010). \"The path to personalized medicine\". New England Journal of Medicine 363 (4): 301\u2013304. doi:10.1056\/NEJMp1006304. PMID 20551152.   \n\n\u2191 Fernald, G.H.; Capriotti, E.; Daneshjou, R.; Karczewski, K.J.; Altman, R.B. (2011). \"Bioinformatics challenges for personalized medicine\". Bioinformatics 27 (13): 1741\u20131748. doi:10.1093\/bioinformatics\/btr295. PMC PMC3117361. PMID 21596790. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3117361 .   \n\n\u2191 Manolio, T. (2010). \"Genomewide association studies and assessment of the risk of disease\". New England Journal of Medicine 363 (2): 166\u2013176. doi:10.1056\/NEJMra0905980. PMID 20647212.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In all cases the DOI, PubMed ID, and\/or ISBN were missing and were added to make the reference more useful.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\">https:\/\/www.limswiki.org\/index.php\/Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:52.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,305 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","43a50f3587a6a8e1ea355835ec07b4a1_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Beyond_information_retrieval_and_electronic_health_record_use_Competencies_in_clinical_informatics_for_medical_education skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Beyond information retrieval and electronic health record use: Competencies in clinical informatics for medical education<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Physicians in the 21st century will increasingly interact in diverse ways with information systems, requiring competence in many aspects of <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_informatics\" title=\"Clinical informatics\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"bda8123083aecb94afe79afec9ae4686\">clinical informatics<\/a>. In recent years, many medical school curricula have added content in <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> retrieval (search) and basic use of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health record<\/a>. However, this omits the growing number of other ways that physicians are interacting with information that includes activities such as <a href=\"https:\/\/www.limswiki.org\/index.php\/Clinical_decision_support_system\" title=\"Clinical decision support system\" target=\"_blank\" class=\"wiki-link\" data-key=\"095141425468d057aa977016869ca37d\">clinical decision support<\/a>, quality measurement and improvement, personal health records, telemedicine, and personalized medicine. We describe a process whereby six faculty members representing different perspectives came together to define competencies in clinical informatics for a curriculum transformation process occurring at Oregon Health & Science University. From the broad competencies, we also developed specific learning objectives and milestones, an implementation schedule, and mapping to general competency domains. We present our work to encourage debate and refinement as well as facilitate evaluation in this area.\n<\/p><p><b>Keywords:<\/b> curriculum transformation, clinical decision support, patient safety, health care quality, patient engagement\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Physicians and medical students have been using <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_information_technology\" title=\"Health information technology\" target=\"_blank\" class=\"wiki-link\" data-key=\"9c8ef822470559f757db89f3fa234cc0\">health information technology<\/a> (HIT) for decades. During this time, the role of HIT has changed dramatically from a useful tool for data access and occasional information retrieval to a ubiquitous presence that permeates health care and medical practice in myriad ways. As the use of HIT has accelerated and the underlying science of biomedical and <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_informatics\" title=\"Health informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"055eb51f53cfdbacc08ed150b266c9f4\">health informatics<\/a> has advanced, medical education has lagged behind, leaving physicians and students alike to learn largely on their own how to make use of these tools. While some medical schools have introduced limited aspects of biomedical informatics into their curricula, most of this is focused on training to perform basic tasks such as accessing knowledge resources or basic use of an electronic health record (EHR).\n<\/p><p>Twenty-first century clinicians face a much more digital world than their predecessors. The quantity of biomedical knowledge continues to expand, with an attendant increase in the primary scientific literature.<sup id=\"rdp-ebb-cite_ref-BastianSeventyFive2010_1-0\" class=\"reference\"><a href=\"#cite_note-BastianSeventyFive2010-1\" rel=\"external_link\">[1]<\/a><\/sup> Secondary sources that summarize this information proliferate as well, not only for use by clinicians but also by patients and consumers. Programs such as the \u201cmeaningful use\u201d financial incentives of the <a href=\"https:\/\/www.limswiki.org\/index.php\/Health_Information_Technology_for_Economic_and_Clinical_Health_Act\" title=\"Health Information Technology for Economic and Clinical Health Act\" target=\"_blank\" class=\"wiki-link\" data-key=\"89c9e20984bbda2e628b6d1d28ec3ad5\">Health Information for Technology and Clinical Health Act<\/a> require comprehensive adoption and use of the EHR.<sup id=\"rdp-ebb-cite_ref-BlumenthalWiring2011_2-0\" class=\"reference\"><a href=\"#cite_note-BlumenthalWiring2011-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BlumenthalImp2011_3-0\" class=\"reference\"><a href=\"#cite_note-BlumenthalImp2011-3\" rel=\"external_link\">[3]<\/a><\/sup> Patients, especially the aging \u201cbaby boomer\u201d generation, want to interact with the health care system the same way they interact with airlines, banks, and retailers, ie, through digital means using technologies such as the personal health record.<sup id=\"rdp-ebb-cite_ref-MillerPHR2009_4-0\" class=\"reference\"><a href=\"#cite_note-MillerPHR2009-4\" rel=\"external_link\">[4]<\/a><\/sup> Patients, payers, and purchasers demand more accountability in health care quality, safety, and cost,<sup id=\"rdp-ebb-cite_ref-BerwickTriple2008_5-0\" class=\"reference\"><a href=\"#cite_note-BerwickTriple2008-5\" rel=\"external_link\">[5]<\/a><\/sup> leading to an expectation of measurement and reporting of quality of care as a routine part of participation in new delivery mechanisms such as primary care medical homes and accountable care organizations. At the same time, telemedicine applications extend the reach of health care systems and clinicians in both rural and urban settings. Ideally, for all of these applications, patient data must move readily across organizational boundaries via health information exchange, while privacy and confidentiality are protected.<sup id=\"rdp-ebb-cite_ref-KupermanHealth2011_6-0\" class=\"reference\"><a href=\"#cite_note-KupermanHealth2011-6\" rel=\"external_link\">[6]<\/a><\/sup> The growing quantity of clinical and administrative data in these systems also affords an opportunity for advanced analysis that can enable better deployment of resources and coordination of care, facilitation of personalized and precision medicine, and advancement of clinical and translational research.<sup id=\"rdp-ebb-cite_ref-SafranToward2007_7-0\" class=\"reference\"><a href=\"#cite_note-SafranToward2007-7\" rel=\"external_link\">[7]<\/a><\/sup> Together, these advances are moving health care toward the global vision of the learning health system put forth by the Institute of Medicine (IOM).<sup id=\"rdp-ebb-cite_ref-FriedmanAchieving2010_8-0\" class=\"reference\"><a href=\"#cite_note-FriedmanAchieving2010-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmithBest2012_9-0\" class=\"reference\"><a href=\"#cite_note-SmithBest2012-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/p><p>The importance of these developments is further evidenced by the recent establishment of a new medical subspecialty of clinical informatics.<sup id=\"rdp-ebb-cite_ref-LehmannClinical2013_10-0\" class=\"reference\"><a href=\"#cite_note-LehmannClinical2013-10\" rel=\"external_link\">[10]<\/a><\/sup> Practicing physicians are now beginning to become board-certified in this new subspecialty, with fellowship programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) to soon be established. This underscores the need for medical students to be introduced to the concepts and competencies of this new medical subspecialty as part of the undergraduate curriculum.\n<\/p><p>In this paper, we describe our work to expand the undergraduate medical curriculum to include a comprehensive set of competencies in clinical informatics. We go beyond the usual focus on information retrieval from knowledge sources and data access through the EHR, aiming to address the expanded and diverse roles of HIT in contemporary health care delivery, personal health, public health, and clinical and <a href=\"https:\/\/www.limswiki.org\/index.php\/Translational_research\" title=\"Translational research\" target=\"_blank\" class=\"wiki-link\" data-key=\"b5462c2278492de7f21a60e40299b634\">translational research<\/a>. We describe the development of a novel biomedical informatics curriculum that is being implemented within a comprehensive transformation of the undergraduate curriculum at our institution. Our future plans include evaluation of the impact of this work, and we encourage others to participate in this evaluation and refinement process.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Previous_work_developing_competencies_in_informatics\">Previous work developing competencies in informatics<\/span><\/h2>\n<p>The need to include biomedical informatics in medical education has been recognized in previous publications of competencies, curricula, and other learning materials for <a href=\"https:\/\/www.limswiki.org\/index.php\/Informatics\" title=\"Informatics\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"ea0ff624ac3a644c35d2b51d39047bdf\">informatics<\/a>. One early, high-profile resource was part of the Medical Student Objectives Project of the Association of American Medical Colleges, which framed proposed objectives in terms of five key roles of physicians: clinicians, life-long learners, educators, researchers, and managers.<sup id=\"rdp-ebb-cite_ref-PanelContemp1999_11-0\" class=\"reference\"><a href=\"#cite_note-PanelContemp1999-11\" rel=\"external_link\">[11]<\/a><\/sup> Although current medical students believe these objectives are important to their education<sup id=\"rdp-ebb-cite_ref-BeaudoinMedical2013_12-0\" class=\"reference\"><a href=\"#cite_note-BeaudoinMedical2013-12\" rel=\"external_link\">[12]<\/a><\/sup>, uptake of these objectives has been modest.<sup id=\"rdp-ebb-cite_ref-McGowanEducating2007_13-0\" class=\"reference\"><a href=\"#cite_note-McGowanEducating2007-13\" rel=\"external_link\">[13]<\/a><\/sup> More recently, the Alliance for Clinical Education issued a policy statement calling for competencies related to the EHR<sup id=\"rdp-ebb-cite_ref-HammoudMed2012_14-0\" class=\"reference\"><a href=\"#cite_note-HammoudMed2012-14\" rel=\"external_link\">[14]<\/a><\/sup>, although also noting that many challenges to implementing such competencies exist.<sup id=\"rdp-ebb-cite_ref-HammoudOpp2012_15-0\" class=\"reference\"><a href=\"#cite_note-HammoudOpp2012-15\" rel=\"external_link\">[15]<\/a><\/sup> Others have also noted the challenges to education about the EHR, citing problems analogous to those of clinicians incorporating EHRs into practice, such as integration into workflows, and concerns about unanticipated and undesirable impacts.<sup id=\"rdp-ebb-cite_ref-EllawayMed2013_16-0\" class=\"reference\"><a href=\"#cite_note-EllawayMed2013-16\" rel=\"external_link\">[16]<\/a><\/sup> It is clear, however, that competencies in the use of the EHR are aligned with more general physician educational competencies<sup id=\"rdp-ebb-cite_ref-TierneyMed2013_17-0\" class=\"reference\"><a href=\"#cite_note-TierneyMed2013-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PagelerRefoc2013_18-0\" class=\"reference\"><a href=\"#cite_note-PagelerRefoc2013-18\" rel=\"external_link\">[18]<\/a><\/sup>, and that informatics curricula can be incorporated into undergraduate medical training<sup id=\"rdp-ebb-cite_ref-SilvermanTheEvo2012_19-0\" class=\"reference\"><a href=\"#cite_note-SilvermanTheEvo2012-19\" rel=\"external_link\">[19]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MilanoSim2014_20-0\" class=\"reference\"><a href=\"#cite_note-MilanoSim2014-20\" rel=\"external_link\">[20]<\/a><\/sup> and evaluated.<sup id=\"rdp-ebb-cite_ref-ReisTheImp2013_21-0\" class=\"reference\"><a href=\"#cite_note-ReisTheImp2013-21\" rel=\"external_link\">[21]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>In 2012, the Oregon Health & Science University, School of Medicine began a comprehensive transformation of the undergraduate medical curriculum for the 21st century. From the outset, the Dean and school leadership recognized the importance of informatics to physicians and health care in the future. With participation and support from the Senior Associate Dean for Education (GM), a team of five physician-faculty was convened to lead this effort, including the following:\n<\/p>\n<ul><li> An informatician involved nationally in informatics education leadership and development of the new clinical informatics subspecialty (WH)<\/li>\n<li> An internist-informatician who is chair of the medical school curriculum committee and actively engaged in the new curriculum development team (PG)<\/li>\n<li> A family physician and innovator in using simulation for teaching and assessing EHR use by medical students (FB)<\/li>\n<li> A critical care physician investigator using EHR simulation to assess residents\u2019 ability to detect important patterns in critical care data (JG)<\/li>\n<li> An informatician and former internal medicine residency associate program director also involved in the previous two above activities (VM)<\/li><\/ul>\n<p>Using an exploratory qualitative methodology based on action research and working with the curriculum transformation leadership, we developed a broad set of competencies in informatics for medical education. With each competency, we developed a set of learning objectives, a designation of when it would be most appropriate to introduce the concept during the medical school curriculum, and how each mapped to the broader ACGME competencies.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Developing_competencies_and_learning_objectives\">Developing competencies and learning objectives<\/span><\/h3>\n<p>Before beginning to identify the competencies for student learning, we needed to develop the rationale for including these topics in the curriculum. This led us to map the competencies to the six ACGME core competency domains<sup id=\"rdp-ebb-cite_ref-EnglanderToward2013_22-0\" class=\"reference\"><a href=\"#cite_note-EnglanderToward2013-22\" rel=\"external_link\">[22]<\/a><\/sup> and develop explicit learning objectives within each competency, followed by their implementation in the emerging curriculum. Discerning the rationale for our competencies was a narrative process, with their implementation enumerating the specific areas deemed important to incorporate into the medical school curriculum, as it was being concurrently designed.\n<\/p><p>Our first step was to provide the rationale to include various topic areas in the curriculum. A first critical concept we agreed upon was that informatics is not the same as computer literacy. Computer literacy is one of many requirements to use informatics successfully, but knowing how to use a computing device (personal computer, tablet, or smartphone) is not the same as having skills in informatics, ie, using that device to improve health, health care delivery, public health, or research.\n<\/p><p>We concurred that one fundamental skill currently being taught in many medical schools for 21st century clinicians was something taught in many medical schools since the late 20th century, namely, how to find information to apply to patient care. The needed informatics skills start, and do not stop, with the knowledge of how to enter simple queries into common general (eg, Google\u2122, <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.google.com\" target=\"_blank\">http:\/\/www.google.com<\/a>) or medical (eg, PubMed, <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmed.gov\" target=\"_blank\">http:\/\/www.pubmed.gov<\/a>) search engines. Clinicians must also know how to formulate a clinical question as an answerable one, and then be able to select the appropriate resource and make optimal use of it. This includes knowing what content is in different search systems. He or she must also know about specialized resources such as the Agency for Healthcare Research and Quality Guidelines Clearinghouse (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.guideline.gov\" target=\"_blank\">http:\/\/www.guideline.gov<\/a>) and the <a href=\"https:\/\/www.limswiki.org\/index.php\/Centers_for_Disease_Control_and_Prevention\" title=\"Centers for Disease Control and Prevention\" target=\"_blank\" class=\"wiki-link\" data-key=\"176aa9c9513251c328d864d1e724e814\">Centers for Disease Control and Prevention<\/a> travel site (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/wwwnc.cdc.gov\/travel\" target=\"_blank\">http:\/\/wwwnc.cdc.gov\/travel<\/a>). Clinicians should also have an understanding of the major commercial publishers as well as what professional societies offer. Once clinicians know how to use a search site, they must be able to phrase an appropriate query and know how to maximize the search using advanced query features. To take maximum advantage of available knowledge resources, it is necessary to understand the advanced search features that are available, whether in general web resources such as Google or dedicated biomedical knowledge resources such as PubMed.\n<\/p><p>Once information is retrieved, clinicians must know how to critically appraise the information retrieved and decide, if warranted, to apply it to the patient or population. As with searching, the type of appraisal varies with the search engine used. With output from general search engines like Google, the trustworthiness of the information must be assessed. Clinicians must also know how to make optimal use of patient data and information. They must know how to use informatics to achieve the triple aim of better health, better care, and lower cost.<sup id=\"rdp-ebb-cite_ref-BerwickTriple2008_5-1\" class=\"reference\"><a href=\"#cite_note-BerwickTriple2008-5\" rel=\"external_link\">[5]<\/a><\/sup> An appropriate framework for applying this emanates from the IOM concept of the \u201clearning health system\u201d<sup id=\"rdp-ebb-cite_ref-FriedmanAchieving2010_8-1\" class=\"reference\"><a href=\"#cite_note-FriedmanAchieving2010-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmithBest2012_9-1\" class=\"reference\"><a href=\"#cite_note-SmithBest2012-9\" rel=\"external_link\">[9]<\/a><\/sup>, which presents a compelling vision for a health care system that is patient-centered, population-based, and promotes learning from data.\n<\/p><p>Being able to use data and information also means understanding that the patient health record is more than \u201ccharting\u201d, and that its value goes beyond being able to read it. Clinicians must be facile with all aspects of the EHR, able to easily move from one vendor system to another and to understand the critical role of health information exchange in making any single record as complete as possible.<sup id=\"rdp-ebb-cite_ref-KupermanHealth2011_6-1\" class=\"reference\"><a href=\"#cite_note-KupermanHealth2011-6\" rel=\"external_link\">[6]<\/a><\/sup> This issue is magnified by the increasing amount of data the clinicians must view and manage.<sup id=\"rdp-ebb-cite_ref-Manor-ShulmanQuant2008_23-0\" class=\"reference\"><a href=\"#cite_note-Manor-ShulmanQuant2008-23\" rel=\"external_link\">[23]<\/a><\/sup> Further, as the health care focus moves from one of quantity to one of value, clinicians must pay attention to quality, patient safety, and cost. This requires coordination of care, in not just providing medical procedures, nursing interventions, therapies, etc in isolation, but also teamwork and communication.<sup id=\"rdp-ebb-cite_ref-DohertyPrinc2013_24-0\" class=\"reference\"><a href=\"#cite_note-DohertyPrinc2013-24\" rel=\"external_link\">[24]<\/a><\/sup>\n<\/p><p>From this vantage, the health record is no longer a passive collection of information used mainly to justify billing. Rather, it is a source of data, organized into coherent information that allows the health care team to deliver the best, safest, and most cost-effective care. This means that the 21st century clinician must have a basic understanding of informatics issues, such as capturing data that is correct and complete as well as consistent in its expression.<sup id=\"rdp-ebb-cite_ref-HershCaveats2013_25-0\" class=\"reference\"><a href=\"#cite_note-HershCaveats2013-25\" rel=\"external_link\">[25]<\/a><\/sup> He or she must be able to work in partnership with informatics professionals to achieve what we know is so critical in the application of informatics: adhering to standards, achieving system interoperability, appropriately and optimally implementing clinical decision support, and maintaining security to assure privacy and confidentiality.<sup id=\"rdp-ebb-cite_ref-HershAStim2009_26-0\" class=\"reference\"><a href=\"#cite_note-HershAStim2009-26\" rel=\"external_link\">[26]<\/a><\/sup>\n<\/p><p>Clinicians also require understanding of areas of quality measurement and improvement, especially as it applies to their direct practice.<sup id=\"rdp-ebb-cite_ref-BerwickTriple2008_5-2\" class=\"reference\"><a href=\"#cite_note-BerwickTriple2008-5\" rel=\"external_link\">[5]<\/a><\/sup> The clinician must understand concepts of health care quality and how the EHR and other sources of data are increasingly used to measure it. They must also know how to work with clinical leaders to select, implement, and improve quality measures. This should be introduced as part of the larger learning health system.<sup id=\"rdp-ebb-cite_ref-FriedmanAchieving2010_8-2\" class=\"reference\"><a href=\"#cite_note-FriedmanAchieving2010-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SmithBest2012_9-2\" class=\"reference\"><a href=\"#cite_note-SmithBest2012-9\" rel=\"external_link\">[9]<\/a><\/sup> This discussion should also include the related issues of patient safety and medical errors<sup id=\"rdp-ebb-cite_ref-ShekelleTheTop2013_27-0\" class=\"reference\"><a href=\"#cite_note-ShekelleTheTop2013-27\" rel=\"external_link\">[27]<\/a><\/sup>, including safety of HIT.<sup id=\"rdp-ebb-cite_ref-CommHealth2012_28-0\" class=\"reference\"><a href=\"#cite_note-CommHealth2012-28\" rel=\"external_link\">[28]<\/a><\/sup>\n<\/p><p>Patient engagement is another component for safe, effective, and coordinated care.<sup id=\"rdp-ebb-cite_ref-RicciardiANat2013_29-0\" class=\"reference\"><a href=\"#cite_note-RicciardiANat2013-29\" rel=\"external_link\">[29]<\/a><\/sup> Not only does patient engagement improve health care, but patients, including Internet-savvy baby boomers, will increasingly demand it. Patients want health care that adapts the online features of other modern industries, allowing them the ability to view their own data and interact with their clinicians and health care system (eg, online scheduling of appointments, prescription refills, and even consultations that are appropriate via the Internet or other electronic media).<sup id=\"rdp-ebb-cite_ref-RicciardiANat2013_29-1\" class=\"reference\"><a href=\"#cite_note-RicciardiANat2013-29\" rel=\"external_link\">[29]<\/a><\/sup> These will likely take the form of a personal health record, accessible from a patient portal that allows access to all information, not just that from the system of the provider organization.<sup id=\"rdp-ebb-cite_ref-MillerPHR2009_4-1\" class=\"reference\"><a href=\"#cite_note-MillerPHR2009-4\" rel=\"external_link\">[4]<\/a><\/sup> Because of this natural evolution of health care, clinicians must also have knowledge and understanding of the appropriate use of telemedicine and telehealth (such as e-visits), both for remote locations and as a convenient option locally in patients\u2019 homes and other settings.\n<\/p><p>Complementary to the patient-centric view, clinicians must also understand population-based care<sup id=\"rdp-ebb-cite_ref-InstPop2012_30-0\" class=\"reference\"><a href=\"#cite_note-InstPop2012-30\" rel=\"external_link\">[30]<\/a><\/sup> and the informatics underlying it.<sup id=\"rdp-ebb-cite_ref-DiamondColl2009_31-0\" class=\"reference\"><a href=\"#cite_note-DiamondColl2009-31\" rel=\"external_link\">[31]<\/a><\/sup> The clinician and their team care for populations of patients; they must be able to view their care needs and results across their patient population. When a new test or treatment is determined to be highly effective, the clinician must be able to quickly identify patients who are candidates for it. They must also be able to identify outliers in their populations who require intervention, such as those with excessively high blood pressure or blood sugar, missed appointments or screen tests, or those at risk for hospital (re)admission.\n<\/p><p>Another area where 21st century clinicians must understand the key issues is in bioinformatics, especially as it relates to personalized medicine.<sup id=\"rdp-ebb-cite_ref-HamburgThePath2010_32-0\" class=\"reference\"><a href=\"#cite_note-HamburgThePath2010-32\" rel=\"external_link\">[32]<\/a><\/sup> Modern clinicians need not understand complicated gene sequencing algorithms, but they should have an understanding of how genomics and related areas are transforming our understanding of maintaining health and diagnosing and treating disease.<sup id=\"rdp-ebb-cite_ref-FernaldBio2011_33-0\" class=\"reference\"><a href=\"#cite_note-FernaldBio2011-33\" rel=\"external_link\">[33]<\/a><\/sup> Now that the vision of personalized medicine is coming to fruition, clinicians will increasingly rely on the help of decision support and other tools for assistance in applying it to individual patients. He or she should have a basic understanding of genome-wide association studies and their ramifications.<sup id=\"rdp-ebb-cite_ref-ManolioGenome2010_34-0\" class=\"reference\"><a href=\"#cite_note-ManolioGenome2010-34\" rel=\"external_link\">[34]<\/a><\/sup>\n<\/p><p>Clinicians must also understand the strengths and limitations of clinical and translational research. They must understand the differences between and the value of experimental and observational studies. Ideally, students will have participated in research while in their training. But even if not, they should understand issues such as data quality, study design, and the limitations that come from the sharp focus perspective of a clinical study. Students should participate in the learning health system laid out in the vision of groups like the IOM.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implementing_competencies_and_objectives\">Implementing competencies and objectives<\/span><\/h3>\n<p>Based on the above narrative, our group of diverse clinician-educators developed through an iterative process a set of 13 competencies in clinical informatics (Table 1). Each competency was mapped to one or more of the six ACGME general competency domains. We then developed more detailed learning objectives and milestones within each competency. We also categorized each learning objective for its presence in the early (at the beginning), middle (during the preclinical portion), or late (during clinical experiences) portion of the curriculum.\t\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Hersh_AdvancesMedEdPrac2014_2014-5.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"6c945e24f13e31c16638a651bc9f991c\"><img alt=\"Tab1 Hersh AdvancesMedEdPrac2014 2014-5.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f3\/Tab1_Hersh_AdvancesMedEdPrac2014_2014-5.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 1<\/b> Competencies in clinical informatics and specific learning objective\/milestone within each<br \/>\n<p><b>Notes<\/b>: All learning objectives\/milestones are designated for time to start (early: start of medical school; mid: later in preclinical years; and late: during clinical experiences); Each is also mapped into one or more of the ACGME core competencies: patient care and procedural skills (PCPS), medical knowledge (MK), practice-based learning and improvement (PBLI), interpersonal and communication skills (ICS), professionalism (Prof), and systems-based practice (SBP).<br \/>\n<\/p>\n<b>Abbreviations<\/b>: ACGME, Accreditation Council for Graduate Medical Education; CDS, clinical decision support; HIE, health information exchange; HIPAA, Health Insurance Portability and Accountability Act.<\/blockquote>\n<p>These learning objectives are now being woven into Oregon Health & Science University\u2019s new curriculum (Table 2). As seen in Table 2, we have designated where the specific learning activities will be placed in the major portions of the curriculum from orientation through preclinical sciences, clinical experiences, and intersession in the fourth year.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab2_Hersh_AdvancesMedEdPrac2014_2014-5.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"8b3894e6a443d53456c67f93b24f48c1\"><img alt=\"Tab2 Hersh AdvancesMedEdPrac2014 2014-5.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/90\/Tab2_Hersh_AdvancesMedEdPrac2014_2014-5.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 2<\/b> Informatics in various portions of curriculum in the preclinical (orientation, fundamentals, and preclinical sciences) and clinical (clinical experiences and intersession) years<br \/>\n<b>Abbreviations<\/b>: CDS, clinical decision support; DDI, drug\u2013drug interaction; EBM, evidence-based medicine; EHR, electronic health record; EKG, electrocardiogram; HIE, health information exchange; HIPAA, Health Insurance Portability and Accountability Act; IT, information technology; OHSU, Oregon Health & Science University; PHR, personal health record; VA, Veterans Affairs. <\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Conclusion_and_future_directions\">Conclusion and future directions<\/span><\/h2>\n<p>Our analysis has shown there are a substantial number of informatics competencies and a large body of associated knowledge that the 21st century clinician needs to learn and apply. From a pedagogical standpoint, there are also issues in how to organize, deliver, and assess this content. Certainly, one approach is to provide this content as a separate course, isolated from the rest of the curriculum. However, a better approach would be to tightly and comprehensively integrate informatics concepts longitudinally into the learning curriculum since clinical informatics is emerging as a core competency of medical practice, applicable in all basic science disciplines and clinical specialties.\n<\/p><p>A next major step for this work will be to develop evaluation activities for the competencies and learning activities. These will vary based on institutional factors (class size, whether students are asynchronous, etc), faculty preference, and funding limitations (desire to use simulation for many things, but cost may be prohibitive). As such, different learning activities will require different evaluation methods. For example, the competency \u201cProvide care using population health management approaches\u201d might be best measured by having students describe or select effective system change options based on immunizations rates of the clinic to which they are assigned, while the competency \u201cProvide clinical care via telemedicine and refer those for whom it is necessary\u201d might best be measured with an objective structured clinical examination simulation.\n<\/p><p>The presence of these competencies also indicates a need for educators who are specialists in informatics to (collaboratively with clinical educators) design the learning and deliver learning experiences that are appropriate for lecture, group discussions, self-paced and self-directed methods, and other settings. But informatics is one of those topics that is best infused throughout the curriculum, especially in clinical settings where it is being used.\n<\/p><p>There are a number of future steps for this work. First, we recognize that this set of competencies represents the views of one medical school\u2019s informatics faculty and must be validated with informatics experts from other institutions. Second, we must evaluate our own implementation of this curriculum to determine how these competencies are delivered to a medical student audience. Finally, we must evaluate this entire process with students from our institution and others to determine which competencies and learning experiences are most valuable for them in their future clinical practice.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgments\">Acknowledgments<\/span><\/h2>\n<p>Dr Biagioli was supported in part by NIH Grant 1R25CA158571. Drs Hersh and Mejicano were supported in part by the Accelerating Change in Medical Education grant of the American Medical Association.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Disclosures\">Disclosures<\/span><\/h2>\n<p>The authors report no conflicts of interest in this work.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-BastianSeventyFive2010-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BastianSeventyFive2010_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bastian, H.; Glasziou, P.; Chalmers, I. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2943439\" target=\"_blank\">\"Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?\"<\/a>. <i>PLoS Medicine<\/i> <b>7<\/b> (9): e1000326. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pmed.1000326\" target=\"_blank\">10.1371\/journal.pmed.1000326<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2943439\/\" target=\"_blank\">PMC2943439<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20877712\" target=\"_blank\">20877712<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2943439\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2943439<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Seventy-five+trials+and+eleven+systematic+reviews+a+day%3A+how+will+we+ever+keep+up%3F&rft.jtitle=PLoS+Medicine&rft.aulast=Bastian%2C+H.%3B+Glasziou%2C+P.%3B+Chalmers%2C+I.&rft.au=Bastian%2C+H.%3B+Glasziou%2C+P.%3B+Chalmers%2C+I.&rft.date=2010&rft.volume=7&rft.issue=9&rft.pages=e1000326&rft_id=info:doi\/10.1371%2Fjournal.pmed.1000326&rft_id=info:pmc\/PMC2943439&rft_id=info:pmid\/20877712&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2943439&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlumenthalWiring2011-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlumenthalWiring2011_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blumenthal, D. (2011). \"Wiring the health system \u2013 origins and provisions of a new federal program\". <i>The New England Journal of Medicine<\/i> <b>365<\/b> (24): 2323\u20132329. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMsr1110507\" target=\"_blank\">10.1056\/NEJMsr1110507<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22168647\" target=\"_blank\">22168647<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Wiring+the+health+system+%E2%80%93+origins+and+provisions+of+a+new+federal+program&rft.jtitle=The+New+England+Journal+of+Medicine&rft.aulast=Blumenthal%2C+D.&rft.au=Blumenthal%2C+D.&rft.date=2011&rft.volume=365&rft.issue=24&rft.pages=2323%E2%80%932329&rft_id=info:doi\/10.1056%2FNEJMsr1110507&rft_id=info:pmid\/22168647&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlumenthalImp2011-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlumenthalImp2011_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blumenthal, D. (2011). \"Implementation of the federal health information technology initiative\". <i>The New England Journal of Medicine<\/i> <b>365<\/b> (25): 2426\u20132431. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMsr1112158\" target=\"_blank\">10.1056\/NEJMsr1112158<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22187990\" target=\"_blank\">22187990<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Implementation+of+the+federal+health+information+technology+initiative&rft.jtitle=The+New+England+Journal+of+Medicine&rft.aulast=Blumenthal%2C+D.&rft.au=Blumenthal%2C+D.&rft.date=2011&rft.volume=365&rft.issue=25&rft.pages=2426%E2%80%932431&rft_id=info:doi\/10.1056%2FNEJMsr1112158&rft_id=info:pmid\/22187990&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MillerPHR2009-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MillerPHR2009_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-MillerPHR2009_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Miller, H.; Yasnoff, W.; Burde, H. (2009). <i>Personal Health Records: The Essential Missing Element in 21st Century Healthcare<\/i>. Chicago, IL: Healthcare Information and Management Systems Society. pp. 178. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780980069761.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Personal+Health+Records%3A+The+Essential+Missing+Element+in+21st+Century+Healthcare&rft.aulast=Miller%2C+H.%3B+Yasnoff%2C+W.%3B+Burde%2C+H.&rft.au=Miller%2C+H.%3B+Yasnoff%2C+W.%3B+Burde%2C+H.&rft.date=2009&rft.pages=pp.%26nbsp%3B178&rft.place=Chicago%2C+IL&rft.pub=Healthcare+Information+and+Management+Systems+Society&rft.isbn=9780980069761&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BerwickTriple2008-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BerwickTriple2008_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-BerwickTriple2008_5-1\" rel=\"external_link\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-BerwickTriple2008_5-2\" rel=\"external_link\">5.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berwick, D.M.; Nolan, T.W.; Whittington, J. (2008). \"The triple aim: care, health, and cost\". <i>Health Affairs<\/i> <b>27<\/b> (3): 759\u2013769. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.27.3.759\" target=\"_blank\">10.1377\/hlthaff.27.3.759<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18474969\" target=\"_blank\">18474969<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+triple+aim%3A+care%2C+health%2C+and+cost&rft.jtitle=Health+Affairs&rft.aulast=Berwick%2C+D.M.%3B+Nolan%2C+T.W.%3B+Whittington%2C+J.&rft.au=Berwick%2C+D.M.%3B+Nolan%2C+T.W.%3B+Whittington%2C+J.&rft.date=2008&rft.volume=27&rft.issue=3&rft.pages=759%E2%80%93769&rft_id=info:doi\/10.1377%2Fhlthaff.27.3.759&rft_id=info:pmid\/18474969&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KupermanHealth2011-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KupermanHealth2011_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-KupermanHealth2011_6-1\" rel=\"external_link\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kuperman, G.J. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168299\" target=\"_blank\">\"Health-information exchange: why are we doing it, and what are we doing?\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>18<\/b> (5): 678\u2013682. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1136%2Famiajnl-2010-000021\" target=\"_blank\">10.1136\/amiajnl-2010-000021<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3168299\/\" target=\"_blank\">PMC3168299<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21676940\" target=\"_blank\">21676940<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168299\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3168299<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Health-information+exchange%3A+why+are+we+doing+it%2C+and+what+are+we+doing%3F&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Kuperman%2C+G.J.&rft.au=Kuperman%2C+G.J.&rft.date=2011&rft.volume=18&rft.issue=5&rft.pages=678%E2%80%93682&rft_id=info:doi\/10.1136%2Famiajnl-2010-000021&rft_id=info:pmc\/PMC3168299&rft_id=info:pmid\/21676940&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3168299&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SafranToward2007-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SafranToward2007_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Safran, C.; Bloomrosen, M.; Hammond, W.E.; et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823\" target=\"_blank\">\"Toward a national framework for the secondary use of health data: an American Medical Informatics Association White Paper\"<\/a>. <i>Journal of the American Medical Informatics Association<\/i> <b>14<\/b> (1): 1\u20139. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1197%2Fjamia.M2273\" target=\"_blank\">10.1197\/jamia.M2273<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2329823\/\" target=\"_blank\">PMC2329823<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17077452\" target=\"_blank\">17077452<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2329823<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+national+framework+for+the+secondary+use+of+health+data%3A+an+American+Medical+Informatics+Association+White+Paper&rft.jtitle=Journal+of+the+American+Medical+Informatics+Association&rft.aulast=Safran%2C+C.%3B+Bloomrosen%2C+M.%3B+Hammond%2C+W.E.%3B+et+al.&rft.au=Safran%2C+C.%3B+Bloomrosen%2C+M.%3B+Hammond%2C+W.E.%3B+et+al.&rft.date=2007&rft.volume=14&rft.issue=1&rft.pages=1%E2%80%939&rft_id=info:doi\/10.1197%2Fjamia.M2273&rft_id=info:pmc\/PMC2329823&rft_id=info:pmid\/17077452&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2329823&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FriedmanAchieving2010-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FriedmanAchieving2010_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-FriedmanAchieving2010_8-1\" rel=\"external_link\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-FriedmanAchieving2010_8-2\" rel=\"external_link\">8.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Friedman, C.P.; Wong, A.K.; Blumenthal, D. (2010). \"Achieving a nationwide learning health system\". <i>Science Translation Medicine<\/i> <b>2<\/b> (57): 57cm29. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscitranslmed.3001456\" target=\"_blank\">10.1126\/scitranslmed.3001456<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Achieving+a+nationwide+learning+health+system&rft.jtitle=Science+Translation+Medicine&rft.aulast=Friedman%2C+C.P.%3B+Wong%2C+A.K.%3B+Blumenthal%2C+D.&rft.au=Friedman%2C+C.P.%3B+Wong%2C+A.K.%3B+Blumenthal%2C+D.&rft.date=2010&rft.volume=2&rft.issue=57&rft.pages=57cm29&rft_id=info:doi\/10.1126%2Fscitranslmed.3001456&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SmithBest2012-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SmithBest2012_9-0\" rel=\"external_link\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-SmithBest2012_9-1\" rel=\"external_link\">9.1<\/a><\/sup> <sup><a href=\"#cite_ref-SmithBest2012_9-2\" rel=\"external_link\">9.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Smith, M.; Saunders, R.; Stuckhardt, L.; McGinnis, J.M., ed. (2012). <i>Best Care at Lower Cost: The Path to Continuously Learning Health Care in America<\/i>. Washington, DC: National Academies Press. pp. 416. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780309260732.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Best+Care+at+Lower+Cost%3A+The+Path+to+Continuously+Learning+Health+Care+in+America&rft.date=2012&rft.pages=pp.%26nbsp%3B416&rft.place=Washington%2C+DC&rft.pub=National+Academies+Press&rft.isbn=9780309260732&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LehmannClinical2013-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LehmannClinical2013_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lehmann, C.U.; Shorte, V.; Gundlapalli, A.V. (2013). \"Clinical informatics sub-specialty board certification\". <i>Pediatrics in Review<\/i> <b>34<\/b> (11): 525\u2013530. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1542%2Fpir.34-11-525\" target=\"_blank\">10.1542\/pir.34-11-525<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24187144\" target=\"_blank\">24187144<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Clinical+informatics+sub-specialty+board+certification&rft.jtitle=Pediatrics+in+Review&rft.aulast=Lehmann%2C+C.U.%3B+Shorte%2C+V.%3B+Gundlapalli%2C+A.V.&rft.au=Lehmann%2C+C.U.%3B+Shorte%2C+V.%3B+Gundlapalli%2C+A.V.&rft.date=2013&rft.volume=34&rft.issue=11&rft.pages=525%E2%80%93530&rft_id=info:doi\/10.1542%2Fpir.34-11-525&rft_id=info:pmid\/24187144&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PanelContemp1999-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PanelContemp1999_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">The Informatics Panel and the Population Health Perspective Panel (1999). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/journals.lww.com\/academicmedicine\/Abstract\/1999\/02000\/Contemporary_issues_in_medicine_medical.12.aspx\" target=\"_blank\">\"Contemporary issues in medicine \u2013 medical informatics and population health: report II of the Medical School Objectives Project\"<\/a>. <i>Academic Medicine<\/i> <b>74<\/b> (2): 130\u2013141<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/journals.lww.com\/academicmedicine\/Abstract\/1999\/02000\/Contemporary_issues_in_medicine_medical.12.aspx\" target=\"_blank\">http:\/\/journals.lww.com\/academicmedicine\/Abstract\/1999\/02000\/Contemporary_issues_in_medicine_medical.12.aspx<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Contemporary+issues+in+medicine+%E2%80%93+medical+informatics+and+population+health%3A+report+II+of+the+Medical+School+Objectives+Project&rft.jtitle=Academic+Medicine&rft.aulast=The+Informatics+Panel+and+the+Population+Health+Perspective+Panel&rft.au=The+Informatics+Panel+and+the+Population+Health+Perspective+Panel&rft.date=1999&rft.volume=74&rft.issue=2&rft.pages=130%E2%80%93141&rft_id=http%3A%2F%2Fjournals.lww.com%2Facademicmedicine%2FAbstract%2F1999%2F02000%2FContemporary_issues_in_medicine_medical.12.aspx&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BeaudoinMedical2013-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BeaudoinMedical2013_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Beaudoin, D.; Richardson, S.J.; Sheng, X.; Mitchell, J. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4205536\" target=\"_blank\">\"Medical students\u2019 perspectives on biomedical informatics learning objectives\"<\/a>. <i>International Journal of Medical Education<\/i> (4): 1\u20138. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.5116%2Fijme.50ce.316b\" target=\"_blank\">10.5116\/ijme.50ce.316b<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4205536\/\" target=\"_blank\">PMC4205536<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4205536\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4205536<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+students%E2%80%99+perspectives+on+biomedical+informatics+learning+objectives&rft.jtitle=International+Journal+of+Medical+Education&rft.aulast=Beaudoin%2C+D.%3B+Richardson%2C+S.J.%3B+Sheng%2C+X.%3B+Mitchell%2C+J.&rft.au=Beaudoin%2C+D.%3B+Richardson%2C+S.J.%3B+Sheng%2C+X.%3B+Mitchell%2C+J.&rft.date=2013&rft.issue=4&rft.pages=1%E2%80%938&rft_id=info:doi\/10.5116%2Fijme.50ce.316b&rft_id=info:pmc\/PMC4205536&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4205536&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McGowanEducating2007-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McGowanEducating2007_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McGowan, J.; Passiment, M.; Hoffman, H. (2007). \"Educating medical students as competent users of health information technologies: the MSOP data\". <i>Studies in Health Technology and Informatics<\/i> <b>129<\/b> (Pt. 2): 1414-8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17911947\" target=\"_blank\">17911947<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Educating+medical+students+as+competent+users+of+health+information+technologies%3A+the+MSOP+data&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=McGowan%2C+J.%3B+Passiment%2C+M.%3B+Hoffman%2C+H.&rft.au=McGowan%2C+J.%3B+Passiment%2C+M.%3B+Hoffman%2C+H.&rft.date=2007&rft.volume=129&rft.issue=Pt.+2&rft.pages=1414-8&rft_id=info:pmid\/17911947&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HammoudMed2012-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HammoudMed2012_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hammoud, M.M.; Dalymple, J.L.; Christner, J.G.; et al. (2012). \"Medical student documentation in electronic health records: a collaborative statement from the Alliance for Clinical Education\". <i>Teaching and Learning in Medicine<\/i> <b>24<\/b> (3): 257\u2013266. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F10401334.2012.692284\" target=\"_blank\">10.1080\/10401334.2012.692284<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22775791\" target=\"_blank\">22775791<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+student+documentation+in+electronic+health+records%3A+a+collaborative+statement+from+the+Alliance+for+Clinical+Education&rft.jtitle=Teaching+and+Learning+in+Medicine&rft.aulast=Hammoud%2C+M.M.%3B+Dalymple%2C+J.L.%3B+Christner%2C+J.G.%3B+et+al.&rft.au=Hammoud%2C+M.M.%3B+Dalymple%2C+J.L.%3B+Christner%2C+J.G.%3B+et+al.&rft.date=2012&rft.volume=24&rft.issue=3&rft.pages=257%E2%80%93266&rft_id=info:doi\/10.1080%2F10401334.2012.692284&rft_id=info:pmid\/22775791&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HammoudOpp2012-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HammoudOpp2012_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hammoud, M.M.; Margo, K.; Christner, J.G.;, Fisher, J.; Fischer, S.H.; Pangaro, L.N. (2012). \"Opportunities and challenges in integrating electronic health records into undergraduate medical education: a national survey of clerkship directors\". <i>Teaching and Learning in Medicine<\/i> <b>24<\/b> (3): 219\u2013224. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F10401334.2012.692267\" target=\"_blank\">10.1080\/10401334.2012.692267<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22775785\" target=\"_blank\">22775785<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Opportunities+and+challenges+in+integrating+electronic+health+records+into+undergraduate+medical+education%3A+a+national+survey+of+clerkship+directors&rft.jtitle=Teaching+and+Learning+in+Medicine&rft.aulast=Hammoud%2C+M.M.%3B+Margo%2C+K.%3B+Christner%2C+J.G.%3B%2C+Fisher%2C+J.%3B+Fischer%2C+S.H.%3B+Pangaro%2C+L.N.&rft.au=Hammoud%2C+M.M.%3B+Margo%2C+K.%3B+Christner%2C+J.G.%3B%2C+Fisher%2C+J.%3B+Fischer%2C+S.H.%3B+Pangaro%2C+L.N.&rft.date=2012&rft.volume=24&rft.issue=3&rft.pages=219%E2%80%93224&rft_id=info:doi\/10.1080%2F10401334.2012.692267&rft_id=info:pmid\/22775785&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EllawayMed2013-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EllawayMed2013_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ellaway, R.H.; Graves, L.; Greene, P.S. (2013). \"Medical education in an electronic health record-mediated world\". <i>Medical Teacher<\/i> <b>35<\/b> (4): 282\u2013286. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3109%2F0142159X.2013.773396\" target=\"_blank\">10.3109\/0142159X.2013.773396<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23464893\" target=\"_blank\">23464893<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+education+in+an+electronic+health+record-mediated+world&rft.jtitle=Medical+Teacher&rft.aulast=Ellaway%2C+R.H.%3B+Graves%2C+L.%3B+Greene%2C+P.S.&rft.au=Ellaway%2C+R.H.%3B+Graves%2C+L.%3B+Greene%2C+P.S.&rft.date=2013&rft.volume=35&rft.issue=4&rft.pages=282%E2%80%93286&rft_id=info:doi\/10.3109%2F0142159X.2013.773396&rft_id=info:pmid\/23464893&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TierneyMed2013-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TierneyMed2013_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Tierney, M.J.; Pageler, N.M.; Kahana, M.; Pantaleoni, J.L.; Longhurst, C.A. (2013). \"Medical education in the electronic medical record (EMR) era: benefits, challenges, and future directions\". <i>Academic Medicine<\/i> <b>88<\/b> (6): 748\u2013752. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FACM.0b013e3182905ceb\" target=\"_blank\">10.1097\/ACM.0b013e3182905ceb<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23619078\" target=\"_blank\">23619078<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Medical+education+in+the+electronic+medical+record+%28EMR%29+era%3A+benefits%2C+challenges%2C+and+future+directions&rft.jtitle=Academic+Medicine&rft.aulast=Tierney%2C+M.J.%3B+Pageler%2C+N.M.%3B+Kahana%2C+M.%3B+Pantaleoni%2C+J.L.%3B+Longhurst%2C+C.A.&rft.au=Tierney%2C+M.J.%3B+Pageler%2C+N.M.%3B+Kahana%2C+M.%3B+Pantaleoni%2C+J.L.%3B+Longhurst%2C+C.A.&rft.date=2013&rft.volume=88&rft.issue=6&rft.pages=748%E2%80%93752&rft_id=info:doi\/10.1097%2FACM.0b013e3182905ceb&rft_id=info:pmid\/23619078&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PagelerRefoc2013-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PagelerRefoc2013_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pageler, N.M.; Friedman, C.P.; Longhurst, C.A. (2013). \"Refocusing medical education in the EMR era\". <i>Journal of the American Medical Association<\/i> <b>310<\/b> (21): 2249\u20132250. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2013.282326\" target=\"_blank\">10.1001\/jama.2013.282326<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24302083\" target=\"_blank\">24302083<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Refocusing+medical+education+in+the+EMR+era&rft.jtitle=Journal+of+the+American+Medical+Association&rft.aulast=Pageler%2C+N.M.%3B+Friedman%2C+C.P.%3B+Longhurst%2C+C.A.&rft.au=Pageler%2C+N.M.%3B+Friedman%2C+C.P.%3B+Longhurst%2C+C.A.&rft.date=2013&rft.volume=310&rft.issue=21&rft.pages=2249%E2%80%932250&rft_id=info:doi\/10.1001%2Fjama.2013.282326&rft_id=info:pmid\/24302083&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SilvermanTheEvo2012-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SilvermanTheEvo2012_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Silverman, H.; Cohen, T.; Fridsma, D. (2012). \"The evolution of a novel biomedical informatics curriculum for medical students\". <i>Academic Medicine<\/i> <b>87<\/b> (1): 84\u201390. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FACM.0b013e31823a599e\" target=\"_blank\">10.1097\/ACM.0b013e31823a599e<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22104049\" target=\"_blank\">22104049<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+evolution+of+a+novel+biomedical+informatics+curriculum+for+medical+students&rft.jtitle=Academic+Medicine&rft.aulast=Silverman%2C+H.%3B+Cohen%2C+T.%3B+Fridsma%2C+D.&rft.au=Silverman%2C+H.%3B+Cohen%2C+T.%3B+Fridsma%2C+D.&rft.date=2012&rft.volume=87&rft.issue=1&rft.pages=84%E2%80%9390&rft_id=info:doi\/10.1097%2FACM.0b013e31823a599e&rft_id=info:pmid\/22104049&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MilanoSim2014-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MilanoSim2014_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Milano, C.E.; Hardman, J.A.; Plesiu, A.; Rdesinski, R.E.; Biagioli, F.E. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4035239\" target=\"_blank\">\"Simulated electronic health record (Sim-EHR) curriculum: teaching EHR skills and use of the EHR for disease management and prevention\"<\/a>. <i>Academic Medicine<\/i> <b>89<\/b> (3): 399\u2013403. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FACM.0000000000000149\" target=\"_blank\">10.1097\/ACM.0000000000000149<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4035239\/\" target=\"_blank\">PMC4035239<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24448035\" target=\"_blank\">24448035<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4035239\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4035239<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Simulated+electronic+health+record+%28Sim-EHR%29+curriculum%3A+teaching+EHR+skills+and+use+of+the+EHR+for+disease+management+and+prevention&rft.jtitle=Academic+Medicine&rft.aulast=Milano%2C+C.E.%3B+Hardman%2C+J.A.%3B+Plesiu%2C+A.%3B+Rdesinski%2C+R.E.%3B+Biagioli%2C+F.E.&rft.au=Milano%2C+C.E.%3B+Hardman%2C+J.A.%3B+Plesiu%2C+A.%3B+Rdesinski%2C+R.E.%3B+Biagioli%2C+F.E.&rft.date=2014&rft.volume=89&rft.issue=3&rft.pages=399%E2%80%93403&rft_id=info:doi\/10.1097%2FACM.0000000000000149&rft_id=info:pmc\/PMC4035239&rft_id=info:pmid\/24448035&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4035239&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ReisTheImp2013-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ReisTheImp2013_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Reis, S.; Sagi, D.; Eisenberg, O.; et al. (2013). \"The impact of residents\u2019 training in Electronic Medical Record (EMR) use on their competence: report of a pragmatic trial\". <i>Patient Education and Counseling<\/i> <b>93<\/b> (3): 515\u2013521. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.pec.2013.08.007\" target=\"_blank\">10.1016\/j.pec.2013.08.007<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24011647\" target=\"_blank\">24011647<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+impact+of+residents%E2%80%99+training+in+Electronic+Medical+Record+%28EMR%29+use+on+their+competence%3A+report+of+a+pragmatic+trial&rft.jtitle=Patient+Education+and+Counseling&rft.aulast=Reis%2C+S.%3B+Sagi%2C+D.%3B+Eisenberg%2C+O.%3B+et+al.&rft.au=Reis%2C+S.%3B+Sagi%2C+D.%3B+Eisenberg%2C+O.%3B+et+al.&rft.date=2013&rft.volume=93&rft.issue=3&rft.pages=515%E2%80%93521&rft_id=info:doi\/10.1016%2Fj.pec.2013.08.007&rft_id=info:pmid\/24011647&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EnglanderToward2013-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EnglanderToward2013_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Englander, R.; Cameron, T.; Ballard, A.J.; Dodge, J.; Bull, J.; Aschenbrener, C.A. (2013). \"Toward a common taxonomy of competency domains for the health professions and competencies for physicians\". <i>Academic Medicine<\/i> <b>88<\/b> (8): 1088\u20131094. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FACM.0b013e31829a3b2b\" target=\"_blank\">10.1097\/ACM.0b013e31829a3b2b<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23807109\" target=\"_blank\">23807109<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+common+taxonomy+of+competency+domains+for+the+health+professions+and+competencies+for+physicians&rft.jtitle=Academic+Medicine&rft.aulast=Englander%2C+R.%3B+Cameron%2C+T.%3B+Ballard%2C+A.J.%3B+Dodge%2C+J.%3B+Bull%2C+J.%3B+Aschenbrener%2C+C.A.&rft.au=Englander%2C+R.%3B+Cameron%2C+T.%3B+Ballard%2C+A.J.%3B+Dodge%2C+J.%3B+Bull%2C+J.%3B+Aschenbrener%2C+C.A.&rft.date=2013&rft.volume=88&rft.issue=8&rft.pages=1088%E2%80%931094&rft_id=info:doi\/10.1097%2FACM.0b013e31829a3b2b&rft_id=info:pmid\/23807109&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Manor-ShulmanQuant2008-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Manor-ShulmanQuant2008_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Manor-Shulman, O.; Beyene, J.; Frndova, H.; Parshuram, C.S. (2008). \"Quantifying the volume of documented clinical information in critical illness\". <i>Journal of Critical Care<\/i> <b>23<\/b> (2): 245\u2013250. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jcrc.2007.06.003\" target=\"_blank\">10.1016\/j.jcrc.2007.06.003<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18538218\" target=\"_blank\">18538218<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quantifying+the+volume+of+documented+clinical+information+in+critical+illness&rft.jtitle=Journal+of+Critical+Care&rft.aulast=Manor-Shulman%2C+O.%3B+Beyene%2C+J.%3B+Frndova%2C+H.%3B+Parshuram%2C+C.S.&rft.au=Manor-Shulman%2C+O.%3B+Beyene%2C+J.%3B+Frndova%2C+H.%3B+Parshuram%2C+C.S.&rft.date=2008&rft.volume=23&rft.issue=2&rft.pages=245%E2%80%93250&rft_id=info:doi\/10.1016%2Fj.jcrc.2007.06.003&rft_id=info:pmid\/18538218&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DohertyPrinc2013-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DohertyPrinc2013_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Doherty, R.B.; Crowley, R.A.; Health and Public Policy Committee of the American College of Physicians (2013). \"Principles supporting dynamic clinical care teams: an American College of Physicians position paper\". <i>Annals of Internal Medicine<\/i> <b>159<\/b> (9): 620\u2013626. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7326%2F0003-4819-159-9-201311050-00710\" target=\"_blank\">10.7326\/0003-4819-159-9-201311050-00710<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24042251\" target=\"_blank\">24042251<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Principles+supporting+dynamic+clinical+care+teams%3A+an+American+College+of+Physicians+position+paper&rft.jtitle=Annals+of+Internal+Medicine&rft.aulast=Doherty%2C+R.B.%3B+Crowley%2C+R.A.%3B+Health+and+Public+Policy+Committee+of+the+American+College+of+Physicians&rft.au=Doherty%2C+R.B.%3B+Crowley%2C+R.A.%3B+Health+and+Public+Policy+Committee+of+the+American+College+of+Physicians&rft.date=2013&rft.volume=159&rft.issue=9&rft.pages=620%E2%80%93626&rft_id=info:doi\/10.7326%2F0003-4819-159-9-201311050-00710&rft_id=info:pmid\/24042251&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HershCaveats2013-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HershCaveats2013_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hersh, W.R.; Weiner, M.G.; Embi, P.J.; et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3748381\" target=\"_blank\">\"Caveats for the use of operational electronic health record data in comparative effectiveness research\"<\/a>. <i>Medical Care<\/i> <b>51<\/b> (8 Suppl 3): S30\u2013S37. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1097%2FMLR.0b013e31829b1dbd\" target=\"_blank\">10.1097\/MLR.0b013e31829b1dbd<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3748381\/\" target=\"_blank\">PMC3748381<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23774517\" target=\"_blank\">23774517<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3748381\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3748381<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Caveats+for+the+use+of+operational+electronic+health+record+data+in+comparative+effectiveness+research&rft.jtitle=Medical+Care&rft.aulast=Hersh%2C+W.R.%3B+Weiner%2C+M.G.%3B+Embi%2C+P.J.%3B+et+al.&rft.au=Hersh%2C+W.R.%3B+Weiner%2C+M.G.%3B+Embi%2C+P.J.%3B+et+al.&rft.date=2013&rft.volume=51&rft.issue=8+Suppl+3&rft.pages=S30%E2%80%93S37&rft_id=info:doi\/10.1097%2FMLR.0b013e31829b1dbd&rft_id=info:pmc\/PMC3748381&rft_id=info:pmid\/23774517&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3748381&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HershAStim2009-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HershAStim2009_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hersh, W.R. (2009). \"A stimulus to define informatics and health information technology\". <i>BMC Medical Informatics & Decision Making<\/i> (9): 24. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1472-6947-9-24\" target=\"_blank\">10.1186\/1472-6947-9-24<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+stimulus+to+define+informatics+and+health+information+technology&rft.jtitle=BMC+Medical+Informatics+%26+Decision+Making&rft.aulast=Hersh%2C+W.R.&rft.au=Hersh%2C+W.R.&rft.date=2009&rft.issue=9&rft.pages=24&rft_id=info:doi\/10.1186%2F1472-6947-9-24&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShekelleTheTop2013-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShekelleTheTop2013_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shekelle, P.G.; Pronovost, P.J.; Wachter, R.M.; et al. (2013). \"The top patient safety strategies that can be encouraged for adoption now\". <i>Annals of Internal Medicine<\/i> <b>158<\/b> (5 Pt 2): 365\u2013368. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7326%2F0003-4819-158-5-201303051-00001\" target=\"_blank\">10.7326\/0003-4819-158-5-201303051-00001<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23460091\" target=\"_blank\">23460091<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+top+patient+safety+strategies+that+can+be+encouraged+for+adoption+now&rft.jtitle=Annals+of+Internal+Medicine&rft.aulast=Shekelle%2C+P.G.%3B+Pronovost%2C+P.J.%3B+Wachter%2C+R.M.%3B+et+al.&rft.au=Shekelle%2C+P.G.%3B+Pronovost%2C+P.J.%3B+Wachter%2C+R.M.%3B+et+al.&rft.date=2013&rft.volume=158&rft.issue=5+Pt+2&rft.pages=365%E2%80%93368&rft_id=info:doi\/10.7326%2F0003-4819-158-5-201303051-00001&rft_id=info:pmid\/23460091&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CommHealth2012-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CommHealth2012_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Committee on Patient Safety and Health Information Technology; Board on Health Care Services; Institute of Medicine (2012). <i>Health IT and Patient Safety: Building Safer Systems for Better Care<\/i>. Washington, DC: National Academies Press. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780309221122.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Health+IT+and+Patient+Safety%3A+Building+Safer+Systems+for+Better+Care&rft.aulast=Committee+on+Patient+Safety+and+Health+Information+Technology%3B+Board+on+Health+Care+Services%3B+Institute+of+Medicine&rft.au=Committee+on+Patient+Safety+and+Health+Information+Technology%3B+Board+on+Health+Care+Services%3B+Institute+of+Medicine&rft.date=2012&rft.place=Washington%2C+DC&rft.pub=National+Academies+Press&rft.isbn=9780309221122&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RicciardiANat2013-29\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RicciardiANat2013_29-0\" rel=\"external_link\">29.0<\/a><\/sup> <sup><a href=\"#cite_ref-RicciardiANat2013_29-1\" rel=\"external_link\">29.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ricciardi, L.; Mostashari, F.; Murphy, J.; Daniel, J.G.; Siminerio, E.P. (2013). \"A national action plan to support consumer engagement via e-health\". <i>Health Affairs<\/i> <b>32<\/b> (2): 376\u2013384. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.2012.1216\" target=\"_blank\">10.1377\/hlthaff.2012.1216<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23381531\" target=\"_blank\">23381531<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+national+action+plan+to+support+consumer+engagement+via+e-health&rft.jtitle=Health+Affairs&rft.aulast=Ricciardi%2C+L.%3B+Mostashari%2C+F.%3B+Murphy%2C+J.%3B+Daniel%2C+J.G.%3B+Siminerio%2C+E.P.&rft.au=Ricciardi%2C+L.%3B+Mostashari%2C+F.%3B+Murphy%2C+J.%3B+Daniel%2C+J.G.%3B+Siminerio%2C+E.P.&rft.date=2013&rft.volume=32&rft.issue=2&rft.pages=376%E2%80%93384&rft_id=info:doi\/10.1377%2Fhlthaff.2012.1216&rft_id=info:pmid\/23381531&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-InstPop2012-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-InstPop2012_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Institute for Health Technology Transformation (2012) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ihealthtran.com\/pdf\/PHMReport.pdf\" target=\"_blank\"><i>Population Health Management \u2013 A Roadmap for Provider-Based Automation in a New Era of Healthcare<\/i><\/a>. New York, NY: Institute for Health Technology Transformation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ihealthtran.com\/pdf\/PHMReport.pdf\" target=\"_blank\">http:\/\/ihealthtran.com\/pdf\/PHMReport.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Population+Health+Management+%E2%80%93+A+Roadmap+for+Provider-Based+Automation+in+a+New+Era+of+Healthcare&rft.aulast=Institute+for+Health+Technology+Transformation&rft.au=Institute+for+Health+Technology+Transformation&rft.date=2012&rft.place=New+York%2C+NY&rft.pub=Institute+for+Health+Technology+Transformation&rft_id=http%3A%2F%2Fihealthtran.com%2Fpdf%2FPHMReport.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DiamondColl2009-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DiamondColl2009_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Diamond, C.C.; Mostashari, F.; Shirky, C. (2009). \"A national action plan to support consumer engagement via e-health\". <i>Health Affairs<\/i> <b>28<\/b> (2): 454\u2013466. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1377%2Fhlthaff.28.2.454\" target=\"_blank\">10.1377\/hlthaff.28.2.454<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19276005\" target=\"_blank\">19276005<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+national+action+plan+to+support+consumer+engagement+via+e-health&rft.jtitle=Health+Affairs&rft.aulast=Diamond%2C+C.C.%3B+Mostashari%2C+F.%3B+Shirky%2C+C.&rft.au=Diamond%2C+C.C.%3B+Mostashari%2C+F.%3B+Shirky%2C+C.&rft.date=2009&rft.volume=28&rft.issue=2&rft.pages=454%E2%80%93466&rft_id=info:doi\/10.1377%2Fhlthaff.28.2.454&rft_id=info:pmid\/19276005&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HamburgThePath2010-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HamburgThePath2010_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hamburg, M.A.; Collins, F.S. (2010). \"The path to personalized medicine\". <i>New England Journal of Medicine<\/i> <b>363<\/b> (4): 301\u2013304. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMp1006304\" target=\"_blank\">10.1056\/NEJMp1006304<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20551152\" target=\"_blank\">20551152<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+path+to+personalized+medicine&rft.jtitle=New+England+Journal+of+Medicine&rft.aulast=Hamburg%2C+M.A.%3B+Collins%2C+F.S.&rft.au=Hamburg%2C+M.A.%3B+Collins%2C+F.S.&rft.date=2010&rft.volume=363&rft.issue=4&rft.pages=301%E2%80%93304&rft_id=info:doi\/10.1056%2FNEJMp1006304&rft_id=info:pmid\/20551152&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FernaldBio2011-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FernaldBio2011_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fernald, G.H.; Capriotti, E.; Daneshjou, R.; Karczewski, K.J.; Altman, R.B. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3117361\" target=\"_blank\">\"Bioinformatics challenges for personalized medicine\"<\/a>. <i>Bioinformatics<\/i> <b>27<\/b> (13): 1741\u20131748. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtr295\" target=\"_blank\">10.1093\/bioinformatics\/btr295<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3117361\/\" target=\"_blank\">PMC3117361<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21596790\" target=\"_blank\">21596790<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3117361\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3117361<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+challenges+for+personalized+medicine&rft.jtitle=Bioinformatics&rft.aulast=Fernald%2C+G.H.%3B+Capriotti%2C+E.%3B+Daneshjou%2C+R.%3B+Karczewski%2C+K.J.%3B+Altman%2C+R.B.&rft.au=Fernald%2C+G.H.%3B+Capriotti%2C+E.%3B+Daneshjou%2C+R.%3B+Karczewski%2C+K.J.%3B+Altman%2C+R.B.&rft.date=2011&rft.volume=27&rft.issue=13&rft.pages=1741%E2%80%931748&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtr295&rft_id=info:pmc\/PMC3117361&rft_id=info:pmid\/21596790&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3117361&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ManolioGenome2010-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ManolioGenome2010_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Manolio, T. (2010). \"Genomewide association studies and assessment of the risk of disease\". <i>New England Journal of Medicine<\/i> <b>363<\/b> (2): 166\u2013176. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMra0905980\" target=\"_blank\">10.1056\/NEJMra0905980<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20647212\" target=\"_blank\">20647212<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genomewide+association+studies+and+assessment+of+the+risk+of+disease&rft.jtitle=New+England+Journal+of+Medicine&rft.aulast=Manolio%2C+T.&rft.au=Manolio%2C+T.&rft.date=2010&rft.volume=363&rft.issue=2&rft.pages=166%E2%80%93176&rft_id=info:doi\/10.1056%2FNEJMra0905980&rft_id=info:pmid\/20647212&rfr_id=info:sid\/en.wikipedia.org:Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In all cases the DOI, PubMed ID, and\/or ISBN were missing and were added to make the reference more useful.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210226\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.785 seconds\nReal time usage: 0.817 seconds\nPreprocessor visited node count: 27575\/1000000\nPreprocessor generated node count: 35781\/1000000\nPost\u2010expand include size: 228951\/2097152 bytes\nTemplate argument size: 73326\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 785.772 1 - -total\n 87.25% 685.607 1 - Template:Reflist\n 77.19% 606.507 34 - Template:Citation\/core\n 72.52% 569.811 30 - Template:Cite_journal\n 10.13% 79.594 64 - Template:Citation\/identifier\n 8.72% 68.534 4 - Template:Cite_book\n 7.70% 60.494 1 - Template:Infobox_journal_article\n 7.36% 57.862 1 - Template:Infobox\n 4.37% 34.312 80 - Template:Infobox\/row\n 4.05% 31.841 34 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7709-0!*!0!!en!5!* and timestamp 20181213210225 and revision id 23406\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education\">https:\/\/www.limswiki.org\/index.php\/Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","43a50f3587a6a8e1ea355835ec07b4a1_images":["https:\/\/www.limswiki.org\/images\/f\/f3\/Tab1_Hersh_AdvancesMedEdPrac2014_2014-5.jpg","https:\/\/www.limswiki.org\/images\/9\/90\/Tab2_Hersh_AdvancesMedEdPrac2014_2014-5.jpg"],"43a50f3587a6a8e1ea355835ec07b4a1_timestamp":1544734945,"548bc0dcb34959573b02fdcd77beb67e_type":"article","548bc0dcb34959573b02fdcd77beb67e_title":"Basics of case report form designing in clinical research (Bellary et al. 2014)","548bc0dcb34959573b02fdcd77beb67e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Basics_of_case_report_form_designing_in_clinical_research","548bc0dcb34959573b02fdcd77beb67e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Basics of case report form designing in clinical research\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nBasics of case report form designing in clinical researchJournal\n \nPerspectives in Clinical ResearchAuthor(s)\n \nBellary, Shantala; Krishnankutty, Binny; Latha, M.S.Author affiliation(s)\n \nDr. Reddy's Laboratories Ltd., Hyderabad, India; USV Limited, Mumbai, IndiaPrimary contact\n \nEmail: lathasu@gmail.comYear published\n \n2014Volume and issue\n \n5 (4)Page(s)\n \n159\u2013166DOI\n \n10.4103\/2229-3485.140555ISSN\n \n2229-3485Distribution license\n \nCreative Commons Attribution-Noncommercial-Share Alike 3.0 UnportedWebsite\n \nhttp:\/\/www.picronline.orgDownload\n \nhttp:\/\/www.picronline.org\/temp\/PerspectClinRes54159-5195597_142555.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Paper case report form versus electronic case report form \n4 Standard case report form design \n5 Well designed case report form versus poorly designed case report form \n6 Standard case report form templates \n7 Case report form connectivity\/Well referenced case report forms \n8 Challenges in case report form designing \n9 Case report form completion guidelines \n10 Conclusion \n11 Acknowledgement \n12 References \n13 Notes \n\n\n\nAbstract \nCase report form (CRF) is a specialized document in clinical research. It should be study protocol driven, robust in content and have material to collect the study specific data. Though paper CRFs are still used largely, use of electronic CRFs (eCRFs) are gaining popularity due to the advantages they offer such as improved data quality, online discrepancy management and faster database lock etc. Main objectives behind CRF development are preserving and maintaining quality and integrity of data. CRF design should be standardized to address the needs of all users such as investigator, site coordinator, study monitor, data entry personnel, medical coder and statistician. Data should be organized in a format that facilitates and simplifies data analysis. Collection of large amount of data will result in wasted resources in collecting and processing it and in many circumstances, will not be utilized for analysis. Apart from that, standard guidelines should be followed while designing the CRF. CRF completion manual should be provided to the site personnel to promote accurate data entry by them. These measures will result in reduced query generations and improved data integrity. It is recommended to establish and maintain a library of templates of standard CRF modules as they are time saving and cost-effective. This article is an attempt to describe the methods of CRF designing in clinical research and discusses the challenges encountered in this process.\nKeywords: Case report form, completion guidelines, case report form design, electronic case report form, standard templates\n\nIntroduction \nA case report form (CRF) is designed to collect the patient data in a clinical trial; its development represents a significant part of the clinical trial and can affect study success.[1] Site personnel capture the subject's data on the CRF, which is collected during their participation in a clinical trial. The International Conference on Harmonization Guidelines for Good Clinical Practice de\ufb01ne the CRF as: A printed, optical or electronic document designed to record all of the protocol - required information to be reported to the sponsor on each trial subject.[2]\nCase report form designing requires enormous planning and attention to minute detail. Designing a CRF is crucial in a clinical trial as it will aid in assessing the safety and efficacy of the medicinal product accurately. CRF should be designed for optimal collection of data in accordance with the study protocol compliance, regulatory requirements and shall enable the researcher test the hypothesis or answer the trial related questions.\nA well-designed CRF should represent the essential contents of the study protocol and in an ideal situation, CRF is designed once the study protocol is finalized. It can be prepared either concurrently along with the protocol development, but may result in many versions, and hence needs to be version controlled. Timing of the design process will also play an important role as both the approaches have pros and cons.\nIt is increasingly recognized that the design of the CRF (paper form\/electronic form) is a key quality step in ensuring the data required by the protocol, regulatory compliance and\/or safety needs\/comments, study-specific hypothesis attributes, site work flow, and cross-checking of data items within a form or across different forms are addressed.[3] The CRF used in clinical research reduces messy clinical realities to round integers and categorical answers.[4]\nThis article is an attempt to describe the methods of CRF designing in clinical research, discusses the challenges encountered and measures to be taken to prevent the occurrence of issues in its development.\n\nPaper case report form versus electronic case report form \nThere are two types of CRFs used in clinical research, that is, traditional paper CRF and improvised electronic CRF (eCRF). Paper CRF is the traditional way of data capture and a better option if studies are small or vary in design, whereas eCRFs are considered if studies are large with similar designs.[5]\nIn the current global scenario, eCRFs are preferred over paper CRFs as they are less time-consuming, and also encourage the sponsor\/pharmaceutical company to carry out large multicentric studies at the same time due to the ease of administration. It is designed in such a way that data entry can be done with zero\/minimal errors. Moreover, the regulatory authorities are readily accepting submissions in which validated electronic data capture (EDC) systems are used.[6]\nWhile designing an eCRF, repetitive data such as protocol ID, site code, subject ID, and patient initials will be generated by the system automatically from the first page to all others, thus ensuring no duplication of CRF pages.\nIn eCRF, linking the data between two related pages of CRFs becomes easy and quick. They have built-in edit checks tagged to each data field as well as to the CRF as a whole. Therefore, majority of data cleaning activities will take place during the completion of the eCRFs, thus reducing the time and effort required by data management personnel. Instant query resolution reduces the time spent on obtaining the clarification from the site\/investigator and hence, clean data is obtained much quickly, resulting in timely database lock, faster regulatory submission, and subsequent approval.\n\nStandard case report form design \nDesigning a paper CRF is a tedious job that could result in data errors and wrong conclusions, requiring meticulous attention to minimize duplication of CRF pages. Chances of error during data transfer from the source document to paper CRF are common. Moreover, for studies with large sample size if traditional method of data collection through paper CRFs is opted, then manual data cleaning may be a major concern. However, this method may not require user training and system validation as in the case of EDC systems, where such things are essential before implementing it. Despite their many advantages, eCRFs have not been accepted widely. Main reasons behind this are lack of available on-site technology, investigators' lack of motivation, complexity of installation, and maintenance of the software and high investment cost.\nDesigning a CRF is an art that should to be based on scientific practices and the design should be implemented keeping the end-user (the one who enters data in the CRF) in mind. While designing, all important sections of the CRF should be included with care; always it is worth to remember that insufficient\/inaccurate data collection would prove expensive during analysis. Hence, it is advisable to have a standard operating procedure for CRF preparation and to follow best practices of CRF designing.\nPrimary objective of CRF designing is to gather complete and accurate data by avoiding duplication and facilitating transcription of data from source documents onto the CRF. CRF should be designed with the primary safety and efficacy endpoints as the main goal of data collection.[6] Ideally, it should be well-structured, easy to complete without much assistance and should collect data of the highest quality. Always minimum amount of data needed to answer the study hypotheses should be collected avoiding collection of elaborate, unimportant information. For ordinal data, to ensure uniformity and clarity among raters, adequate explanation should be provided adjacent to the CRF fields. Capturing the same piece of data in more than one place (duplication) on the CRF should also be avoided. In other words, CRF should collect data in sufficient detail without ambiguity and at the same time, should avoid redundancy and avoid capture of unwanted details. Hence, striking the perfect chords to ensure balance between effective data collection and structuring the CRF to support accurate data entry is essential. Collecting the data in the coded form whenever possible is ideal as it facilitates data entry (at CRF and at the database levels) and helps the statistician in data interpretation and analysis.\nImportant part of the CRF is an informative header and footer, which can be customized.[7]\nIn general, the header includes protocol ID, site code, subject ID, and patient initials. Whereas, the footer includes investigator's signature, date of signature, version number, and page number.\nIn order to enhance easy reading\/understanding and accurate data entry, an uncrowded CRF layout should be preferred. Placing too many details on the same page, makes the CRF look cluttered and makes data entry difficult, which eventually leads to increase in data discrepancies.\nCase report form design should be standardized to address the needs of all those who handle the data such as investigator, data manager, biostatistician, clinical research monitor\/coordinator, database developer\/programmer and data entry personnel etc. An effective CRF design would always be user friendly. Moreover, it should capture legible, consistent and valid data, thereby, reducing query generations.[7] While designing the CRFs, design standards should be adhered to for improving the quality of data collected. Hence, data should be organized in a format that facilitates data analysis and makes it simplified.\nThe following points are to be borne in mind while designing a CRF:\n\n Use of consistent formats, font style and font sizes throughout the CRF booklet\n Selection of portrait versus landscape versus combination layouts\n Use of clear and concise questions, prompts, and instructions\n Visual cues, such as boxes that clearly indicate place and format of data to be recorded should be provided to the person recording the data as much as possible\n Using the option of \"circling of answers\" should be limited as it's hard to interpret; instead check boxes would be appropriate\n Clear guidance about skip patterns like what to skip and what not to skip should be mentioned at appropriate places\n Skips (are instructions provided in the CRF page to maintain the connectivity between pages) should be kept to a minimum by the placement of questions to avoid confusions\n Provide boxes or separate lines to hold the answers. This indirectly informs the data recorder where to write\/enter the response and helps to differentiate it visually from the entry fields for other questions\n Separate the columns with thick lines\n Provide bold and italicized instructions\n Minimize free text responses\n Position only specified density of questions on each page\n Page numbering if necessary, should be consistent throughout\n Avoid using \"check all that apply\" as it forces assumptions about the clinical data\n Specify the unit of measurement\n Indicate the number of decimal places to be recorded\n Use standard data format (e.g., dd\/mm\/yyyy) throughout the CRF\n Use precoded answer sets such as yes\/no, male\/female, method of administration of medicine, and severity of adverse event (AE) (mild\/moderate\/severe) wherever possible\n Not to split modules\/sections (a set of one or more related groups of questions that pertain to a single clinical study visit) like, for example, AE section should not be split and laid across pages such that information related to a single AE will have to be collected from different pages\n Use \"no carbon required (NCR)\" copies to ensure exact replica of CRF\n Use instructions including page numbers where data has to be entered (e.g., during a follow-up visit, the investigator is supposed to record whether any AE has occurred and if occurred, details of the AE has to be recorded in the AE module. Hence, the field corresponding to this question on the module for the particular visit would be having the options \"yes\" or \"no\". There should be an instruction \"If 'yes', please provide the information in the AEs page (page no. XX)\" as shown in Figure 1. \n\r\n\n\n\n\nFigure 1. A sample case report form (CRF) page. An adverse event page\r\nof CRF is depicted showing codes, and skips questions\nWell designed case report form versus poorly designed case report form \nTable 1 provides the comparison between well-designed and poorly designed CRF. In case of poorly designed CRF, by placing a single line for required response results in variations in the investigator's responses from site to site. On the other hand, separate lines and boxes are provided in the well-designed form, which gives the visual cues about what is expected as a response and thereby, reduces the unnecessary queries. Usually, boxes are used for entering dates and the date format (i.e., dd\/mm\/yyyy ) should be consistent on all pages of CRF. Units and decimal points should be displayed for vital sign records, which clarify the user about the expected values and also facilitates the data interpretation and reduces manipulation during analysis procedures. Figure 2 and Figure 3 are examples of poorly designed CRFs. Poor CRF design results in frequent database modification thus affecting the study timelines. Data need to be collected in a way that does not introduce bias or errors. Collection of a large amount of data will result in wasted resources in collecting and processing it. Questions in the CRF should be clear and unambiguous to avoid unnecessary confusions.[8]\n\n\n\nTable 1. Illustrating a well-designed and poorly designed data fields imparting\r\nthe significance of visual cues to help the site personnel to understand the format\n\n\n\nFigure 2. Example of a poorly designed case report form\n\n\n\nFigure 3. Illustrating the missing indicator question\nIn some circumstances, data can be obtained using derivation procedures; collection of derived data again on the CRF should be avoided to minimize calculation errors. For example, age can be calculated using date of birth. Body mass index can be calculated using height and weight of the subject, only the latter two should be captured.\nIn conditions where same parameters are to be recorded at multiple visits, it is recommended to use the same CRF module for each visit to reduce the number of query generation. For example, vital Signs and body systems in the physical examination (PE) module can be collected in the same order each time.\nIn some places, answers are coded in order to simplify the data collection. When codes are used to obtain an answer for a question, consistency in codes should be maintained throughout the CRF booklet and there should not be any variation in the answer for the same question.\nFor example, yes\/no answers are coded as 1 = yes and 2 = no (preferred coding) as shown in Figure 4. If the codes are assigned in this order, the same order should be practiced throughout the CRF. Nowhere in the same CRF \"1\" should be coded for \"no\" and \"2\" should be coded for \"yes\". \n\r\n\n\n\n\nFigure 4. Coding on the case report form module\nEven the location of these codes on the CRF should be consistent; same is shown in Figure 4. Clear instructions should be provided to the user where ever necessary; otherwise, it will have a significant impact on the data management activities like database designing, data cleaning, data validation, and data extraction due to poor understanding of the site personnel about the expected responses. It is advisable to use indicator questions wherever needed to avoid assumptions about the data. Use indicator questions in connection to a set of other questions, and the response to the indicator question would decide on whether the associated set of questions needs to be answered or not. For example, in an AE question group, an indicator question could be, \"Did any AE occur after the last visit?\"\nIf the response is \"yes\", the remaining questions pertaining to the details of the AE(s) (such as severity, seriousness, causality, date of onset, date of resolution, and action taken) require responses. If the response is \"no\", the rest of the question group is not answered. Incorporation of an indicator question and skips are shown in Figure 1. Ideally, CRF booklet should have a chart reflecting the expected list of assessments as per schedule specified in the protocol.\n\nStandard case report form templates \nSome of the data requirements such as demography, PE, AEs are same across studies, so standard CRF templates should be developed which can be customized accordingly. These templates are of great help while conducting multiple studies in the same research area. These templates will have the same design principles that help the user to enter data with ease since the design is familiar to them; there is no need for special training on these modules of CRFs.\nA \"library\" of standard templates should be established and maintained by the sponsor\/contract research organizations, pharmaceutical companies in order to maintain uniformity in the CRF design and to save time. Most commonly used standard CRF templates are inclusion criteria, exclusion criteria, demography, medical history, PE, AE, concomitant medication and study outcome modules, whereas, the modules which captures efficacy data are not unique. Their design varies from study to study depending on the protocol specifications.\n\nCase report form connectivity\/Well referenced case report forms \nLinking of CRF (paper CRF and eCRF) pages wherever necessary is known as CRF connectivity. Each CRF booklet is assigned with unique subject ID and it is the duty of site personnel to make sure that same ID is entered on all pages of CRF booklet. Consistently entered subject ID will help in tracking the missing CRF pages. The fields such as protocol ID, site code, subject ID, and patient initials make database designing easier and helps linking CRF pages to the study database. The fields like protocol ID and visit labels are informative features as they provide brief descriptions of the study and the schedule of assessments, respectively. The CRF version number is a critical field that prevents an incorrect CRF page being used. All pages of the CRF booklet should be numbered in sequential order, which will help in identifying queries through data validation procedures and manual reviews. Page numbering not only provides the site personnel with a quick reference to specific pages, but also helps to design the database in a structured manner. Especially, in case of eCRF, retrieving of CRFs becomes challenging if proper programming is not carried out. CRF connectivity is crucial when statistical analysis plan (SAP) is complex and these fields will be of help in statistical analysis.\n\nChallenges in case report form designing \nCommonly encountered challenges in CRF designing are consistency in the design, collection of precise data and user-friendliness. These challenges can be overcome by proper planning by a team of data management personnel, biostatisticians, clinicians, and medical writers. Objectives should be defined clearly before designing. Consistent design is a crucial aspect as it reduces the number of mistakes in data entry. It is of great advantage when using them across various studies. Maintaining standard CRF templates would resolve this issue. Collection of extraneous data is another issue and measures should be taken to avoid it, as processing this becomes tedious. In such instances, ensuring accuracy and quality become major challenges. Attention should be paid to avoid duplication. Design the CRF to avoid referential and redundant data collection. For example, collecting calculated fields\/derivable data should be avoided and to ensure that data collection is cost-effective. Designing user-friendly CRF to reduce data entry errors is again a challenge. Simple\/standard designs should be incorporated wherever possible.\nUser feedback mechanism should be built into the CRF design and maintenance process. Best practices should be applied to improve the data quality and save time with CRF design. Providing CRF completion guideline aids in minimizing the challenges in data capture and data entry.\n\nCase report form completion guidelines \nA CRF completion guideline is a document to assist the investigator to complete the CRF in a step by step manner and is drafted concurrently in line with the CRF and protocol. Figure 5 shows sample page of CRF completion guideline. There is no standard template for CRF completion guidelines as it is study specific. It should be prepared in such a way that it enables the site personnel to complete the CRFs with ease and legibility. CRF completion manual should provide clear instructions to site personnel for accurate completion of CRFs along with clear expectations including proper instructions on handling unknown data. For example, if exact date is unknown, then use a preferred notation in the place of missing value (i.e., UK\/UNK\/2012). The language used should be simple with clear instructions, concise, and easy to understand. \n\n\n\nFigure 5. Sample page of case report form completion guideline\n\r\n\nCase report form completion guidelines document, while bridging the gap between the study protocol and the data collection process, explains the activities involved in CRF completion, correction, signing, and data handling.[7] It provides unambiguous instructions on CRF completion in all practical scenarios. For example, if data were wrongly entered and the site personnel wants to correct it, the instruction provided would be to strike-through the incorrect data with a single line, put the initials (of the person who makes the change) with date and to write the correct entry in the margin against the corresponding line. Similarly, instructions will be provided for each data entry field on each page in the CRF booklet. This helps to ensure completion of all required data fields and enhances the data flow.[7] CRF completion guidelines could be a separate document or could be a part of the CRF booklet giving page by page instructions. If it is included as part of the CRF, it is advisable to print instructions on the page facing the CRF page (back side of the previous page) as the investigator can easily take instructions and simultaneously fill the CRF page. CRF completion guidelines document should have version control and amendments should be done as and when required.\n\nConclusion \nCase report form design is the initial step in translating the protocol into standard questionnaires and is paramount to a successful clinical trial. Standard CRF should be designed in such a way that it helps the collection of consistent and valid data, ultimately resulting in submission of data to regulatory authorities and its acceptance. Regardless of the time and effort spent conducting the trial, the correct data points (response to a CRF question\/data is entered) must be collected; otherwise, a meaningful analysis may not be possible. Therefore, a sound SAP should be used as a tool to develop and judge the adequacy of the CRF, which should be available to guide on what data points need to be captured on the CRF. To avoid future amendments, it is important to have design principles in mind well in advance before CRF designing is initiated. These standard guidelines will contribute in preparing a well-designed CRF for data acquisition.\n\nAcknowledgement \nWe would like to acknowledge the technical support offered by Mr. Vinoth T. and Mr. Sagi Subbaraju that has helped us during the preparation of this article.\n\nReferences \n\n\u2191 Nahm, M.; Shepherd, J.; Buzenberg, A.; Rostami, R.; Corcoran, A.; McCall, J. et al. (2011). \"Design and implementation of an institutional case report form library\". Clinical Trials 8 (1): 94\u2013102. doi:10.1177\/1740774510391916. PMC PMC3494996. PMID 21163853. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3494996 .   \n\n\u2191 ICH Steering Committee (1996). \"Guidance for Industry - E6 Good Clinical Practice: Consolidated Guidance\" (PDF). U.S. Food and Drug Administration. http:\/\/www.fda.gov\/downloads\/Drugs\/Guidances\/ucm073122.pdf . Retrieved 11 June 2013 .   \n\n\u2191 Lu, Z.; Su, J. (June 2010). \"Design and implementation of an institutional case report form library\". Open Access Journal of Clinical Trials 2010 (2): 93\u2014105. doi:10.2147\/OAJCT.S8172.   \n\n\u2191 Latimer, P. (May 2008). \"Case report form insanity\" (PDF). Journal of Clinical Research Best Practices 4 (5). http:\/\/www.firstclinical.com\/journal\/2008\/0805_Insanity.pdf . Retrieved 17 June 2013 .   \n\n\u2191 Stark, N.J. (October 2011). \"Data management in device studies\" (PDF). Journal of Clinical Research Best Practices 7 (10). http:\/\/www.firstclinical.com\/journal\/2011\/1110_Data_Management.pdf . Retrieved 17 June 2013 .   \n\n\u2191 6.0 6.1 \"Good Clinical Data Management Practices\" (PDF). Society for Clinical Data Management. October 2009. http:\/\/www.swaggerinfosolutions.pro\/sridevi\/New%20Folder\/ICRI\/GCDMP_complete_OCT2009.pdf . Retrieved 17 June 2013 .   \n\n\u2191 7.0 7.1 7.2 7.3 Moon, K.K. (Fall 2006). \"Techniques for designing case report forms in clinical trials: Considerations for efficient data management and statistical analysis\" (PDF). ScianNews 9 (1). Archived from the original on 12 May 2013. https:\/\/web.archive.org\/web\/20130512162105\/http:\/\/www.mycro.ca\/media\/1533\/sciannews91b.pdf .   \n\n\u2191 CDISC CDASH Core and Domain Teams (01 October 2008). \"Clinical Data Acquisition Standards Harmonization (CDASH)\" (PDF). CDISC, Inc. http:\/\/www.cdisc.org\/system\/files\/all\/standard\/application\/pdf\/cdash_std_1_0_2008_10_01.pdf . Retrieved 11 June 2013 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Basics_of_case_report_form_designing_in_clinical_research\">https:\/\/www.limswiki.org\/index.php\/Journal:Basics_of_case_report_form_designing_in_clinical_research<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on clinical researchLIMSwiki journal articles on health informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:51.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,589 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","548bc0dcb34959573b02fdcd77beb67e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Basics_of_case_report_form_designing_in_clinical_research skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Basics of case report form designing in clinical research<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Case report form (CRF) is a specialized document in clinical research. It should be study protocol driven, robust in content and have material to collect the study specific data. Though paper CRFs are still used largely, use of electronic CRFs (eCRFs) are gaining popularity due to the advantages they offer such as improved data quality, online discrepancy management and faster database lock etc. Main objectives behind CRF development are preserving and maintaining quality and integrity of data. CRF design should be standardized to address the needs of all users such as investigator, site coordinator, study monitor, data entry personnel, medical coder and statistician. Data should be organized in a format that facilitates and simplifies data analysis. Collection of large amount of data will result in wasted resources in collecting and processing it and in many circumstances, will not be utilized for analysis. Apart from that, standard guidelines should be followed while designing the CRF. CRF completion manual should be provided to the site personnel to promote accurate data entry by them. These measures will result in reduced query generations and improved data integrity. It is recommended to establish and maintain a library of templates of standard CRF modules as they are time saving and cost-effective. This article is an attempt to describe the methods of CRF designing in clinical research and discusses the challenges encountered in this process.\n<\/p><p><b>Keywords<\/b>: Case report form, completion guidelines, case report form design, electronic case report form, standard templates\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>A case report form (CRF) is designed to collect the patient data in a clinical trial; its development represents a significant part of the clinical trial and can affect study success.<sup id=\"rdp-ebb-cite_ref-NahmDes11_1-0\" class=\"reference\"><a href=\"#cite_note-NahmDes11-1\" rel=\"external_link\">[1]<\/a><\/sup> Site personnel capture the subject's data on the CRF, which is collected during their participation in a clinical trial. The International Conference on Harmonization Guidelines for Good Clinical Practice de\ufb01ne the CRF as: A printed, optical or electronic document designed to record all of the protocol - required <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> to be reported to the sponsor on each trial subject.<sup id=\"rdp-ebb-cite_ref-ICHGuidance96_2-0\" class=\"reference\"><a href=\"#cite_note-ICHGuidance96-2\" rel=\"external_link\">[2]<\/a><\/sup>\n<\/p><p>Case report form designing requires enormous planning and attention to minute detail. Designing a CRF is crucial in a clinical trial as it will aid in assessing the safety and efficacy of the medicinal product accurately. CRF should be designed for optimal collection of data in accordance with the study protocol compliance, regulatory requirements and shall enable the researcher test the hypothesis or answer the trial related questions.\n<\/p><p>A well-designed CRF should represent the essential contents of the study protocol and in an ideal situation, CRF is designed once the study protocol is finalized. It can be prepared either concurrently along with the protocol development, but may result in many versions, and hence needs to be version controlled. Timing of the design process will also play an important role as both the approaches have pros and cons.\n<\/p><p>It is increasingly recognized that the design of the CRF (paper form\/electronic form) is a key quality step in ensuring the data required by the protocol, regulatory compliance and\/or safety needs\/comments, study-specific hypothesis attributes, site work flow, and cross-checking of data items within a form or across different forms are addressed.<sup id=\"rdp-ebb-cite_ref-LuClinical10_3-0\" class=\"reference\"><a href=\"#cite_note-LuClinical10-3\" rel=\"external_link\">[3]<\/a><\/sup> The CRF used in clinical research reduces messy clinical realities to round integers and categorical answers.<sup id=\"rdp-ebb-cite_ref-LatimerCase08_4-0\" class=\"reference\"><a href=\"#cite_note-LatimerCase08-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/p><p>This article is an attempt to describe the methods of CRF designing in clinical research, discusses the challenges encountered and measures to be taken to prevent the occurrence of issues in its development.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Paper_case_report_form_versus_electronic_case_report_form\">Paper case report form versus electronic case report form<\/span><\/h2>\n<p>There are two types of CRFs used in clinical research, that is, traditional paper CRF and improvised electronic CRF (eCRF). Paper CRF is the traditional way of data capture and a better option if studies are small or vary in design, whereas eCRFs are considered if studies are large with similar designs.<sup id=\"rdp-ebb-cite_ref-StarkData11_5-0\" class=\"reference\"><a href=\"#cite_note-StarkData11-5\" rel=\"external_link\">[5]<\/a><\/sup>\n<\/p><p>In the current global scenario, eCRFs are preferred over paper CRFs as they are less time-consuming, and also encourage the sponsor\/pharmaceutical company to carry out large multicentric studies at the same time due to the ease of administration. It is designed in such a way that data entry can be done with zero\/minimal errors. Moreover, the regulatory authorities are readily accepting submissions in which validated electronic data capture (EDC) systems are used.<sup id=\"rdp-ebb-cite_ref-SCDMGood09_6-0\" class=\"reference\"><a href=\"#cite_note-SCDMGood09-6\" rel=\"external_link\">[6]<\/a><\/sup>\n<\/p><p>While designing an eCRF, repetitive data such as protocol ID, site code, subject ID, and patient initials will be generated by the system automatically from the first page to all others, thus ensuring no duplication of CRF pages.\n<\/p><p>In eCRF, linking the data between two related pages of CRFs becomes easy and quick. They have built-in edit checks tagged to each data field as well as to the CRF as a whole. Therefore, majority of data cleaning activities will take place during the completion of the eCRFs, thus reducing the time and effort required by data management personnel. Instant query resolution reduces the time spent on obtaining the clarification from the site\/investigator and hence, clean data is obtained much quickly, resulting in timely database lock, faster regulatory submission, and subsequent approval.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Standard_case_report_form_design\">Standard case report form design<\/span><\/h2>\n<p>Designing a paper CRF is a tedious job that could result in data errors and wrong conclusions, requiring meticulous attention to minimize duplication of CRF pages. Chances of error during data transfer from the source document to paper CRF are common. Moreover, for studies with large sample size if traditional method of data collection through paper CRFs is opted, then manual data cleaning may be a major concern. However, this method may not require user training and system validation as in the case of EDC systems, where such things are essential before implementing it. Despite their many advantages, eCRFs have not been accepted widely. Main reasons behind this are lack of available on-site technology, investigators' lack of motivation, complexity of installation, and maintenance of the software and high investment cost.\n<\/p><p>Designing a CRF is an art that should to be based on scientific practices and the design should be implemented keeping the end-user (the one who enters data in the CRF) in mind. While designing, all important sections of the CRF should be included with care; always it is worth to remember that insufficient\/inaccurate data collection would prove expensive during analysis. Hence, it is advisable to have a standard operating procedure for CRF preparation and to follow best practices of CRF designing.\n<\/p><p>Primary objective of CRF designing is to gather complete and accurate data by avoiding duplication and facilitating transcription of data from source documents onto the CRF. CRF should be designed with the primary safety and efficacy endpoints as the main goal of data collection.<sup id=\"rdp-ebb-cite_ref-SCDMGood09_6-1\" class=\"reference\"><a href=\"#cite_note-SCDMGood09-6\" rel=\"external_link\">[6]<\/a><\/sup> Ideally, it should be well-structured, easy to complete without much assistance and should collect data of the highest quality. Always minimum amount of data needed to answer the study hypotheses should be collected avoiding collection of elaborate, unimportant information. For ordinal data, to ensure uniformity and clarity among raters, adequate explanation should be provided adjacent to the CRF fields. Capturing the same piece of data in more than one place (duplication) on the CRF should also be avoided. In other words, CRF should collect data in sufficient detail without ambiguity and at the same time, should avoid redundancy and avoid capture of unwanted details. Hence, striking the perfect chords to ensure balance between effective data collection and structuring the CRF to support accurate data entry is essential. Collecting the data in the coded form whenever possible is ideal as it facilitates data entry (at CRF and at the database levels) and helps the statistician in data interpretation and analysis.\n<\/p><p>Important part of the CRF is an informative header and footer, which can be customized.<sup id=\"rdp-ebb-cite_ref-MoonTech06_7-0\" class=\"reference\"><a href=\"#cite_note-MoonTech06-7\" rel=\"external_link\">[7]<\/a><\/sup>\n<\/p><p>In general, the header includes protocol ID, site code, subject ID, and patient initials. Whereas, the footer includes investigator's signature, date of signature, version number, and page number.\n<\/p><p>In order to enhance easy reading\/understanding and accurate data entry, an uncrowded CRF layout should be preferred. Placing too many details on the same page, makes the CRF look cluttered and makes data entry difficult, which eventually leads to increase in data discrepancies.\n<\/p><p>Case report form design should be standardized to address the needs of all those who handle the data such as investigator, data manager, biostatistician, clinical research monitor\/coordinator, database developer\/programmer and data entry personnel etc. An effective CRF design would always be user friendly. Moreover, it should capture legible, consistent and valid data, thereby, reducing query generations.<sup id=\"rdp-ebb-cite_ref-MoonTech06_7-1\" class=\"reference\"><a href=\"#cite_note-MoonTech06-7\" rel=\"external_link\">[7]<\/a><\/sup> While designing the CRFs, design standards should be adhered to for improving the quality of data collected. Hence, data should be organized in a format that facilitates data analysis and makes it simplified.\n<\/p><p>The following points are to be borne in mind while designing a CRF:\n<\/p>\n<ul><li> Use of consistent formats, font style and font sizes throughout the CRF booklet<\/li>\n<li> Selection of portrait versus landscape versus combination layouts<\/li>\n<li> Use of clear and concise questions, prompts, and instructions<\/li>\n<li> Visual cues, such as boxes that clearly indicate place and format of data to be recorded should be provided to the person recording the data as much as possible<\/li>\n<li> Using the option of \"circling of answers\" should be limited as it's hard to interpret; instead check boxes would be appropriate<\/li>\n<li> Clear guidance about skip patterns like what to skip and what not to skip should be mentioned at appropriate places<\/li>\n<li> Skips (are instructions provided in the CRF page to maintain the connectivity between pages) should be kept to a minimum by the placement of questions to avoid confusions<\/li>\n<li> Provide boxes or separate lines to hold the answers. This indirectly informs the data recorder where to write\/enter the response and helps to differentiate it visually from the entry fields for other questions<\/li>\n<li> Separate the columns with thick lines<\/li>\n<li> Provide bold and italicized instructions<\/li>\n<li> Minimize free text responses<\/li>\n<li> Position only specified density of questions on each page<\/li>\n<li> Page numbering if necessary, should be consistent throughout<\/li>\n<li> Avoid using \"check all that apply\" as it forces assumptions about the clinical data<\/li>\n<li> Specify the unit of measurement<\/li>\n<li> Indicate the number of decimal places to be recorded<\/li>\n<li> Use standard data format (e.g., dd\/mm\/yyyy) throughout the CRF<\/li>\n<li> Use precoded answer sets such as yes\/no, male\/female, method of administration of medicine, and severity of adverse event (AE) (mild\/moderate\/severe) wherever possible<\/li>\n<li> Not to split modules\/sections (a set of one or more related groups of questions that pertain to a single clinical study visit) like, for example, AE section should not be split and laid across pages such that information related to a single AE will have to be collected from different pages<\/li>\n<li> Use \"no carbon required (NCR)\" copies to ensure exact replica of CRF<\/li>\n<li> Use instructions including page numbers where data has to be entered (e.g., during a follow-up visit, the investigator is supposed to record whether any AE has occurred and if occurred, details of the AE has to be recorded in the AE module. Hence, the field corresponding to this question on the module for the particular visit would be having the options \"yes\" or \"no\". There should be an instruction \"If 'yes', please provide the information in the AEs page (page no. XX)\" as shown in Figure 1. <\/li><\/ul>\n<p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Bellary_PerspectivesClinRes2014_5-4.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"40686be403ea918c4e7356c644767af0\"><img alt=\"Fig1 Bellary PerspectivesClinRes2014 5-4.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f5\/Fig1_Bellary_PerspectivesClinRes2014_5-4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> A sample case report form (CRF) page. An adverse event page<br \/>of CRF is depicted showing codes, and skips questions<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Well_designed_case_report_form_versus_poorly_designed_case_report_form\">Well designed case report form versus poorly designed case report form<\/span><\/h2>\n<p>Table 1 provides the comparison between well-designed and poorly designed CRF. In case of poorly designed CRF, by placing a single line for required response results in variations in the investigator's responses from site to site. On the other hand, separate lines and boxes are provided in the well-designed form, which gives the visual cues about what is expected as a response and thereby, reduces the unnecessary queries. Usually, boxes are used for entering dates and the date format (i.e., dd\/mm\/yyyy ) should be consistent on all pages of CRF. Units and decimal points should be displayed for vital sign records, which clarify the user about the expected values and also facilitates the data interpretation and reduces manipulation during analysis procedures. Figure 2 and Figure 3 are examples of poorly designed CRFs. Poor CRF design results in frequent database modification thus affecting the study timelines. Data need to be collected in a way that does not introduce bias or errors. Collection of a large amount of data will result in wasted resources in collecting and processing it. Questions in the CRF should be clear and unambiguous to avoid unnecessary confusions.<sup id=\"rdp-ebb-cite_ref-CDISCClinical08_8-0\" class=\"reference\"><a href=\"#cite_note-CDISCClinical08-8\" rel=\"external_link\">[8]<\/a><\/sup>\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Tab1_Bellary_PerspectivesClinRes2014_5-4.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5929378c4a7058a43eae4d6c3917512c\"><img alt=\"Tab1 Bellary PerspectivesClinRes2014 5-4.jpg\" src=\"https:\/\/www.limswiki.org\/images\/1\/1d\/Tab1_Bellary_PerspectivesClinRes2014_5-4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 1.<\/b> Illustrating a well-designed and poorly designed data fields imparting<br \/>the significance of visual cues to help the site personnel to understand the format<\/blockquote>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Bellary_PerspectivesClinRes2014_5-4.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"9f5eee44d9bdcfe4f87e92a9f30b1e08\"><img alt=\"Fig2 Bellary PerspectivesClinRes2014 5-4.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/ff\/Fig2_Bellary_PerspectivesClinRes2014_5-4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> Example of a poorly designed case report form<\/blockquote>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Bellary_PerspectivesClinRes2014_5-4.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"83a8e46ea0f8ab3c254b36934356a353\"><img alt=\"Fig3 Bellary PerspectivesClinRes2014 5-4.jpg\" src=\"https:\/\/www.limswiki.org\/images\/5\/5e\/Fig3_Bellary_PerspectivesClinRes2014_5-4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3.<\/b> Illustrating the missing indicator question<\/blockquote>\n<p>In some circumstances, data can be obtained using derivation procedures; collection of derived data again on the CRF should be avoided to minimize calculation errors. For example, age can be calculated using date of birth. Body mass index can be calculated using height and weight of the subject, only the latter two should be captured.\n<\/p><p>In conditions where same parameters are to be recorded at multiple visits, it is recommended to use the same CRF module for each visit to reduce the number of query generation. For example, vital Signs and body systems in the physical examination (PE) module can be collected in the same order each time.\n<\/p><p>In some places, answers are coded in order to simplify the data collection. When codes are used to obtain an answer for a question, consistency in codes should be maintained throughout the CRF booklet and there should not be any variation in the answer for the same question.\n<\/p><p>For example, yes\/no answers are coded as 1 = yes and 2 = no (preferred coding) as shown in Figure 4. If the codes are assigned in this order, the same order should be practiced throughout the CRF. Nowhere in the same CRF \"1\" should be coded for \"no\" and \"2\" should be coded for \"yes\". \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Bellary_PerspectivesClinRes2014_5-4.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"df9091b39878a47ab20a7b52fb0597ed\"><img alt=\"Fig4 Bellary PerspectivesClinRes2014 5-4.jpg\" src=\"https:\/\/www.limswiki.org\/images\/d\/d6\/Fig4_Bellary_PerspectivesClinRes2014_5-4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4.<\/b> Coding on the case report form module<\/blockquote>\n<p>Even the location of these codes on the CRF should be consistent; same is shown in Figure 4. Clear instructions should be provided to the user where ever necessary; otherwise, it will have a significant impact on the data management activities like database designing, data cleaning, data validation, and data extraction due to poor understanding of the site personnel about the expected responses. It is advisable to use indicator questions wherever needed to avoid assumptions about the data. Use indicator questions in connection to a set of other questions, and the response to the indicator question would decide on whether the associated set of questions needs to be answered or not. For example, in an AE question group, an indicator question could be, \"Did any AE occur after the last visit?\"\n<\/p><p>If the response is \"yes\", the remaining questions pertaining to the details of the AE(s) (such as severity, seriousness, causality, date of onset, date of resolution, and action taken) require responses. If the response is \"no\", the rest of the question group is not answered. Incorporation of an indicator question and skips are shown in Figure 1. Ideally, CRF booklet should have a chart reflecting the expected list of assessments as per schedule specified in the protocol.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Standard_case_report_form_templates\">Standard case report form templates<\/span><\/h2>\n<p>Some of the data requirements such as demography, PE, AEs are same across studies, so standard CRF templates should be developed which can be customized accordingly. These templates are of great help while conducting multiple studies in the same research area. These templates will have the same design principles that help the user to enter data with ease since the design is familiar to them; there is no need for special training on these modules of CRFs.\n<\/p><p>A \"library\" of standard templates should be established and maintained by the sponsor\/contract research organizations, pharmaceutical companies in order to maintain uniformity in the CRF design and to save time. Most commonly used standard CRF templates are inclusion criteria, exclusion criteria, demography, medical history, PE, AE, concomitant medication and study outcome modules, whereas, the modules which captures efficacy data are not unique. Their design varies from study to study depending on the protocol specifications.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Case_report_form_connectivity.2FWell_referenced_case_report_forms\">Case report form connectivity\/Well referenced case report forms<\/span><\/h2>\n<p>Linking of CRF (paper CRF and eCRF) pages wherever necessary is known as CRF connectivity. Each CRF booklet is assigned with unique subject ID and it is the duty of site personnel to make sure that same ID is entered on all pages of CRF booklet. Consistently entered subject ID will help in tracking the missing CRF pages. The fields such as protocol ID, site code, subject ID, and patient initials make database designing easier and helps linking CRF pages to the study database. The fields like protocol ID and visit labels are informative features as they provide brief descriptions of the study and the schedule of assessments, respectively. The CRF version number is a critical field that prevents an incorrect CRF page being used. All pages of the CRF booklet should be numbered in sequential order, which will help in identifying queries through data validation procedures and manual reviews. Page numbering not only provides the site personnel with a quick reference to specific pages, but also helps to design the database in a structured manner. Especially, in case of eCRF, retrieving of CRFs becomes challenging if proper programming is not carried out. CRF connectivity is crucial when statistical analysis plan (SAP) is complex and these fields will be of help in statistical analysis.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Challenges_in_case_report_form_designing\">Challenges in case report form designing<\/span><\/h2>\n<p>Commonly encountered challenges in CRF designing are consistency in the design, collection of precise data and user-friendliness. These challenges can be overcome by proper planning by a team of data management personnel, biostatisticians, clinicians, and medical writers. Objectives should be defined clearly before designing. Consistent design is a crucial aspect as it reduces the number of mistakes in data entry. It is of great advantage when using them across various studies. Maintaining standard CRF templates would resolve this issue. Collection of extraneous data is another issue and measures should be taken to avoid it, as processing this becomes tedious. In such instances, ensuring accuracy and quality become major challenges. Attention should be paid to avoid duplication. Design the CRF to avoid referential and redundant data collection. For example, collecting calculated fields\/derivable data should be avoided and to ensure that data collection is cost-effective. Designing user-friendly CRF to reduce data entry errors is again a challenge. Simple\/standard designs should be incorporated wherever possible.\n<\/p><p>User feedback mechanism should be built into the CRF design and maintenance process. Best practices should be applied to improve the data quality and save time with CRF design. Providing CRF completion guideline aids in minimizing the challenges in data capture and data entry.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Case_report_form_completion_guidelines\">Case report form completion guidelines<\/span><\/h2>\n<p>A CRF completion guideline is a document to assist the investigator to complete the CRF in a step by step manner and is drafted concurrently in line with the CRF and protocol. Figure 5 shows sample page of CRF completion guideline. There is no standard template for CRF completion guidelines as it is study specific. It should be prepared in such a way that it enables the site personnel to complete the CRFs with ease and legibility. CRF completion manual should provide clear instructions to site personnel for accurate completion of CRFs along with clear expectations including proper instructions on handling unknown data. For example, if exact date is unknown, then use a preferred notation in the place of missing value (i.e., UK\/UNK\/2012). The language used should be simple with clear instructions, concise, and easy to understand. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Bellary_PerspectivesClinRes2014_5-4.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"3b5483493369bc5a9ba2691d2a2e8b15\"><img alt=\"Fig5 Bellary PerspectivesClinRes2014 5-4.jpg\" src=\"https:\/\/www.limswiki.org\/images\/1\/17\/Fig5_Bellary_PerspectivesClinRes2014_5-4.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 5.<\/b> Sample page of case report form completion guideline<\/blockquote>\n<p><br \/>\nCase report form completion guidelines document, while bridging the gap between the study protocol and the data collection process, explains the activities involved in CRF completion, correction, signing, and data handling.<sup id=\"rdp-ebb-cite_ref-MoonTech06_7-2\" class=\"reference\"><a href=\"#cite_note-MoonTech06-7\" rel=\"external_link\">[7]<\/a><\/sup> It provides unambiguous instructions on CRF completion in all practical scenarios. For example, if data were wrongly entered and the site personnel wants to correct it, the instruction provided would be to strike-through the incorrect data with a single line, put the initials (of the person who makes the change) with date and to write the correct entry in the margin against the corresponding line. Similarly, instructions will be provided for each data entry field on each page in the CRF booklet. This helps to ensure completion of all required data fields and enhances the data flow.<sup id=\"rdp-ebb-cite_ref-MoonTech06_7-3\" class=\"reference\"><a href=\"#cite_note-MoonTech06-7\" rel=\"external_link\">[7]<\/a><\/sup> CRF completion guidelines could be a separate document or could be a part of the CRF booklet giving page by page instructions. If it is included as part of the CRF, it is advisable to print instructions on the page facing the CRF page (back side of the previous page) as the investigator can easily take instructions and simultaneously fill the CRF page. CRF completion guidelines document should have version control and amendments should be done as and when required.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h2>\n<p>Case report form design is the initial step in translating the protocol into standard questionnaires and is paramount to a successful clinical trial. Standard CRF should be designed in such a way that it helps the collection of consistent and valid data, ultimately resulting in submission of data to regulatory authorities and its acceptance. Regardless of the time and effort spent conducting the trial, the correct data points (response to a CRF question\/data is entered) must be collected; otherwise, a meaningful analysis may not be possible. Therefore, a sound SAP should be used as a tool to develop and judge the adequacy of the CRF, which should be available to guide on what data points need to be captured on the CRF. To avoid future amendments, it is important to have design principles in mind well in advance before CRF designing is initiated. These standard guidelines will contribute in preparing a well-designed CRF for data acquisition.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgement\">Acknowledgement<\/span><\/h2>\n<p>We would like to acknowledge the technical support offered by Mr. Vinoth T. and Mr. Sagi Subbaraju that has helped us during the preparation of this article.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<ol class=\"references\">\n<li id=\"cite_note-NahmDes11-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NahmDes11_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nahm, M.; Shepherd, J.; Buzenberg, A.; Rostami, R.; Corcoran, A.; McCall, J. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3494996\" target=\"_blank\">\"Design and implementation of an institutional case report form library\"<\/a>. <i>Clinical Trials<\/i> <b>8<\/b> (1): 94\u2013102. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1177%2F1740774510391916\" target=\"_blank\">10.1177\/1740774510391916<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3494996\/\" target=\"_blank\">PMC3494996<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21163853\" target=\"_blank\">21163853<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3494996\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3494996<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+and+implementation+of+an+institutional+case+report+form+library&rft.jtitle=Clinical+Trials&rft.aulast=Nahm%2C+M.%3B+Shepherd%2C+J.%3B+Buzenberg%2C+A.%3B+Rostami%2C+R.%3B+Corcoran%2C+A.%3B+McCall%2C+J.+et+al.&rft.au=Nahm%2C+M.%3B+Shepherd%2C+J.%3B+Buzenberg%2C+A.%3B+Rostami%2C+R.%3B+Corcoran%2C+A.%3B+McCall%2C+J.+et+al.&rft.date=2011&rft.volume=8&rft.issue=1&rft.pages=94%E2%80%93102&rft_id=info:doi\/10.1177%2F1740774510391916&rft_id=info:pmc\/PMC3494996&rft_id=info:pmid\/21163853&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3494996&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ICHGuidance96-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ICHGuidance96_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">ICH Steering Committee (1996). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.fda.gov\/downloads\/Drugs\/Guidances\/ucm073122.pdf\" target=\"_blank\">\"Guidance for Industry - E6 Good Clinical Practice: Consolidated Guidance\"<\/a> (PDF). U.S. Food and Drug Administration<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.fda.gov\/downloads\/Drugs\/Guidances\/ucm073122.pdf\" target=\"_blank\">http:\/\/www.fda.gov\/downloads\/Drugs\/Guidances\/ucm073122.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 June 2013<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Guidance+for+Industry+-+E6+Good+Clinical+Practice%3A+Consolidated+Guidance&rft.atitle=&rft.aulast=ICH+Steering+Committee&rft.au=ICH+Steering+Committee&rft.date=1996&rft.pub=U.S.+Food+and+Drug+Administration&rft_id=http%3A%2F%2Fwww.fda.gov%2Fdownloads%2FDrugs%2FGuidances%2Fucm073122.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LuClinical10-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LuClinical10_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lu, Z.; Su, J. (June 2010). \"Design and implementation of an institutional case report form library\". <i>Open Access Journal of Clinical Trials<\/i> <b>2010<\/b> (2): 93\u2014105. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2147%2FOAJCT.S8172\" target=\"_blank\">10.2147\/OAJCT.S8172<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+and+implementation+of+an+institutional+case+report+form+library&rft.jtitle=Open+Access+Journal+of+Clinical+Trials&rft.aulast=Lu%2C+Z.%3B+Su%2C+J.&rft.au=Lu%2C+Z.%3B+Su%2C+J.&rft.date=June+2010&rft.volume=2010&rft.issue=2&rft.pages=93%E2%80%94105&rft_id=info:doi\/10.2147%2FOAJCT.S8172&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LatimerCase08-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LatimerCase08_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Latimer, P. (May 2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.firstclinical.com\/journal\/2008\/0805_Insanity.pdf\" target=\"_blank\">\"Case report form insanity\"<\/a> (PDF). <i>Journal of Clinical Research Best Practices<\/i> <b>4<\/b> (5)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.firstclinical.com\/journal\/2008\/0805_Insanity.pdf\" target=\"_blank\">http:\/\/www.firstclinical.com\/journal\/2008\/0805_Insanity.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 June 2013<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Case+report+form+insanity&rft.jtitle=Journal+of+Clinical+Research+Best+Practices&rft.aulast=Latimer%2C+P.&rft.au=Latimer%2C+P.&rft.date=May+2008&rft.volume=4&rft.issue=5&rft_id=http%3A%2F%2Fwww.firstclinical.com%2Fjournal%2F2008%2F0805_Insanity.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StarkData11-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StarkData11_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stark, N.J. (October 2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.firstclinical.com\/journal\/2011\/1110_Data_Management.pdf\" target=\"_blank\">\"Data management in device studies\"<\/a> (PDF). <i>Journal of Clinical Research Best Practices<\/i> <b>7<\/b> (10)<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.firstclinical.com\/journal\/2011\/1110_Data_Management.pdf\" target=\"_blank\">http:\/\/www.firstclinical.com\/journal\/2011\/1110_Data_Management.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 June 2013<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+management+in+device+studies&rft.jtitle=Journal+of+Clinical+Research+Best+Practices&rft.aulast=Stark%2C+N.J.&rft.au=Stark%2C+N.J.&rft.date=October+2011&rft.volume=7&rft.issue=10&rft_id=http%3A%2F%2Fwww.firstclinical.com%2Fjournal%2F2011%2F1110_Data_Management.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SCDMGood09-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SCDMGood09_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-SCDMGood09_6-1\" rel=\"external_link\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.swaggerinfosolutions.pro\/sridevi\/New%20Folder\/ICRI\/GCDMP_complete_OCT2009.pdf\" target=\"_blank\">\"Good Clinical Data Management Practices\"<\/a> (PDF). Society for Clinical Data Management. October 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.swaggerinfosolutions.pro\/sridevi\/New%20Folder\/ICRI\/GCDMP_complete_OCT2009.pdf\" target=\"_blank\">http:\/\/www.swaggerinfosolutions.pro\/sridevi\/New%20Folder\/ICRI\/GCDMP_complete_OCT2009.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 17 June 2013<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Good+Clinical+Data+Management+Practices&rft.atitle=&rft.date=October+2009&rft.pub=Society+for+Clinical+Data+Management&rft_id=http%3A%2F%2Fwww.swaggerinfosolutions.pro%2Fsridevi%2FNew%2520Folder%2FICRI%2FGCDMP_complete_OCT2009.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MoonTech06-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MoonTech06_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-MoonTech06_7-1\" rel=\"external_link\">7.1<\/a><\/sup> <sup><a href=\"#cite_ref-MoonTech06_7-2\" rel=\"external_link\">7.2<\/a><\/sup> <sup><a href=\"#cite_ref-MoonTech06_7-3\" rel=\"external_link\">7.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moon, K.K. (Fall 2006). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/web.archive.org\/web\/20130512162105\/http:\/\/www.mycro.ca\/media\/1533\/sciannews91b.pdf\" target=\"_blank\">\"Techniques for designing case report forms in clinical trials: Considerations for efficient data management and statistical analysis\"<\/a> (PDF). <i>ScianNews<\/i> <b>9<\/b> (1). Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.mycro.ca\/media\/1533\/sciannews91b.pdf\" target=\"_blank\">the original<\/a> on 12 May 2013<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/web.archive.org\/web\/20130512162105\/http:\/\/www.mycro.ca\/media\/1533\/sciannews91b.pdf\" target=\"_blank\">https:\/\/web.archive.org\/web\/20130512162105\/http:\/\/www.mycro.ca\/media\/1533\/sciannews91b.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Techniques+for+designing+case+report+forms+in+clinical+trials%3A+Considerations+for+efficient+data+management+and+statistical+analysis&rft.jtitle=ScianNews&rft.aulast=Moon%2C+K.K.&rft.au=Moon%2C+K.K.&rft.date=Fall+2006&rft.volume=9&rft.issue=1&rft_id=https%3A%2F%2Fweb.archive.org%2Fweb%2F20130512162105%2Fhttp%3A%2F%2Fwww.mycro.ca%2Fmedia%2F1533%2Fsciannews91b.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CDISCClinical08-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CDISCClinical08_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">CDISC CDASH Core and Domain Teams (01 October 2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cdisc.org\/system\/files\/all\/standard\/application\/pdf\/cdash_std_1_0_2008_10_01.pdf\" target=\"_blank\">\"Clinical Data Acquisition Standards Harmonization (CDASH)\"<\/a> (PDF). CDISC, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cdisc.org\/system\/files\/all\/standard\/application\/pdf\/cdash_std_1_0_2008_10_01.pdf\" target=\"_blank\">http:\/\/www.cdisc.org\/system\/files\/all\/standard\/application\/pdf\/cdash_std_1_0_2008_10_01.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 11 June 2013<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Clinical+Data+Acquisition+Standards+Harmonization+%28CDASH%29&rft.atitle=&rft.aulast=CDISC+CDASH+Core+and+Domain+Teams&rft.au=CDISC+CDASH+Core+and+Domain+Teams&rft.date=01+October+2008&rft.pub=CDISC%2C+Inc&rft_id=http%3A%2F%2Fwww.cdisc.org%2Fsystem%2Ffiles%2Fall%2Fstandard%2Fapplication%2Fpdf%2Fcdash_std_1_0_2008_10_01.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Basics_of_case_report_form_designing_in_clinical_research\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210225\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.255 seconds\nReal time usage: 0.300 seconds\nPreprocessor visited node count: 7560\/1000000\nPreprocessor generated node count: 29708\/1000000\nPost\u2010expand include size: 52779\/2097152 bytes\nTemplate argument size: 19277\/2097152 bytes\nHighest expansion depth: 17\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 242.762 1 - -total\n 58.48% 141.958 8 - Template:Citation\/core\n 45.71% 110.978 5 - Template:Cite_journal\n 25.61% 62.182 1 - Template:Infobox_journal_article\n 24.43% 59.299 1 - Template:Infobox\n 18.62% 45.201 3 - Template:Cite_web\n 14.58% 35.389 80 - Template:Infobox\/row\n 4.25% 10.323 9 - Template:Citation\/make_link\n 4.12% 10.010 4 - Template:Citation\/identifier\n 1.49% 3.623 9 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7676-0!*!0!!en!5!* and timestamp 20181213210224 and revision id 23404\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Basics_of_case_report_form_designing_in_clinical_research\">https:\/\/www.limswiki.org\/index.php\/Journal:Basics_of_case_report_form_designing_in_clinical_research<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","548bc0dcb34959573b02fdcd77beb67e_images":["https:\/\/www.limswiki.org\/images\/f\/f5\/Fig1_Bellary_PerspectivesClinRes2014_5-4.jpg","https:\/\/www.limswiki.org\/images\/1\/1d\/Tab1_Bellary_PerspectivesClinRes2014_5-4.jpg","https:\/\/www.limswiki.org\/images\/f\/ff\/Fig2_Bellary_PerspectivesClinRes2014_5-4.jpg","https:\/\/www.limswiki.org\/images\/5\/5e\/Fig3_Bellary_PerspectivesClinRes2014_5-4.jpg","https:\/\/www.limswiki.org\/images\/d\/d6\/Fig4_Bellary_PerspectivesClinRes2014_5-4.jpg","https:\/\/www.limswiki.org\/images\/1\/17\/Fig5_Bellary_PerspectivesClinRes2014_5-4.jpg"],"548bc0dcb34959573b02fdcd77beb67e_timestamp":1544734944,"89de2c69e566bc890426667a52566bb6_type":"article","89de2c69e566bc890426667a52566bb6_title":"Analyzing huge pathology images with open source software (Deroulers et al. 2013)","89de2c69e566bc890426667a52566bb6_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_huge_pathology_images_with_open_source_software","89de2c69e566bc890426667a52566bb6_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Analyzing huge pathology images with open source software\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nAnalyzing huge pathology images with open source softwareJournal\n \nDiagnostic PathologyAuthor(s)\n \nDeroulers, Christophe; Ameisen, David; Badoual, Mathilde; Gerin, Chlo\u00e9; Granier, Alexandre; Lartaud, MarcAuthor affiliation(s)\n \nUniversit\u00e9 Paris Diderot, Universit\u00e9 Paris-Sud, Montpellier RIO Imaging, CIRADPrimary contact\n \nEmail: deroulers@imnc.in2p3.frYear published\n \n2013Volume and issue\n \n8Page(s)\n \n92DOI\n \n10.1186\/1746-1596-8-92ISSN\n \n1746-1596Distribution license\n \nCreative Commons Attribution 2.0 GenericWebsite\n \nhttp:\/\/www.diagnosticpathology.org\/content\/8\/1\/92Download\n \nhttp:\/\/www.diagnosticpathology.org\/content\/pdf\/1746-1596-8-92.pdf (PDF)\n\nContents\n\n1 Abstract \n\n1.1 Background \n1.2 Results \n1.3 Conclusions \n1.4 Virtual slides \n\n\n2 Background \n3 Implementation \n\n3.1 Overview \n3.2 Basic functions \n3.3 Mosaics \n3.4 Usage \n\n3.4.1 Standalone \n3.4.2 ImageJ integration \n\n\n\n\n4 Results and discussion \n\n4.1 Performance \n4.2 Making a mosaic from a huge image \n4.3 Converting NDPI into TIFF \n4.4 Extracting a small region from a huge image \n4.5 From a NDPI file \n\n\n5 Applications \n\n5.1 Integration in digital pathology image servers or virtual slide systems \n5.2 Exploiting a large set of digital slides \n5.3 Study of a whole slide of brain tissue invaded by an oligodendroglioma \n\n\n6 Conclusions \n7 Availability and requirements \n8 Competing interests \n9 Authors\u2019 contributions \n10 Acknowledgements \n11 References \n12 Notes \n\n\n\nAbstract \nBackground \nDigital pathology images are increasingly used both for diagnosis and research, because slide scanners are nowadays broadly available and because the quantitative study of these images yields new insights in systems biology. However, such virtual slides build up a technical challenge since the images occupy often several gigabytes and cannot be fully opened in a computer\u2019s memory. Moreover, there is no standard format. Therefore, most common open source tools such as ImageJ fail at treating them, and the others require expensive hardware while still being prohibitively slow.\n\nResults \nWe have developed several cross-platform open source software tools to overcome these limitations. The NDPITools provide a way to transform microscopy images initially in the loosely supported NDPI format into one or several standard TIFF files, and to create mosaics (division of huge images into small ones, with or without overlap) in various TIFF and JPEG formats. They can be driven through ImageJ plugins. The LargeTIFFTools achieve similar functionality for huge TIFF images which do not fit into RAM. We test the performance of these tools on several digital slides and compare them, when applicable, to standard software. A statistical study of the cells in a tissue sample from an oligodendroglioma was performed on an average laptop computer to demonstrate the efficiency of the tools.\n\nConclusions \nOur open source software enables dealing with huge images with standard software on average computers. They are cross-platform, independent of proprietary libraries and very modular, allowing them to be used in other open source projects. They have excellent performance in terms of execution speed and RAM requirements. They open promising perspectives both to the clinician who wants to study a single slide and to the research team or data centre who do image analysis of many slides on a computer cluster.\n\nVirtual slides \nThe virtual slide(s) for this article can be found here: http:\/\/www.diagnosticpathology.diagnomx.eu\/vs\/5955513929846272\nKeywords: Digital pathology; Image processing; Virtual slides; Systems biology; ImageJ; NDPI\n\nBackground \nVirtual microscopy has become routinely used over the last few years for the transmission of pathology images (the so-called virtual slides), for both telepathology and teaching.[1][2] In more and more hospitals, virtual slides are even attached to the patient\u2019s file.[3][4] They have also a great potential for research, especially in the context of multidisciplinary projects involving e.g. mathematicians and clinicians who do not work at the same location. Quantitative histology is a promising new field, involving computer-based morphometry or statistical analysis of tissues.[5][6][7][8][9] A growing number of works report the pertinence of such images for diagnosis and classification of diseases, e.g. tumours.[10][11][12][13][14] Databases of clinical cases[15] will include more and more digitized tissue images. This growing use of virtual microscopy is accompanied by the development of integrated image analysis systems offering both virtual slide scanning and automatic image analysis, which makes integration into the daily practice of pathologists easier. See Ref. 16[16] for a review of some of these systems. \nModern slide scanners produce high magnification microscopy images of excellent quality[1], for instance at the so-called \u201c40x\u201d magnification. They allow much better visualization and analysis than lower magnification images. As an example, Figure 1 shows two portions of a slide at different magnifications, 10x and 40x. The benefit of the high magnification for both diagnosis and automated image analysis is clear. For instance, the state of the chromatin inside the nucleus and the cell morphology, better seen at high magnification, are essential to help the clinician distinguish tumorous and non-tumorous cells. An accurate, non-pixelated determination of the perimeters of the cell nuclei is needed for morphometry and statistics. \n\n\n\n\n\n\n\n\n\n Figure 1. A sample slide. (a): macroscopic view of the whole slide (the black rectangle on the left is 1x2 cm). (b,c): Influence of the magnification on the quality of results. (b): a portion of the slide scanned at magnification level 10x. The white contours show the result of an automatic detection of the dark cell nuclei with the ImageJ software. A significant fraction of the cell nuclei is missed and the contours are rather pixelated. (c): the same portion of the slide scanned at magnification 40x. The white contours show the result of the same automatic detection. Almost all cell nuclei are detected and the shapes of the contours are much more precise. Scale bar: 4 \u03bcm.\n\n\n\nHowever, this technique involves the manipulation of huge images (of the order of 10 billions of pixels for a full-size slide at magnification 40x with a single focus level) for which the approach taken by most standard software, loading and decompressing the full image into RAM, is impossible (a single slice of a full-size slide needs of the order of 30 GiB of RAM). As a result, standard open-source software such as ImageJ[17], ImageMagick[18] or GraphicsMagick[19] completely fails or is prohibitively slow when used on these images. Of course, commercially available software exists[16], but it is usually quite expensive, and very often restricted to a single operating system. It uses proprietary source code, which is a problem if one wants to control or check the algorithms and their parameters when doing image analysis for research. \nIn addition, many automated microscopes or slide scanners store the images which they produce into proprietary or poorly documented file formats, and the software provided by vendors is often specific to some operating system. This leads to several concerns. First, it makes research based on digital pathology technically more difficult. Even when a project is led on a single site, one has often to use clusters of computers to achieve large-scale studies of many full-size slides from several patients.[20] Since clusters of computers are typically run by open source software such as Linux, pathology images stored in non-standard file formats are a problem. Furthermore, research projects are now commonly performed in parallel in several sites, not to say in several countries, thanks to technology such as Grid[21], and there is ongoing efforts towards the interoperability of information systems used in pathology.[3][22] Second, proprietary formats may hinder the development of shared clinical databases[15] and access of the general public to knowledge, whereas the citizen should receive benefit of public investments. Finally, they may also raise financial concerns and conflicts of interest.[23]\nThere have been recent attempts to define open, documented, vendor-independent software[24][25], which partly address this problem. However, very large images stored in the NDPI file format produced by some slide scanners manufactured by Hamamatsu, such as the NanoZoomer, are not yet fully supported by such software. For instance, LOCI Bio-Formats[25] is presently unable to open images, one dimension of which is larger than 65k, and does not deal properly with NDPI files of more than 4 GiB. OpenSlide[24] does not currently support the NDPI format. NDPI-Splitter[26]needs to be run on Windows and depends on a proprietary library.\nTo address these problems, we have developed open source tools which achieve two main goals: reading and converting images in the NDPI file format into standard open formats such as TIFF, and splitting a huge image, without decompressing it entirely into RAM, into a mosaic of much smaller pieces (tiles), each of which can be easily opened or processed by standard software. All this is realized with high treatment speed on all platforms. \n\nImplementation \nOverview \nThe main software is implemented in the C programming language as separate, command-line driven executables. It is independent of any proprietary library. This ensures portability on a large number of platforms (we have tested several versions of Mac OS X, Linux and Windows), modularity and ease of integration into scripts or other software projects.\nIt is complemented by a set of plugins for the public domain software ImageJ[17], implemented in Java, which call the main executables in an automatic way to enable an interactive use.\nThe LargeTIFFTools and NDPITools are based on the open source TIFF[27] and JPEG[28] or libjpeg-turbo[29] libraries. The NDPITools plugins for ImageJ are based on the Java API of ImageJ[17][30] and on the open source software Image-IO [31], and use the Java Advanced Imaging 1.1.3 library.[32]\n\nBasic functions \nThe basic functions are the following. They can be performed even on a computer with a modest amount of RAM (see below the \u201cPerformance\u201d discussion).\n1. splitting a tiled TIFF file into multiple TIFF files, one for each of the tiles (tiffsplittiles program);\n2. extracting (\u201ccropping\u201d) quickly a given rectangle of a supposedly tiled TIFF file into a TIFF or JPEG file (tifffastcrop program);\n3. splitting one or several TIFF file(s), possibly very large, into mosaic(s), without fully decompressing them in memory (tiffmakemosaic program); \n4. converting a NDPI file into a standard multiple-image TIFF file, tiled if necessary, using upon request the BigTIFF format introduced in version 4.0.0 of the TIFF library[28][33][34], and encoding magnification and focus levels as TIFF \u201cimage description\u201d fields (ndpi2tiff program);\n5. creating a standard TIFF file for all or part of the magnification levels and focus levels present in the given NDPI file (the user can ask for specific magnification and focus levels and for a specific rectangular region of the image), and, upon request, creating a mosaic for each image which doesn\u2019t fit into RAM or for all images (ndpisplit program). The names of the created files are built on the name of the source file and incorporate the magnification and focus levels (and, in the case of mosaic pieces, the coordinates inside the mosaic). \n\nMosaics \nA mosaic is a set of TIFF or JPEG files (the pieces) which would reproduce the original image if reassembled together, but of manageable size by standard software. The user can either specify the maximum amount of RAM which a mosaic piece should need to be uncompressed (default: 1024 MiB), or directly specify the size of each piece. In the first case, the size of each piece is determined by the software. A given amount of overlap between mosaic pieces can be requested, either in pixels or as a percentage of the image size. This is useful e.g. for cell counting, not to miss cells which lie on the limit between two adjacent pieces. \n\nUsage \nStandalone \nOur tools can be used through the command line (POSIX-like shell or Windows command interpreter), and therefore can be very easily integrated into scripts or other programs. Depending on the tool, the paths and file names of one or several files, in NDPI or TIFF format, have to be provided. Options can be added with their arguments on the command line to modify the behavior of the programs from its default. They are explained in the messages printed by the programs run without arguments, in Unix-style man pages, and on the web pages of the project (see below in the Availability and requirements Section).\nUnder the Windows OS, one can click-and-drag the NDPI file icon onto the icon of ndpi2tiff or ndpisplit. We provide precompiled binaries where frequently-used options are turned on by default: e.g. ndpisplit-mJ.exe produces a mosaic in JPEG format as with option -mJ. The conversion result or mosaic can be found in the same directory as the original NDPI image.\n\nImageJ integration \nIn addition to command line use, the ndpisplit program can be driven through the NDPITools plugins in ImageJ with a point-and-click interface, so that previewing the content of a NDPI file at low resolution, selecting a portion, extracting it at high resolution and finally opening it in ImageJ to apply further treatments can be done in an easy and graphical way. Figure 2 shows a screen shot of ImageJ 1.47 m after extraction of a rectangular zone from a NDPI file. Figure 3 explains what happens when the NDPI file contains several levels of focalization: the preview image is displayed as a stack. \n\n\n\n\n\n\n\n\n\n Figure 2. A typical session using ImageJ and the NDPITools plugins. A NDPI file has been opened with the NDPITools plugins and it is displayed as a preview image (image at largest resolution which still fits into the computer\u2019s screen) \u2014 top window. A rectangular region has been selected and extracted as a TIFF image, then opened \u2014 bottom window.\n\n\n\n\n\n\n\n\n\n\n\n\n Figure 3. Preview image of a NDPI file with several focalization levels in ImageJ. The NDPI file 08.ndpi contains images at 5 different focalization levels. Therefore, its preview image is displayed as a stack of 5 images.\n\n\n\nWhen producing a mosaic, the user can request that pieces be JPEG files. Since the File > Open command of versions 1.x of ImageJ is unable to open TIFF files with JPEG compression (one has to use plugins), this is way to produce mosaics which can be opened by click-and-drag onto the window or icon of ImageJ while still saving disk space thanks to efficient compression. Figure 4 shows how the mosaic production options can be set inside ImageJ through the NDPITools plugins. \n\n\n\n\n\n\n\n\n\n Figure 4. Dialog box for customized extraction in ImageJ from an NDPI file with production of a mosaic. The dialog box shows some options which can be customized while producing a mosaic from a rectangular selection of a NDPI file preview image (here, using the file previewed in Figure 3).\n\n\n\nResults and discussion \nPerformance \nWe compare the performance of our tools on several fundamental tasks to standard, broadly available software in representative examples and on broadly available computers.\n\nMaking a mosaic from a huge image \nWe chose an 8-bit RGB colour JPEG-compressed TIFF file of 103168\u00d763232 pixels originating in the digitization of a pathology slide. The original file weighted 975.01 MiB. Loading this image entirely into RAM would need at least 3\u00d7103168\u00d763232=18.2 GiB and is presently intractable on most if not all desktop and laptop computers of reasonable cost.\nThe task was to produce, from this image, a mosaic of 64 pieces so that each one needs less than 512 MiB RAM to open.\nOn a 3.2 GHz Intel Core i3 IMac computer with 16 GB of RAM, the convert command from ImageMagick (version 6.8.0-7 with quantum size 8 bits) was unable to complete the request. GraphicsMagick (gm convert-crop; version 1.3.17 with quantum size 8 bits) completed the request in 70 min, using 25 GiB of disk space. tiffmakemosaic from our LargeTIFFTools completed the request in 2.5 min.\nTo ascertain that this task can be equally achieved even on computers with a modest RAM amount, we performed the same task on a 6-year-old 2.66 GHz Core2Duo Intel IMac with 2 GiB RAM. The task was completed in 9.0 min. \n\nConverting NDPI into TIFF \nSplitting a NDPI file into TIFF files. A pathology sample (6.7 cm<super>2<\/super> of tissue) was scanned at magnification 40x and with 11 focus levels (every 2 microns) by a NanoZoomer, resulting in a 6.5 GiB file in proprietary NDPI format (called file a.ndpi hereafter). On a 2.6 GHz Intel Core i7 Mac Mini computer with 16 GiB RAM, ndpisplit extracted all 55 images (11 focus levels and 5 magnifications) as independent, single-image TIFF files with JPEG compression in 7.11 min. The size of the largest images was 180224\u00d770144. The speed was limited only by the rate of I\/O transfers since the CPU usage of this task was 1.38 min, out of which the system used 1.30 min. Executing again the same task straight after the first execution took only 0.57 min because the NDPI file was still in the cache of the operating system.\nTo ascertain that this task can be equally achieved even on computers with a modest RAM amount, we made a try on a 6-year-old 2.66 GHz Core2Duo Intel PC with 2 GiB RAM running 32-bits Windows XP Pro SP3. The original file shown in Figure 1, called b.ndpi, and weighting 2.07 GiB (largest image: 103168\u00d763232 pixels), was split into independent TIFF files in 2.2 min without swapping.\nIn comparison, the LOCI Bio-Formats plugins for ImageJ[25], in its version 4.4.6 with ImageJ 1.43 m, was not able to open the images in file a.ndpi even at low resolution. \nConverting a NDPI file into a multiple-images TIFF file. Alternatively, the same proprietary-format file a.ndpi was converted into a multiple-images TIFF file with ndpi2tiff. On the same computer as before, the conversion time was 7.0 min. Here again, the speed of the process is limited only by the rate of I\/O transfers since the conversion took only 30 s if performed when the NDPI file was still in the cache of the operating system.\nSince the resulting TIFF file could not store all 55 images in less than 4 GiB, we passed the option -8 on the command line to ndpi2tiff to request using the BigTIFF format extension. The specifications of this extension to the TIFF standard, discussed and published before 2008[33][34], are supported by LibTIFF as of version 4.0.0[27], and therefore by the abundant image viewing and manipulation software which relies on LibTIFF. If the use of the BigTIFF format extension would have impeded the further exploitation of the produced TIFF file, we could have simply used ndpisplit as above. Or we could have called the ndpi2tiff command several times, each time requesting extraction of a subset of all images by specifying image numbers after the file name, separated with commas, as in a.ndpi,0,1,2,3,4.\n\nExtracting a small region from a huge image \nThis task can be useful to visualize at full resolution a region of interest which the user has selected on a low-magnification preview image. Therefore, it should be performed as quickly as possible.\nFrom a TIFF file\nThe task was to extract a rectangular region of size 256\u00d7256 pixels situated at the bottom right corner of huge TIFF images and to save it as an independent file. The source images were single-image TIFF files using JPEG compression. Table 1 compares the time needed to complete the task with tifffastcrop from our LargeTIFFTools and with several software tools, on increasingly large TIFF files. Tests were performed on a 2.6 GHz Intel Core i7 Mac Mini computer with 16 GB of RAM and used GraphicsMagick 1.3.17, ImageMagick 6.8.0-7 and the utility tiffcrop from LibTIFF 4.0.3. Noticeably, when treating the largest image, GraphicsMagick needs 50 GiB of free disk space, whereas tifffastcrop doesn\u2019t need it. \n\n\n\n\n\n\n\nTable 1. Speed comparison of software to extract a 256\u00d7256 rectangle from a huge TIFF image\n\n\n Image size (px)\n\n 11264\u00d74384\n\n 45056\u00d717536\n\n 180224\u00d770144\n\n\ntifffastcrop\n\n0.30 s\n\n0.30 s\n\n0.30 s\n\n\nGraphicsMagick\n\n0.74 s\n\n23.6 s\n\n>80 min\n\n\nImageMagick\n\n1.18 s\n\n236 s\n\nfailed\n\n\ntiffcrop\n\n0.50 s\n\nfailed\n\nseg. fault\n\n\nTime needed (or indication of failure when the task was not completed) by several software tools to extract a rectangular region of size 256\u00d7256 pixels situated at the bottom right corner of huge TIFF images, and to save it as an independent file. The input images were single-image tiled TIFF files using JPEG compression. Their dimensions are indicated in the top row. The computer used was a 2.6 GHz Intel Core i7 Mac Mini with 16 GiB of RAM and more than 100 GiB of free hard disk. The tested software tools were, from top to bottom, tifffastcrop from our LargeTIFFTools, GraphicsMagick 1.3.17, ImageMagick 6.8.0-7 and the utility tiffcrop from LibTIFF 4.0.3.\n\n\n\nFrom a NDPI file \nThe task was to extract a rectangular region of size 256\u00d7256 from one of the largest images of the file a.ndpi (size 180224\u00d770144). On a 2.6 GHz Intel Core i7 Mac Mini computer with 16 GB of RAM, the execution time was 0.12 s for one extract, and in average 0.06 s per extract in a series of 20 extracts with locations drawn uniformly at random inside the whole image. \n\nApplications \nIntegration in digital pathology image servers or virtual slide systems \nThe NDPITools are being used in several other software projects:\n\u2022in a system for automatic blur detection[2][4].\n\u2022in WIDE[22], to deal with NDPI files. WIDE is an open-source biological and digital pathology image archiving and visualization system, which allows the remote user to see images stored in a remote library in a browser. In particular, thanks to the feature of high-speed extraction of a rectangular region by ndpisplit, WIDE saves costly disk space since it doesn\u2019t need to store TIFF files converted from NDPI files in addition to the latter. \n\nExploiting a large set of digital slides \nIn the framework of a study about invasive low-grade oligodendrogliomas reported elsewhere[8], we had to deal with 303 NDPI files, occupying 122 GiB. On a 3.2 GHz Intel Core i3 IMac computer with 16 GB RAM, we used ndpisplit in a batch work to convert them into standard TIFF files, which took only a few hours. The experimental -s option of ndpisplit was used to remove the blank filling between scanned regions, resulting in an important disk space saving and in smaller TIFF files (one for each scanned region) which where easier to manipulate afterwards. Then, for each sample, Preview.app and ImageJ were used to inspect the resulting images and manually select the regions of clinical interest. The corresponding extracts of the high magnification images were the subject of automated cell counting and other quantitative analyses using ImageJ. In particular, we collected quantitative data about edema or tissue hyperhydration[8]. This quantity needed a specific image analysis procedure which is not offered by standard morphometry software and, unlike cell density estimates, could not be retrieved by sampling a few fields of view in the microscope. Therefore, virtual microscopy and our tools were essential in this study. \n\nStudy of a whole slide of brain tissue invaded by an oligodendroglioma \nTo demonstrate the possibility to do research on huge images even with a modest computer, we chose a 3-year-old MacBook Pro laptop computer with 2.66 GHz Intel Core 2 Duo and 4 GiB of RAM. We used ImageJ and the NDPITools to perform statistics on the upper piece of tissue on the slide shown in Figure 1.\nSince the digital slide b.ndpi weighted 2.07 GiB, with a high resolution image of 103168\u00d763232 pixels, it was not possible to do the study in a straightforward way. We opened the file b.ndpi as a preview image with the command Plugins > NDPITools > Preview NDPI... and selected on it the left tissue sample. Then we used the command Plugins > NDPITools > Custom extract to TIFF \/ Mosaic... and asked for extraction as a mosaic of 16 JPEG files, each one needing less than 1 GiB of RAM to open, and with an overlap of 60 pixels. This was completed within a few minutes. Then we applied an ImageJ macro to each of the 16 pieces to identify the dark cell nuclei (those with high chromatin content), based on thresholding the luminosity values of the pixels, as shown in Figure 1. It produced text files with the coordinates and size of each cell nucleus.\nOut of the 154240 identified nuclei, 1951 were positioned on the overlapping regions between pieces. Using the overlap feature of our tools enabled to properly detect these nuclei, since they would have been cut by the boundary of the pieces of the mosaic in absence of overlap. We avoided double counting by identifying the pairs of nuclei situated in the overlapping regions and which were separated by a distance smaller than their radius.\nAs shown in earlier studies[7][10][11], these data can be used for research and diagnosis purposes. As an example, Figure 5 shows the distribution of the distance of each cell nucleus to its nearest neighbor. Thanks to the very high number of analyzed cell nuclei, this distribution is obtained with an excellent precision. \n\n\n\n\n\n\n\n\n\n Figure 5. The positions of the 154240 identified nuclei were obtained from the analysis with ImageJ of the digital slide on a laptop computer. Since the slide was too large to fit into the computer\u2019s memory, it was turned into a mosaic of 16 pieces with overlap of 60 pixels, and each piece underwent automated analysis independently. Then the results were aggregated. The graph shows the probability density function of the distance of a cell nucleus to its nearest neighbor in the whole sample.\n\n\n\nConclusions \nThe LargeTIFFTools, NDPITools and NDPITools plugins for ImageJ achieve efficiently some fundamental functions on large images and in particular digital slides, for which standard open source software fails or performs badly. They enable both the clinician to examine a single slide and the bioinformatics research team to perform large-scale analysis of many slides, possibly on computer grids.[20]\nTo date, the LargeTIFFTools have been downloaded from more than 388 different IP addresses, the NDPITools from more than 1361 addresses, and the ImageJ plugins from more than 235 addresses. Table 2 lists the distribution of the target platforms among the downloads of the binary files. It shows a broad usage of the different platforms by the community, emphasizing the importance of cross-platform, open source tools. \n\n\n\n\n\n\n\nTable 2. Downloads of the NDPITools\n\n\n Windows (32 bits)\n\n Windows(64 bits)\n\n Linux\n\n Mac OS X\n\n\n483\n\n542\n\n217\n\n285\n\n\nDistribution of the downloads (unique IP address) of the precompiled binaries of the NDPITools between March 2012 and April 2013.\n\n\n\nWe have explained how the software was used to study some microscopic properties of brain tissue when invaded by an oligodendroglioma, and we have given an illustrative application to the analysis of a whole-size pathology slide. This suggests other promising applications. \n\nAvailability and requirements \na. LargeTIFFTools\n\u2022Project name: LargeTIFFTools\n\u2022Project home page: http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/largetifftools\/\n\u2022Operating system(s): Platform independent\n\u2022Programming language: C\n\u2022Other requirements: libjpeg, libtiff\n\u2022License: GNU GPLv3\n\r\n\nb. NDPITools\n\u2022Project name: NDPITools\n\u2022Project home page: http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/ndpitools\/\n\u2022Operating system(s): Platform independent\n\u2022Programming language: C\n\u2022Other requirements: \u2014\n\u2022License: GNU GPLv3\nFor the convenience of users, precompiled binaries are provided for Windows (32 and 64 bits), Mac OS X and Linux.\n\r\n\nc. NDPITools plugins for ImageJ\n\u2022Project name: NDPITools plugins for ImageJ\n\u2022Project home page: http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/ndpitools\/\n\u2022Operating system(s): Platform independent\n\u2022Programming language: Java\n\u2022Other requirements: ImageJ 1.31s or higher, Ant, JAI 1.1.3\n\n\u2022License: GNU GPLv3\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors\u2019 contributions \nCD wrote the paper. ML conceived and implemented a first version of the integration into ImageJ as a toolset of macros. CD implemented the software and wrote the documentation. CG, AG and ML contributed suggestions to the software. CD, DA, AG and ML performed software tests. CD, MB, CG, AG and ML selected and provided histological samples. CD performed the statistical analysis of the sample slide. All authors reviewed the manuscript. All authors read and approved the final manuscript.\n\nAcknowledgements \nWe thank F. Bouhidel and P. Bertheau for their help with the slide scanner of the Pathology Laboratory of the Saint-Louis Hospital in Paris, and C. Klein (Imaging facility, Cordeliers Research Center \u2013 INSERM U872, Paris) for tests and suggestions.\nThe computer, CPU, operating system, and programming language names quoted in this article are trademarks of their respective owners.\n\nReferences \n\n\n\u2191 1.0 1.1 Diamond, J.; McCleary, D. (2009). \"Virtual Microscopy\". In Hannon-Fletcher, M.; Maxwell, P.. Advanced Techniques in Diagnostic Cellular Pathology. Chichester UK: John Wiley & Sons, Ltd. ISBN 9780470515976.   \n\n\u2191 2.0 2.1 Ameisen, D.; Yun\u00e8s, J.B.; Deroulers, C.; Perrier, V.; Bouhidel, F.; Battistella, M.; Legr\u00e8s, L.; Janin, A.; Bertheau, P. (2013). \"Stack or Trash? Fast quality assessment of virtual slides\". Diagnostic Pathology 8 (Suppl 1): S23. doi:10.1186\/1746-1596-8-S1-S23. PMC PMC3849546. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849546 .   \n\n\u2191 3.0 3.1 Garc\u00eda Rojo, M.; Castro, A.M.; Gon\u00e7alves, L. (2011). \"COST action \u201cEuroTelepath\u201d: digital pathology integration in electronic health record, including primary care centres\". Diagnostic Pathology 6 (Suppl 1): S6. doi:10.1186\/1746-1596-6-S1-S6. PMC PMC3073224. PMID 21489201. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073224 .   \n\n\u2191 4.0 4.1 Ameisen, D. (2013). \"Int\u00e9gration des lames virtuelles dans le dossier patient \u00e9lectronique\". PhD thesis. Univ Paris Diderot-Paris 7.   \n\n\u2191 Collan, Y.; Torkkeli, T.; Personen, E.; Jantunen, E.; Kosma, V.M. (1987). \"Application of morphometry in tumor pathology\". Analytical and Quantitative Cytology and Histology 9 (2): 79\u201388. PMID 3300687.   \n\n\u2191 Wolfe, P.; Murphy, J.; McGinley, J.; Zhu, Z.; Jiang, W.; Gottschall, E.; Thompson, H. (2004). \"Using nuclear morphometry to discriminate the tumorigenic potential of cells: A comparison of statistical methods\". Cancer Epidemiology, Biomarkers & Prevention 13 (6): 976\u2013988. PMID 15184254.   \n\n\u2191 7.0 7.1 G\u00fcrcan, M.N.; Boucheron, L.E.; Can, A.; Madabhushi, A.; Rajpoot, N.M.; Yener, B. (2009). \"Histopathological image analysis: a review\". IEEE Reviews in Biomedical Engineering 2009 (2): 147\u2013171. doi:10.1109\/RBME.2009.2034865. PMC PMC2910932. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2910932 .   \n\n\u2191 8.0 8.1 8.2 Gerin, C.; Pallud, J.; Deroulers, C.; Varlet, P.; Oppenheim, C.; Roux, F.X.; Chr\u00e9tien, F.; Thomas, S.R.; Grammaticos, B.; Badoual, M. (2013). \"Quantitative characterization of the imaging limits of diffuse low-grade oligodendrogliomas\". Neuro-Oncology 15 (10): 1379-88. doi:10.1093\/neuonc\/not072. PMC PMC3779035. PMID 23771168. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3779035 .   \n\n\u2191 Wienert, S.; Heim, D.; Kotani, M.; Lindequist, B.; Stenzinger, A.; Ishii, M.; Hufnagl, P.; Beil, M.; Dietel, M.; Denkert, C.; Klauschen, F. (2013). \"CognitionMaster: an object-based image analysis framework\". Diagnostic Pathology 8: 34. doi:10.1186\/1746-1596-8-34. PMC PMC3626931. PMID 23445542. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626931 .   \n\n\u2191 10.0 10.1 Gunduz, C.; Yener, B.; Gultekin, S.H. (2004). \"The cell graphs of cancer\". Bioinformatics 20 (Suppl 1): i145-i151. doi:10.1093\/bioinformatics\/bth933. PMID 15262793.   \n\n\u2191 11.0 11.1 Gunduz, C.; Gultekin, S.H.; Yener, B. (2005). \"Augmented cell-graphs for automated cancer diagnosis\". Bioinformatics 21 (Suppl 2): ii7-ii12. doi:10.1093\/bioinformatics\/bti1100. PMID 16204128.   \n\n\u2191 West, N.P.; Dattani, M.; McShane, P.; Hutchins, G.; Grabsch, J.; Mueller, W.; Treanor, D.; Quirke, P.; Grabsch, H. (2010). \"The proportion of tumour cells is an independent predictor for survival in colorectal cancer patients\". British Journal of Cancer 102 (10): 1519\u20131523. doi:10.1038\/sj.bjc.6605674. PMC PMC2869173. PMID 20407439. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2869173 .   \n\n\u2191 Chang, H.; Han, J.; Borowsky, A.; Loss, L.; Gray, J.W.; Spellman, P.T.; Parvin, B. (2013). \"Invariant delineation of nuclear architecture in Glioblastoma multiforme for clinical and molecular association\". IEEE Transactions on Medical Imaging 32 (4): 670\u2013682. doi:10.1109\/TMI.2012.2231420. PMC PMC3728287. PMID 23221815. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3728287 .   \n\n\u2191 Kayser, K.; Radziszowski, D.; Bzdyl, P.; Sommer, R.; Kayser, G. (2006). \"Towards an automated virtual slide screening: theoretical considerations and practical experiences of automated tissue-based virtual diagnosis to be implemented in the internet\". Diagnostic Pathology 1: 10. doi:10.1186\/1746-1596-1-10. PMC PMC1524814. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524814 .   \n\n\u2191 15.0 15.1 PLGA Foundation (2012). \"Meta analysis low grade glioma database project\". Archived from the original on 11 July 2013. http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.fightplga.org\/research\/PLGA-Sponsored_Projects\/MetaAnalysis&refdoi=10.1186\/1746-1596-8-92 .   \n\n\u2191 16.0 16.1 Garc\u00eda Rojo, M.; Bueno, G.; Slodkowska, J. (2009). \"Review of imaging solutions for integrated quantitative immunohistochemistry in the Pathology daily practice\". Folia Histochemica et Cytobiologica 47 (3): 349\u2013354. doi:10.2478\/v10042-008-0114-4. PMID 20164017.   \n\n\u2191 17.0 17.1 17.2 Rasband, W.S. (2012). \"ImageJ\". http:\/\/imagej.nih.gov\/ij\/ .   \n\n\u2191 ImageMagick Studio, LLC (2013). \"ImageMagick\". http:\/\/www.imagemagick.org\/ .   \n\n\u2191 GraphicsMagick Group (2013). \"GraphicsMagick\". http:\/\/www.graphicsmagick.org\/ .   \n\n\u2191 20.0 20.1 Kong, J.; Cooper, L.A.D.; Wang, F.; Chisolm, C.; Moreno, C.S.; Kurc, T.M.; Widener, P.M.; Brat, D.J.; Saltz, J.H. (2011). \"A comprehensive framework for classification of nuclei in digital microscopy imaging: An application to diffuse gliomas\". IEEE International Symposium on Biomedical Imaging 2011 Mar 30: 2128\u20132131. doi:10.1109\/ISBI.2011.5872833. PMC PMC3256584. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3256584 .   \n\n\u2191 Kayser, K.; G\u00f6rtler, J.; Borkenfeld, S.; Kayser, G. (2011). \"Grid computing in image analysis\". Diagnostic Pathology 6 (Suppl 1): S12. PMC PMC3073205. PMID 21516880. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073205 .   \n\n\u2191 22.0 22.1 Granier, A.; Olivier, M.; Laborie, S.; Vaudescal, S.; Baecker, V.; Tran-Aupiais, C. (2013). \"WIDE (Web Images and Data Environment)\". Archived from the original on 11 July 2013. http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.mri.cnrs.fr\/index.php?m=81&refdoi=10.1186\/1746-1596-8-92 .   \n\n\u2191 Kayser, K. (2012). \"Introduction of virtual microscopy in routine surgical pathology \u2014 a hypothesis and personal view from Europe\". Diagnostic Pathology 7: 48. doi:10.1186\/1746-1596-7-48. PMC PMC3441330. PMID 22546238. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3441330 .   \n\n\u2191 24.0 24.1 Goode, A.; Satyanarayanan, M. (2008). \"A vendor-neutral library and viewer for whole-slide images\" (PDF). Technical Report CMU-CS-08-136. School of Computer Science, Carnegie Mellon University. Archived from the original on 11 July 2013. http:\/\/www.webcitation.org\/query.php?url=http:\/\/reports-archive.adm.cs.cmu.edu\/anon\/2008\/CMU-CS-08-136.pdf&refdoi=10.1186\/1746-1596-8-92 .   \n\n\u2191 25.0 25.1 25.2 Linkert, M.; Rueden, C.T.; Allan, C.; Burel, J.M.; Moore, W.; Patterson, A.; Loranger, B.; Moore, J.; Neves, C.; MacDonald, D.; Tarkowska, A.; Sticco, C.; Hill, E.; Rossner, M.; Eliceiri, K.W.; Swedlow, J.R. (2010). \"Metadata matters: access to image data in the real world\". Journal of Cell Biology 198 (5): 777-782. doi:10.1083\/jcb.201004104. PMC PMC2878938. PMID 20513764. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938 .   \n\n\u2191 Khushi, M.; Edwards, G.; de Marcos, D.A.; Carpenter, J.E.; Graham, J.D.; Clarke, C.L. (2013). \"Open source tools for management and archiving of digital microscopy data to allow integration with patient pathology and treatment information\". Diagnostic Pathology 8: 22. doi:10.1186\/1746-1596-8-22. PMC PMC3575263. PMID 23402499. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3575263 .   \n\n\u2191 27.0 27.1 Sam Leffler, S.; the authors of LibTIFF (2012). \"LibTIFF \u2013 TIFF Library and Utilities\". http:\/\/www.remotesensing.org\/libtiff\/ .   \n\n\u2191 28.0 28.1 Lane, T.G.; Vollbeding, G. (2013). \"The Independent JPEG Group\u2019s JPEG software\". http:\/\/www.ijg.org\/ .   \n\n\u2191 Lane, T.G.; Vollbeding, G.; the authors of the libjpeg-turbo software (2012). \"libjpeg-turbo\". http:\/\/libjpeg-turbo.virtualgl.org\/ .   \n\n\u2191 Schneider, C.A.; Rasband, W.S.; Eliceiri, K.W. (2012). \"NIH Image to ImageJ: 25 years of image analysis\". Nature Methods 9: 671-675. doi:10.1038\/nmeth.2089. PMID 22930834.   \n\n\u2191 Sacha, J. (2004). \"Image IO Plugin Bundle\". http:\/\/ij-plugins.sourceforge.net\/plugins\/imageio\/ .   \n\n\u2191 Sun Microsystems, Inc (2006). \"Java Advanced Library 1.1.3\". Archived from the original on 11 July 2013. http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.oracle.com\/technetwork\/java\/current-142188.html&refdoi=10.1186\/1746-1596-8-92 .   \n\n\u2191 33.0 33.1 \"BigTIFF Design\". 2012. http:\/\/www.remotesensing.org\/libtiff\/bigtiffdesign.html .   \n\n\u2191 34.0 34.1 \"The BigTIFF File Format Proposal\". 2008. http:\/\/www.awaresystems.be\/imaging\/tiff\/bigtiff.html .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_huge_pathology_images_with_open_source_software\">https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_huge_pathology_images_with_open_source_software<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on health informaticsLIMSwiki journal articles on imaging informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 12 January 2016, at 16:38.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,389 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","89de2c69e566bc890426667a52566bb6_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Analyzing_huge_pathology_images_with_open_source_software skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Analyzing huge pathology images with open source software<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h3>\n<p>Digital pathology images are increasingly used both for diagnosis and research, because slide scanners are nowadays broadly available and because the quantitative study of these images yields new insights in systems biology. However, such virtual slides build up a technical challenge since the images occupy often several gigabytes and cannot be fully opened in a computer\u2019s memory. Moreover, there is no standard format. Therefore, most common open source tools such as ImageJ fail at treating them, and the others require expensive hardware while still being prohibitively slow.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h3>\n<p>We have developed several cross-platform open source software tools to overcome these limitations. The NDPITools provide a way to transform microscopy images initially in the loosely supported NDPI format into one or several standard TIFF files, and to create mosaics (division of huge images into small ones, with or without overlap) in various TIFF and JPEG formats. They can be driven through ImageJ plugins. The LargeTIFFTools achieve similar functionality for huge TIFF images which do not fit into RAM. We test the performance of these tools on several digital slides and compare them, when applicable, to standard software. A statistical study of the cells in a tissue sample from an oligodendroglioma was performed on an average laptop computer to demonstrate the efficiency of the tools.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h3>\n<p>Our open source software enables dealing with huge images with standard software on average computers. They are cross-platform, independent of proprietary libraries and very modular, allowing them to be used in other open source projects. They have excellent performance in terms of execution speed and RAM requirements. They open promising perspectives both to the clinician who wants to study a single slide and to the research team or data centre who do image analysis of many slides on a computer cluster.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Virtual_slides\">Virtual slides<\/span><\/h3>\n<p>The virtual slide(s) for this article can be found here: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.diagnosticpathology.diagnomx.eu\/vs\/5955513929846272\" target=\"_blank\">http:\/\/www.diagnosticpathology.diagnomx.eu\/vs\/5955513929846272<\/a>\n<\/p><p><b>Keywords<\/b>: Digital pathology; Image processing; Virtual slides; Systems biology; ImageJ; NDPI\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background_2\">Background<\/span><\/h2>\n<p>Virtual microscopy has become routinely used over the last few years for the transmission of pathology images (the so-called virtual slides), for both telepathology and teaching.<sup id=\"rdp-ebb-cite_ref-DiamondVirt09_1-0\" class=\"reference\"><a href=\"#cite_note-DiamondVirt09-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AmeisenStack13_2-0\" class=\"reference\"><a href=\"#cite_note-AmeisenStack13-2\" rel=\"external_link\">[2]<\/a><\/sup> In more and more hospitals, virtual slides are even attached to the patient\u2019s file.<sup id=\"rdp-ebb-cite_ref-GarciaCOST11_3-0\" class=\"reference\"><a href=\"#cite_note-GarciaCOST11-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AmeisenInt13_4-0\" class=\"reference\"><a href=\"#cite_note-AmeisenInt13-4\" rel=\"external_link\">[4]<\/a><\/sup> They have also a great potential for research, especially in the context of multidisciplinary projects involving e.g. mathematicians and clinicians who do not work at the same location. Quantitative histology is a promising new field, involving computer-based morphometry or statistical analysis of tissues.<sup id=\"rdp-ebb-cite_ref-CollanApp87_5-0\" class=\"reference\"><a href=\"#cite_note-CollanApp87-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WolfeUsing04_6-0\" class=\"reference\"><a href=\"#cite_note-WolfeUsing04-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GurcanHist09_7-0\" class=\"reference\"><a href=\"#cite_note-GurcanHist09-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GerinQuant13_8-0\" class=\"reference\"><a href=\"#cite_note-GerinQuant13-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WienertCog13_9-0\" class=\"reference\"><a href=\"#cite_note-WienertCog13-9\" rel=\"external_link\">[9]<\/a><\/sup> A growing number of works report the pertinence of such images for diagnosis and classification of diseases, e.g. tumours.<sup id=\"rdp-ebb-cite_ref-GunduzTheCell04_10-0\" class=\"reference\"><a href=\"#cite_note-GunduzTheCell04-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GunduzAug05_11-0\" class=\"reference\"><a href=\"#cite_note-GunduzAug05-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WestThePro10_12-0\" class=\"reference\"><a href=\"#cite_note-WestThePro10-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ChangInv13_13-0\" class=\"reference\"><a href=\"#cite_note-ChangInv13-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KayserTowards06_14-0\" class=\"reference\"><a href=\"#cite_note-KayserTowards06-14\" rel=\"external_link\">[14]<\/a><\/sup> Databases of clinical cases<sup id=\"rdp-ebb-cite_ref-PLGAMeta12_15-0\" class=\"reference\"><a href=\"#cite_note-PLGAMeta12-15\" rel=\"external_link\">[15]<\/a><\/sup> will include more and more digitized tissue images. This growing use of virtual microscopy is accompanied by the development of integrated image analysis systems offering both virtual slide scanning and automatic image analysis, which makes integration into the daily practice of pathologists easier. See Ref. 16<sup id=\"rdp-ebb-cite_ref-GarciaRev09_16-0\" class=\"reference\"><a href=\"#cite_note-GarciaRev09-16\" rel=\"external_link\">[16]<\/a><\/sup> for a review of some of these systems. \n<\/p><p>Modern slide scanners produce high magnification microscopy images of excellent quality<sup id=\"rdp-ebb-cite_ref-DiamondVirt09_1-1\" class=\"reference\"><a href=\"#cite_note-DiamondVirt09-1\" rel=\"external_link\">[1]<\/a><\/sup>, for instance at the so-called \u201c40x\u201d magnification. They allow much better visualization and analysis than lower magnification images. As an example, Figure 1 shows two portions of a slide at different magnifications, 10x and 40x. The benefit of the high magnification for both diagnosis and automated image analysis is clear. For instance, the state of the chromatin inside the nucleus and the cell morphology, better seen at high magnification, are essential to help the clinician distinguish tumorous and non-tumorous cells. An accurate, non-pixelated determination of the perimeters of the cell nuclei is needed for morphometry and statistics. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Deroulers_DiagnosticPath2013_8.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"d7f8c6943fe81b0d551db9b095c8b720\"><img alt=\"Fig1 Deroulers DiagnosticPath2013 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/3e\/Fig1_Deroulers_DiagnosticPath2013_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1. A sample slide. (a)<\/b>: macroscopic view of the whole slide (the black rectangle on the left is 1x2 cm). <b>(b,c)<\/b>: Influence of the magnification on the quality of results. <b>(b)<\/b>: a portion of the slide scanned at magnification level 10x. The white contours show the result of an automatic detection of the dark cell nuclei with the ImageJ software. A significant fraction of the cell nuclei is missed and the contours are rather pixelated. <b>(c)<\/b>: the same portion of the slide scanned at magnification 40x. The white contours show the result of the same automatic detection. Almost all cell nuclei are detected and the shapes of the contours are much more precise. Scale bar: 4 <i>\u03bc<\/i>m.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>However, this technique involves the manipulation of huge images (of the order of 10 billions of pixels for a full-size slide at magnification 40x with a single focus level) for which the approach taken by most standard software, loading and decompressing the full image into RAM, is impossible (a single slice of a full-size slide needs of the order of 30 GiB of RAM). As a result, standard open-source software such as ImageJ<sup id=\"rdp-ebb-cite_ref-RasbandImageJ12_17-0\" class=\"reference\"><a href=\"#cite_note-RasbandImageJ12-17\" rel=\"external_link\">[17]<\/a><\/sup>, ImageMagick<sup id=\"rdp-ebb-cite_ref-IMSImage13_18-0\" class=\"reference\"><a href=\"#cite_note-IMSImage13-18\" rel=\"external_link\">[18]<\/a><\/sup> or GraphicsMagick<sup id=\"rdp-ebb-cite_ref-GMGGraphics13_19-0\" class=\"reference\"><a href=\"#cite_note-GMGGraphics13-19\" rel=\"external_link\">[19]<\/a><\/sup> completely fails or is prohibitively slow when used on these images. Of course, commercially available software exists<sup id=\"rdp-ebb-cite_ref-GarciaRev09_16-1\" class=\"reference\"><a href=\"#cite_note-GarciaRev09-16\" rel=\"external_link\">[16]<\/a><\/sup>, but it is usually quite expensive, and very often restricted to a single operating system. It uses proprietary source code, which is a problem if one wants to control or check the algorithms and their parameters when doing image analysis for research. \n<\/p><p>In addition, many automated microscopes or slide scanners store the images which they produce into proprietary or poorly documented file formats, and the software provided by vendors is often specific to some operating system. This leads to several concerns. First, it makes research based on digital pathology technically more difficult. Even when a project is led on a single site, one has often to use clusters of computers to achieve large-scale studies of many full-size slides from several patients.<sup id=\"rdp-ebb-cite_ref-KongAComp11_20-0\" class=\"reference\"><a href=\"#cite_note-KongAComp11-20\" rel=\"external_link\">[20]<\/a><\/sup> Since clusters of computers are typically run by open source software such as Linux, pathology images stored in non-standard file formats are a problem. Furthermore, research projects are now commonly performed in parallel in several sites, not to say in several countries, thanks to technology such as Grid<sup id=\"rdp-ebb-cite_ref-KayserGrid11_21-0\" class=\"reference\"><a href=\"#cite_note-KayserGrid11-21\" rel=\"external_link\">[21]<\/a><\/sup>, and there is ongoing efforts towards the interoperability of information systems used in pathology.<sup id=\"rdp-ebb-cite_ref-GarciaCOST11_3-1\" class=\"reference\"><a href=\"#cite_note-GarciaCOST11-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GranierWIDE13_22-0\" class=\"reference\"><a href=\"#cite_note-GranierWIDE13-22\" rel=\"external_link\">[22]<\/a><\/sup> Second, proprietary formats may hinder the development of shared clinical databases<sup id=\"rdp-ebb-cite_ref-PLGAMeta12_15-1\" class=\"reference\"><a href=\"#cite_note-PLGAMeta12-15\" rel=\"external_link\">[15]<\/a><\/sup> and access of the general public to knowledge, whereas the citizen should receive benefit of public investments. Finally, they may also raise financial concerns and conflicts of interest.<sup id=\"rdp-ebb-cite_ref-KayserIntro12_23-0\" class=\"reference\"><a href=\"#cite_note-KayserIntro12-23\" rel=\"external_link\">[23]<\/a><\/sup>\n<\/p><p>There have been recent attempts to define open, documented, vendor-independent software<sup id=\"rdp-ebb-cite_ref-GoodeAVendor08_24-0\" class=\"reference\"><a href=\"#cite_note-GoodeAVendor08-24\" rel=\"external_link\">[24]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LinkertMeta10_25-0\" class=\"reference\"><a href=\"#cite_note-LinkertMeta10-25\" rel=\"external_link\">[25]<\/a><\/sup>, which partly address this problem. However, very large images stored in the NDPI file format produced by some slide scanners manufactured by Hamamatsu, such as the NanoZoomer, are not yet fully supported by such software. For instance, LOCI Bio-Formats<sup id=\"rdp-ebb-cite_ref-LinkertMeta10_25-1\" class=\"reference\"><a href=\"#cite_note-LinkertMeta10-25\" rel=\"external_link\">[25]<\/a><\/sup> is presently unable to open images, one dimension of which is larger than 65k, and does not deal properly with NDPI files of more than 4 GiB. OpenSlide<sup id=\"rdp-ebb-cite_ref-GoodeAVendor08_24-1\" class=\"reference\"><a href=\"#cite_note-GoodeAVendor08-24\" rel=\"external_link\">[24]<\/a><\/sup> does not currently support the NDPI format. NDPI-Splitter<sup id=\"rdp-ebb-cite_ref-KhushiOpen13_26-0\" class=\"reference\"><a href=\"#cite_note-KhushiOpen13-26\" rel=\"external_link\">[26]<\/a><\/sup>needs to be run on Windows and depends on a proprietary library.\n<\/p><p>To address these problems, we have developed open source tools which achieve two main goals: reading and converting images in the NDPI file format into standard open formats such as TIFF, and splitting a huge image, without decompressing it entirely into RAM, into a mosaic of much smaller pieces (tiles), each of which can be easily opened or processed by standard software. All this is realized with high treatment speed on all platforms. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Overview\">Overview<\/span><\/h3>\n<p>The main software is implemented in the C programming language as separate, command-line driven executables. It is independent of any proprietary library. This ensures portability on a large number of platforms (we have tested several versions of Mac OS X, Linux and Windows), modularity and ease of integration into scripts or other software projects.\n<\/p><p>It is complemented by a set of plugins for the public domain software ImageJ<sup id=\"rdp-ebb-cite_ref-RasbandImageJ12_17-1\" class=\"reference\"><a href=\"#cite_note-RasbandImageJ12-17\" rel=\"external_link\">[17]<\/a><\/sup>, implemented in Java, which call the main executables in an automatic way to enable an interactive use.\n<\/p><p>The LargeTIFFTools and NDPITools are based on the open source TIFF<sup id=\"rdp-ebb-cite_ref-LefflerLib12_27-0\" class=\"reference\"><a href=\"#cite_note-LefflerLib12-27\" rel=\"external_link\">[27]<\/a><\/sup> and JPEG<sup id=\"rdp-ebb-cite_ref-LaneTheInd13_28-0\" class=\"reference\"><a href=\"#cite_note-LaneTheInd13-28\" rel=\"external_link\">[28]<\/a><\/sup> or libjpeg-turbo<sup id=\"rdp-ebb-cite_ref-Lanelibjpeg12_29-0\" class=\"reference\"><a href=\"#cite_note-Lanelibjpeg12-29\" rel=\"external_link\">[29]<\/a><\/sup> libraries. The NDPITools plugins for ImageJ are based on the Java API of ImageJ<sup id=\"rdp-ebb-cite_ref-RasbandImageJ12_17-2\" class=\"reference\"><a href=\"#cite_note-RasbandImageJ12-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SchneiderNIH12_30-0\" class=\"reference\"><a href=\"#cite_note-SchneiderNIH12-30\" rel=\"external_link\">[30]<\/a><\/sup> and on the open source software Image-IO <sup id=\"rdp-ebb-cite_ref-SachaImage04_31-0\" class=\"reference\"><a href=\"#cite_note-SachaImage04-31\" rel=\"external_link\">[31]<\/a><\/sup>, and use the Java Advanced Imaging 1.1.3 library.<sup id=\"rdp-ebb-cite_ref-SunJava06_32-0\" class=\"reference\"><a href=\"#cite_note-SunJava06-32\" rel=\"external_link\">[32]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Basic_functions\">Basic functions<\/span><\/h3>\n<p>The basic functions are the following. They can be performed even on a computer with a modest amount of RAM (see below the \u201cPerformance\u201d discussion).\n<\/p><p>1. splitting a tiled TIFF file into multiple TIFF files, one for each of the tiles (<tt>tiffsplittiles<\/tt> program);\n<\/p><p>2. extracting (\u201ccropping\u201d) quickly a given rectangle of a supposedly tiled TIFF file into a TIFF or JPEG file (<tt>tifffastcrop<\/tt> program);\n<\/p><p>3. splitting one or several TIFF file(s), possibly very large, into mosaic(s), without fully decompressing them in memory (<tt>tiffmakemosaic<\/tt> program); \n<\/p><p>4. converting a NDPI file into a standard multiple-image TIFF file, tiled if necessary, using upon request the BigTIFF format introduced in version 4.0.0 of the TIFF library<sup id=\"rdp-ebb-cite_ref-LaneTheInd13_28-1\" class=\"reference\"><a href=\"#cite_note-LaneTheInd13-28\" rel=\"external_link\">[28]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BTDesign12_33-0\" class=\"reference\"><a href=\"#cite_note-BTDesign12-33\" rel=\"external_link\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BTProp08_34-0\" class=\"reference\"><a href=\"#cite_note-BTProp08-34\" rel=\"external_link\">[34]<\/a><\/sup>, and encoding magnification and focus levels as TIFF \u201cimage description\u201d fields (<tt>ndpi2tiff<\/tt> program);\n<\/p><p>5. creating a standard TIFF file for all or part of the magnification levels and focus levels present in the given NDPI file (the user can ask for specific magnification and focus levels and for a specific rectangular region of the image), and, upon request, creating a mosaic for each image which doesn\u2019t fit into RAM or for all images (<tt>ndpisplit<\/tt> program). The names of the created files are built on the name of the source file and incorporate the magnification and focus levels (and, in the case of mosaic pieces, the coordinates inside the mosaic). \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Mosaics\">Mosaics<\/span><\/h3>\n<p>A mosaic is a set of TIFF or JPEG files (the <i>pieces<\/i>) which would reproduce the original image if reassembled together, but of manageable size by standard software. The user can either specify the maximum amount of RAM which a mosaic piece should need to be uncompressed (default: 1024 MiB), or directly specify the size of each piece. In the first case, the size of each piece is determined by the software. A given amount of overlap between mosaic pieces can be requested, either in pixels or as a percentage of the image size. This is useful e.g. for cell counting, not to miss cells which lie on the limit between two adjacent pieces. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Usage\">Usage<\/span><\/h3>\n<h4><span class=\"mw-headline\" id=\"Standalone\">Standalone<\/span><\/h4>\n<p>Our tools can be used through the command line (POSIX-like shell or Windows command interpreter), and therefore can be very easily integrated into scripts or other programs. Depending on the tool, the paths and file names of one or several files, in NDPI or TIFF format, have to be provided. Options can be added with their arguments on the command line to modify the behavior of the programs from its default. They are explained in the messages printed by the programs run without arguments, in Unix-style man pages, and on the web pages of the project (see below in the <i>Availability and requirements<\/i> Section).\n<\/p><p>Under the Windows OS, one can click-and-drag the NDPI file icon onto the icon of <tt>ndpi2tiff<\/tt> or <tt>ndpisplit<\/tt>. We provide precompiled binaries where frequently-used options are turned on by default: e.g. <tt>ndpisplit-mJ.exe<\/tt> produces a mosaic in JPEG format as with option <tt>-mJ<\/tt>. The conversion result or mosaic can be found in the same directory as the original NDPI image.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"ImageJ_integration\">ImageJ integration<\/span><\/h4>\n<p>In addition to command line use, the <tt>ndpisplit<\/tt> program can be driven through the <tt>NDPITools<\/tt> plugins in ImageJ with a point-and-click interface, so that previewing the content of a NDPI file at low resolution, selecting a portion, extracting it at high resolution and finally opening it in ImageJ to apply further treatments can be done in an easy and graphical way. Figure 2 shows a screen shot of ImageJ 1.47 m after extraction of a rectangular zone from a NDPI file. Figure 3 explains what happens when the NDPI file contains several levels of focalization: the preview image is displayed as a stack. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Deroulers_DiagnosticPath2013_8.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"ac56ade8f465db87112cc168fc313198\"><img alt=\"Fig2 Deroulers DiagnosticPath2013 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/ca\/Fig2_Deroulers_DiagnosticPath2013_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2. A typical session using ImageJ and the NDPITools plugins<\/b>. A NDPI file has been opened with the NDPITools plugins and it is displayed as a preview image (image at largest resolution which still fits into the computer\u2019s screen) \u2014 top window. A rectangular region has been selected and extracted as a TIFF image, then opened \u2014 bottom window.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Deroulers_DiagnosticPath2013_8.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"76f84f37c820d4eec2f047fc64aa6ac9\"><img alt=\"Fig3 Deroulers DiagnosticPath2013 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/90\/Fig3_Deroulers_DiagnosticPath2013_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3. Preview image of a NDPI file with several focalization levels in ImageJ<\/b>. The NDPI file <tt>08.ndpi<\/tt> contains images at 5 different focalization levels. Therefore, its preview image is displayed as a stack of 5 images.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>When producing a mosaic, the user can request that pieces be JPEG files. Since the <tt>File > Open<\/tt> command of versions 1.x of ImageJ is unable to open TIFF files with JPEG compression (one has to use plugins), this is way to produce mosaics which can be opened by click-and-drag onto the window or icon of ImageJ while still saving disk space thanks to efficient compression. Figure 4 shows how the mosaic production options can be set inside ImageJ through the NDPITools plugins. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Deroulers_DiagnosticPath2013_8.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5af0868794f86b288e2da5082729c635\"><img alt=\"Fig4 Deroulers DiagnosticPath2013 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/d\/d5\/Fig4_Deroulers_DiagnosticPath2013_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4. Dialog box for customized extraction in ImageJ from an NDPI file with production of a mosaic<\/b>. The dialog box shows some options which can be customized while producing a mosaic from a rectangular selection of a NDPI file preview image (here, using the file previewed in Figure 3).<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Performance\">Performance<\/span><\/h3>\n<p>We compare the performance of our tools on several fundamental tasks to standard, broadly available software in representative examples and on broadly available computers.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Making_a_mosaic_from_a_huge_image\">Making a mosaic from a huge image<\/span><\/h3>\n<p>We chose an 8-bit RGB colour JPEG-compressed TIFF file of 103168\u00d763232 pixels originating in the digitization of a pathology slide. The original file weighted 975.01 MiB. Loading this image entirely into RAM would need at least 3\u00d7103168\u00d763232=18.2 GiB and is presently intractable on most if not all desktop and laptop computers of reasonable cost.\n<\/p><p>The task was to produce, from this image, a mosaic of 64 pieces so that each one needs less than 512 MiB RAM to open.\n<\/p><p>On a 3.2 GHz Intel Core i3 IMac computer with 16 GB of RAM, the convert command from ImageMagick (version 6.8.0-7 with quantum size 8 bits) was unable to complete the request. GraphicsMagick (<tt>gm convert-crop<\/tt>; version 1.3.17 with quantum size 8 bits) completed the request in 70 min, using 25 GiB of disk space. <tt>tiffmakemosaic<\/tt> from our LargeTIFFTools completed the request in 2.5 min.\n<\/p><p>To ascertain that this task can be equally achieved even on computers with a modest RAM amount, we performed the same task on a 6-year-old 2.66 GHz Core2Duo Intel IMac with 2 GiB RAM. The task was completed in 9.0 min. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Converting_NDPI_into_TIFF\">Converting NDPI into TIFF<\/span><\/h3>\n<p><i>Splitting a NDPI file into TIFF files.<\/i> A pathology sample (6.7 cm<super>2<\/super> of tissue) was scanned at magnification 40x and with 11 focus levels (every 2 microns) by a NanoZoomer, resulting in a 6.5 GiB file in proprietary NDPI format (called file <tt>a.ndpi<\/tt> hereafter). On a 2.6 GHz Intel Core i7 Mac Mini computer with 16 GiB RAM, <tt>ndpisplit<\/tt> extracted all 55 images (11 focus levels and 5 magnifications) as independent, single-image TIFF files with JPEG compression in 7.11 min. The size of the largest images was 180224\u00d770144. The speed was limited only by the rate of I\/O transfers since the CPU usage of this task was 1.38 min, out of which the system used 1.30 min. Executing again the same task straight after the first execution took only 0.57 min because the NDPI file was still in the cache of the operating system.\n<\/p><p>To ascertain that this task can be equally achieved even on computers with a modest RAM amount, we made a try on a 6-year-old 2.66 GHz Core2Duo Intel PC with 2 GiB RAM running 32-bits Windows XP Pro SP3. The original file shown in Figure 1, called <tt>b.ndpi<\/tt>, and weighting 2.07 GiB (largest image: 103168\u00d763232 pixels), was split into independent TIFF files in 2.2 min without swapping.\n<\/p><p>In comparison, the LOCI Bio-Formats plugins for ImageJ<sup id=\"rdp-ebb-cite_ref-LinkertMeta10_25-2\" class=\"reference\"><a href=\"#cite_note-LinkertMeta10-25\" rel=\"external_link\">[25]<\/a><\/sup>, in its version 4.4.6 with ImageJ 1.43 m, was not able to open the images in file <tt>a.ndpi<\/tt> even at low resolution. \n<\/p><p><i>Converting a NDPI file into a multiple-images TIFF file.<\/i> Alternatively, the same proprietary-format file <tt>a.ndpi<\/tt> was converted into a multiple-images TIFF file with <tt>ndpi2tiff<\/tt>. On the same computer as before, the conversion time was 7.0 min. Here again, the speed of the process is limited only by the rate of I\/O transfers since the conversion took only 30 s if performed when the NDPI file was still in the cache of the operating system.\n<\/p><p>Since the resulting TIFF file could not store all 55 images in less than 4 GiB, we passed the option <tt>-8<\/tt> on the command line to <tt>ndpi2tiff<\/tt> to request using the BigTIFF format extension. The specifications of this extension to the TIFF standard, discussed and published before 2008<sup id=\"rdp-ebb-cite_ref-BTDesign12_33-1\" class=\"reference\"><a href=\"#cite_note-BTDesign12-33\" rel=\"external_link\">[33]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BTProp08_34-1\" class=\"reference\"><a href=\"#cite_note-BTProp08-34\" rel=\"external_link\">[34]<\/a><\/sup>, are supported by LibTIFF as of version 4.0.0<sup id=\"rdp-ebb-cite_ref-LefflerLib12_27-1\" class=\"reference\"><a href=\"#cite_note-LefflerLib12-27\" rel=\"external_link\">[27]<\/a><\/sup>, and therefore by the abundant image viewing and manipulation software which relies on LibTIFF. If the use of the BigTIFF format extension would have impeded the further exploitation of the produced TIFF file, we could have simply used <tt>ndpisplit<\/tt> as above. Or we could have called the <tt>ndpi2tiff<\/tt> command several times, each time requesting extraction of a subset of all images by specifying image numbers after the file name, separated with commas, as in <tt>a.ndpi,0,1,2,3,4<\/tt>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Extracting_a_small_region_from_a_huge_image\">Extracting a small region from a huge image<\/span><\/h3>\n<p>This task can be useful to visualize at full resolution a region of interest which the user has selected on a low-magnification preview image. Therefore, it should be performed as quickly as possible.\nFrom a TIFF file\n<\/p><p>The task was to extract a rectangular region of size 256\u00d7256 pixels situated at the bottom right corner of huge TIFF images and to save it as an independent file. The source images were single-image TIFF files using JPEG compression. Table 1 compares the time needed to complete the task with <tt>tifffastcrop<\/tt> from our LargeTIFFTools and with several software tools, on increasingly large TIFF files. Tests were performed on a 2.6 GHz Intel Core i7 Mac Mini computer with 16 GB of RAM and used GraphicsMagick 1.3.17, ImageMagick 6.8.0-7 and the utility <tt>tiffcrop<\/tt> from LibTIFF 4.0.3. Noticeably, when treating the largest image, GraphicsMagick needs 50 GiB of free disk space, whereas <tt>tifffastcrop<\/tt> doesn\u2019t need it. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 1. Speed comparison of software to extract a 256\u00d7256 rectangle from a huge TIFF image<\/b>\n<\/td><\/tr>\n<tr>\n<th> Image size (px)\n<\/th>\n<th> 11264\u00d74384\n<\/th>\n<th> 45056\u00d717536\n<\/th>\n<th> 180224\u00d770144\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><tt>tifffastcrop<\/tt>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.30 s\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.30 s\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.30 s\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GraphicsMagick\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.74 s\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23.6 s\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">>80 min\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ImageMagick\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.18 s\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">236 s\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">failed\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><tt>tiffcrop<\/tt>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.50 s\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">failed\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">seg. fault\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\">Time needed (or indication of failure when the task was not completed) by several software tools to extract a rectangular region of size <b>256\u00d7256<\/b> pixels situated at the bottom right corner of huge TIFF images, and to save it as an independent file. The input images were single-image tiled TIFF files using JPEG compression. Their dimensions are indicated in the top row. The computer used was a 2.6 GHz Intel Core i7 Mac Mini with 16 GiB of RAM and more than 100 GiB of free hard disk. The tested software tools were, from top to bottom, <tt>tifffastcrop<\/tt> from our LargeTIFFTools, GraphicsMagick 1.3.17, ImageMagick 6.8.0-7 and the utility <tt>tiffcrop<\/tt> from LibTIFF 4.0.3.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"From_a_NDPI_file\">From a NDPI file<\/span><\/h3>\n<p>The task was to extract a rectangular region of size 256\u00d7256 from one of the largest images of the file <tt>a.ndpi<\/tt> (size 180224\u00d770144). On a 2.6 GHz Intel Core i7 Mac Mini computer with 16 GB of RAM, the execution time was 0.12 s for one extract, and in average 0.06 s per extract in a series of 20 extracts with locations drawn uniformly at random inside the whole image. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Applications\">Applications<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Integration_in_digital_pathology_image_servers_or_virtual_slide_systems\">Integration in digital pathology image servers or virtual slide systems<\/span><\/h3>\n<p>The NDPITools are being used in several other software projects:\n<\/p><p>\u2022in a system for automatic blur detection<sup id=\"rdp-ebb-cite_ref-AmeisenStack13_2-1\" class=\"reference\"><a href=\"#cite_note-AmeisenStack13-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AmeisenInt13_4-1\" class=\"reference\"><a href=\"#cite_note-AmeisenInt13-4\" rel=\"external_link\">[4]<\/a><\/sup>.\n<\/p><p>\u2022in WIDE<sup id=\"rdp-ebb-cite_ref-GranierWIDE13_22-1\" class=\"reference\"><a href=\"#cite_note-GranierWIDE13-22\" rel=\"external_link\">[22]<\/a><\/sup>, to deal with NDPI files. WIDE is an open-source biological and digital pathology image archiving and visualization system, which allows the remote user to see images stored in a remote library in a browser. In particular, thanks to the feature of high-speed extraction of a rectangular region by <tt>ndpisplit<\/tt>, WIDE saves costly disk space since it doesn\u2019t need to store TIFF files converted from NDPI files in addition to the latter. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Exploiting_a_large_set_of_digital_slides\">Exploiting a large set of digital slides<\/span><\/h3>\n<p>In the framework of a study about invasive low-grade oligodendrogliomas reported elsewhere<sup id=\"rdp-ebb-cite_ref-GerinQuant13_8-1\" class=\"reference\"><a href=\"#cite_note-GerinQuant13-8\" rel=\"external_link\">[8]<\/a><\/sup>, we had to deal with 303 NDPI files, occupying 122 GiB. On a 3.2 GHz Intel Core i3 IMac computer with 16 GB RAM, we used ndpisplit in a batch work to convert them into standard TIFF files, which took only a few hours. The experimental <tt>-s<\/tt> option of <tt>ndpisplit<\/tt> was used to remove the blank filling between scanned regions, resulting in an important disk space saving and in smaller TIFF files (one for each scanned region) which where easier to manipulate afterwards. Then, for each sample, Preview.app and ImageJ were used to inspect the resulting images and manually select the regions of clinical interest. The corresponding extracts of the high magnification images were the subject of automated cell counting and other quantitative analyses using ImageJ. In particular, we collected quantitative data about edema or tissue hyperhydration<sup id=\"rdp-ebb-cite_ref-GerinQuant13_8-2\" class=\"reference\"><a href=\"#cite_note-GerinQuant13-8\" rel=\"external_link\">[8]<\/a><\/sup>. This quantity needed a specific image analysis procedure which is not offered by standard morphometry software and, unlike cell density estimates, could not be retrieved by sampling a few fields of view in the microscope. Therefore, virtual microscopy and our tools were essential in this study. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Study_of_a_whole_slide_of_brain_tissue_invaded_by_an_oligodendroglioma\">Study of a whole slide of brain tissue invaded by an oligodendroglioma<\/span><\/h3>\n<p>To demonstrate the possibility to do research on huge images even with a modest computer, we chose a 3-year-old MacBook Pro laptop computer with 2.66 GHz Intel Core 2 Duo and 4 GiB of RAM. We used ImageJ and the NDPITools to perform statistics on the upper piece of tissue on the slide shown in Figure 1.\n<\/p><p>Since the digital slide <tt>b.ndpi<\/tt> weighted 2.07 GiB, with a high resolution image of 103168\u00d763232 pixels, it was not possible to do the study in a straightforward way. We opened the file <tt>b.ndpi<\/tt> as a preview image with the command <tt>Plugins > NDPITools > Preview NDPI...<\/tt> and selected on it the left tissue sample. Then we used the command <tt>Plugins > NDPITools > Custom extract to TIFF \/ Mosaic...<\/tt> and asked for extraction as a mosaic of 16 JPEG files, each one needing less than 1 GiB of RAM to open, and with an overlap of 60 pixels. This was completed within a few minutes. Then we applied an ImageJ macro to each of the 16 pieces to identify the dark cell nuclei (those with high chromatin content), based on thresholding the luminosity values of the pixels, as shown in Figure 1. It produced text files with the coordinates and size of each cell nucleus.\n<\/p><p>Out of the 154240 identified nuclei, 1951 were positioned on the overlapping regions between pieces. Using the overlap feature of our tools enabled to properly detect these nuclei, since they would have been cut by the boundary of the pieces of the mosaic in absence of overlap. We avoided double counting by identifying the pairs of nuclei situated in the overlapping regions and which were separated by a distance smaller than their radius.\n<\/p><p>As shown in earlier studies<sup id=\"rdp-ebb-cite_ref-GurcanHist09_7-1\" class=\"reference\"><a href=\"#cite_note-GurcanHist09-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GunduzTheCell04_10-1\" class=\"reference\"><a href=\"#cite_note-GunduzTheCell04-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GunduzAug05_11-1\" class=\"reference\"><a href=\"#cite_note-GunduzAug05-11\" rel=\"external_link\">[11]<\/a><\/sup>, these data can be used for research and diagnosis purposes. As an example, Figure 5 shows the distribution of the distance of each cell nucleus to its nearest neighbor. Thanks to the very high number of analyzed cell nuclei, this distribution is obtained with an excellent precision. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Deroulers_DiagnosticPath2013_8.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"981a8e074b31a994912bbf30057a083a\"><img alt=\"Fig5 Deroulers DiagnosticPath2013 8.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/ff\/Fig5_Deroulers_DiagnosticPath2013_8.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 5.<\/b> The positions of the 154240 identified nuclei were obtained from the analysis with ImageJ of the digital slide on a laptop computer. Since the slide was too large to fit into the computer\u2019s memory, it was turned into a mosaic of 16 pieces with overlap of 60 pixels, and each piece underwent automated analysis independently. Then the results were aggregated. The graph shows the probability density function of the distance of a cell nucleus to its nearest neighbor in the whole sample.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Conclusions_2\">Conclusions<\/span><\/h2>\n<p>The LargeTIFFTools, NDPITools and NDPITools plugins for ImageJ achieve efficiently some fundamental functions on large images and in particular digital slides, for which standard open source software fails or performs badly. They enable both the clinician to examine a single slide and the bioinformatics research team to perform large-scale analysis of many slides, possibly on computer grids.<sup id=\"rdp-ebb-cite_ref-KongAComp11_20-1\" class=\"reference\"><a href=\"#cite_note-KongAComp11-20\" rel=\"external_link\">[20]<\/a><\/sup>\n<\/p><p>To date, the LargeTIFFTools have been downloaded from more than 388 different IP addresses, the NDPITools from more than 1361 addresses, and the ImageJ plugins from more than 235 addresses. Table 2 lists the distribution of the target platforms among the downloads of the binary files. It shows a broad usage of the different platforms by the community, emphasizing the importance of cross-platform, open source tools. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\"><b>Table 2. Downloads of the NDPITools<\/b>\n<\/td><\/tr>\n<tr>\n<th> Windows (32 bits)\n<\/th>\n<th> Windows(64 bits)\n<\/th>\n<th> Linux\n<\/th>\n<th> Mac OS X\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">483\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">542\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">217\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">285\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"4\">Distribution of the downloads (unique IP address) of the precompiled binaries of the NDPITools between March 2012 and April 2013.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We have explained how the software was used to study some microscopic properties of brain tissue when invaded by an oligodendroglioma, and we have given an illustrative application to the analysis of a whole-size pathology slide. This suggests other promising applications. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p>a. LargeTIFFTools\n<\/p><p>\u2022<b>Project name<\/b>: LargeTIFFTools\n<\/p><p>\u2022<b>Project home page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/largetifftools\/\" target=\"_blank\">http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/largetifftools\/<\/a>\n<\/p><p>\u2022<b>Operating system(s)<\/b>: Platform independent\n<\/p><p>\u2022<b>Programming language<\/b>: C\n<\/p><p>\u2022<b>Other requirements<\/b>: libjpeg, libtiff\n<\/p><p>\u2022<b>License<\/b>: GNU GPLv3\n<\/p><p><br \/>\nb. NDPITools\n<\/p><p>\u2022<b>Project name<\/b>: NDPITools\n<\/p><p>\u2022<b>Project home page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/ndpitools\/\" target=\"_blank\">http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/ndpitools\/<\/a>\n<\/p><p>\u2022<b>Operating system(s)<\/b>: Platform independent\n<\/p><p>\u2022<b>Programming language<\/b>: C\n<\/p><p>\u2022<b>Other requirements<\/b>: \u2014\n<\/p><p>\u2022<b>License<\/b>: GNU GPLv3\n<\/p><p>For the convenience of users, precompiled binaries are provided for Windows (32 and 64 bits), Mac OS X and Linux.\n<\/p><p><br \/>\nc. NDPITools plugins for ImageJ\n<\/p><p>\u2022<b>Project name<\/b>: NDPITools plugins for ImageJ\n<\/p><p>\u2022<b>Project home page<\/b>: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/ndpitools\/\" target=\"_blank\">http:\/\/www.imnc.in2p3.fr\/pagesperso\/deroulers\/software\/ndpitools\/<\/a>\n<\/p><p>\u2022<b>Operating system(s)<\/b>: Platform independent\n<\/p><p>\u2022<b>Programming language<\/b>: Java\n<\/p><p>\u2022<b>Other requirements<\/b>: ImageJ 1.31s or higher, Ant, JAI 1.1.3\n\n\u2022License<b>: GNU GPLv3<\/b>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h2>\n<p>CD wrote the paper. ML conceived and implemented a first version of the integration into ImageJ as a toolset of macros. CD implemented the software and wrote the documentation. CG, AG and ML contributed suggestions to the software. CD, DA, AG and ML performed software tests. CD, MB, CG, AG and ML selected and provided histological samples. CD performed the statistical analysis of the sample slide. All authors reviewed the manuscript. All authors read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We thank F. Bouhidel and P. Bertheau for their help with the slide scanner of the Pathology Laboratory of the Saint-Louis Hospital in Paris, and C. Klein (Imaging facility, Cordeliers Research Center \u2013 INSERM U872, Paris) for tests and suggestions.\n<\/p><p>The computer, CPU, operating system, and programming language names quoted in this article are trademarks of their respective owners.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-DiamondVirt09-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DiamondVirt09_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-DiamondVirt09_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Diamond, J.; McCleary, D. (2009). \"Virtual Microscopy\". In Hannon-Fletcher, M.; Maxwell, P.. <i>Advanced Techniques in Diagnostic Cellular Pathology<\/i>. Chichester UK: John Wiley & Sons, Ltd. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780470515976.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Virtual+Microscopy&rft.atitle=Advanced+Techniques+in+Diagnostic+Cellular+Pathology&rft.aulast=Diamond%2C+J.%3B+McCleary%2C+D.&rft.au=Diamond%2C+J.%3B+McCleary%2C+D.&rft.date=2009&rft.place=Chichester+UK&rft.pub=John+Wiley+%26+Sons%2C+Ltd&rft.isbn=9780470515976&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AmeisenStack13-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AmeisenStack13_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-AmeisenStack13_2-1\" rel=\"external_link\">2.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ameisen, D.; Yun\u00e8s, J.B.; Deroulers, C.; Perrier, V.; Bouhidel, F.; Battistella, M.; Legr\u00e8s, L.; Janin, A.; Bertheau, P. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849546\" target=\"_blank\">\"Stack or Trash? Fast quality assessment of virtual slides\"<\/a>. <i>Diagnostic Pathology<\/i> <b>8<\/b> (Suppl 1): S23. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1746-1596-8-S1-S23\" target=\"_blank\">10.1186\/1746-1596-8-S1-S23<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3849546\/\" target=\"_blank\">PMC3849546<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849546\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3849546<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Stack+or+Trash%3F+Fast+quality+assessment+of+virtual+slides&rft.jtitle=Diagnostic+Pathology&rft.aulast=Ameisen%2C+D.%3B+Yun%C3%A8s%2C+J.B.%3B+Deroulers%2C+C.%3B+Perrier%2C+V.%3B+Bouhidel%2C+F.%3B+Battistella%2C+M.%3B+Legr%C3%A8s%2C+L.%3B+Janin%2C+A.%3B+Bertheau%2C+P.&rft.au=Ameisen%2C+D.%3B+Yun%C3%A8s%2C+J.B.%3B+Deroulers%2C+C.%3B+Perrier%2C+V.%3B+Bouhidel%2C+F.%3B+Battistella%2C+M.%3B+Legr%C3%A8s%2C+L.%3B+Janin%2C+A.%3B+Bertheau%2C+P.&rft.date=2013&rft.volume=8&rft.issue=Suppl+1&rft.pages=S23&rft_id=info:doi\/10.1186%2F1746-1596-8-S1-S23&rft_id=info:pmc\/PMC3849546&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3849546&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GarciaCOST11-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GarciaCOST11_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-GarciaCOST11_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Garc\u00eda Rojo, M.; Castro, A.M.; Gon\u00e7alves, L. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073224\" target=\"_blank\">\"COST action \u201cEuroTelepath\u201d: digital pathology integration in electronic health record, including primary care centres\"<\/a>. <i>Diagnostic Pathology<\/i> <b>6<\/b> (Suppl 1): S6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1746-1596-6-S1-S6\" target=\"_blank\">10.1186\/1746-1596-6-S1-S6<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3073224\/\" target=\"_blank\">PMC3073224<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21489201\" target=\"_blank\">21489201<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073224\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073224<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=COST+action+%E2%80%9CEuroTelepath%E2%80%9D%3A+digital+pathology+integration+in+electronic+health+record%2C+including+primary+care+centres&rft.jtitle=Diagnostic+Pathology&rft.aulast=Garc%C3%ADa+Rojo%2C+M.%3B+Castro%2C+A.M.%3B+Gon%C3%A7alves%2C+L.&rft.au=Garc%C3%ADa+Rojo%2C+M.%3B+Castro%2C+A.M.%3B+Gon%C3%A7alves%2C+L.&rft.date=2011&rft.volume=6&rft.issue=Suppl+1&rft.pages=S6&rft_id=info:doi\/10.1186%2F1746-1596-6-S1-S6&rft_id=info:pmc\/PMC3073224&rft_id=info:pmid\/21489201&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3073224&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AmeisenInt13-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AmeisenInt13_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-AmeisenInt13_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Ameisen, D. (2013). \"Int\u00e9gration des lames virtuelles dans le dossier patient \u00e9lectronique\". <i>PhD thesis<\/i>. Univ Paris Diderot-Paris 7.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Int%C3%A9gration+des+lames+virtuelles+dans+le+dossier+patient+%C3%A9lectronique&rft.atitle=PhD+thesis&rft.aulast=Ameisen%2C+D.&rft.au=Ameisen%2C+D.&rft.date=2013&rft.pub=Univ+Paris+Diderot-Paris+7&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CollanApp87-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CollanApp87_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Collan, Y.; Torkkeli, T.; Personen, E.; Jantunen, E.; Kosma, V.M. (1987). \"Application of morphometry in tumor pathology\". <i>Analytical and Quantitative Cytology and Histology<\/i> <b>9<\/b> (2): 79\u201388. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/3300687\" target=\"_blank\">3300687<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Application+of+morphometry+in+tumor+pathology&rft.jtitle=Analytical+and+Quantitative+Cytology+and+Histology&rft.aulast=Collan%2C+Y.%3B+Torkkeli%2C+T.%3B+Personen%2C+E.%3B+Jantunen%2C+E.%3B+Kosma%2C+V.M.&rft.au=Collan%2C+Y.%3B+Torkkeli%2C+T.%3B+Personen%2C+E.%3B+Jantunen%2C+E.%3B+Kosma%2C+V.M.&rft.date=1987&rft.volume=9&rft.issue=2&rft.pages=79%E2%80%9388&rft_id=info:pmid\/3300687&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WolfeUsing04-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WolfeUsing04_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wolfe, P.; Murphy, J.; McGinley, J.; Zhu, Z.; Jiang, W.; Gottschall, E.; Thompson, H. (2004). \"Using nuclear morphometry to discriminate the tumorigenic potential of cells: A comparison of statistical methods\". <i>Cancer Epidemiology, Biomarkers & Prevention<\/i> <b>13<\/b> (6): 976\u2013988. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15184254\" target=\"_blank\">15184254<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Using+nuclear+morphometry+to+discriminate+the+tumorigenic+potential+of+cells%3A+A+comparison+of+statistical+methods&rft.jtitle=Cancer+Epidemiology%2C+Biomarkers+%26+Prevention&rft.aulast=Wolfe%2C+P.%3B+Murphy%2C+J.%3B+McGinley%2C+J.%3B+Zhu%2C+Z.%3B+Jiang%2C+W.%3B+Gottschall%2C+E.%3B+Thompson%2C+H.&rft.au=Wolfe%2C+P.%3B+Murphy%2C+J.%3B+McGinley%2C+J.%3B+Zhu%2C+Z.%3B+Jiang%2C+W.%3B+Gottschall%2C+E.%3B+Thompson%2C+H.&rft.date=2004&rft.volume=13&rft.issue=6&rft.pages=976%E2%80%93988&rft_id=info:pmid\/15184254&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GurcanHist09-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GurcanHist09_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-GurcanHist09_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">G\u00fcrcan, M.N.; Boucheron, L.E.; Can, A.; Madabhushi, A.; Rajpoot, N.M.; Yener, B. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2910932\" target=\"_blank\">\"Histopathological image analysis: a review\"<\/a>. <i>IEEE Reviews in Biomedical Engineering<\/i> <b>2009<\/b> (2): 147\u2013171. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FRBME.2009.2034865\" target=\"_blank\">10.1109\/RBME.2009.2034865<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2910932\/\" target=\"_blank\">PMC2910932<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2910932\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2910932<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Histopathological+image+analysis%3A+a+review&rft.jtitle=IEEE+Reviews+in+Biomedical+Engineering&rft.aulast=G%C3%BCrcan%2C+M.N.%3B+Boucheron%2C+L.E.%3B+Can%2C+A.%3B+Madabhushi%2C+A.%3B+Rajpoot%2C+N.M.%3B+Yener%2C+B.&rft.au=G%C3%BCrcan%2C+M.N.%3B+Boucheron%2C+L.E.%3B+Can%2C+A.%3B+Madabhushi%2C+A.%3B+Rajpoot%2C+N.M.%3B+Yener%2C+B.&rft.date=2009&rft.volume=2009&rft.issue=2&rft.pages=147%E2%80%93171&rft_id=info:doi\/10.1109%2FRBME.2009.2034865&rft_id=info:pmc\/PMC2910932&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2910932&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GerinQuant13-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GerinQuant13_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-GerinQuant13_8-1\" rel=\"external_link\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-GerinQuant13_8-2\" rel=\"external_link\">8.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gerin, C.; Pallud, J.; Deroulers, C.; Varlet, P.; Oppenheim, C.; Roux, F.X.; Chr\u00e9tien, F.; Thomas, S.R.; Grammaticos, B.; Badoual, M. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3779035\" target=\"_blank\">\"Quantitative characterization of the imaging limits of diffuse low-grade oligodendrogliomas\"<\/a>. <i>Neuro-Oncology<\/i> <b>15<\/b> (10): 1379-88. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fneuonc%2Fnot072\" target=\"_blank\">10.1093\/neuonc\/not072<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3779035\/\" target=\"_blank\">PMC3779035<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23771168\" target=\"_blank\">23771168<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3779035\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3779035<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Quantitative+characterization+of+the+imaging+limits+of+diffuse+low-grade+oligodendrogliomas&rft.jtitle=Neuro-Oncology&rft.aulast=Gerin%2C+C.%3B+Pallud%2C+J.%3B+Deroulers%2C+C.%3B+Varlet%2C+P.%3B+Oppenheim%2C+C.%3B+Roux%2C+F.X.%3B+Chr%C3%A9tien%2C+F.%3B+Thomas%2C+S.R.%3B+Grammaticos%2C+B.%3B+Badoual%2C+M.&rft.au=Gerin%2C+C.%3B+Pallud%2C+J.%3B+Deroulers%2C+C.%3B+Varlet%2C+P.%3B+Oppenheim%2C+C.%3B+Roux%2C+F.X.%3B+Chr%C3%A9tien%2C+F.%3B+Thomas%2C+S.R.%3B+Grammaticos%2C+B.%3B+Badoual%2C+M.&rft.date=2013&rft.volume=15&rft.issue=10&rft.pages=1379-88&rft_id=info:doi\/10.1093%2Fneuonc%2Fnot072&rft_id=info:pmc\/PMC3779035&rft_id=info:pmid\/23771168&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3779035&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WienertCog13-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WienertCog13_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wienert, S.; Heim, D.; Kotani, M.; Lindequist, B.; Stenzinger, A.; Ishii, M.; Hufnagl, P.; Beil, M.; Dietel, M.; Denkert, C.; Klauschen, F. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626931\" target=\"_blank\">\"CognitionMaster: an object-based image analysis framework\"<\/a>. <i>Diagnostic Pathology<\/i> <b>8<\/b>: 34. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1746-1596-8-34\" target=\"_blank\">10.1186\/1746-1596-8-34<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3626931\/\" target=\"_blank\">PMC3626931<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23445542\" target=\"_blank\">23445542<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626931\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3626931<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=CognitionMaster%3A+an+object-based+image+analysis+framework&rft.jtitle=Diagnostic+Pathology&rft.aulast=Wienert%2C+S.%3B+Heim%2C+D.%3B+Kotani%2C+M.%3B+Lindequist%2C+B.%3B+Stenzinger%2C+A.%3B+Ishii%2C+M.%3B+Hufnagl%2C+P.%3B+Beil%2C+M.%3B+Dietel%2C+M.%3B+Denkert%2C+C.%3B+Klauschen%2C+F.&rft.au=Wienert%2C+S.%3B+Heim%2C+D.%3B+Kotani%2C+M.%3B+Lindequist%2C+B.%3B+Stenzinger%2C+A.%3B+Ishii%2C+M.%3B+Hufnagl%2C+P.%3B+Beil%2C+M.%3B+Dietel%2C+M.%3B+Denkert%2C+C.%3B+Klauschen%2C+F.&rft.date=2013&rft.volume=8&rft.pages=34&rft_id=info:doi\/10.1186%2F1746-1596-8-34&rft_id=info:pmc\/PMC3626931&rft_id=info:pmid\/23445542&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3626931&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GunduzTheCell04-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GunduzTheCell04_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-GunduzTheCell04_10-1\" rel=\"external_link\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gunduz, C.; Yener, B.; Gultekin, S.H. (2004). \"The cell graphs of cancer\". <i>Bioinformatics<\/i> <b>20<\/b> (Suppl 1): i145-i151. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbth933\" target=\"_blank\">10.1093\/bioinformatics\/bth933<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15262793\" target=\"_blank\">15262793<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+cell+graphs+of+cancer&rft.jtitle=Bioinformatics&rft.aulast=Gunduz%2C+C.%3B+Yener%2C+B.%3B+Gultekin%2C+S.H.&rft.au=Gunduz%2C+C.%3B+Yener%2C+B.%3B+Gultekin%2C+S.H.&rft.date=2004&rft.volume=20&rft.issue=Suppl+1&rft.pages=i145-i151&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbth933&rft_id=info:pmid\/15262793&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GunduzAug05-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GunduzAug05_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-GunduzAug05_11-1\" rel=\"external_link\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gunduz, C.; Gultekin, S.H.; Yener, B. (2005). \"Augmented cell-graphs for automated cancer diagnosis\". <i>Bioinformatics<\/i> <b>21<\/b> (Suppl 2): ii7-ii12. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbti1100\" target=\"_blank\">10.1093\/bioinformatics\/bti1100<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16204128\" target=\"_blank\">16204128<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Augmented+cell-graphs+for+automated+cancer+diagnosis&rft.jtitle=Bioinformatics&rft.aulast=Gunduz%2C+C.%3B+Gultekin%2C+S.H.%3B+Yener%2C+B.&rft.au=Gunduz%2C+C.%3B+Gultekin%2C+S.H.%3B+Yener%2C+B.&rft.date=2005&rft.volume=21&rft.issue=Suppl+2&rft.pages=ii7-ii12&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbti1100&rft_id=info:pmid\/16204128&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WestThePro10-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WestThePro10_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">West, N.P.; Dattani, M.; McShane, P.; Hutchins, G.; Grabsch, J.; Mueller, W.; Treanor, D.; Quirke, P.; Grabsch, H. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2869173\" target=\"_blank\">\"The proportion of tumour cells is an independent predictor for survival in colorectal cancer patients\"<\/a>. <i>British Journal of Cancer<\/i> <b>102<\/b> (10): 1519\u20131523. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fsj.bjc.6605674\" target=\"_blank\">10.1038\/sj.bjc.6605674<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2869173\/\" target=\"_blank\">PMC2869173<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20407439\" target=\"_blank\">20407439<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2869173\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2869173<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+proportion+of+tumour+cells+is+an+independent+predictor+for+survival+in+colorectal+cancer+patients&rft.jtitle=British+Journal+of+Cancer&rft.aulast=West%2C+N.P.%3B+Dattani%2C+M.%3B+McShane%2C+P.%3B+Hutchins%2C+G.%3B+Grabsch%2C+J.%3B+Mueller%2C+W.%3B+Treanor%2C+D.%3B+Quirke%2C+P.%3B+Grabsch%2C+H.&rft.au=West%2C+N.P.%3B+Dattani%2C+M.%3B+McShane%2C+P.%3B+Hutchins%2C+G.%3B+Grabsch%2C+J.%3B+Mueller%2C+W.%3B+Treanor%2C+D.%3B+Quirke%2C+P.%3B+Grabsch%2C+H.&rft.date=2010&rft.volume=102&rft.issue=10&rft.pages=1519%E2%80%931523&rft_id=info:doi\/10.1038%2Fsj.bjc.6605674&rft_id=info:pmc\/PMC2869173&rft_id=info:pmid\/20407439&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2869173&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChangInv13-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChangInv13_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chang, H.; Han, J.; Borowsky, A.; Loss, L.; Gray, J.W.; Spellman, P.T.; Parvin, B. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3728287\" target=\"_blank\">\"Invariant delineation of nuclear architecture in Glioblastoma multiforme for clinical and molecular association\"<\/a>. <i>IEEE Transactions on Medical Imaging<\/i> <b>32<\/b> (4): 670\u2013682. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FTMI.2012.2231420\" target=\"_blank\">10.1109\/TMI.2012.2231420<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3728287\/\" target=\"_blank\">PMC3728287<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23221815\" target=\"_blank\">23221815<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3728287\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3728287<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Invariant+delineation+of+nuclear+architecture+in+Glioblastoma+multiforme+for+clinical+and+molecular+association&rft.jtitle=IEEE+Transactions+on+Medical+Imaging&rft.aulast=Chang%2C+H.%3B+Han%2C+J.%3B+Borowsky%2C+A.%3B+Loss%2C+L.%3B+Gray%2C+J.W.%3B+Spellman%2C+P.T.%3B+Parvin%2C+B.&rft.au=Chang%2C+H.%3B+Han%2C+J.%3B+Borowsky%2C+A.%3B+Loss%2C+L.%3B+Gray%2C+J.W.%3B+Spellman%2C+P.T.%3B+Parvin%2C+B.&rft.date=2013&rft.volume=32&rft.issue=4&rft.pages=670%E2%80%93682&rft_id=info:doi\/10.1109%2FTMI.2012.2231420&rft_id=info:pmc\/PMC3728287&rft_id=info:pmid\/23221815&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3728287&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KayserTowards06-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KayserTowards06_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kayser, K.; Radziszowski, D.; Bzdyl, P.; Sommer, R.; Kayser, G. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524814\" target=\"_blank\">\"Towards an automated virtual slide screening: theoretical considerations and practical experiences of automated tissue-based virtual diagnosis to be implemented in the internet\"<\/a>. <i>Diagnostic Pathology<\/i> <b>1<\/b>: 10. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1746-1596-1-10\" target=\"_blank\">10.1186\/1746-1596-1-10<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1524814\/\" target=\"_blank\">PMC1524814<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524814\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1524814<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Towards+an+automated+virtual+slide+screening%3A+theoretical+considerations+and+practical+experiences+of+automated+tissue-based+virtual+diagnosis+to+be+implemented+in+the+internet&rft.jtitle=Diagnostic+Pathology&rft.aulast=Kayser%2C+K.%3B+Radziszowski%2C+D.%3B+Bzdyl%2C+P.%3B+Sommer%2C+R.%3B+Kayser%2C+G.&rft.au=Kayser%2C+K.%3B+Radziszowski%2C+D.%3B+Bzdyl%2C+P.%3B+Sommer%2C+R.%3B+Kayser%2C+G.&rft.date=2006&rft.volume=1&rft.pages=10&rft_id=info:doi\/10.1186%2F1746-1596-1-10&rft_id=info:pmc\/PMC1524814&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1524814&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PLGAMeta12-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PLGAMeta12_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-PLGAMeta12_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">PLGA Foundation (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.fightplga.org\/research\/PLGA-Sponsored_Projects\/MetaAnalysis&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">\"Meta analysis low grade glioma database project\"<\/a>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.fightplga.org\/research\/PLGA-Sponsored_Projects\/MetaAnalysis\" target=\"_blank\">the original<\/a> on 11 July 2013<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.fightplga.org\/research\/PLGA-Sponsored_Projects\/MetaAnalysis&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.fightplga.org\/research\/PLGA-Sponsored_Projects\/MetaAnalysis&refdoi=10.1186\/1746-1596-8-92<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Meta+analysis+low+grade+glioma+database+project&rft.atitle=&rft.aulast=PLGA+Foundation&rft.au=PLGA+Foundation&rft.date=2012&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Fwww.fightplga.org%2Fresearch%2FPLGA-Sponsored_Projects%2FMetaAnalysis%26refdoi%3D10.1186%2F1746-1596-8-92&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GarciaRev09-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GarciaRev09_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-GarciaRev09_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Garc\u00eda Rojo, M.; Bueno, G.; Slodkowska, J. (2009). \"Review of imaging solutions for integrated quantitative immunohistochemistry in the Pathology daily practice\". <i>Folia Histochemica et Cytobiologica<\/i> <b>47<\/b> (3): 349\u2013354. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.2478%2Fv10042-008-0114-4\" target=\"_blank\">10.2478\/v10042-008-0114-4<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20164017\" target=\"_blank\">20164017<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Review+of+imaging+solutions+for+integrated+quantitative+immunohistochemistry+in+the+Pathology+daily+practice&rft.jtitle=Folia+Histochemica+et+Cytobiologica&rft.aulast=Garc%C3%ADa+Rojo%2C+M.%3B+Bueno%2C+G.%3B+Slodkowska%2C+J.&rft.au=Garc%C3%ADa+Rojo%2C+M.%3B+Bueno%2C+G.%3B+Slodkowska%2C+J.&rft.date=2009&rft.volume=47&rft.issue=3&rft.pages=349%E2%80%93354&rft_id=info:doi\/10.2478%2Fv10042-008-0114-4&rft_id=info:pmid\/20164017&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RasbandImageJ12-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RasbandImageJ12_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-RasbandImageJ12_17-1\" rel=\"external_link\">17.1<\/a><\/sup> <sup><a href=\"#cite_ref-RasbandImageJ12_17-2\" rel=\"external_link\">17.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Rasband, W.S. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/imagej.nih.gov\/ij\/\" target=\"_blank\">\"ImageJ\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/imagej.nih.gov\/ij\/\" target=\"_blank\">http:\/\/imagej.nih.gov\/ij\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ImageJ&rft.atitle=&rft.aulast=Rasband%2C+W.S.&rft.au=Rasband%2C+W.S.&rft.date=2012&rft_id=http%3A%2F%2Fimagej.nih.gov%2Fij%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IMSImage13-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IMSImage13_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">ImageMagick Studio, LLC (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.imagemagick.org\/\" target=\"_blank\">\"ImageMagick\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.imagemagick.org\/\" target=\"_blank\">http:\/\/www.imagemagick.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ImageMagick&rft.atitle=&rft.aulast=ImageMagick+Studio%2C+LLC&rft.au=ImageMagick+Studio%2C+LLC&rft.date=2013&rft_id=http%3A%2F%2Fwww.imagemagick.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GMGGraphics13-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GMGGraphics13_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">GraphicsMagick Group (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.graphicsmagick.org\/\" target=\"_blank\">\"GraphicsMagick\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.graphicsmagick.org\/\" target=\"_blank\">http:\/\/www.graphicsmagick.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GraphicsMagick&rft.atitle=&rft.aulast=GraphicsMagick+Group&rft.au=GraphicsMagick+Group&rft.date=2013&rft_id=http%3A%2F%2Fwww.graphicsmagick.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KongAComp11-20\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KongAComp11_20-0\" rel=\"external_link\">20.0<\/a><\/sup> <sup><a href=\"#cite_ref-KongAComp11_20-1\" rel=\"external_link\">20.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kong, J.; Cooper, L.A.D.; Wang, F.; Chisolm, C.; Moreno, C.S.; Kurc, T.M.; Widener, P.M.; Brat, D.J.; Saltz, J.H. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3256584\" target=\"_blank\">\"A comprehensive framework for classification of nuclei in digital microscopy imaging: An application to diffuse gliomas\"<\/a>. <i>IEEE International Symposium on Biomedical Imaging<\/i> <b>2011 Mar 30<\/b>: 2128\u20132131. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FISBI.2011.5872833\" target=\"_blank\">10.1109\/ISBI.2011.5872833<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3256584\/\" target=\"_blank\">PMC3256584<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3256584\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3256584<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comprehensive+framework+for+classification+of+nuclei+in+digital+microscopy+imaging%3A+An+application+to+diffuse+gliomas&rft.jtitle=IEEE+International+Symposium+on+Biomedical+Imaging&rft.aulast=Kong%2C+J.%3B+Cooper%2C+L.A.D.%3B+Wang%2C+F.%3B+Chisolm%2C+C.%3B+Moreno%2C+C.S.%3B+Kurc%2C+T.M.%3B+Widener%2C+P.M.%3B+Brat%2C+D.J.%3B+Saltz%2C+J.H.&rft.au=Kong%2C+J.%3B+Cooper%2C+L.A.D.%3B+Wang%2C+F.%3B+Chisolm%2C+C.%3B+Moreno%2C+C.S.%3B+Kurc%2C+T.M.%3B+Widener%2C+P.M.%3B+Brat%2C+D.J.%3B+Saltz%2C+J.H.&rft.date=2011&rft.volume=2011+Mar+30&rft.pages=2128%E2%80%932131&rft_id=info:doi\/10.1109%2FISBI.2011.5872833&rft_id=info:pmc\/PMC3256584&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3256584&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KayserGrid11-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KayserGrid11_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kayser, K.; G\u00f6rtler, J.; Borkenfeld, S.; Kayser, G. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073205\" target=\"_blank\">\"Grid computing in image analysis\"<\/a>. <i>Diagnostic Pathology<\/i> <b>6<\/b> (Suppl 1): S12. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3073205\/\" target=\"_blank\">PMC3073205<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21516880\" target=\"_blank\">21516880<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073205\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3073205<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Grid+computing+in+image+analysis&rft.jtitle=Diagnostic+Pathology&rft.aulast=Kayser%2C+K.%3B+G%C3%B6rtler%2C+J.%3B+Borkenfeld%2C+S.%3B+Kayser%2C+G.&rft.au=Kayser%2C+K.%3B+G%C3%B6rtler%2C+J.%3B+Borkenfeld%2C+S.%3B+Kayser%2C+G.&rft.date=2011&rft.volume=6&rft.issue=Suppl+1&rft.pages=S12&rft_id=info:pmc\/PMC3073205&rft_id=info:pmid\/21516880&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3073205&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GranierWIDE13-22\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GranierWIDE13_22-0\" rel=\"external_link\">22.0<\/a><\/sup> <sup><a href=\"#cite_ref-GranierWIDE13_22-1\" rel=\"external_link\">22.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Granier, A.; Olivier, M.; Laborie, S.; Vaudescal, S.; Baecker, V.; Tran-Aupiais, C. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.mri.cnrs.fr\/index.php?m=81&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">\"WIDE (Web Images and Data Environment)\"<\/a>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.mri.cnrs.fr\/index.php?m=81\" target=\"_blank\">the original<\/a> on 11 July 2013<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.mri.cnrs.fr\/index.php?m=81&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.mri.cnrs.fr\/index.php?m=81&refdoi=10.1186\/1746-1596-8-92<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=WIDE+%28Web+Images+and+Data+Environment%29&rft.atitle=&rft.aulast=Granier%2C+A.%3B+Olivier%2C+M.%3B+Laborie%2C+S.%3B+Vaudescal%2C+S.%3B+Baecker%2C+V.%3B+Tran-Aupiais%2C+C.&rft.au=Granier%2C+A.%3B+Olivier%2C+M.%3B+Laborie%2C+S.%3B+Vaudescal%2C+S.%3B+Baecker%2C+V.%3B+Tran-Aupiais%2C+C.&rft.date=2013&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Fwww.mri.cnrs.fr%2Findex.php%3Fm%3D81%26refdoi%3D10.1186%2F1746-1596-8-92&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KayserIntro12-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KayserIntro12_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kayser, K. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3441330\" target=\"_blank\">\"Introduction of virtual microscopy in routine surgical pathology \u2014 a hypothesis and personal view from Europe\"<\/a>. <i>Diagnostic Pathology<\/i> <b>7<\/b>: 48. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1746-1596-7-48\" target=\"_blank\">10.1186\/1746-1596-7-48<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3441330\/\" target=\"_blank\">PMC3441330<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22546238\" target=\"_blank\">22546238<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3441330\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3441330<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Introduction+of+virtual+microscopy+in+routine+surgical+pathology+%E2%80%94+a+hypothesis+and+personal+view+from+Europe&rft.jtitle=Diagnostic+Pathology&rft.aulast=Kayser%2C+K.&rft.au=Kayser%2C+K.&rft.date=2012&rft.volume=7&rft.pages=48&rft_id=info:doi\/10.1186%2F1746-1596-7-48&rft_id=info:pmc\/PMC3441330&rft_id=info:pmid\/22546238&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3441330&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoodeAVendor08-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GoodeAVendor08_24-0\" rel=\"external_link\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-GoodeAVendor08_24-1\" rel=\"external_link\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Goode, A.; Satyanarayanan, M. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/reports-archive.adm.cs.cmu.edu\/anon\/2008\/CMU-CS-08-136.pdf&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">\"A vendor-neutral library and viewer for whole-slide images\"<\/a> (PDF). <i>Technical Report CMU-CS-08-136<\/i>. School of Computer Science, Carnegie Mellon University. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/reports-archive.adm.cs.cmu.edu\/anon\/2008\/CMU-CS-08-136.pdf\" target=\"_blank\">the original<\/a> on 11 July 2013<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/reports-archive.adm.cs.cmu.edu\/anon\/2008\/CMU-CS-08-136.pdf&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/reports-archive.adm.cs.cmu.edu\/anon\/2008\/CMU-CS-08-136.pdf&refdoi=10.1186\/1746-1596-8-92<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+vendor-neutral+library+and+viewer+for+whole-slide+images&rft.atitle=Technical+Report+CMU-CS-08-136&rft.aulast=Goode%2C+A.%3B+Satyanarayanan%2C+M.&rft.au=Goode%2C+A.%3B+Satyanarayanan%2C+M.&rft.date=2008&rft.pub=School+of+Computer+Science%2C+Carnegie+Mellon+University&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Freports-archive.adm.cs.cmu.edu%2Fanon%2F2008%2FCMU-CS-08-136.pdf%26refdoi%3D10.1186%2F1746-1596-8-92&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LinkertMeta10-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LinkertMeta10_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-LinkertMeta10_25-1\" rel=\"external_link\">25.1<\/a><\/sup> <sup><a href=\"#cite_ref-LinkertMeta10_25-2\" rel=\"external_link\">25.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Linkert, M.; Rueden, C.T.; Allan, C.; Burel, J.M.; Moore, W.; Patterson, A.; Loranger, B.; Moore, J.; Neves, C.; MacDonald, D.; Tarkowska, A.; Sticco, C.; Hill, E.; Rossner, M.; Eliceiri, K.W.; Swedlow, J.R. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938\" target=\"_blank\">\"Metadata matters: access to image data in the real world\"<\/a>. <i>Journal of Cell Biology<\/i> <b>198<\/b> (5): 777-782. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1083%2Fjcb.201004104\" target=\"_blank\">10.1083\/jcb.201004104<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2878938\/\" target=\"_blank\">PMC2878938<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20513764\" target=\"_blank\">20513764<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Metadata+matters%3A+access+to+image+data+in+the+real+world&rft.jtitle=Journal+of+Cell+Biology&rft.aulast=Linkert%2C+M.%3B+Rueden%2C+C.T.%3B+Allan%2C+C.%3B+Burel%2C+J.M.%3B+Moore%2C+W.%3B+Patterson%2C+A.%3B+Loranger%2C+B.%3B+Moore%2C+J.%3B+Neves%2C+C.%3B+MacDonald%2C+D.%3B+Tarkowska%2C+A.%3B+Sticco%2C+C.%3B+Hill%2C+E.%3B+Rossner%2C+M.%3B+Eliceiri%2C+K.W.%3B+Swedlow%2C+J.R.&rft.au=Linkert%2C+M.%3B+Rueden%2C+C.T.%3B+Allan%2C+C.%3B+Burel%2C+J.M.%3B+Moore%2C+W.%3B+Patterson%2C+A.%3B+Loranger%2C+B.%3B+Moore%2C+J.%3B+Neves%2C+C.%3B+MacDonald%2C+D.%3B+Tarkowska%2C+A.%3B+Sticco%2C+C.%3B+Hill%2C+E.%3B+Rossner%2C+M.%3B+Eliceiri%2C+K.W.%3B+Swedlow%2C+J.R.&rft.date=2010&rft.volume=198&rft.issue=5&rft.pages=777-782&rft_id=info:doi\/10.1083%2Fjcb.201004104&rft_id=info:pmc\/PMC2878938&rft_id=info:pmid\/20513764&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2878938&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KhushiOpen13-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KhushiOpen13_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Khushi, M.; Edwards, G.; de Marcos, D.A.; Carpenter, J.E.; Graham, J.D.; Clarke, C.L. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3575263\" target=\"_blank\">\"Open source tools for management and archiving of digital microscopy data to allow integration with patient pathology and treatment information\"<\/a>. <i>Diagnostic Pathology<\/i> <b>8<\/b>: 22. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1746-1596-8-22\" target=\"_blank\">10.1186\/1746-1596-8-22<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3575263\/\" target=\"_blank\">PMC3575263<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23402499\" target=\"_blank\">23402499<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3575263\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3575263<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+source+tools+for+management+and+archiving+of+digital+microscopy+data+to+allow+integration+with+patient+pathology+and+treatment+information&rft.jtitle=Diagnostic+Pathology&rft.aulast=Khushi%2C+M.%3B+Edwards%2C+G.%3B+de+Marcos%2C+D.A.%3B+Carpenter%2C+J.E.%3B+Graham%2C+J.D.%3B+Clarke%2C+C.L.&rft.au=Khushi%2C+M.%3B+Edwards%2C+G.%3B+de+Marcos%2C+D.A.%3B+Carpenter%2C+J.E.%3B+Graham%2C+J.D.%3B+Clarke%2C+C.L.&rft.date=2013&rft.volume=8&rft.pages=22&rft_id=info:doi\/10.1186%2F1746-1596-8-22&rft_id=info:pmc\/PMC3575263&rft_id=info:pmid\/23402499&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3575263&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LefflerLib12-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LefflerLib12_27-0\" rel=\"external_link\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-LefflerLib12_27-1\" rel=\"external_link\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Sam Leffler, S.; the authors of LibTIFF (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.remotesensing.org\/libtiff\/\" target=\"_blank\">\"LibTIFF \u2013 TIFF Library and Utilities\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.remotesensing.org\/libtiff\/\" target=\"_blank\">http:\/\/www.remotesensing.org\/libtiff\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=LibTIFF+%E2%80%93+TIFF+Library+and+Utilities&rft.atitle=&rft.aulast=Sam+Leffler%2C+S.%3B+the+authors+of+LibTIFF&rft.au=Sam+Leffler%2C+S.%3B+the+authors+of+LibTIFF&rft.date=2012&rft_id=http%3A%2F%2Fwww.remotesensing.org%2Flibtiff%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LaneTheInd13-28\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LaneTheInd13_28-0\" rel=\"external_link\">28.0<\/a><\/sup> <sup><a href=\"#cite_ref-LaneTheInd13_28-1\" rel=\"external_link\">28.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lane, T.G.; Vollbeding, G. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ijg.org\/\" target=\"_blank\">\"The Independent JPEG Group\u2019s JPEG software\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ijg.org\/\" target=\"_blank\">http:\/\/www.ijg.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Independent+JPEG+Group%E2%80%99s+JPEG+software&rft.atitle=&rft.aulast=Lane%2C+T.G.%3B+Vollbeding%2C+G.&rft.au=Lane%2C+T.G.%3B+Vollbeding%2C+G.&rft.date=2013&rft_id=http%3A%2F%2Fwww.ijg.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Lanelibjpeg12-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Lanelibjpeg12_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Lane, T.G.; Vollbeding, G.; the authors of the libjpeg-turbo software (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/libjpeg-turbo.virtualgl.org\/\" target=\"_blank\">\"libjpeg-turbo\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/libjpeg-turbo.virtualgl.org\/\" target=\"_blank\">http:\/\/libjpeg-turbo.virtualgl.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=libjpeg-turbo&rft.atitle=&rft.aulast=Lane%2C+T.G.%3B+Vollbeding%2C+G.%3B+the+authors+of+the+libjpeg-turbo+software&rft.au=Lane%2C+T.G.%3B+Vollbeding%2C+G.%3B+the+authors+of+the+libjpeg-turbo+software&rft.date=2012&rft_id=http%3A%2F%2Flibjpeg-turbo.virtualgl.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchneiderNIH12-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchneiderNIH12_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schneider, C.A.; Rasband, W.S.; Eliceiri, K.W. (2012). \"NIH Image to ImageJ: 25 years of image analysis\". <i>Nature Methods<\/i> <b>9<\/b>: 671-675. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth.2089\" target=\"_blank\">10.1038\/nmeth.2089<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22930834\" target=\"_blank\">22930834<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NIH+Image+to+ImageJ%3A+25+years+of+image+analysis&rft.jtitle=Nature+Methods&rft.aulast=Schneider%2C+C.A.%3B+Rasband%2C+W.S.%3B+Eliceiri%2C+K.W.&rft.au=Schneider%2C+C.A.%3B+Rasband%2C+W.S.%3B+Eliceiri%2C+K.W.&rft.date=2012&rft.volume=9&rft.pages=671-675&rft_id=info:doi\/10.1038%2Fnmeth.2089&rft_id=info:pmid\/22930834&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SachaImage04-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SachaImage04_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Sacha, J. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ij-plugins.sourceforge.net\/plugins\/imageio\/\" target=\"_blank\">\"Image IO Plugin Bundle\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ij-plugins.sourceforge.net\/plugins\/imageio\/\" target=\"_blank\">http:\/\/ij-plugins.sourceforge.net\/plugins\/imageio\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Image+IO+Plugin+Bundle&rft.atitle=&rft.aulast=Sacha%2C+J.&rft.au=Sacha%2C+J.&rft.date=2004&rft_id=http%3A%2F%2Fij-plugins.sourceforge.net%2Fplugins%2Fimageio%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SunJava06-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SunJava06_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Sun Microsystems, Inc (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.oracle.com\/technetwork\/java\/current-142188.html&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">\"Java Advanced Library 1.1.3\"<\/a>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.oracle.com\/technetwork\/java\/current-142188.html\" target=\"_blank\">the original<\/a> on 11 July 2013<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.oracle.com\/technetwork\/java\/current-142188.html&refdoi=10.1186\/1746-1596-8-92\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.oracle.com\/technetwork\/java\/current-142188.html&refdoi=10.1186\/1746-1596-8-92<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Java+Advanced+Library+1.1.3&rft.atitle=&rft.aulast=Sun+Microsystems%2C+Inc&rft.au=Sun+Microsystems%2C+Inc&rft.date=2006&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Fwww.oracle.com%2Ftechnetwork%2Fjava%2Fcurrent-142188.html%26refdoi%3D10.1186%2F1746-1596-8-92&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BTDesign12-33\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BTDesign12_33-0\" rel=\"external_link\">33.0<\/a><\/sup> <sup><a href=\"#cite_ref-BTDesign12_33-1\" rel=\"external_link\">33.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.remotesensing.org\/libtiff\/bigtiffdesign.html\" target=\"_blank\">\"BigTIFF Design\"<\/a>. 2012<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.remotesensing.org\/libtiff\/bigtiffdesign.html\" target=\"_blank\">http:\/\/www.remotesensing.org\/libtiff\/bigtiffdesign.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=BigTIFF+Design&rft.atitle=&rft.date=2012&rft_id=http%3A%2F%2Fwww.remotesensing.org%2Flibtiff%2Fbigtiffdesign.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BTProp08-34\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BTProp08_34-0\" rel=\"external_link\">34.0<\/a><\/sup> <sup><a href=\"#cite_ref-BTProp08_34-1\" rel=\"external_link\">34.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.awaresystems.be\/imaging\/tiff\/bigtiff.html\" target=\"_blank\">\"The BigTIFF File Format Proposal\"<\/a>. 2008<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.awaresystems.be\/imaging\/tiff\/bigtiff.html\" target=\"_blank\">http:\/\/www.awaresystems.be\/imaging\/tiff\/bigtiff.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+BigTIFF+File+Format+Proposal&rft.atitle=&rft.date=2008&rft_id=http%3A%2F%2Fwww.awaresystems.be%2Fimaging%2Ftiff%2Fbigtiff.html&rfr_id=info:sid\/en.wikipedia.org:Journal:Analyzing_huge_pathology_images_with_open_source_software\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210224\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.813 seconds\nReal time usage: 0.858 seconds\nPreprocessor visited node count: 26335\/1000000\nPreprocessor generated node count: 38382\/1000000\nPost\u2010expand include size: 217225\/2097152 bytes\nTemplate argument size: 79908\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 808.089 1 - -total\n 85.59% 691.656 1 - Template:Reflist\n 74.81% 604.559 34 - Template:Citation\/core\n 47.80% 386.263 19 - Template:Cite_journal\n 25.44% 205.594 14 - Template:Cite_web\n 8.48% 68.506 45 - Template:Citation\/identifier\n 8.07% 65.204 1 - Template:Infobox_journal_article\n 7.69% 62.170 1 - Template:Infobox\n 6.02% 48.621 1 - Template:Cite_book\n 4.58% 37.040 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7712-0!*!*!!en!5!* and timestamp 20181213210223 and revision id 23626\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_huge_pathology_images_with_open_source_software\">https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_huge_pathology_images_with_open_source_software<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","89de2c69e566bc890426667a52566bb6_images":["https:\/\/www.limswiki.org\/images\/3\/3e\/Fig1_Deroulers_DiagnosticPath2013_8.jpg","https:\/\/www.limswiki.org\/images\/c\/ca\/Fig2_Deroulers_DiagnosticPath2013_8.jpg","https:\/\/www.limswiki.org\/images\/9\/90\/Fig3_Deroulers_DiagnosticPath2013_8.jpg","https:\/\/www.limswiki.org\/images\/d\/d5\/Fig4_Deroulers_DiagnosticPath2013_8.jpg","https:\/\/www.limswiki.org\/images\/f\/ff\/Fig5_Deroulers_DiagnosticPath2013_8.jpg"],"89de2c69e566bc890426667a52566bb6_timestamp":1544734943,"e85e2a721257f9447a4d6f2238effb07_type":"article","e85e2a721257f9447a4d6f2238effb07_title":"University-level practical activities in bioinformatics benefit voluntary groups of pupils in the last 2 years of school (Barker et al. 2015)","e85e2a721257f9447a4d6f2238effb07_url":"https:\/\/www.limswiki.org\/index.php\/Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school","e85e2a721257f9447a4d6f2238effb07_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:University-level practical activities in bioinformatics benefit voluntary groups of pupils in the last 2 years of school\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nUniversity-level practical activities in bioinformatics benefit voluntary groups of pupils in the last 2 years of schoolJournal\n \nInternational Journal of STEM EducationAuthor(s)\n \nBarker, Daniel; Alderson, Rosanna G.; McDonagh, James L.; Plaisier, Heleen;\r\nComrie, Muriel M.; Duncan, Leigh; Muirhead, Gavin T.P.; Sweeney, Stuart D.Author affiliation(s)\n \nUniversity of St. Andrews, University of Manchester, Kilgraston School, Forfar Academy, Portlethen AcademyPrimary contact\n \nEmail: db60@st-andrews.ac.ukYear published\n \n2015Volume and issue\n \n2Page(s)\n \n17DOI\n \n10.1186\/s40594-015-0030-zISSN\n \n2196-7822Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.stemeducationjournal.com\/content\/2\/1\/17Download\n \nhttp:\/\/www.stemeducationjournal.com\/content\/pdf\/s40594-015-0030-z.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Findings \n\n2.1 Introduction \n2.2 Methods \n2.3 Results \n2.4 Discussion \n2.5 Relevance to the new higher curriculum for biology and human biology \n2.6 The Raspberry Pi \u2014 a flexible, general-purpose computer \n2.7 Conclusion \n\n\n3 Abbreviations \n4 Competing interests \n5 Authors' contributions \n6 Additional files \n7 Acknowledgements \n8 References \n9 Notes \n\n\n\nAbstract \nBackground: Bioinformatics \u2014 the use of computers in biology \u2014 is of major and increasing importance to biological sciences and medicine. We conducted a preliminary investigation of the value of bringing practical, university-level bioinformatics education to the school level. We conducted voluntary activities for pupils at two schools in Scotland (years S5 and S6; pupils aged 15\u201317). We used material originally developed for an optional final-year undergraduate module and now incorporated into 4273\u03c0, a resource for teaching and learning bioinformatics on the low-cost Raspberry Pi computer.\nResults: Pupils\u2019 feedback forms suggested our activities were beneficial. During the course of the activity, they provide strong evidence of increase in the following: pupils\u2019 perception of the value of computers within biology; their knowledge of the Linux operating system and the Raspberry Pi; their willingness to use computers rather than phones or tablets; their ability to program a computer and their ability to analyse DNA sequences with a computer. We found no strong evidence of negative effects.\nConclusions: Our preliminary study supports the feasibility of bringing university-level, practical bioinformatics activities to school pupils.\nKeywords: Bioinformatics; Computational biology; Secondary school; Raspberry Pi; Open access teaching material; Case study\n\nFindings \nIntroduction \nProgress in Science, Technology, Engineering, Mathematics and Medicine (STEMM) subjects is increasingly dominated by computational analyses. In biological sciences, for example, the exceptional pace of recent advances in technology for DNA and genome sequencing has created a demand for computationally able researchers, to analyse the large amounts of data produced. A field specialising in application of computation to biological problems has emerged, known as bioinformatics. The development of bioinformatics is discussed by Hogeweg[1], and university-level bioinformatics education has been reviewed by Magana et al.[2]\nDNA sequences and related data are available at low cost (for new sequencing work) or free in online databases such as GenBank (Benson et al.[3]), Ensembl (Cunningham et al.[4]) and hundreds of others (Galperin et al.[5]). Software for bioinformatics research is usually free, for example the very widely used sequence database search software, BLAST (Altschul et al.[6]). Free resources are also available for bioinformatics teaching and learning, for example 4273\u03c0 (Barker et al.[7]), Bioinform\u00e1tica na escola (Marques et al.[8]), GOBLET (Corpas et al.[9]), Bioinformatics@school and the EvoEd Digital Library. These publicly available data, software and materials present excellent opportunities for relatively low-cost teaching. \nThere has been a recent, encouraging increase in exposure of school pupils to bioinformatics (e.g. Gallagher et al.[10]; Lewitter and Bourne[11]; McQueen et al.[12]; Kovarik et al.[13]; Machluf and Yarden[14]; Wood and Gebhardt[15]; Marques et al.[8]; Toby and Pope[16]). Genomics and associated topics have started to appear in many official school curricula, for example in Scotland (see \u201cDiscussion\u201d, below), the Netherlands (College voor Examens, p. 17[17]) and the USA (Wefer and Sheppard[18]). From a different angle, computer science is now a major part of the primary school curriculum for England. This is in line with a \u201cback to basics\u201d approach to computing currently emerging, as opposed to more traditional information and communications technology (ICT). In the UK, this change has been particularly associated with the low-cost Raspberry Pi computer, which is suitable for educational projects in electronics and engineering as well as general use and has sold over 5 million units. However, a practical link between computers and STEMM \u2014 which we will refer to as computational science, as opposed to computer science \u2014 still does not feature strongly on the UK school curriculum. DNA sequencing has a pervasive and increasing influence across traditionally disparate subject areas, including biochemistry, biomedical research, clinical medicine, evolutionary biology, ecology, neuroscience and anthropology. DNA sequencing is used to diagnose genetic and infectious diseases, discover drugs, characterise environments, monitor the progress of cancers, identify species and reveal evolutionary patterns. We consider increased amounts of practical bioinformatics at school to be a priority. \nMotivated by the increasing importance of bioinformatics to the life sciences and its appearance on school curricula, we conducted a preliminary investigation of the benefits of bringing university-level bioinformatics teaching material to voluntary groups of children in the last 2 years of school in Scotland (S5 and S6; pupils aged 15\u201317). The material was originally developed for an optional, final-year undergraduate module at the University of St Andrews, BL4273 Bioinformatics for Biologists. To better match bioinformatics as it is actually used in research at universities, institutes and industry, the material uses the Linux operating system, in this case a variant of Rasbpian Linux running on low-cost Raspberry Pi hardware. This material has been released under an open access licence, as part of 4273\u03c0 (Barker et al.[7]). Our proposition was that school pupils can benefit from practical, undergraduate-level bioinformatics teaching material. Compared to the undergraduates for whom this material was originally developed, school pupils are less experienced and knowledgeable about biology in general. However, their levels of practical bioinformatics experience are broadly similar: zero in the case of the school pupils, and approximately ten actual contact hours among undergraduates at the time of starting the module. \nMany of the skills developed in our activities, and 4273\u03c0 or bioinformatics in general, are generic skills in computational science. For example, although the programming language taught in the \u201cINTRO\u201d component \u2014 Perl \u2014 is particularly widely used in bioinformatics (e.g. Stajich et al.[19]; Stabenau et al.[20]), it is structurally similar to other programming languages widely used in science, including C, Fortran, Java, Python and R. Use of the command-line, emphasised in 4273\u03c0, is also essential in computational physics, computational chemistry and, indeed, computer science. Although computational chemistry is not yet part of the Higher qualification in Chemistry, several simulations are suggested by the Scottish Qualifications Authority.[21] Computational skills, as taught in 4273\u03c0, will be valuable to students taking chemistry, physics and other STEMM subjects at university.\nJudged by pupil self-assessment forms, our preliminary trial was a success, though caution is required due to the small sample size. We will continue developing peer-reviewed bioinformatics material, targeted at school pupils and\/or undergraduates, and applying it in practice. This will simultaneously lead to expansion of the 4273\u03c0 resource and the gathering of larger, more complex and conclusive educational data at a future date. 4273\u03c0 itself, and links to relevant social media groups, may be found at http:\/\/4273pi.org. \n\nMethods \nTwo activities were carried out, each using a voluntary group of seven pupils studying science from a single school in Scotland. One group was from Kilgraston, an independent girls\u2019 school, and the other was from Forfar Academy, a comprehensive school. In the case of Kilgraston, five pupils were at S5 and two were at S6 level, and instruction and assistance were provided by D.B., M.M.C., G.T.P.M. and H.P. In the case of Forfar Academy, all pupils were at S5 level, of whom two where girls and five were boys, and instruction and assistance were provided by R.G.A., D.B., L.D., J.L.M. and S.D.S. Generally, university staff or PhD students provided detailed instruction on the bioinformatics activity, and school staff highlighted links to material already taught and the curriculum. With a combination of university staff or PhD students and school staff, students were guided through the practical material of two components (modules) of 4273\u03c0 Bioinformatics for Biologists. At Kilgraston, the event was held at the school, occupying an entire day on which no other classes were scheduled. With Forfar Academy, the event was held at the University of St Andrews, where students participated in an afternoon and evening session, primarily held in the same room used, at other times, by undergraduates on the BL4273 module. Refreshment breaks were included, using the school\u2019s usual facilities (Kilgraston) or the Bell Pettigrew Museum (St Andrews). In total, the teaching time was approximately 4 h. \nRaspberry Pi Model B hardware was used, one per student (plus one connected to a projector for demonstration). Prior to the first event, at Kilgraston, tasks were selected from existing material in discussion between D.B. and H.P. (familiar with 4273\u03c0) and M.M.C. (familiar with the school curriculum). For both groups of pupils, the first task corresponded to the \u201cINTRO\u201d component of 4273\u03c0 Bioinformatics for Biologists, providing an introduction to the Raspberry Pi computer hardware, the Linux command-line, BLAST sequence similarity search software and Programming in the Perl language. The second task corresponded to the \u201cDNA\u201d component, involving an introduction to the FlyBase database (Dos Santos et al.[22]) and genome annotation with BLAST (Altschul et al.[6]), GeneWise (Birney et al.[23]) and SNAP (Korf[24]). Hard-copy handouts were provided. The handouts for Kilgraston and Forfar Academy were identical in content apart from date, location of the event, staff details and location of files (~\/kilgraston or ~\/forfar_academy). For the record, the specific handouts used are available as Additional file 1 (Kilgraston) and Additional file 2 (Forfar Academy), but with names and contact details redacted. The latest, open access versions of these will be found in 4273\u03c0. \nHard-copy, paired, \u201cbefore\u201d (prior to the use of the computers) and \u201cafter\u201d questionnaires were used for pupils and school staff, involving questions on a 1\u20135 Likert scale for self-assessment of attitudes and free text (Table 1; Additional file 3). In preparing the questionnaires, for those questions on a Likert scale, the sequence of questions was randomised and the sense of each question (\u201c1\u201d corresponding to \u201cgood\u201d on our subjective scale, vs \u201c1\u201d corresponding to \u201cbad\u201d) was randomised. The same sequence and sense were used for each questionnaire handed out; within each group (pupils or staff), the sequence of paired questions was the same \u201cbefore\u201d and \u201cafter\u201d. Results of the questions on the Likert scale were summarised per question as a bar chart, and as a likelihood ratio sign test for evidence of systematic change over the course of the activity. We apply a likelihood approach to statistical inference (Birnbaum[25]; Edwards[26]; Royall[27]; Barker[28]). In common with other approaches to statistical inference, this provides no absolute threshold beyond which evidence is considered conclusive. By convention, we define \u201cstrong\u201d evidence as a log (ln) likelihood ratio, \u0394\u2113, of at least 2, or a likelihood ratio of at least 8 (Edwards, pp. 199\u2013202[26]; Royall[27]). Were \u0394\u2113 converted to a p value under the assumptions of a likelihood ratio test (Wilks[29]), then for one free parameter \u0394\u2113\u2009\u2265\u20092 corresponds to p\u2009\u2264\u20090.046, approximately the traditional threshold for statistical significance prior to any correction for multiple testing (i.e. p\u2009<\u20090.05). Calculations were performed in R (R Development Core Team[30]). \n\n\n\n\n\n\n\nTable 1. Analysis of before and after questions for pupils on a Likert scale (1 \u201cstrongly disagree\u201d to 5 \u201cstrongly agree\u201d). For each pair of responses for each question, for each pupil, the change from before to after was noted, if any, and was reduced to a binary variable, indicating increase or decrease on the Likert scale. The proportion of changes that were increases and the proportion that were decreases constitute our maximum likelihood (ML) estimates for the probability of increase and probability of decrease, conditional on a change occurring. Whether the majority of changes are in the direction of improvement is indicated, with the direction indicating improvement being a subjective judgement by the authors in the context of this study. Assuming a binomial distribution, for each question separately, the likelihood of the observed changes was calculated, firstly, assuming the ML estimates for probability of increase and decrease obtained from the data; and secondly, assuming an extrinsic hypothesis that the probability of increase and probability of decrease are equal (0.5). From these likelihoods, the likelihood ratio and its natural logarithm (\u0394\u2113) were calculated. Rows in italics show strong evidence of overall change in a specific direction over the course of the activity (\u0394\u2113\u2009\u2265\u20092). N\u2009=\u200912 pupils submitted both \u201cbefore\u201d and \u201cafter\u201d questionnaires\n\n\n Question\n\n Increases\n\n Decreases\n\n Increases proportion\n\n Decreases proportion\n\n Most changes were improvements?\n\n Likelihood ratio\n\n \u0394\u2113\n\n\n1. I will end up working in science.\n\n1\n\n1\n\n0.5\n\n0.5\n\nNo\n\n1\n\n0\n\n\n2. I think computers are useful within biology.\n\n5\n\n0\n\n1\n\n0\n\nYes\n\n32\n\n3.47\n\n\n3. I have heard of Linux.\n\n10\n\n0\n\n1\n\n0\n\nYes\n\n1024\n\n6.93\n\n\n4. I do not expect to enjoy [after: did not enjoy] the activity today\n\n4\n\n3\n\n0.57\n\n0.43\n\nNo\n\n1.07\n\n0.07\n\n\n5. I know more about computers than most adults do.\n\n4\n\n2\n\n0.67\n\n0.33\n\nYes\n\n1.40\n\n0.34\n\n\n6. I know more about computers than my teachers do.\n\n2\n\n0\n\n1\n\n0\n\nYes\n\n4\n\n1.39\n\n\n7. I am not intending to go to university.\n\n0\n\n0\n\nn\/a\n\nn\/a\n\nn\/a\n\n1\n\n0\n\n\n8. I have not heard of the Raspberry Pi.\n\n1\n\n10\n\n0.09\n\n0.91\n\nYes\n\n71.78\n\n4.27\n\n\n9. I think computers are useful within sciences other than biology.\n\n2\n\n0\n\n1\n\n0\n\nYes\n\n4\n\n1.39\n\n\n10. I would rather not use a computer, I would prefer to use a phone or a tablet.\n\n0\n\n3\n\n0\n\n1\n\nYes\n\n8\n\n2.08\n\n\n11. I cannot program a computer.\n\n1\n\n8\n\n0.11\n\n0.89\n\nYes\n\n22.17\n\n3.10\n\n\n12. I am not interested in biology.\n\n1\n\n0\n\n1\n\n0\n\nNo\n\n2\n\n0.69\n\n\n13. I do not enjoy using computers for fun.\n\n1\n\n2\n\n0.33\n\n0.67\n\nYes\n\n1.19\n\n0.17\n\n\n14. I am good at using a computer to analyse DNA sequences.\n\n8\n\n1\n\n0.89\n\n0.11\n\nYes\n\n22.17\n\n3.10\n\n\nn\/a not available\n\n\n\nResults \nQuestionnaire responses, with paired before and after questionnaire answers, are available as Additional file 3 and summarised in Table 1. The spread of answers to pupil questionnaires, on the Likert scale, is presented in Fig. 1. Because individual pupils may apply different criteria for each specific question (i.e. categories are simultaneously pupil-specific and question-specific), we regard the separation into five categories as unsuitable for either a continuous approximation or ranking. However, the direction of change on the Likert scale is comparable throughout the data, on the weak assumption that each pupil applies his or her own criteria for a given question consistently both before and after. Apart from changes over the course of the activity, whether the weight of answers is in the \u201cdisagree\u201d categories (1 and 2) vs the \u201cagree\u201d categories (4 and 5) may also have some general meaning. \n\n\n\n\n\n\n\n\n\n Figure 1. For the pupil questions on a Likert scale (Table 1), the summary of answers before the activities (white) and after (black). N\u2009=\u200914 (before), N\u2009=\u200912 (after). Dashed arrows indicate the direction of change considered to be \u201cimprovement\u201d. Solid arrows, below these, indicate the observed direction of most changes. Brief mnemonics (in italics) summarise question content. For the full text of questions, see Table 1.\n\n\n\nTo an extent, all our questions are expected, if anything, to improve as a result of the activities. For example, a significant part of the first component (\u201cINTRO\u201d) is devoted to programming. Hence, it would represent either disaster for the educational approach, or perhaps humour on the part of pupils, if pupils tended to agree more with \u201cI cannot program a computer\u201d after the event than before. (The question has some similarity to a \u201ccontrol\u201d in a laboratory experiment). Fortunately, in line with the educational approach being sound, evidence of disastrous effects was weak or absent.\nWith Question 1, \u201cI will end up working in science\u201d, most pupils agree and there is no systematic change during the activity. This is as expected, because pupils with no interest in science would be unlikely to volunteer to take part. Question 2, \u201cI think computers are useful within biology\u201d, is more specifically related to the activity itself. Again, most pupils agree both before and after, but here we see strong evidence of improvement (log likelihood ratio, \u0394\u2113\u2009=\u20093.47). The usefulness of computers in other sciences (Question 9, \u201cI think computers are useful within sciences other than biology\u201d) shows little change as a result of the activity. As part of the volunteers\u2019 positive mentality, we also see that most pupils expected to enjoy the activity and, in practice, did so (Question 4, \u201cI do not expect to enjoy [after: did not enjoy] the activity today\u201d). For this question, there is a fairly high level of change over the course of the activity, with four pupils reporting changes on the Likert scale in the direction opposite to \u201cimprovement\u201d, and three in the direction of \u201cimprovement\u201d (Table 1). All pupils have a strong desire to continue studies at university, which was entirely unchanged by participating in the activity (Question 7, \u201cI am not intending to go to university\u201d). Both before and after, they have a strong interest in biology (Question 12, \u201cI am not interested in biology\u201d). One pupil reports a change in the direction opposite to improvement, whose implications are difficult to assess without a larger sample. All appear interested in computers (Question 13, \u201cI do not enjoy using computers for fun\u201d). \nQuestions 3 (\u201cI have heard of Linux\u201d) and 8 (\u201cI have not heard of the Raspberry Pi\u201d) reveal that, before the activity, most pupils have not heard of the Linux operating system or the Raspberry Pi, though more have heard of the Raspberry Pi than Linux. Improvement over the course of the activity is almost guaranteed due to its content, and indeed there is strong evidence for this (\u0394\u2113\u2009=\u20096.93 for Linux, \u0394\u2113\u2009=\u20094.27 for the Raspberry Pi).\nMore complex and perhaps more educationally relevant (Question 11, \u201cI cannot program a computer\u201d), students show strong evidence of improvement from a position of not regarding themselves able to program a computer, to being positive or at least uncertain (\u0394\u2113\u2009=\u20093.10). From a position of considering themselves generally poor at using a computer to analyse DNA sequences (Question 14, \u201cI am good at using a computer to analyse DNA sequences\u201d), there is strong evidence of improvement towards a more positive assessment of their own abilities (\u0394\u2113\u2009=\u20093.10), although no pupil ended up strongly confident (Fig. 1). We speculate this would improve further with repeated activities.\nQuestions 5 (\u201cI know more about computers than most adults do\u201d) and 6 (\u201cI know more about computers than my teachers do\u201d) are logically correlated, and both show weak evidence of improvement as a result of the activities. We prefer to interpret this as evidence of increased computational confidence among the pupils; however, heightened perception of adult incompetence cannot be ruled out.\nQuestion 10 (\u201cI would rather not use a computer, I would rather use a phone or a tablet\u201d) indicates a base-line level of openness to the educational method used. Our activity, in contrast to much computational activity by young people, is computer-based, using the Raspberry Pi. Pupils\u2019 views are generally moderate both before and after, but with borderline strong evidence of a strengthening tendency to prefer a computer as a result of the activity (\u0394\u2113\u2009=\u20092.08).\nPupil free-text answers to Q15 (\u201cWhat was the best part of the activity today, and why\u201d) included \u201cgetting the hands-on experience with computers, as it is not something we ever get to do at school\u201d; \u201cI liked the programming\u2014making files and commands\u2014because it was really interesting to see how computers actually work\u201d; \u201cThe simpler programming stages as they were at an accomplishable level & well explained\u201d; \u201cLearning how computers are used in biology and getting to try it ourselves because it was interesting and a new experience.\u201d; \u201cThe best part was going on flybase because I found it really interesting\u201d. Free-text answers to Q16 (\u201cWhat was the worst part of the activity today, and why?\u201d) included \u201cTyping everything during programme because it took ages and a tiny mistake or omission could cause problems.\u201d; \u201cThe worst part was flybase as it was difficult to understand.\u201d; \u201cI enjoyed the entire day and there wasn\u2019t one part I didn\u2019t enjoy. It was really interesting.\u201d; \u201ctyping in all of the commands to the LXterminal as it took a long time and it was easy to make mistakes.\u201d; \u201cI didn\u2019t enjoy typing stuff into LxTerminal because I was awful at it\u201d; \u201cThe DNA sequencing on the internet was confusing and poorly explained in the handbook\u201d; and \u201cI don\u2019t think the instructions were very clear\u201d (Additional file 3).\nIt seems the activities stretched pupils intellectually and practically, which was regarded both positively and negatively among the pupils. This is illustrated, for example, by opposite views on the Flybase database, and may also be reflected in changing views on the enjoyability of the activities (Question 4). Typing programs and using the command-line within the terminal was unpopular. However, this is a crucial part of computational science, which cannot be bypassed, and which remains difficult even for professional bioinformaticians. The practical handouts clearly did not suit all pupils. These attempt a practical balance between brevity and total coverage. In future, these could be improved by inclusion of diagrams and screenshots.\nSchool staff free-text answers to the question \u201cWhat was the best part of the activity today, and why?\u201d included \u201cThe pupils being given the opportunity to contextualise their theory based work in class into a practical application using the software. They gained a great deal of practice in making the connection between proteins\u2009+\u2009amino acids being the result of DNA sequences.\u201d And: \u201cWatching pupils who had never written code program Pis. Seeing their excitement when the code worked.\u201d Also: \u201cThe pupils were all fully engaged & gained a great deal from the seminar. I have also gained an insight into bioinformatics which will be vital for teaching the next part of the new higher on genomics\u201d (Additional file 3).\n\nDiscussion \nOur results were for a small sample and cannot represent the full range of voluntary groups one could draw from schools even in Scotland. But it is encouraging for the approach taken that almost all changes in pupil questionnaires, over the course of the activity, were in the direction that we regard as indicating improvement.\nFor all pupil questions showing strong evidence of change over the course of the activity (\u0394\u2113\u2009\u2265\u20092), the tendency was in the direction of improvement. Also, free-text responses from pupils were generally positive, though with a suggestion that the handouts could be improved (\u201cResults\u201d, above). On this basis, we judge the preliminary investigation a success. The evidence strongly suggests that, in their own assessment, the pupils benefited from the bioinformatics activities undertaken. This complements other research, involving successful bioinformatics activities developed specifically for school level (e.g. Marques et al.[8]). \nGallagher et al.[10]) also reported a successful experience bringing bioinformatics to high school level, but with three challenges. We did not experience any of these. Firstly, some of the students in their study questioned the relevance of computation to biology. Gallagher et al. propose that part of their difficulty was use of computer scientists to teach lessons. In our case, lessons were co-taught by university biologists or biologists and chemists, alongside school staff, lending support to this proposition. However, other factors may be relevant, for example the use of whole-class teaching by Gallagher et al., as opposed to our voluntary groups; and the passage of time since their study. Secondly, some of the students in the study by Gallagher et al. doubted the value of learning about algorithms. Since our material was practical, with little detail on algorithms, the objection would\u2014a priori\u2014be unlikely. Thirdly, Gallagher et al. found some students doubted the relevance of bioinformatics material if this was not covered in official tests. In our case, there is relevance to the syllabus (see \u201cRelevance to the new Higher curriculum for Biology and Human Biology\u201d, below), and this was highlighted during the activities by school staff. With a different audience but similar educational aim, Wood and Gebhardt[15] also emphasise the importance of relevance to the curriculum. They report successfully updating high school teachers on the topic of bioinformatics via hands-on experience, practical demonstrations and lectures. One of their considerations for those wishing to replicate their approach, \u201cWork within the practical limitations of the classroom\u201d, may be alleviated by hardware such as the Raspberry Pi (see \u201cThe Raspberry Pi\u2014a flexible, general-purpose computer\u201d, below). \nHow far can we generalise from our results? Further work is warranted, for example using a broader range of teaching material; involvement of a larger number of voluntary groups from a wider range of schools, for a more varied sample and greater statistical power; use of less enthusiastic staff; a lower staff\/student ratio; a whole-class activity, not only a voluntary class sample; validation of the assessment instrument; direct measurement of student knowledge acquisition, in addition to self-assessment; interviews with staff and students and field observations. Where there is a surfeit of material available among our university colleagues \u2014 for example, introductory teaching material on the widely used program BLAST (Altschul et al.[6]) \u2014 one can imagine educational experiments, allowing an evidence-based decision on which material is more successful, perhaps in different geographical regions, countries, educational systems or types of school or with pupils of different age, academic achievement levels, social background or gender. These are important topics for the future.\nBioinformatics has always been relatively \u201copen\u201d as a subject, with public deposition of DNA sequence data associated with research publications and a wealth of free software. Although the algorithms do not change rapidly, the precise details of the software do. For these two reasons\u2014the tradition of openness within the field and the rapid obsolescence of specific details\u2014we consider the subject particularly amenable to online-only, open access teaching material. This costs nothing to obtain; for those writing the material, it can be relatively easily updated in light of developments in software and data. The material we used was from one open access bioinformatics educational effort, 4273\u03c0 (Barker et al.[7]). We hope to expand this resource to cover a wider range of bioinformatics topics, whilst continuing to ensure that all material is tested in practice and peer-reviewed. \n\nRelevance to the new higher curriculum for biology and human biology \nThe new Scottish Qualifications Authority Curriculum for Excellence Higher Syllabus for both Higher Biology and Higher Human Biology includes DNA technologies. Part of the content focuses on the use of computer software in both Biology (Scottish Qualifications Authority, pp. 10, 16[31]) and Human Biology (Scottish Qualifications Authority, pp. 14, 15, 56[32]). The suggested learning activity \u201cUse genome data to identify stop and start codons and known protein coding sequences\u201d (Scottish Qualifications Authority[32]) is directly addressed by the \u201cDNA\u201d component of 4273\u03c0, as used in the current study. Other components are also relevant, for example to phylogeny and comparative genomics in Higher Biology (Scottish Qualifications Authority, p. 16[31]). In addition, the new Curriculum for Excellence Advanced Higher Biology has content related to protein structure (covered, to an extent, by the \u201cENZYME\u201d component of 4273\u03c0) and scientific investigative approaches (Scottish Qualifications Authority[33]).\nBased on our experiences as outlined in this paper, and in agreement with Gallagher et al.[10], workshops taught alongside university staff seem an efficient means of professional development for teachers, who may be presented with curriculum content which is new to them and outwith their experience. From the pupils\u2019 point of view, having access to workshops and seminars run by university staff also gives some experience of the university experience, particularly when\u2014as in the current case\u2014the teaching material is precisely at the undergraduate level. \n\nThe Raspberry Pi \u2014 a flexible, general-purpose computer \nThe use of Raspberry Pi computers allows for a range of new software and experiences for the pupils, without any major impact on school\u2019s ICT staff and systems. By bringing in complete software plus hardware systems, there is no need for specific software to be installed on school computers (some of which may require existing security restrictions to be relaxed). As the first part of the activity was to assemble the Pi, for the activity held at Kilgraston, the only prerequisite for the school was to check the school\u2019s VGA monitors would work with the Pi and check there were enough power sockets.\nPupils may be more confident using the Raspberry Pi, with the knowledge that if they \u201cbroke\u201d the software set-up, instructors could just swap its SD card. This allows a risk-taking approach to learning, and the possibility to learn by ones mistakes in a way that is prevented in managed computer classrooms. (Despite this, in the two school activities in this study, and in 3 years of teaching BL4273 Bioinformatics for Biologists at the University of St Andrews, we have never had to reinitialise an SD card due to such an accident. We plan to incorporate greater software risks in future material). It also gives pupils experience of Linux, the operating system they may well use should they choose to further their career in computational science. \n\nConclusion \nIn our current study, we have demonstrated that practical bioinformatics material initially developed for an optional, final-year undergraduate module can be used successfully with school pupils at sixth form level (S5 and S6 in Scotland). Covering practical bioinformatics at this level in all schools may be an attainable goal, with subsequent benefits to higher education and research, in both biology and other sciences. \n\nAbbreviations \nBLAST: Basic Local Alignment Search Tool\nGOBLET: Global Organisation for Bioinformatics Learning, Education and Training\nICT: information and communications technology\nML: maximum likelihood\nSD: secure digital\nSTEMM: Science, Technology, Engineering, Mathematics and Medicine\nUK: United Kingdom\nUSA: United States of America\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors' contributions \nDB and HP conceived of the school activities. DB, RGA, JLM, HP, MMC, LD, GTPM and SDS carried out the school activities. DB analysed results with assistance from RGA and JLM. DB wrote the manuscript with input from RGA, JLM, HP and other authors. All authors read and approved the final manuscript.\n\nAdditional files \n\n\n\n\n\n\n Additional file 1. Teaching material and blank questionnaires used for the event with Kilgraston School\n\n\nNames and contact details have been redacted.\nFormat: ZIP; Size: 431KB Download file\n\n\n\n\n\n\n Additional file 2. Teaching material and blank questionnaires used for the event with Forfar Academy\n\n\nNames and contact details have been redacted.\nFormat: ZIP; Size: 478KB Download file\n\n\n\n\n\n\n Additional file 3. Data from questionnaires (LibreOffice Calc spreadsheet)\n\n\nFor details of contents, see the \u201cNotes\u201d worksheet.\nFormat: ODS; Size: 81KB Download file\n\n\n\n\nAcknowledgements \nThis work was supported in part by the Science and Technology Facilities Council under grant ST\/M000435\/1 to DB. We thank all participants in our activities for their help, and thank Ian McDonald for technical assistance.\n\nReferences \n\n\n\u2191 Hogeweg, P. (2011). \"The roots of bioinformatics in theoretical biology\". PLOS Computational Biology 7 (3): e1002021. doi:10.1371\/journal.pcbi.1002021. PMC PMC3068925. PMID 21483479. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3068925 .   \n\n\u2191 Magana, A.J.; Taleyarkhan, M.; Alvarado, D.R.; Kane, M.; Springer, J.; Clase, K. (2014). \"A survey of scholarly literature describing the field of bioinformatics education and bioinformatics educational research\". CBE \u2013 Life Sciences Education 13 (4): 607\u201323. doi:10.1187\/cbe.13-10-0193. PMC PMC4255348. PMID 25452484. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4255348 .   \n\n\u2191 Benson, D.A.; Clark, K.; Karsch-Mizrachi, I.; Lipman, D.J.; Ostell, J.; Sayers, E.W. (2015). \"GenBank\". Nucleic Acids Research 43 (Database issue): D30\u2013D35. doi:10.1093\/nar\/gku1216. PMC PMC4383990. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383990 .   \n\n\u2191 Cunningham, F.; Amode, M.R.; Barrell, D., et al. (2015). \"Ensembl 2015\". Nucleic Acids Research 43 (Database issue): D662-D669. doi:10.1093\/nar\/gku1010.   \n\n\u2191 Galperin, M.Y.; Rigden, D.J.; Fern\u00e1ndez-Su\u00e1rez, X.M. (2015). \"The 2015 Nucleic Acids Research Database Issue and Molecular Biology Database Collection\". Nucleic Acids Research 43 (Database issue): D1-D5. doi:10.1093\/nar\/gku1241. PMC PMC4383995. PMID 25593347. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383995 .   \n\n\u2191 6.0 6.1 6.2 Altschul, S.F.; Madden, T.L.; Sch\u00e4ffer, A.A.; Zhang, J.; Zhang, Z.; Miller, W.; Lipman, D.J. (1997). \"Gapped BLAST and PSI-BLAST: a new generation of protein database search programs\". Nucleic Acids Research 25 (17): 3389-3402. doi:10.1093\/nar\/25.17.3389. PMC PMC146917. PMID 9254694. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917 .   \n\n\u2191 7.0 7.1 7.2 Barker, D.; Ferrier, D.E.K.; Holland, P.W.H.; Mitchell, J.B.O.; Plaisier, H.; Ritchie, M.G.; Smart, S.D. (2013). \"4273\u03c0: Bioinformatics education on low cost ARM hardware\". BMC Bioinformatics 14: 243. doi:10.1186\/1471-2105-14-243. PMC PMC3751261. PMID 23937194. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3751261 .   \n\n\u2191 8.0 8.1 8.2 Marques, I.; Almeida, P.; Alves, R.; Jo\u00e3o Dias, M.; Godinho, A.; Pereira-Leal, J.B. (2014). \"Bioinformatics Projects Supporting Life-Sciences Learning in High Schools\". PLOS Computational Biology 10 (1): e1003404. doi:10.1371\/journal.pcbi.1003404. PMC PMC3900377. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3900377 .   \n\n\u2191 Corpas, M.; Jimenez, R.C.; Bongcam-Rudloff, E.; et al. (2015). \"The GOBLET training portal: a global repository of bioinformatics training materials, courses and trainers\". Bioinformatics 31 (1): 140\u2013142. doi:10.1093\/bioinformatics\/btu601. PMC PMC4271145. PMID 25189782. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4271145 .   \n\n\u2191 10.0 10.1 10.2 Gallagher, S.R.; Coon, W.; Donley, K.; Scott, A.; Goldberg, D.S. (2011). \"A first attempt to bring computational biology into advanced high school biology classrooms\". PLOS Computational Biology 7 (10): e1002244. doi:10.1371\/journal.pcbi.1002244. PMC PMC3203055. PMID 22046118. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203055 .   \n\n\u2191 Lewitter, F.; Bourne, P.E. (2011). \"Teaching bioinformatics at the secondary school level\". PLOS Computational Biology 7 (10): e1002242. doi:10.1371\/journal.pcbi.1002242. PMC PMC3203059. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203059 .   \n\n\u2191 McQueen, J.; Wright, J.J.; Fox, J.A. (2012). \"Design and implementation of a genomics field trip program aimed at secondary school students\". PLOS Computational Biology 8 (8): e1002636. doi:10.1371\/journal.pcbi.1002636. PMC PMC3431290. PMID 22956895. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3431290 .   \n\n\u2191 Kovarik, D.N.; Patterson, D.G.; Cohen, C.; Sanders, E.A.; Peterson, K.A.; Porter, S.G.; Chowning, J.T. (2013). \"Bioinformatics education in high school: implications for promoting science, technology, engineering, and mathematics careers\". CBE \u2013 Life Sciences Education 12 (3): 441\u201359. doi:10.1187\/cbe.12-11-0193. PMC PMC3763012. PMID 24006393. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3763012 .   \n\n\u2191 Machluf, Y.; Yarden, A. (2013). \"Integrating bioinformatics into senior high school: design principles and implications\". Briefings in Bioinformatics 14 (5): 648-60. doi:10.1093\/bib\/bbt030. PMID 23665511.   \n\n\u2191 15.0 15.1 Wood, L.; Gebhardt, P. (2013). \"Bioinformatics goes to school \u2013 new avenues for teaching contemporary biology\". PLOS Computational Biology 9 (6): e1003089. doi:10.1371\/journal.pcbi.1003089. PMC PMC3681668. PMID 23785266. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3681668 .   \n\n\u2191 Toby, I.; Pope, A. (2014). \"Bioinformatics tools available for K-12 students to engage in research\". Aviation, Space, and Environmental Medicine 85 (4): 484-485. doi:10.3357\/ASEM.3953.2014. PMID 24754215.   \n\n\u2191 College voor Examens (April 2014). \"Biologie VWO. Syllabus Centraal Examen 2016\" (PDF). pp. 59. http:\/\/www.examenblad.nl\/examenstof\/syllabus-2016-biologie-vwo-nader\/2016\/vwo\/f=\/biologie_vwo_2016_def_voor_hervaststelling.pdf . Retrieved 21 October 2015 .   \n\n\u2191 Wefer, S.H.; Sheppard, K. (2008). \"Bioinformatics in high school biology curricula: a study of state science standards\". CBE \u2013 Life Sciences Education 7 (1): 155\u2013162. doi:10.1187\/cbe.07-05-0026. PMC PMC2262119. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2262119 .   \n\n\u2191 Stajich, J.E.; Block, D.; Boulez, K.; et al. (2002). \"The BioPerl toolkit: Perl modules for the life sciences\". Genome Research 12 (10): 1611\u20131618. doi:10.1101\/gr.361602. PMC PMC187536. PMID 12368254. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187536 .   \n\n\u2191 Stabenau, A.; McVicker, G.; Melsopp, C.; Proctor, G.; Clamp, M.; Birney, E. (2004). \"The Ensembl core software libraries\". Genome Research 14 (5): 929\u2013933. doi:10.1101\/gr.1857204. PMC PMC479122. PMID 15123588. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479122 .   \n\n\u2191 Scottish Qualifications Authority (2015). \"Higher Chemistry Course Support Notes\" (PDF). pp. 98. http:\/\/www.sqa.org.uk\/files_ccc\/CfE_CourseUnitSupportNotes_Higher_Sciences_Chemistry.pdf . Retrieved 20 October 2015 .   \n\n\u2191 Dos Santos, G.; Schroeder, A.J.; Goodman, J.L.; Strelets, V.B.; Crosby, M.A.; Thurmond, J.; Emmert, D.B.; Gelbart, W.M.; The FlyBase Consortium (2015). \"FlyBase: introduction of the Drosophila melanogaster Release 6 reference genome assembly and large-scale migration of genome annotations\". Nucleic Acids Research 43 (Database issue): D690-7. doi:10.1093\/nar\/gku1099. PMC PMC4383921. PMID 25398896. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383921 .   \n\n\u2191 Birney, E.; Clamp, M.; Durbin, R. (2004). \"GeneWise and Genomewise\". Genome Research 14 (5): 988\u201395. doi:10.1101\/gr.1865504. PMC PMC479130. PMID 15123596. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130 .   \n\n\u2191 Korf, I. (2004). \"Gene finding in novel genomes\". BMC Bioinformatics 14 (5): 59. doi:10.1186\/1471-2105-5-59. PMC PMC421630. PMID 15144565. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630 .   \n\n\u2191 Birnbaum, A. (1962). \"On the Foundations of Statistical Inference\". Journal of the American Statistical Association 57 (298): 269-306. doi:10.1080\/01621459.1962.10480660.   \n\n\u2191 26.0 26.1 Edwards, A.W.F. (1992). Likelihood. Baltimore, MD: Johns Hopkins University Press. pp. 296. ISBN 9780801844430.   \n\n\u2191 27.0 27.1 Royall, R. (1997). Statistical evidence: a likelihood paradigm. Boca Raton, FL: Chapman and Hall\/CRC. pp. 191. ISBN 9780412044113.   \n\n\u2191 Barker, D. (2015). \"Seeing the wood for the trees: philosophical aspects of classical, Bayesian and likelihood approaches in statistical inference and some implications for phylogenetic analysis\". Biology & Philosophy 30 (4): 505\u2013525. doi:10.1007\/s10539-014-9455-x.   \n\n\u2191 Wilks, S.S. (1938). \"The large-sample distribution of the likelihood ratio for testing composite hypotheses\". Annals of Mathematical Statistics 9 (1): 60-62. doi:10.1214\/aoms\/1177732360.   \n\n\u2191 R Development Core Team (2015). \"R: A language and environment for statistical computing\". The R Foundation for Statistical Computing. https:\/\/www.r-project.org\/ . Retrieved 20 July 2015 .   \n\n\u2191 31.0 31.1 Scottish Qualifications Authority (2014). \"Higher Biology Course Support Notes\". http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology . Retrieved 20 October 2015 .   \n\n\u2191 32.0 32.1 Scottish Qualifications Authority (2014). \"Higher Human Biology Course Support Notes\". http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Human%20Biology . Retrieved 20 October 2015 .   \n\n\u2191 Scottish Qualifications Authority (2015). \"Advanced Higher Biology Course\/Unit Support Notes\". http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology . Retrieved 20 October 2015 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In several cases citation information was missing and was added to make the reference more useful.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\">https:\/\/www.limswiki.org\/index.php\/Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on education\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 22:34.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,734 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","e85e2a721257f9447a4d6f2238effb07_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:University-level practical activities in bioinformatics benefit voluntary groups of pupils in the last 2 years of school<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">Bioinformatics<\/a> \u2014 the use of computers in biology \u2014 is of major and increasing importance to biological sciences and medicine. We conducted a preliminary investigation of the value of bringing practical, university-level bioinformatics education to the school level. We conducted voluntary activities for pupils at two schools in Scotland (years S5 and S6; pupils aged 15\u201317). We used material originally developed for an optional final-year undergraduate module and now incorporated into 4273<i>\u03c0<\/i>, a resource for teaching and learning bioinformatics on the low-cost Raspberry Pi computer.\n<\/p><p><b>Results:<\/b> Pupils\u2019 feedback forms suggested our activities were beneficial. During the course of the activity, they provide strong evidence of increase in the following: pupils\u2019 perception of the value of computers within biology; their knowledge of the Linux operating system and the Raspberry Pi; their willingness to use computers rather than phones or tablets; their ability to program a computer and their ability to analyse DNA sequences with a computer. We found no strong evidence of negative effects.\n<\/p><p><b>Conclusions:<\/b> Our preliminary study supports the feasibility of bringing university-level, practical bioinformatics activities to school pupils.\n<\/p><p><b>Keywords:<\/b> Bioinformatics; Computational biology; Secondary school; Raspberry Pi; Open access teaching material; Case study\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Findings\">Findings<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h3>\n<p>Progress in Science, Technology, Engineering, Mathematics and Medicine (STEMM) subjects is increasingly dominated by computational analyses. In biological sciences, for example, the exceptional pace of recent advances in technology for DNA and genome sequencing has created a demand for computationally able researchers, to analyse the large amounts of data produced. A field specialising in application of computation to biological problems has emerged, known as bioinformatics. The development of bioinformatics is discussed by Hogeweg<sup id=\"rdp-ebb-cite_ref-HogwegTheRoo_1-0\" class=\"reference\"><a href=\"#cite_note-HogwegTheRoo-1\" rel=\"external_link\">[1]<\/a><\/sup>, and university-level bioinformatics education has been reviewed by Magana et al.<sup id=\"rdp-ebb-cite_ref-MaganaASur_2-0\" class=\"reference\"><a href=\"#cite_note-MaganaASur-2\" rel=\"external_link\">[2]<\/a><\/sup>\n<\/p><p>DNA sequences and related data are available at low cost (for new sequencing work) or free in online databases such as GenBank (Benson et al.<sup id=\"rdp-ebb-cite_ref-BensonGen_3-0\" class=\"reference\"><a href=\"#cite_note-BensonGen-3\" rel=\"external_link\">[3]<\/a><\/sup>), Ensembl (Cunningham et al.<sup id=\"rdp-ebb-cite_ref-CunninghamEns_4-0\" class=\"reference\"><a href=\"#cite_note-CunninghamEns-4\" rel=\"external_link\">[4]<\/a><\/sup>) and hundreds of others (Galperin et al.<sup id=\"rdp-ebb-cite_ref-GalperinThe2015_5-0\" class=\"reference\"><a href=\"#cite_note-GalperinThe2015-5\" rel=\"external_link\">[5]<\/a><\/sup>). Software for bioinformatics research is usually free, for example the very widely used sequence database search software, BLAST (Altschul et al.<sup id=\"rdp-ebb-cite_ref-AltschulGap_6-0\" class=\"reference\"><a href=\"#cite_note-AltschulGap-6\" rel=\"external_link\">[6]<\/a><\/sup>). Free resources are also available for bioinformatics teaching and learning, for example 4273<i>\u03c0<\/i> (Barker et al.<sup id=\"rdp-ebb-cite_ref-Barker4273n_7-0\" class=\"reference\"><a href=\"#cite_note-Barker4273n-7\" rel=\"external_link\">[7]<\/a><\/sup>), Bioinform\u00e1tica na escola (Marques et al.<sup id=\"rdp-ebb-cite_ref-MarquesBio_8-0\" class=\"reference\"><a href=\"#cite_note-MarquesBio-8\" rel=\"external_link\">[8]<\/a><\/sup>), GOBLET (Corpas et al.<sup id=\"rdp-ebb-cite_ref-CorpasTheGOB_9-0\" class=\"reference\"><a href=\"#cite_note-CorpasTheGOB-9\" rel=\"external_link\">[9]<\/a><\/sup>), <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.nbic.nl\/nl\/education\/high-school-programmes\/bioinformaticsschool\" target=\"_blank\">Bioinformatics@school<\/a> and the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/evoed.evolutionsociety.org\" target=\"_blank\">EvoEd Digital Library<\/a>. These publicly available data, software and materials present excellent opportunities for relatively low-cost teaching. \n<\/p><p>There has been a recent, encouraging increase in exposure of school pupils to bioinformatics (e.g. Gallagher et al.<sup id=\"rdp-ebb-cite_ref-GallagherAFirst_10-0\" class=\"reference\"><a href=\"#cite_note-GallagherAFirst-10\" rel=\"external_link\">[10]<\/a><\/sup>; Lewitter and Bourne<sup id=\"rdp-ebb-cite_ref-LewitterTeach_11-0\" class=\"reference\"><a href=\"#cite_note-LewitterTeach-11\" rel=\"external_link\">[11]<\/a><\/sup>; McQueen et al.<sup id=\"rdp-ebb-cite_ref-McQueenDesign_12-0\" class=\"reference\"><a href=\"#cite_note-McQueenDesign-12\" rel=\"external_link\">[12]<\/a><\/sup>; Kovarik et al.<sup id=\"rdp-ebb-cite_ref-KovarikBio_13-0\" class=\"reference\"><a href=\"#cite_note-KovarikBio-13\" rel=\"external_link\">[13]<\/a><\/sup>; Machluf and Yarden<sup id=\"rdp-ebb-cite_ref-MachlufInt_14-0\" class=\"reference\"><a href=\"#cite_note-MachlufInt-14\" rel=\"external_link\">[14]<\/a><\/sup>; Wood and Gebhardt<sup id=\"rdp-ebb-cite_ref-WoodBio_15-0\" class=\"reference\"><a href=\"#cite_note-WoodBio-15\" rel=\"external_link\">[15]<\/a><\/sup>; Marques et al.<sup id=\"rdp-ebb-cite_ref-MarquesBio_8-1\" class=\"reference\"><a href=\"#cite_note-MarquesBio-8\" rel=\"external_link\">[8]<\/a><\/sup>; Toby and Pope<sup id=\"rdp-ebb-cite_ref-TobyBio_16-0\" class=\"reference\"><a href=\"#cite_note-TobyBio-16\" rel=\"external_link\">[16]<\/a><\/sup>). Genomics and associated topics have started to appear in many official school curricula, for example in Scotland (see \u201cDiscussion\u201d, below), the Netherlands (College voor Examens, p. 17<sup id=\"rdp-ebb-cite_ref-CollegeBio_17-0\" class=\"reference\"><a href=\"#cite_note-CollegeBio-17\" rel=\"external_link\">[17]<\/a><\/sup>) and the USA (Wefer and Sheppard<sup id=\"rdp-ebb-cite_ref-WeferBio_18-0\" class=\"reference\"><a href=\"#cite_note-WeferBio-18\" rel=\"external_link\">[18]<\/a><\/sup>). From a different angle, computer science is now a major part of the <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.gov.uk\/government\/publications\/national-curriculum-in-england-computing-programmes-of-study\" target=\"_blank\">primary school curriculum for England<\/a>. This is in line with a \u201cback to basics\u201d approach to computing currently emerging, as opposed to more traditional <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> and communications technology (ICT). In the UK, this change has been particularly associated with the low-cost <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.raspberrypi.org\" target=\"_blank\">Raspberry Pi computer<\/a>, which is suitable for educational projects in electronics and engineering as well as general use and has <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wired.co.uk\/news\/archive\/2015-02\/18\/raspberry-pi-5-million\" target=\"_blank\">sold over 5 million units<\/a>. However, a practical link between computers and STEMM \u2014 which we will refer to as computational science, as opposed to computer science \u2014 still does not feature strongly on the UK school curriculum. DNA sequencing has a pervasive and increasing influence across traditionally disparate subject areas, including biochemistry, biomedical research, clinical medicine, evolutionary biology, ecology, neuroscience and anthropology. DNA sequencing is used to diagnose genetic and infectious diseases, discover drugs, characterise environments, monitor the progress of cancers, identify species and reveal evolutionary patterns. We consider increased amounts of practical bioinformatics at school to be a priority. \n<\/p><p>Motivated by the increasing importance of bioinformatics to the life sciences and its appearance on school curricula, we conducted a preliminary investigation of the benefits of bringing university-level bioinformatics teaching material to voluntary groups of children in the last 2 years of school in Scotland (S5 and S6; pupils aged 15\u201317). The material was originally developed for an optional, final-year undergraduate module at the University of St Andrews, <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.st-andrews.ac.uk\/coursecatalogue\/ug\/2015-2016\" target=\"_blank\">BL4273 Bioinformatics for Biologists<\/a>. To better match bioinformatics as it is actually used in research at universities, institutes and industry, the material uses the Linux operating system, in this case a variant of Rasbpian Linux running on low-cost Raspberry Pi hardware. This material has been released under an open access licence, as part of <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/4273pi.org\" target=\"_blank\">4273<i>\u03c0<\/i><\/a> (Barker et al.<sup id=\"rdp-ebb-cite_ref-Barker4273n_7-1\" class=\"reference\"><a href=\"#cite_note-Barker4273n-7\" rel=\"external_link\">[7]<\/a><\/sup>). Our proposition was that school pupils can benefit from practical, undergraduate-level bioinformatics teaching material. Compared to the undergraduates for whom this material was originally developed, school pupils are less experienced and knowledgeable about biology in general. However, their levels of practical bioinformatics experience are broadly similar: zero in the case of the school pupils, and approximately ten actual contact hours among undergraduates at the time of starting the module. \n<\/p><p>Many of the skills developed in our activities, and 4273<i>\u03c0<\/i> or bioinformatics in general, are generic skills in computational science. For example, although the programming language taught in the \u201cINTRO\u201d component \u2014 Perl \u2014 is particularly widely used in bioinformatics (e.g. Stajich et al.<sup id=\"rdp-ebb-cite_ref-StajichTheBio_19-0\" class=\"reference\"><a href=\"#cite_note-StajichTheBio-19\" rel=\"external_link\">[19]<\/a><\/sup>; Stabenau et al.<sup id=\"rdp-ebb-cite_ref-StabenauTheEns_20-0\" class=\"reference\"><a href=\"#cite_note-StabenauTheEns-20\" rel=\"external_link\">[20]<\/a><\/sup>), it is structurally similar to other programming languages widely used in science, including C, Fortran, Java, Python and R. Use of the command-line, emphasised in 4273<i>\u03c0<\/i>, is also essential in computational physics, computational chemistry and, indeed, computer science. Although computational chemistry is not yet part of the Higher qualification in Chemistry, several simulations are suggested by the Scottish Qualifications Authority.<sup id=\"rdp-ebb-cite_ref-ScottHigher_21-0\" class=\"reference\"><a href=\"#cite_note-ScottHigher-21\" rel=\"external_link\">[21]<\/a><\/sup> Computational skills, as taught in 4273<i>\u03c0<\/i>, will be valuable to students taking chemistry, physics and other STEMM subjects at university.\n<\/p><p>Judged by pupil self-assessment forms, our preliminary trial was a success, though caution is required due to the small sample size. We will continue developing peer-reviewed bioinformatics material, targeted at school pupils and\/or undergraduates, and applying it in practice. This will simultaneously lead to expansion of the 4273<i>\u03c0<\/i> resource and the gathering of larger, more complex and conclusive educational data at a future date. 4273<i>\u03c0<\/i> itself, and links to relevant social media groups, may be found at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/4273pi.org\" target=\"_blank\">http:\/\/4273pi.org<\/a>. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h3>\n<p>Two activities were carried out, each using a voluntary group of seven pupils studying science from a single school in Scotland. One group was from Kilgraston, an independent girls\u2019 school, and the other was from Forfar Academy, a comprehensive school. In the case of Kilgraston, five pupils were at S5 and two were at S6 level, and instruction and assistance were provided by D.B., M.M.C., G.T.P.M. and H.P. In the case of Forfar Academy, all pupils were at S5 level, of whom two where girls and five were boys, and instruction and assistance were provided by R.G.A., D.B., L.D., J.L.M. and S.D.S. Generally, university staff or PhD students provided detailed instruction on the bioinformatics activity, and school staff highlighted links to material already taught and the curriculum. With a combination of university staff or PhD students and school staff, students were guided through the practical material of two components (modules) of 4273<i>\u03c0<\/i> Bioinformatics for Biologists. At Kilgraston, the event was held at the school, occupying an entire day on which no other classes were scheduled. With Forfar Academy, the event was held at the University of St Andrews, where students participated in an afternoon and evening session, primarily held in the same room used, at other times, by undergraduates on the BL4273 module. Refreshment breaks were included, using the school\u2019s usual facilities (Kilgraston) or the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.st-andrews.ac.uk\/museum\/bellpettigrew\" target=\"_blank\">Bell Pettigrew Museum<\/a> (St Andrews). In total, the teaching time was approximately 4 h. \n<\/p><p>Raspberry Pi Model B hardware was used, one per student (plus one connected to a projector for demonstration). Prior to the first event, at Kilgraston, tasks were selected from existing material in discussion between D.B. and H.P. (familiar with 4273<i>\u03c0<\/i>) and M.M.C. (familiar with the school curriculum). For both groups of pupils, the first task corresponded to the \u201cINTRO\u201d component of 4273<i>\u03c0<\/i> Bioinformatics for Biologists, providing an introduction to the Raspberry Pi computer hardware, the Linux command-line, BLAST sequence similarity search software and Programming in the Perl language. The second task corresponded to the \u201cDNA\u201d component, involving an introduction to the FlyBase database (Dos Santos et al.<sup id=\"rdp-ebb-cite_ref-DosSantosIntro_22-0\" class=\"reference\"><a href=\"#cite_note-DosSantosIntro-22\" rel=\"external_link\">[22]<\/a><\/sup>) and genome annotation with BLAST (Altschul et al.<sup id=\"rdp-ebb-cite_ref-AltschulGap_6-1\" class=\"reference\"><a href=\"#cite_note-AltschulGap-6\" rel=\"external_link\">[6]<\/a><\/sup>), GeneWise (Birney et al.<sup id=\"rdp-ebb-cite_ref-BirneyGene_23-0\" class=\"reference\"><a href=\"#cite_note-BirneyGene-23\" rel=\"external_link\">[23]<\/a><\/sup>) and SNAP (Korf<sup id=\"rdp-ebb-cite_ref-KorfGene_24-0\" class=\"reference\"><a href=\"#cite_note-KorfGene-24\" rel=\"external_link\">[24]<\/a><\/sup>). Hard-copy handouts were provided. The handouts for Kilgraston and Forfar Academy were identical in content apart from date, location of the event, staff details and location of files (~\/kilgraston or ~\/forfar_academy). For the record, the specific handouts used are available as Additional file 1 (Kilgraston) and Additional file 2 (Forfar Academy), but with names and contact details redacted. The latest, open access versions of these will be found in <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/4273pi.org\" target=\"_blank\">4273<i>\u03c0<\/i><\/a>. \n<\/p><p>Hard-copy, paired, \u201cbefore\u201d (prior to the use of the computers) and \u201cafter\u201d questionnaires were used for pupils and school staff, involving questions on a 1\u20135 Likert scale for self-assessment of attitudes and free text (Table 1; Additional file 3). In preparing the questionnaires, for those questions on a Likert scale, the sequence of questions was randomised and the sense of each question (\u201c1\u201d corresponding to \u201cgood\u201d on our subjective scale, vs \u201c1\u201d corresponding to \u201cbad\u201d) was randomised. The same sequence and sense were used for each questionnaire handed out; within each group (pupils or staff), the sequence of paired questions was the same \u201cbefore\u201d and \u201cafter\u201d. Results of the questions on the Likert scale were summarised per question as a bar chart, and as a likelihood ratio sign test for evidence of systematic change over the course of the activity. We apply a likelihood approach to statistical inference (Birnbaum<sup id=\"rdp-ebb-cite_ref-BirnbaumOnThe_25-0\" class=\"reference\"><a href=\"#cite_note-BirnbaumOnThe-25\" rel=\"external_link\">[25]<\/a><\/sup>; Edwards<sup id=\"rdp-ebb-cite_ref-EdwardsLike_26-0\" class=\"reference\"><a href=\"#cite_note-EdwardsLike-26\" rel=\"external_link\">[26]<\/a><\/sup>; Royall<sup id=\"rdp-ebb-cite_ref-RoyallStat_27-0\" class=\"reference\"><a href=\"#cite_note-RoyallStat-27\" rel=\"external_link\">[27]<\/a><\/sup>; Barker<sup id=\"rdp-ebb-cite_ref-BarkerSee_28-0\" class=\"reference\"><a href=\"#cite_note-BarkerSee-28\" rel=\"external_link\">[28]<\/a><\/sup>). In common with other approaches to statistical inference, this provides no absolute threshold beyond which evidence is considered conclusive. By convention, we define \u201cstrong\u201d evidence as a log (ln) likelihood ratio, \u0394\u2113, of at least 2, or a likelihood ratio of at least 8 (Edwards, pp. 199\u2013202<sup id=\"rdp-ebb-cite_ref-EdwardsLike_26-1\" class=\"reference\"><a href=\"#cite_note-EdwardsLike-26\" rel=\"external_link\">[26]<\/a><\/sup>; Royall<sup id=\"rdp-ebb-cite_ref-RoyallStat_27-1\" class=\"reference\"><a href=\"#cite_note-RoyallStat-27\" rel=\"external_link\">[27]<\/a><\/sup>). Were \u0394\u2113 converted to a p value under the assumptions of a likelihood ratio test (Wilks<sup id=\"rdp-ebb-cite_ref-WilksTheLar_29-0\" class=\"reference\"><a href=\"#cite_note-WilksTheLar-29\" rel=\"external_link\">[29]<\/a><\/sup>), then for one free parameter \u0394\u2113\u2009\u2265\u20092 corresponds to p\u2009\u2264\u20090.046, approximately the traditional threshold for statistical significance prior to any correction for multiple testing (i.e. p\u2009<\u20090.05). Calculations were performed in R (R Development Core Team<sup id=\"rdp-ebb-cite_ref-RTheR_30-0\" class=\"reference\"><a href=\"#cite_note-RTheR-30\" rel=\"external_link\">[30]<\/a><\/sup>). \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 1.<\/b> Analysis of before and after questions for pupils on a Likert scale (1 \u201cstrongly disagree\u201d to 5 \u201cstrongly agree\u201d). For each pair of responses for each question, for each pupil, the change from before to after was noted, if any, and was reduced to a binary variable, indicating increase or decrease on the Likert scale. The proportion of changes that were increases and the proportion that were decreases constitute our maximum likelihood (ML) estimates for the probability of increase and probability of decrease, conditional on a change occurring. Whether the majority of changes are in the direction of improvement is indicated, with the direction indicating improvement being a subjective judgement by the authors in the context of this study. Assuming a binomial distribution, for each question separately, the likelihood of the observed changes was calculated, firstly, assuming the ML estimates for probability of increase and decrease obtained from the data; and secondly, assuming an extrinsic hypothesis that the probability of increase and probability of decrease are equal (0.5). From these likelihoods, the likelihood ratio and its natural logarithm (\u0394\u2113) were calculated. Rows in italics show strong evidence of overall change in a specific direction over the course of the activity (\u0394\u2113\u2009\u2265\u20092). N\u2009=\u200912 pupils submitted both \u201cbefore\u201d and \u201cafter\u201d questionnaires\n<\/td><\/tr>\n<tr>\n<th> Question\n<\/th>\n<th> Increases\n<\/th>\n<th> Decreases\n<\/th>\n<th> Increases proportion\n<\/th>\n<th> Decreases proportion\n<\/th>\n<th> Most changes were improvements?\n<\/th>\n<th> Likelihood ratio\n<\/th>\n<th> \u0394\u2113\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1. I will end up working in science.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2. I think computers are useful within biology.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.47\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3. I have heard of Linux.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1024\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6.93\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4. I do not expect to enjoy [after: did not enjoy] the activity today\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.57\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.43\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.07\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.07\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5. I know more about computers than most adults do.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.67\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.40\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.34\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6. I know more about computers than my teachers do.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.39\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7. I am not intending to go to university.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n\/a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n\/a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">n\/a\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8. I have not heard of the Raspberry Pi.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.91\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">71.78\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4.27\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9. I think computers are useful within sciences other than biology.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.39\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">10. I would rather not use a computer, I would prefer to use a phone or a tablet.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.08\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">11. I cannot program a computer.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.89\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">12. I am not interested in biology.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.69\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">13. I do not enjoy using computers for fun.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.33\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.67\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.19\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.17\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14. I am good at using a computer to analyse DNA sequences.\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.89\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.11\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22.17\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.10\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><i>n\/a<\/i> not available\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h3>\n<p>Questionnaire responses, with paired before and after questionnaire answers, are available as Additional file 3 and summarised in Table 1. The spread of answers to pupil questionnaires, on the Likert scale, is presented in Fig. 1. Because individual pupils may apply different criteria for each specific question (i.e. categories are simultaneously pupil-specific and question-specific), we regard the separation into five categories as unsuitable for either a continuous approximation or ranking. However, the direction of change on the Likert scale is comparable throughout the data, on the weak assumption that each pupil applies his or her own criteria for a given question consistently both before and after. Apart from changes over the course of the activity, whether the weight of answers is in the \u201cdisagree\u201d categories (1 and 2) vs the \u201cagree\u201d categories (4 and 5) may also have some general meaning. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Barker_IntJourSTEMEd2015_2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"9d11becc267eaee2c62fd32b55cd9a3a\"><img alt=\"Fig1 Barker IntJourSTEMEd2015 2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/e\/ed\/Fig1_Barker_IntJourSTEMEd2015_2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> For the pupil questions on a Likert scale (Table 1), the summary of answers before the activities (white) and after (black). N\u2009=\u200914 (before), N\u2009=\u200912 (after). Dashed arrows indicate the direction of change considered to be \u201cimprovement\u201d. Solid arrows, below these, indicate the observed direction of most changes. Brief mnemonics (in italics) summarise question content. For the full text of questions, see Table 1.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>To an extent, all our questions are expected, if anything, to improve as a result of the activities. For example, a significant part of the first component (\u201cINTRO\u201d) is devoted to programming. Hence, it would represent either disaster for the educational approach, or perhaps humour on the part of pupils, if pupils tended to agree more with \u201cI cannot program a computer\u201d after the event than before. (The question has some similarity to a \u201ccontrol\u201d in a laboratory experiment). Fortunately, in line with the educational approach being sound, evidence of disastrous effects was weak or absent.\n<\/p><p>With Question 1, \u201cI will end up working in science\u201d, most pupils agree and there is no systematic change during the activity. This is as expected, because pupils with no interest in science would be unlikely to volunteer to take part. Question 2, \u201cI think computers are useful within biology\u201d, is more specifically related to the activity itself. Again, most pupils agree both before and after, but here we see strong evidence of improvement (log likelihood ratio, \u0394\u2113\u2009=\u20093.47). The usefulness of computers in other sciences (Question 9, \u201cI think computers are useful within sciences other than biology\u201d) shows little change as a result of the activity. As part of the volunteers\u2019 positive mentality, we also see that most pupils expected to enjoy the activity and, in practice, did so (Question 4, \u201cI do not expect to enjoy [after: did not enjoy] the activity today\u201d). For this question, there is a fairly high level of change over the course of the activity, with four pupils reporting changes on the Likert scale in the direction opposite to \u201cimprovement\u201d, and three in the direction of \u201cimprovement\u201d (Table 1). All pupils have a strong desire to continue studies at university, which was entirely unchanged by participating in the activity (Question 7, \u201cI am not intending to go to university\u201d). Both before and after, they have a strong interest in biology (Question 12, \u201cI am not interested in biology\u201d). One pupil reports a change in the direction opposite to improvement, whose implications are difficult to assess without a larger sample. All appear interested in computers (Question 13, \u201cI do not enjoy using computers for fun\u201d). \n<\/p><p>Questions 3 (\u201cI have heard of Linux\u201d) and 8 (\u201cI have not heard of the Raspberry Pi\u201d) reveal that, before the activity, most pupils have not heard of the Linux operating system or the Raspberry Pi, though more have heard of the Raspberry Pi than Linux. Improvement over the course of the activity is almost guaranteed due to its content, and indeed there is strong evidence for this (\u0394\u2113\u2009=\u20096.93 for Linux, \u0394\u2113\u2009=\u20094.27 for the Raspberry Pi).\n<\/p><p>More complex and perhaps more educationally relevant (Question 11, \u201cI cannot program a computer\u201d), students show strong evidence of improvement from a position of not regarding themselves able to program a computer, to being positive or at least uncertain (\u0394\u2113\u2009=\u20093.10). From a position of considering themselves generally poor at using a computer to analyse DNA sequences (Question 14, \u201cI am good at using a computer to analyse DNA sequences\u201d), there is strong evidence of improvement towards a more positive assessment of their own abilities (\u0394\u2113\u2009=\u20093.10), although no pupil ended up strongly confident (Fig. 1). We speculate this would improve further with repeated activities.\n<\/p><p>Questions 5 (\u201cI know more about computers than most adults do\u201d) and 6 (\u201cI know more about computers than my teachers do\u201d) are logically correlated, and both show weak evidence of improvement as a result of the activities. We prefer to interpret this as evidence of increased computational confidence among the pupils; however, heightened perception of adult incompetence cannot be ruled out.\n<\/p><p>Question 10 (\u201cI would rather not use a computer, I would rather use a phone or a tablet\u201d) indicates a base-line level of openness to the educational method used. Our activity, in contrast to much computational activity by young people, is computer-based, using the Raspberry Pi. Pupils\u2019 views are generally moderate both before and after, but with borderline strong evidence of a strengthening tendency to prefer a computer as a result of the activity (\u0394\u2113\u2009=\u20092.08).\n<\/p><p>Pupil free-text answers to Q15 (\u201cWhat was the best part of the activity today, and why\u201d) included \u201cgetting the hands-on experience with computers, as it is not something we ever get to do at school\u201d; \u201cI liked the programming\u2014making files and commands\u2014because it was really interesting to see how computers actually work\u201d; \u201cThe simpler programming stages as they were at an accomplishable level & well explained\u201d; \u201cLearning how computers are used in biology and getting to try it ourselves because it was interesting and a new experience.\u201d; \u201cThe best part was going on flybase because I found it really interesting\u201d. Free-text answers to Q16 (\u201cWhat was the worst part of the activity today, and why?\u201d) included \u201cTyping everything during programme because it took ages and a tiny mistake or omission could cause problems.\u201d; \u201cThe worst part was flybase as it was difficult to understand.\u201d; \u201cI enjoyed the entire day and there wasn\u2019t one part I didn\u2019t enjoy. It was really interesting.\u201d; \u201ctyping in all of the commands to the LXterminal as it took a long time and it was easy to make mistakes.\u201d; \u201cI didn\u2019t enjoy typing stuff into LxTerminal because I was awful at it\u201d; \u201cThe DNA sequencing on the internet was confusing and poorly explained in the handbook\u201d; and \u201cI don\u2019t think the instructions were very clear\u201d (Additional file 3).\n<\/p><p>It seems the activities stretched pupils intellectually and practically, which was regarded both positively and negatively among the pupils. This is illustrated, for example, by opposite views on the Flybase database, and may also be reflected in changing views on the enjoyability of the activities (Question 4). Typing programs and using the command-line within the terminal was unpopular. However, this is a crucial part of computational science, which cannot be bypassed, and which remains difficult even for professional bioinformaticians. The practical handouts clearly did not suit all pupils. These attempt a practical balance between brevity and total coverage. In future, these could be improved by inclusion of diagrams and screenshots.\n<\/p><p>School staff free-text answers to the question \u201cWhat was the best part of the activity today, and why?\u201d included \u201cThe pupils being given the opportunity to contextualise their theory based work in class into a practical application using the software. They gained a great deal of practice in making the connection between proteins\u2009+\u2009amino acids being the result of DNA sequences.\u201d And: \u201cWatching pupils who had never written code program Pis. Seeing their excitement when the code worked.\u201d Also: \u201cThe pupils were all fully engaged & gained a great deal from the seminar. I have also gained an insight into bioinformatics which will be vital for teaching the next part of the new higher on genomics\u201d (Additional file 3).\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h3>\n<p>Our results were for a small sample and cannot represent the full range of voluntary groups one could draw from schools even in Scotland. But it is encouraging for the approach taken that almost all changes in pupil questionnaires, over the course of the activity, were in the direction that we regard as indicating improvement.\n<\/p><p>For all pupil questions showing strong evidence of change over the course of the activity (\u0394\u2113\u2009\u2265\u20092), the tendency was in the direction of improvement. Also, free-text responses from pupils were generally positive, though with a suggestion that the handouts could be improved (\u201cResults\u201d, above). On this basis, we judge the preliminary investigation a success. The evidence strongly suggests that, in their own assessment, the pupils benefited from the bioinformatics activities undertaken. This complements other research, involving successful bioinformatics activities developed specifically for school level (e.g. Marques et al.<sup id=\"rdp-ebb-cite_ref-MarquesBio_8-2\" class=\"reference\"><a href=\"#cite_note-MarquesBio-8\" rel=\"external_link\">[8]<\/a><\/sup>). \n<\/p><p>Gallagher et al.<sup id=\"rdp-ebb-cite_ref-GallagherAFirst_10-1\" class=\"reference\"><a href=\"#cite_note-GallagherAFirst-10\" rel=\"external_link\">[10]<\/a><\/sup>) also reported a successful experience bringing bioinformatics to high school level, but with three challenges. We did not experience any of these. Firstly, some of the students in their study questioned the relevance of computation to biology. Gallagher et al. propose that part of their difficulty was use of computer scientists to teach lessons. In our case, lessons were co-taught by university biologists or biologists and chemists, alongside school staff, lending support to this proposition. However, other factors may be relevant, for example the use of whole-class teaching by Gallagher et al., as opposed to our voluntary groups; and the passage of time since their study. Secondly, some of the students in the study by Gallagher et al. doubted the value of learning about algorithms. Since our material was practical, with little detail on algorithms, the objection would\u2014a priori\u2014be unlikely. Thirdly, Gallagher et al. found some students doubted the relevance of bioinformatics material if this was not covered in official tests. In our case, there is relevance to the syllabus (see \u201cRelevance to the new Higher curriculum for Biology and Human Biology\u201d, below), and this was highlighted during the activities by school staff. With a different audience but similar educational aim, Wood and Gebhardt<sup id=\"rdp-ebb-cite_ref-WoodBio_15-1\" class=\"reference\"><a href=\"#cite_note-WoodBio-15\" rel=\"external_link\">[15]<\/a><\/sup> also emphasise the importance of relevance to the curriculum. They report successfully updating high school teachers on the topic of bioinformatics via hands-on experience, practical demonstrations and lectures. One of their considerations for those wishing to replicate their approach, \u201cWork within the practical limitations of the classroom\u201d, may be alleviated by hardware such as the Raspberry Pi (see \u201cThe Raspberry Pi\u2014a flexible, general-purpose computer\u201d, below). \n<\/p><p>How far can we generalise from our results? Further work is warranted, for example using a broader range of teaching material; involvement of a larger number of voluntary groups from a wider range of schools, for a more varied sample and greater statistical power; use of less enthusiastic staff; a lower staff\/student ratio; a whole-class activity, not only a voluntary class sample; validation of the assessment instrument; direct measurement of student knowledge acquisition, in addition to self-assessment; interviews with staff and students and field observations. Where there is a surfeit of material available among our university colleagues \u2014 for example, introductory teaching material on the widely used program BLAST (Altschul et al.<sup id=\"rdp-ebb-cite_ref-AltschulGap_6-2\" class=\"reference\"><a href=\"#cite_note-AltschulGap-6\" rel=\"external_link\">[6]<\/a><\/sup>) \u2014 one can imagine educational experiments, allowing an evidence-based decision on which material is more successful, perhaps in different geographical regions, countries, educational systems or types of school or with pupils of different age, academic achievement levels, social background or gender. These are important topics for the future.\n<\/p><p>Bioinformatics has always been relatively \u201copen\u201d as a subject, with public deposition of DNA sequence data associated with research publications and a wealth of free software. Although the algorithms do not change rapidly, the precise details of the software do. For these two reasons\u2014the tradition of openness within the field and the rapid obsolescence of specific details\u2014we consider the subject particularly amenable to online-only, open access teaching material. This costs nothing to obtain; for those writing the material, it can be relatively easily updated in light of developments in software and data. The material we used was from one open access bioinformatics educational effort, 4273<i>\u03c0<\/i> (Barker et al.<sup id=\"rdp-ebb-cite_ref-Barker4273n_7-2\" class=\"reference\"><a href=\"#cite_note-Barker4273n-7\" rel=\"external_link\">[7]<\/a><\/sup>). We hope to expand this resource to cover a wider range of bioinformatics topics, whilst continuing to ensure that all material is tested in practice and peer-reviewed. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Relevance_to_the_new_higher_curriculum_for_biology_and_human_biology\">Relevance to the new higher curriculum for biology and human biology<\/span><\/h3>\n<p>The new Scottish Qualifications Authority Curriculum for Excellence Higher Syllabus for both Higher Biology and Higher Human Biology includes DNA technologies. Part of the content focuses on the use of computer software in both Biology (Scottish Qualifications Authority, pp. 10, 16<sup id=\"rdp-ebb-cite_ref-ScottHigher2_31-0\" class=\"reference\"><a href=\"#cite_note-ScottHigher2-31\" rel=\"external_link\">[31]<\/a><\/sup>) and Human Biology (Scottish Qualifications Authority, pp. 14, 15, 56<sup id=\"rdp-ebb-cite_ref-ScottHigher3_32-0\" class=\"reference\"><a href=\"#cite_note-ScottHigher3-32\" rel=\"external_link\">[32]<\/a><\/sup>). The suggested learning activity \u201cUse genome data to identify stop and start codons and known protein coding sequences\u201d (Scottish Qualifications Authority<sup id=\"rdp-ebb-cite_ref-ScottHigher3_32-1\" class=\"reference\"><a href=\"#cite_note-ScottHigher3-32\" rel=\"external_link\">[32]<\/a><\/sup>) is directly addressed by the \u201cDNA\u201d component of 4273\u03c0, as used in the current study. Other components are also relevant, for example to phylogeny and comparative genomics in Higher Biology (Scottish Qualifications Authority, p. 16<sup id=\"rdp-ebb-cite_ref-ScottHigher2_31-1\" class=\"reference\"><a href=\"#cite_note-ScottHigher2-31\" rel=\"external_link\">[31]<\/a><\/sup>). In addition, the new Curriculum for Excellence Advanced Higher Biology has content related to protein structure (covered, to an extent, by the \u201cENZYME\u201d component of 4273<i>\u03c0<\/i>) and scientific investigative approaches (Scottish Qualifications Authority<sup id=\"rdp-ebb-cite_ref-ScottHigher4_33-0\" class=\"reference\"><a href=\"#cite_note-ScottHigher4-33\" rel=\"external_link\">[33]<\/a><\/sup>).\n<\/p><p>Based on our experiences as outlined in this paper, and in agreement with Gallagher et al.<sup id=\"rdp-ebb-cite_ref-GallagherAFirst_10-2\" class=\"reference\"><a href=\"#cite_note-GallagherAFirst-10\" rel=\"external_link\">[10]<\/a><\/sup>, workshops taught alongside university staff seem an efficient means of professional development for teachers, who may be presented with curriculum content which is new to them and outwith their experience. From the pupils\u2019 point of view, having access to workshops and seminars run by university staff also gives some experience of the university experience, particularly when\u2014as in the current case\u2014the teaching material is precisely at the undergraduate level. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_Raspberry_Pi_.E2.80.94_a_flexible.2C_general-purpose_computer\">The Raspberry Pi \u2014 a flexible, general-purpose computer<\/span><\/h3>\n<p>The use of Raspberry Pi computers allows for a range of new software and experiences for the pupils, without any major impact on school\u2019s ICT staff and systems. By bringing in complete software plus hardware systems, there is no need for specific software to be installed on school computers (some of which may require existing security restrictions to be relaxed). As the first part of the activity was to assemble the Pi, for the activity held at Kilgraston, the only prerequisite for the school was to <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/4273pi.org\/files\/2015\/10\/hardware_4273pi.pdf\" target=\"_blank\">check the school\u2019s VGA monitors would work with the Pi<\/a> and check there were enough power sockets.\n<\/p><p>Pupils may be more confident using the Raspberry Pi, with the knowledge that if they \u201cbroke\u201d the software set-up, instructors could just swap its SD card. This allows a risk-taking approach to learning, and the possibility to learn by ones mistakes in a way that is prevented in managed computer classrooms. (Despite this, in the two school activities in this study, and in 3 years of teaching BL4273 Bioinformatics for Biologists at the University of St Andrews, we have never had to reinitialise an SD card due to such an accident. We plan to incorporate greater software risks in future material). It also gives pupils experience of Linux, the operating system they may well use should they choose to further their career in computational science. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h3>\n<p>In our current study, we have demonstrated that practical bioinformatics material initially developed for an optional, final-year undergraduate module can be used successfully with school pupils at sixth form level (S5 and S6 in Scotland). Covering practical bioinformatics at this level in all schools may be an attainable goal, with subsequent benefits to higher education and research, in both biology and other sciences. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>BLAST:<\/b> Basic Local Alignment Search Tool\n<\/p><p><b>GOBLET:<\/b> Global Organisation for Bioinformatics Learning, Education and Training\n<\/p><p><b>ICT:<\/b> information and communications technology\n<\/p><p><b>ML:<\/b> maximum likelihood\n<\/p><p><b>SD:<\/b> secure digital\n<\/p><p><b>STEMM:<\/b> Science, Technology, Engineering, Mathematics and Medicine\n<\/p><p><b>UK:<\/b> United Kingdom\n<\/p><p><b>USA:<\/b> United States of America\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.27_contributions\">Authors' contributions<\/span><\/h2>\n<p>DB and HP conceived of the school activities. DB, RGA, JLM, HP, MMC, LD, GTPM and SDS carried out the school activities. DB analysed results with assistance from RGA and JLM. DB wrote the manuscript with input from RGA, JLM, HP and other authors. All authors read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Additional_files\">Additional files<\/span><\/h2>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 1. Teaching material and blank questionnaires used for the event with Kilgraston School\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Names and contact details have been redacted.\n<p>Format: ZIP; Size: 431KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.stemeducationjournal.com\/content\/supplementary\/s40594-015-0030-z-s1.zip\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 2. Teaching material and blank questionnaires used for the event with Forfar Academy\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Names and contact details have been redacted.\n<p>Format: ZIP; Size: 478KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.stemeducationjournal.com\/content\/supplementary\/s40594-015-0030-z-s2.zip\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 3. Data from questionnaires (LibreOffice Calc spreadsheet)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">For details of contents, see the \u201cNotes\u201d worksheet.\n<p>Format: ODS; Size: 81KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.stemeducationjournal.com\/content\/supplementary\/s40594-015-0030-z-s3.ods\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This work was supported in part by the Science and Technology Facilities Council under grant ST\/M000435\/1 to DB. We thank all participants in our activities for their help, and thank Ian McDonald for technical assistance.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-HogwegTheRoo-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HogwegTheRoo_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hogeweg, P. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3068925\" target=\"_blank\">\"The roots of bioinformatics in theoretical biology\"<\/a>. <i>PLOS Computational Biology<\/i> <b>7<\/b> (3): e1002021. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1002021\" target=\"_blank\">10.1371\/journal.pcbi.1002021<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3068925\/\" target=\"_blank\">PMC3068925<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21483479\" target=\"_blank\">21483479<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3068925\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3068925<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+roots+of+bioinformatics+in+theoretical+biology&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Hogeweg%2C+P.&rft.au=Hogeweg%2C+P.&rft.date=2011&rft.volume=7&rft.issue=3&rft.pages=e1002021&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1002021&rft_id=info:pmc\/PMC3068925&rft_id=info:pmid\/21483479&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3068925&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaganaASur-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaganaASur_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Magana, A.J.; Taleyarkhan, M.; Alvarado, D.R.; Kane, M.; Springer, J.; Clase, K. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4255348\" target=\"_blank\">\"A survey of scholarly literature describing the field of bioinformatics education and bioinformatics educational research\"<\/a>. <i>CBE \u2013 Life Sciences Education<\/i> <b>13<\/b> (4): 607\u201323. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1187%2Fcbe.13-10-0193\" target=\"_blank\">10.1187\/cbe.13-10-0193<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4255348\/\" target=\"_blank\">PMC4255348<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25452484\" target=\"_blank\">25452484<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4255348\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4255348<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+survey+of+scholarly+literature+describing+the+field+of+bioinformatics+education+and+bioinformatics+educational+research&rft.jtitle=CBE+%E2%80%93+Life+Sciences+Education&rft.aulast=Magana%2C+A.J.%3B+Taleyarkhan%2C+M.%3B+Alvarado%2C+D.R.%3B+Kane%2C+M.%3B+Springer%2C+J.%3B+Clase%2C+K.&rft.au=Magana%2C+A.J.%3B+Taleyarkhan%2C+M.%3B+Alvarado%2C+D.R.%3B+Kane%2C+M.%3B+Springer%2C+J.%3B+Clase%2C+K.&rft.date=2014&rft.volume=13&rft.issue=4&rft.pages=607%E2%80%9323&rft_id=info:doi\/10.1187%2Fcbe.13-10-0193&rft_id=info:pmc\/PMC4255348&rft_id=info:pmid\/25452484&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4255348&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BensonGen-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BensonGen_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Benson, D.A.; Clark, K.; Karsch-Mizrachi, I.; Lipman, D.J.; Ostell, J.; Sayers, E.W. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383990\" target=\"_blank\">\"GenBank\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (Database issue): D30\u2013D35. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1216\" target=\"_blank\">10.1093\/nar\/gku1216<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383990\/\" target=\"_blank\">PMC4383990<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383990\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383990<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GenBank&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Benson%2C+D.A.%3B+Clark%2C+K.%3B+Karsch-Mizrachi%2C+I.%3B+Lipman%2C+D.J.%3B+Ostell%2C+J.%3B+Sayers%2C+E.W.&rft.au=Benson%2C+D.A.%3B+Clark%2C+K.%3B+Karsch-Mizrachi%2C+I.%3B+Lipman%2C+D.J.%3B+Ostell%2C+J.%3B+Sayers%2C+E.W.&rft.date=2015&rft.volume=43&rft.issue=Database+issue&rft.pages=D30%E2%80%93D35&rft_id=info:doi\/10.1093%2Fnar%2Fgku1216&rft_id=info:pmc\/PMC4383990&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383990&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CunninghamEns-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CunninghamEns_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cunningham, F.; Amode, M.R.; Barrell, D., et al. (2015). \"Ensembl 2015\". <i>Nucleic Acids Research<\/i> <b>43<\/b> (Database issue): D662-D669. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1010\" target=\"_blank\">10.1093\/nar\/gku1010<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ensembl+2015&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Cunningham%2C+F.%3B+Amode%2C+M.R.%3B+Barrell%2C+D.%2C+et+al.&rft.au=Cunningham%2C+F.%3B+Amode%2C+M.R.%3B+Barrell%2C+D.%2C+et+al.&rft.date=2015&rft.volume=43&rft.issue=Database+issue&rft.pages=D662-D669&rft_id=info:doi\/10.1093%2Fnar%2Fgku1010&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GalperinThe2015-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GalperinThe2015_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Galperin, M.Y.; Rigden, D.J.; Fern\u00e1ndez-Su\u00e1rez, X.M. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383995\" target=\"_blank\">\"The 2015 <i>Nucleic Acids Research<\/i> Database Issue and Molecular Biology Database Collection\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (Database issue): D1-D5. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1241\" target=\"_blank\">10.1093\/nar\/gku1241<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383995\/\" target=\"_blank\">PMC4383995<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25593347\" target=\"_blank\">25593347<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383995\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383995<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+2015+%27%27Nucleic+Acids+Research%27%27+Database+Issue+and+Molecular+Biology+Database+Collection&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Galperin%2C+M.Y.%3B+Rigden%2C+D.J.%3B+Fern%C3%A1ndez-Su%C3%A1rez%2C+X.M.&rft.au=Galperin%2C+M.Y.%3B+Rigden%2C+D.J.%3B+Fern%C3%A1ndez-Su%C3%A1rez%2C+X.M.&rft.date=2015&rft.volume=43&rft.issue=Database+issue&rft.pages=D1-D5&rft_id=info:doi\/10.1093%2Fnar%2Fgku1241&rft_id=info:pmc\/PMC4383995&rft_id=info:pmid\/25593347&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383995&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AltschulGap-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AltschulGap_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-AltschulGap_6-1\" rel=\"external_link\">6.1<\/a><\/sup> <sup><a href=\"#cite_ref-AltschulGap_6-2\" rel=\"external_link\">6.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Altschul, S.F.; Madden, T.L.; Sch\u00e4ffer, A.A.; Zhang, J.; Zhang, Z.; Miller, W.; Lipman, D.J. (1997). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917\" target=\"_blank\">\"Gapped BLAST and PSI-BLAST: a new generation of protein database search programs\"<\/a>. <i>Nucleic Acids Research<\/i> <b>25<\/b> (17): 3389-3402. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2F25.17.3389\" target=\"_blank\">10.1093\/nar\/25.17.3389<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC146917\/\" target=\"_blank\">PMC146917<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/9254694\" target=\"_blank\">9254694<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gapped+BLAST+and+PSI-BLAST%3A+a+new+generation+of+protein+database+search+programs&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Altschul%2C+S.F.%3B+Madden%2C+T.L.%3B+Sch%C3%A4ffer%2C+A.A.%3B+Zhang%2C+J.%3B+Zhang%2C+Z.%3B+Miller%2C+W.%3B+Lipman%2C+D.J.&rft.au=Altschul%2C+S.F.%3B+Madden%2C+T.L.%3B+Sch%C3%A4ffer%2C+A.A.%3B+Zhang%2C+J.%3B+Zhang%2C+Z.%3B+Miller%2C+W.%3B+Lipman%2C+D.J.&rft.date=1997&rft.volume=25&rft.issue=17&rft.pages=3389-3402&rft_id=info:doi\/10.1093%2Fnar%2F25.17.3389&rft_id=info:pmc\/PMC146917&rft_id=info:pmid\/9254694&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC146917&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Barker4273n-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Barker4273n_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-Barker4273n_7-1\" rel=\"external_link\">7.1<\/a><\/sup> <sup><a href=\"#cite_ref-Barker4273n_7-2\" rel=\"external_link\">7.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barker, D.; Ferrier, D.E.K.; Holland, P.W.H.; Mitchell, J.B.O.; Plaisier, H.; Ritchie, M.G.; Smart, S.D. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3751261\" target=\"_blank\">\"4273<i>\u03c0<\/i>: Bioinformatics education on low cost ARM hardware\"<\/a>. <i>BMC Bioinformatics<\/i> <b>14<\/b>: 243. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-14-243\" target=\"_blank\">10.1186\/1471-2105-14-243<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3751261\/\" target=\"_blank\">PMC3751261<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23937194\" target=\"_blank\">23937194<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3751261\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3751261<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=4273%27%27%CF%80%27%27%3A+Bioinformatics+education+on+low+cost+ARM+hardware&rft.jtitle=BMC+Bioinformatics&rft.aulast=Barker%2C+D.%3B+Ferrier%2C+D.E.K.%3B+Holland%2C+P.W.H.%3B+Mitchell%2C+J.B.O.%3B+Plaisier%2C+H.%3B+Ritchie%2C+M.G.%3B+Smart%2C+S.D.&rft.au=Barker%2C+D.%3B+Ferrier%2C+D.E.K.%3B+Holland%2C+P.W.H.%3B+Mitchell%2C+J.B.O.%3B+Plaisier%2C+H.%3B+Ritchie%2C+M.G.%3B+Smart%2C+S.D.&rft.date=2013&rft.volume=14&rft.pages=243&rft_id=info:doi\/10.1186%2F1471-2105-14-243&rft_id=info:pmc\/PMC3751261&rft_id=info:pmid\/23937194&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3751261&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarquesBio-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MarquesBio_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-MarquesBio_8-1\" rel=\"external_link\">8.1<\/a><\/sup> <sup><a href=\"#cite_ref-MarquesBio_8-2\" rel=\"external_link\">8.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Marques, I.; Almeida, P.; Alves, R.; Jo\u00e3o Dias, M.; Godinho, A.; Pereira-Leal, J.B. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3900377\" target=\"_blank\">\"Bioinformatics Projects Supporting Life-Sciences Learning in High Schools\"<\/a>. <i>PLOS Computational Biology<\/i> <b>10<\/b> (1): e1003404. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1003404\" target=\"_blank\">10.1371\/journal.pcbi.1003404<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3900377\/\" target=\"_blank\">PMC3900377<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3900377\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3900377<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+Projects+Supporting+Life-Sciences+Learning+in+High+Schools&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Marques%2C+I.%3B+Almeida%2C+P.%3B+Alves%2C+R.%3B+Jo%C3%A3o+Dias%2C+M.%3B+Godinho%2C+A.%3B+Pereira-Leal%2C+J.B.&rft.au=Marques%2C+I.%3B+Almeida%2C+P.%3B+Alves%2C+R.%3B+Jo%C3%A3o+Dias%2C+M.%3B+Godinho%2C+A.%3B+Pereira-Leal%2C+J.B.&rft.date=2014&rft.volume=10&rft.issue=1&rft.pages=e1003404&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1003404&rft_id=info:pmc\/PMC3900377&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3900377&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CorpasTheGOB-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CorpasTheGOB_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Corpas, M.; Jimenez, R.C.; Bongcam-Rudloff, E.; et al. (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4271145\" target=\"_blank\">\"The GOBLET training portal: a global repository of bioinformatics training materials, courses and trainers\"<\/a>. <i>Bioinformatics<\/i> <b>31<\/b> (1): 140\u2013142. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtu601\" target=\"_blank\">10.1093\/bioinformatics\/btu601<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4271145\/\" target=\"_blank\">PMC4271145<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25189782\" target=\"_blank\">25189782<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4271145\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4271145<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+GOBLET+training+portal%3A+a+global+repository+of+bioinformatics+training+materials%2C+courses+and+trainers&rft.jtitle=Bioinformatics&rft.aulast=Corpas%2C+M.%3B+Jimenez%2C+R.C.%3B+Bongcam-Rudloff%2C+E.%3B+et+al.&rft.au=Corpas%2C+M.%3B+Jimenez%2C+R.C.%3B+Bongcam-Rudloff%2C+E.%3B+et+al.&rft.date=2015&rft.volume=31&rft.issue=1&rft.pages=140%E2%80%93142&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtu601&rft_id=info:pmc\/PMC4271145&rft_id=info:pmid\/25189782&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4271145&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GallagherAFirst-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-GallagherAFirst_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-GallagherAFirst_10-1\" rel=\"external_link\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-GallagherAFirst_10-2\" rel=\"external_link\">10.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gallagher, S.R.; Coon, W.; Donley, K.; Scott, A.; Goldberg, D.S. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203055\" target=\"_blank\">\"A first attempt to bring computational biology into advanced high school biology classrooms\"<\/a>. <i>PLOS Computational Biology<\/i> <b>7<\/b> (10): e1002244. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1002244\" target=\"_blank\">10.1371\/journal.pcbi.1002244<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3203055\/\" target=\"_blank\">PMC3203055<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22046118\" target=\"_blank\">22046118<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203055\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203055<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+first+attempt+to+bring+computational+biology+into+advanced+high+school+biology+classrooms&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Gallagher%2C+S.R.%3B+Coon%2C+W.%3B+Donley%2C+K.%3B+Scott%2C+A.%3B+Goldberg%2C+D.S.&rft.au=Gallagher%2C+S.R.%3B+Coon%2C+W.%3B+Donley%2C+K.%3B+Scott%2C+A.%3B+Goldberg%2C+D.S.&rft.date=2011&rft.volume=7&rft.issue=10&rft.pages=e1002244&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1002244&rft_id=info:pmc\/PMC3203055&rft_id=info:pmid\/22046118&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3203055&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LewitterTeach-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LewitterTeach_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lewitter, F.; Bourne, P.E. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203059\" target=\"_blank\">\"Teaching bioinformatics at the secondary school level\"<\/a>. <i>PLOS Computational Biology<\/i> <b>7<\/b> (10): e1002242. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1002242\" target=\"_blank\">10.1371\/journal.pcbi.1002242<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3203059\/\" target=\"_blank\">PMC3203059<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203059\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3203059<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Teaching+bioinformatics+at+the+secondary+school+level&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Lewitter%2C+F.%3B+Bourne%2C+P.E.&rft.au=Lewitter%2C+F.%3B+Bourne%2C+P.E.&rft.date=2011&rft.volume=7&rft.issue=10&rft.pages=e1002242&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1002242&rft_id=info:pmc\/PMC3203059&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3203059&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McQueenDesign-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McQueenDesign_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McQueen, J.; Wright, J.J.; Fox, J.A. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3431290\" target=\"_blank\">\"Design and implementation of a genomics field trip program aimed at secondary school students\"<\/a>. <i>PLOS Computational Biology<\/i> <b>8<\/b> (8): e1002636. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1002636\" target=\"_blank\">10.1371\/journal.pcbi.1002636<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3431290\/\" target=\"_blank\">PMC3431290<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22956895\" target=\"_blank\">22956895<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3431290\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3431290<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Design+and+implementation+of+a+genomics+field+trip+program+aimed+at+secondary+school+students&rft.jtitle=PLOS+Computational+Biology&rft.aulast=McQueen%2C+J.%3B+Wright%2C+J.J.%3B+Fox%2C+J.A.&rft.au=McQueen%2C+J.%3B+Wright%2C+J.J.%3B+Fox%2C+J.A.&rft.date=2012&rft.volume=8&rft.issue=8&rft.pages=e1002636&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1002636&rft_id=info:pmc\/PMC3431290&rft_id=info:pmid\/22956895&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3431290&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KovarikBio-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KovarikBio_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kovarik, D.N.; Patterson, D.G.; Cohen, C.; Sanders, E.A.; Peterson, K.A.; Porter, S.G.; Chowning, J.T. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3763012\" target=\"_blank\">\"Bioinformatics education in high school: implications for promoting science, technology, engineering, and mathematics careers\"<\/a>. <i>CBE \u2013 Life Sciences Education<\/i> <b>12<\/b> (3): 441\u201359. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1187%2Fcbe.12-11-0193\" target=\"_blank\">10.1187\/cbe.12-11-0193<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3763012\/\" target=\"_blank\">PMC3763012<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24006393\" target=\"_blank\">24006393<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3763012\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3763012<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+education+in+high+school%3A+implications+for+promoting+science%2C+technology%2C+engineering%2C+and+mathematics+careers&rft.jtitle=CBE+%E2%80%93+Life+Sciences+Education&rft.aulast=Kovarik%2C+D.N.%3B+Patterson%2C+D.G.%3B+Cohen%2C+C.%3B+Sanders%2C+E.A.%3B+Peterson%2C+K.A.%3B+Porter%2C+S.G.%3B+Chowning%2C+J.T.&rft.au=Kovarik%2C+D.N.%3B+Patterson%2C+D.G.%3B+Cohen%2C+C.%3B+Sanders%2C+E.A.%3B+Peterson%2C+K.A.%3B+Porter%2C+S.G.%3B+Chowning%2C+J.T.&rft.date=2013&rft.volume=12&rft.issue=3&rft.pages=441%E2%80%9359&rft_id=info:doi\/10.1187%2Fcbe.12-11-0193&rft_id=info:pmc\/PMC3763012&rft_id=info:pmid\/24006393&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3763012&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MachlufInt-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MachlufInt_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Machluf, Y.; Yarden, A. (2013). \"Integrating bioinformatics into senior high school: design principles and implications\". <i>Briefings in Bioinformatics<\/i> <b>14<\/b> (5): 648-60. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbt030\" target=\"_blank\">10.1093\/bib\/bbt030<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23665511\" target=\"_blank\">23665511<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrating+bioinformatics+into+senior+high+school%3A+design+principles+and+implications&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Machluf%2C+Y.%3B+Yarden%2C+A.&rft.au=Machluf%2C+Y.%3B+Yarden%2C+A.&rft.date=2013&rft.volume=14&rft.issue=5&rft.pages=648-60&rft_id=info:doi\/10.1093%2Fbib%2Fbbt030&rft_id=info:pmid\/23665511&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoodBio-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WoodBio_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-WoodBio_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wood, L.; Gebhardt, P. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3681668\" target=\"_blank\">\"Bioinformatics goes to school \u2013 new avenues for teaching contemporary biology\"<\/a>. <i>PLOS Computational Biology<\/i> <b>9<\/b> (6): e1003089. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.1003089\" target=\"_blank\">10.1371\/journal.pcbi.1003089<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3681668\/\" target=\"_blank\">PMC3681668<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23785266\" target=\"_blank\">23785266<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3681668\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3681668<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+goes+to+school+%E2%80%93+new+avenues+for+teaching+contemporary+biology&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Wood%2C+L.%3B+Gebhardt%2C+P.&rft.au=Wood%2C+L.%3B+Gebhardt%2C+P.&rft.date=2013&rft.volume=9&rft.issue=6&rft.pages=e1003089&rft_id=info:doi\/10.1371%2Fjournal.pcbi.1003089&rft_id=info:pmc\/PMC3681668&rft_id=info:pmid\/23785266&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3681668&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TobyBio-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TobyBio_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Toby, I.; Pope, A. (2014). \"Bioinformatics tools available for K-12 students to engage in research\". <i>Aviation, Space, and Environmental Medicine<\/i> <b>85<\/b> (4): 484-485. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3357%2FASEM.3953.2014\" target=\"_blank\">10.3357\/ASEM.3953.2014<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24754215\" target=\"_blank\">24754215<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+tools+available+for+K-12+students+to+engage+in+research&rft.jtitle=Aviation%2C+Space%2C+and+Environmental+Medicine&rft.aulast=Toby%2C+I.%3B+Pope%2C+A.&rft.au=Toby%2C+I.%3B+Pope%2C+A.&rft.date=2014&rft.volume=85&rft.issue=4&rft.pages=484-485&rft_id=info:doi\/10.3357%2FASEM.3953.2014&rft_id=info:pmid\/24754215&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CollegeBio-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CollegeBio_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">College voor Examens (April 2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.examenblad.nl\/examenstof\/syllabus-2016-biologie-vwo-nader\/2016\/vwo\/f=\/biologie_vwo_2016_def_voor_hervaststelling.pdf\" target=\"_blank\">\"Biologie VWO. Syllabus Centraal Examen 2016\"<\/a> (PDF). pp. 59<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.examenblad.nl\/examenstof\/syllabus-2016-biologie-vwo-nader\/2016\/vwo\/f=\/biologie_vwo_2016_def_voor_hervaststelling.pdf\" target=\"_blank\">http:\/\/www.examenblad.nl\/examenstof\/syllabus-2016-biologie-vwo-nader\/2016\/vwo\/f=\/biologie_vwo_2016_def_voor_hervaststelling.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 21 October 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Biologie+VWO.+Syllabus+Centraal+Examen+2016&rft.atitle=&rft.aulast=College+voor+Examens&rft.au=College+voor+Examens&rft.date=April+2014&rft.pages=pp.+59&rft_id=http%3A%2F%2Fwww.examenblad.nl%2Fexamenstof%2Fsyllabus-2016-biologie-vwo-nader%2F2016%2Fvwo%2Ff%3D%2Fbiologie_vwo_2016_def_voor_hervaststelling.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeferBio-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WeferBio_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wefer, S.H.; Sheppard, K. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2262119\" target=\"_blank\">\"Bioinformatics in high school biology curricula: a study of state science standards\"<\/a>. <i>CBE \u2013 Life Sciences Education<\/i> <b>7<\/b> (1): 155\u2013162. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1187%2Fcbe.07-05-0026\" target=\"_blank\">10.1187\/cbe.07-05-0026<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2262119\/\" target=\"_blank\">PMC2262119<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2262119\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2262119<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+in+high+school+biology+curricula%3A+a+study+of+state+science+standards&rft.jtitle=CBE+%E2%80%93+Life+Sciences+Education&rft.aulast=Wefer%2C+S.H.%3B+Sheppard%2C+K.&rft.au=Wefer%2C+S.H.%3B+Sheppard%2C+K.&rft.date=2008&rft.volume=7&rft.issue=1&rft.pages=155%E2%80%93162&rft_id=info:doi\/10.1187%2Fcbe.07-05-0026&rft_id=info:pmc\/PMC2262119&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2262119&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StajichTheBio-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StajichTheBio_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stajich, J.E.; Block, D.; Boulez, K.; et al. (2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187536\" target=\"_blank\">\"The BioPerl toolkit: Perl modules for the life sciences\"<\/a>. <i>Genome Research<\/i> <b>12<\/b> (10): 1611\u20131618. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.361602\" target=\"_blank\">10.1101\/gr.361602<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC187536\/\" target=\"_blank\">PMC187536<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12368254\" target=\"_blank\">12368254<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187536\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187536<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+BioPerl+toolkit%3A+Perl+modules+for+the+life+sciences&rft.jtitle=Genome+Research&rft.aulast=Stajich%2C+J.E.%3B+Block%2C+D.%3B+Boulez%2C+K.%3B+et+al.&rft.au=Stajich%2C+J.E.%3B+Block%2C+D.%3B+Boulez%2C+K.%3B+et+al.&rft.date=2002&rft.volume=12&rft.issue=10&rft.pages=1611%E2%80%931618&rft_id=info:doi\/10.1101%2Fgr.361602&rft_id=info:pmc\/PMC187536&rft_id=info:pmid\/12368254&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC187536&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-StabenauTheEns-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-StabenauTheEns_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stabenau, A.; McVicker, G.; Melsopp, C.; Proctor, G.; Clamp, M.; Birney, E. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479122\" target=\"_blank\">\"The Ensembl core software libraries\"<\/a>. <i>Genome Research<\/i> <b>14<\/b> (5): 929\u2013933. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.1857204\" target=\"_blank\">10.1101\/gr.1857204<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC479122\/\" target=\"_blank\">PMC479122<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15123588\" target=\"_blank\">15123588<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479122\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479122<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Ensembl+core+software+libraries&rft.jtitle=Genome+Research&rft.aulast=Stabenau%2C+A.%3B+McVicker%2C+G.%3B+Melsopp%2C+C.%3B+Proctor%2C+G.%3B+Clamp%2C+M.%3B+Birney%2C+E.&rft.au=Stabenau%2C+A.%3B+McVicker%2C+G.%3B+Melsopp%2C+C.%3B+Proctor%2C+G.%3B+Clamp%2C+M.%3B+Birney%2C+E.&rft.date=2004&rft.volume=14&rft.issue=5&rft.pages=929%E2%80%93933&rft_id=info:doi\/10.1101%2Fgr.1857204&rft_id=info:pmc\/PMC479122&rft_id=info:pmid\/15123588&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC479122&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScottHigher-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScottHigher_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Scottish Qualifications Authority (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.sqa.org.uk\/files_ccc\/CfE_CourseUnitSupportNotes_Higher_Sciences_Chemistry.pdf\" target=\"_blank\">\"Higher Chemistry Course Support Notes\"<\/a> (PDF). pp. 98<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sqa.org.uk\/files_ccc\/CfE_CourseUnitSupportNotes_Higher_Sciences_Chemistry.pdf\" target=\"_blank\">http:\/\/www.sqa.org.uk\/files_ccc\/CfE_CourseUnitSupportNotes_Higher_Sciences_Chemistry.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 October 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Higher+Chemistry+Course+Support+Notes&rft.atitle=&rft.aulast=Scottish+Qualifications+Authority&rft.au=Scottish+Qualifications+Authority&rft.date=2015&rft.pages=pp.+98&rft_id=http%3A%2F%2Fwww.sqa.org.uk%2Ffiles_ccc%2FCfE_CourseUnitSupportNotes_Higher_Sciences_Chemistry.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DosSantosIntro-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DosSantosIntro_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dos Santos, G.; Schroeder, A.J.; Goodman, J.L.; Strelets, V.B.; Crosby, M.A.; Thurmond, J.; Emmert, D.B.; Gelbart, W.M.; The FlyBase Consortium (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383921\" target=\"_blank\">\"FlyBase: introduction of the Drosophila melanogaster Release 6 reference genome assembly and large-scale migration of genome annotations\"<\/a>. <i>Nucleic Acids Research<\/i> <b>43<\/b> (Database issue): D690-7. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgku1099\" target=\"_blank\">10.1093\/nar\/gku1099<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4383921\/\" target=\"_blank\">PMC4383921<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25398896\" target=\"_blank\">25398896<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383921\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4383921<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=FlyBase%3A+introduction+of+the+Drosophila+melanogaster+Release+6+reference+genome+assembly+and+large-scale+migration+of+genome+annotations&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Dos+Santos%2C+G.%3B+Schroeder%2C+A.J.%3B+Goodman%2C+J.L.%3B+Strelets%2C+V.B.%3B+Crosby%2C+M.A.%3B+Thurmond%2C+J.%3B+Emmert%2C+D.B.%3B+Gelbart%2C+W.M.%3B+The+FlyBase+Consortium&rft.au=Dos+Santos%2C+G.%3B+Schroeder%2C+A.J.%3B+Goodman%2C+J.L.%3B+Strelets%2C+V.B.%3B+Crosby%2C+M.A.%3B+Thurmond%2C+J.%3B+Emmert%2C+D.B.%3B+Gelbart%2C+W.M.%3B+The+FlyBase+Consortium&rft.date=2015&rft.volume=43&rft.issue=Database+issue&rft.pages=D690-7&rft_id=info:doi\/10.1093%2Fnar%2Fgku1099&rft_id=info:pmc\/PMC4383921&rft_id=info:pmid\/25398896&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4383921&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BirneyGene-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BirneyGene_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Birney, E.; Clamp, M.; Durbin, R. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130\" target=\"_blank\">\"GeneWise and Genomewise\"<\/a>. <i>Genome Research<\/i> <b>14<\/b> (5): 988\u201395. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.1865504\" target=\"_blank\">10.1101\/gr.1865504<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC479130\/\" target=\"_blank\">PMC479130<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15123596\" target=\"_blank\">15123596<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GeneWise+and+Genomewise&rft.jtitle=Genome+Research&rft.aulast=Birney%2C+E.%3B+Clamp%2C+M.%3B+Durbin%2C+R.&rft.au=Birney%2C+E.%3B+Clamp%2C+M.%3B+Durbin%2C+R.&rft.date=2004&rft.volume=14&rft.issue=5&rft.pages=988%E2%80%9395&rft_id=info:doi\/10.1101%2Fgr.1865504&rft_id=info:pmc\/PMC479130&rft_id=info:pmid\/15123596&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC479130&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KorfGene-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KorfGene_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Korf, I. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630\" target=\"_blank\">\"Gene finding in novel genomes\"<\/a>. <i>BMC Bioinformatics<\/i> <b>14<\/b> (5): 59. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-5-59\" target=\"_blank\">10.1186\/1471-2105-5-59<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC421630\/\" target=\"_blank\">PMC421630<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15144565\" target=\"_blank\">15144565<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+finding+in+novel+genomes&rft.jtitle=BMC+Bioinformatics&rft.aulast=Korf%2C+I.&rft.au=Korf%2C+I.&rft.date=2004&rft.volume=14&rft.issue=5&rft.pages=59&rft_id=info:doi\/10.1186%2F1471-2105-5-59&rft_id=info:pmc\/PMC421630&rft_id=info:pmid\/15144565&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC421630&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BirnbaumOnThe-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BirnbaumOnThe_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Birnbaum, A. (1962). \"On the Foundations of Statistical Inference\". <i>Journal of the American Statistical Association<\/i> <b>57<\/b> (298): 269-306. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1080%2F01621459.1962.10480660\" target=\"_blank\">10.1080\/01621459.1962.10480660<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=On+the+Foundations+of+Statistical+Inference&rft.jtitle=Journal+of+the+American+Statistical+Association&rft.aulast=Birnbaum%2C+A.&rft.au=Birnbaum%2C+A.&rft.date=1962&rft.volume=57&rft.issue=298&rft.pages=269-306&rft_id=info:doi\/10.1080%2F01621459.1962.10480660&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EdwardsLike-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-EdwardsLike_26-0\" rel=\"external_link\">26.0<\/a><\/sup> <sup><a href=\"#cite_ref-EdwardsLike_26-1\" rel=\"external_link\">26.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Edwards, A.W.F. (1992). <i>Likelihood<\/i>. Baltimore, MD: Johns Hopkins University Press. pp. 296. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780801844430.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Likelihood&rft.aulast=Edwards%2C+A.W.F.&rft.au=Edwards%2C+A.W.F.&rft.date=1992&rft.pages=pp.%26nbsp%3B296&rft.place=Baltimore%2C+MD&rft.pub=Johns+Hopkins+University+Press&rft.isbn=9780801844430&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoyallStat-27\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-RoyallStat_27-0\" rel=\"external_link\">27.0<\/a><\/sup> <sup><a href=\"#cite_ref-RoyallStat_27-1\" rel=\"external_link\">27.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Royall, R. (1997). <i>Statistical evidence: a likelihood paradigm<\/i>. Boca Raton, FL: Chapman and Hall\/CRC. pp. 191. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780412044113.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Statistical+evidence%3A+a+likelihood+paradigm&rft.aulast=Royall%2C+R.&rft.au=Royall%2C+R.&rft.date=1997&rft.pages=pp.%26nbsp%3B191&rft.place=Boca+Raton%2C+FL&rft.pub=Chapman+and+Hall%2FCRC&rft.isbn=9780412044113&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BarkerSee-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BarkerSee_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Barker, D. (2015). \"Seeing the wood for the trees: philosophical aspects of classical, Bayesian and likelihood approaches in statistical inference and some implications for phylogenetic analysis\". <i>Biology & Philosophy<\/i> <b>30<\/b> (4): 505\u2013525. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs10539-014-9455-x\" target=\"_blank\">10.1007\/s10539-014-9455-x<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Seeing+the+wood+for+the+trees%3A+philosophical+aspects+of+classical%2C+Bayesian+and+likelihood+approaches+in+statistical+inference+and+some+implications+for+phylogenetic+analysis&rft.jtitle=Biology+%26+Philosophy&rft.aulast=Barker%2C+D.&rft.au=Barker%2C+D.&rft.date=2015&rft.volume=30&rft.issue=4&rft.pages=505%E2%80%93525&rft_id=info:doi\/10.1007%2Fs10539-014-9455-x&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WilksTheLar-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WilksTheLar_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wilks, S.S. (1938). \"The large-sample distribution of the likelihood ratio for testing composite hypotheses\". <i>Annals of Mathematical Statistics<\/i> <b>9<\/b> (1): 60-62. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1214%2Faoms%2F1177732360\" target=\"_blank\">10.1214\/aoms\/1177732360<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+large-sample+distribution+of+the+likelihood+ratio+for+testing+composite+hypotheses&rft.jtitle=Annals+of+Mathematical+Statistics&rft.aulast=Wilks%2C+S.S.&rft.au=Wilks%2C+S.S.&rft.date=1938&rft.volume=9&rft.issue=1&rft.pages=60-62&rft_id=info:doi\/10.1214%2Faoms%2F1177732360&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RTheR-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RTheR_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">R Development Core Team (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.r-project.org\/\" target=\"_blank\">\"R: A language and environment for statistical computing\"<\/a>. The R Foundation for Statistical Computing<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.r-project.org\/\" target=\"_blank\">https:\/\/www.r-project.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 July 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=R%3A+A+language+and+environment+for+statistical+computing&rft.atitle=&rft.aulast=R+Development+Core+Team&rft.au=R+Development+Core+Team&rft.date=2015&rft.pub=The+R+Foundation+for+Statistical+Computing&rft_id=https%3A%2F%2Fwww.r-project.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScottHigher2-31\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ScottHigher2_31-0\" rel=\"external_link\">31.0<\/a><\/sup> <sup><a href=\"#cite_ref-ScottHigher2_31-1\" rel=\"external_link\">31.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Scottish Qualifications Authority (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology\" target=\"_blank\">\"Higher Biology Course Support Notes\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology\" target=\"_blank\">http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 October 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Higher+Biology+Course+Support+Notes&rft.atitle=&rft.aulast=Scottish+Qualifications+Authority&rft.au=Scottish+Qualifications+Authority&rft.date=2014&rft_id=http%3A%2F%2Fwww.sqa.org.uk%2Fsqa%2F48674.html%3Fsubject%3DBiology&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScottHigher3-32\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ScottHigher3_32-0\" rel=\"external_link\">32.0<\/a><\/sup> <sup><a href=\"#cite_ref-ScottHigher3_32-1\" rel=\"external_link\">32.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\">Scottish Qualifications Authority (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Human%20Biology\" target=\"_blank\">\"Higher Human Biology Course Support Notes\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Human%20Biology\" target=\"_blank\">http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Human%20Biology<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 October 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Higher+Human+Biology+Course+Support+Notes&rft.atitle=&rft.aulast=Scottish+Qualifications+Authority&rft.au=Scottish+Qualifications+Authority&rft.date=2014&rft_id=http%3A%2F%2Fwww.sqa.org.uk%2Fsqa%2F48674.html%3Fsubject%3DHuman%2520Biology&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScottHigher4-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScottHigher4_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Scottish Qualifications Authority (2015). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology\" target=\"_blank\">\"Advanced Higher Biology Course\/Unit Support Notes\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology\" target=\"_blank\">http:\/\/www.sqa.org.uk\/sqa\/48674.html?subject=Biology<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 20 October 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Advanced+Higher+Biology+Course%2FUnit+Support+Notes&rft.atitle=&rft.aulast=Scottish+Qualifications+Authority&rft.au=Scottish+Qualifications+Authority&rft.date=2015&rft_id=http%3A%2F%2Fwww.sqa.org.uk%2Fsqa%2F48674.html%3Fsubject%3DBiology&rfr_id=info:sid\/en.wikipedia.org:Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In several cases citation information was missing and was added to make the reference more useful.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210223\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.817 seconds\nReal time usage: 0.846 seconds\nPreprocessor visited node count: 27053\/1000000\nPreprocessor generated node count: 38158\/1000000\nPost\u2010expand include size: 243248\/2097152 bytes\nTemplate argument size: 77656\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 811.272 1 - -total\n 87.09% 706.513 1 - Template:Reflist\n 76.39% 619.708 33 - Template:Citation\/core\n 65.37% 530.341 25 - Template:Cite_journal\n 10.91% 88.470 6 - Template:Cite_web\n 10.05% 81.523 63 - Template:Citation\/identifier\n 7.56% 61.336 1 - Template:Infobox_journal_article\n 7.25% 58.845 1 - Template:Infobox\n 4.62% 37.478 33 - Template:Citation\/make_link\n 4.56% 36.960 2 - Template:Cite_book\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7837-0!*!0!!en!5!* and timestamp 20181213210222 and revision id 23438\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school\">https:\/\/www.limswiki.org\/index.php\/Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","e85e2a721257f9447a4d6f2238effb07_images":["https:\/\/www.limswiki.org\/images\/e\/ed\/Fig1_Barker_IntJourSTEMEd2015_2.jpg"],"e85e2a721257f9447a4d6f2238effb07_timestamp":1544734942,"8d2093f9b49a2cde0a0f2b14e9a154a1_type":"article","8d2093f9b49a2cde0a0f2b14e9a154a1_title":"SeqWare Query Engine: Storing and searching sequence data in the cloud (O\u2019Connor et al. 2010)","8d2093f9b49a2cde0a0f2b14e9a154a1_url":"https:\/\/www.limswiki.org\/index.php\/Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud","8d2093f9b49a2cde0a0f2b14e9a154a1_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:SeqWare Query Engine: Storing and searching sequence data in the cloud\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nSeqWare Query Engine: Storing and searching sequence data in the cloudJournal\n \nBMC BioinformaticsAuthor(s)\n \nO\u2019Connor, Brian D.; Merriman, Barry; Nelson, Stanley F.Author affiliation(s)\n \nUniversity of North Carolina; University of California - Los AngelesPrimary contact\n \nEmail: snelson@ucla.eduYear published\n \n2010Volume and issue\n \n11(12)Page(s)\n \nS2DOI\n \n10.1186\/1471-2105-11-S12-S2ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 2.0 GenericWebsite\n \nhttp:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/1471-2105-11-S12-S2Download\n \nhttp:\/\/bmcbioinformatics.biomedcentral.com\/track\/pdf\/10.1186\/1471-2105-11-S12-S2 (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Methods \n\n3.1 Design approach \n3.2 Datasets \n3.3 Programmatic access \n3.4 Web service access \n3.5 Data load tools \n3.6 Analysis tools \n3.7 Performance measurement \n\n\n4 Results \n\n4.1 U87MG genome database \n4.2 Performance \n4.3 Software availability \n\n\n5 Discussion \n6 Conclusions \n7 List of abbreviations \n8 Declarations \n\n8.1 Acknowledgments \n8.2 Authors\u2019 contributions \n8.3 Competing interests \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nBackground: Since the introduction of next-generation DNA sequencers the rapid increase in sequencer throughput, and associated drop in costs, has resulted in more than a dozen human genomes being resequenced over the last few years. These efforts are merely a prelude for a future in which genome resequencing will be commonplace for both biomedical research and clinical applications. The dramatic increase in sequencer output strains all facets of computational infrastructure, especially databases and query interfaces. The advent of cloud computing, and a variety of powerful tools designed to process petascale datasets, provide a compelling solution to these ever increasing demands.\nResults:' In this work, we present the SeqWare Query Engine which has been created using modern cloud computing technologies and designed to support databasing information from thousands of genomes. Our backend implementation was built using the highly scalable, NoSQL HBase database from the Hadoop project. We also created a web-based frontend that provides both a programmatic and interactive query interface and integrates with widely used genome browsers and tools. Using the query engine, users can load and query variants (SNVs, indels, translocations, etc) with a rich level of annotations including coverage and functional consequences. As a proof of concept we loaded several whole genome datasets including the U87MG cell line. We also used a glioblastoma multiforme tumor\/normal pair to both profile performance and provide an example of using the Hadoop MapReduce framework within the query engine. This software is open source and freely available from the SeqWare project (https:\/\/github.com\/SeqWare).\nConclusions: The SeqWare Query Engine provided an easy way to make the U87MG genome accessible to programmers and non-programmers alike. This enabled a faster and more open exploration of results, quicker tuning of parameters for heuristic variant calling filters, and a common data interface to simplify development of analytical tools. The range of data types supported, the ease of querying and integrating with existing tools, and the robust scalability of the underlying cloud-based technologies make SeqWare Query Engine a nature fit for storing and searching ever-growing genome sequence datasets.\n\nBackground \nRecent advances in sequencing technologies have led to a greatly reduced cost and increased throughput.[1] The dramatic reductions in both time and financial costs have shaped the experiments scientists are able to perform and have opened up the possibility of whole human genome resequencing becoming commonplace. Currently over a dozen human genomes have been completed, most using one of the short read, high-throughput technologies that are responsible for this growth in sequencing.[2][3][4][5][6][7][8][9][10][11][12][13][14][15][16] The datatypes produced by these projects are varied, but most report single nucleotide variants (SNVs), small insertions\/deletions (indels, typically <10 bases), structural variants (SVs), and may include additional information such as haplotype phasing and novel sequence assemblies. Paired tumor\/normal samples can additionally be used to identify somatic mutation events by filtering for those variants present in the tumor but not the normal.\nFull genome sequencing, while increasingly common, is just one of many experimental designs that are currently used with this generation of sequencing platforms. Targeted resequencing, whole-exome sequencing, RNA sequencing (RNA-Seq), Chromatin Immunoprecipitation sequencing (ChIP-Seq), and bisulfite sequencing for methylation detection are examples of other important analysis types that require large scale databasing capabilities. Efforts such as the 1000 Genomes project (http:\/\/www.1000genomes.org), the Cancer Genome Atlas (TCGA, http:\/\/cancergenome.nih.gov), and the International Cancer Genome Consortium (http:\/\/www.icgc.org) are each generating a wide variety of such data across hundreds to thousands of samples. The diversity and number of sequencing datasets already produced, in production, or being planned present huge infrastructure challenges for the research community.\nPrimary data, if available, are typically huge, difficult to transfer over public networks, and cumbersome to analyze without significant local computational infrastructure. These include large compute clusters, extensive data storage facilities, dedicated system administrators, and bioinformaticians adept at low-level programming. Highly annotated datasets, such as finished variant calls, are more commonly available, particularly for human datasets. These present a more compact representation of the most salient information, but are typically only available as flat text files in a variety of quasi-standard file formats that require reformatting and processing. This effort is substantial, particularly as the number of datasets grow, and, as a result, is typically undertaken by a small number of researchers that have a personal stake in the data rather than being more widely and easily accessible. In many cases, essential source information has been eliminated for the sake of data reduction, making recalculation impossible. These challenges, in terms of file sizes, diverse formats, limited data retention, and computational requirements, can make writing generic analysis tools complex and difficult. Efforts such as the Variant Call Format (VCF) from the 1000 Genomes Project provide a standard to exchange variant data. But to facilitate the integration of multiple experimental types and increase tool reuse, a common mechanism to both store and query variant calls and other key information from sequencing experiments is highly desirable. Properly databasing this information enables both a common underlying data structure and a search interface to support powerful data mining of sequence-derived information.\nTo date most biological database projects have focused on the storage of heavily annotated model organism reference sequences. For example, efforts such as the UCSC genome databases[17], the Generic Model Organism Database\u2019s Chado schema[18], and the Ensembl database[19] all solve the problem of storing reference genome annotations in a complete and comprehensive way. The focus for these databases is the proper representation of biological data types and genome annotations, but not storing many thousands of genomes worth of variants relative to a given reference. While many biological database schemas currently in wide use could support tens or even hundreds of genomes worth of variant calls, ultimately these systems are limited by the resources of a single database instance. Since they focus on relatively modest amounts of annotation storage, loading hundreds of genomes worth of multi-terabyte sequencing coverage information, for example, would likely overwhelm these traditional database approaches. Yet the appeal of databasing next generation sequence data is clear since it would simplify tool development and allow for useful queries across samples and projects.\nIn this work we introduce the SeqWare Query Engine, a scalable database system intended to represent the full range of data types common to whole genome and other experimental designs for next generation sequence data. HBase was chosen as the underlying backend because of its robust querying abilities using the Hadoop MapReduce environment and its auto-sharding of data across a commodity cluster based on the Hadoop HDFS distributed filesystem (http:\/\/hadoop.apache.org). We also present a web service that wraps the use of MapReduce to allow for sophisticated queries of the database through a simple web interface. The web service can be used interactively or programmatically and makes it possible to easily integrate with genome browsers, such as the UCSC Browser[20], GBrowse[21], or IGV (http:\/\/www.broadinstitute.org\/\u200bigv), and with data analysis tools, such as the UCSC table browser[22], GALAXY[23], and others. The backend and web service can be used together to create databases containing varying levels of annotations, from raw variant calls and coverage to highly annotated and filtered SNV predictions. This flexibility allows the SeqWare Query Engine to scale from raw data analysis and algorithm tuning through highly annotated data dissemination and hosting. The design decision to move away from traditional relational databases in favor of the NoSQL-style of limited, but highly scalable, databases allowed us to support tens of genomes now and thousands of genomes in the future, limited only by the underlying cloud resources.\n\nMethods \nDesign approach \nThe HBase backend for SeqWare Query Engine is based on the increasingly popular design paradigm that focuses on scalability at the expense of full ACID compliance, relational database schemas, and the Structured Query Language (SQL, as reflected in the name \u201cNoSQL\u201d). The result is that, while scalable to thousands of compute nodes, the overall operations permitted on the database are limited. Each records consists of a key and the value, which consists of one or more \u201ccolumn families\u201d that are fixed at table creation time. Each column family can have many \u201clabels\u201d which can be added at any time, and each of these labels can have one or more \u201ctimestamps\u201d (versions). For the query engine database, the genomic start position of each feature was used as the key while four column families served to represent the core data types: variants (SNVs, indels, SVs, and translocations), coverage, features (any location-based annotations on the genome), and coding consequences which link back to the variants entries they report on. The coverage object stores individual base coverages in a hash and covers a user-defined range of bases to minimize storage requirements for this data type. New column families can be added to the database to support new data types beyond those described here. Additional column family labels are created as new genomes are loaded into the database, and timestamps are used to distinguish variants in the same genome at identical locations. Figure 1 shows an example row with two genomes\u2019 data loaded. This design was chosen because it meant identical variants in different genomes are stored within the same row, making comparisons between genomes extremely fast using MapReduce or similar simple, uniform operators (Figure 1a). The diagram also shows how secondary indexes are handled in the HBase backend (Figure 1b). Tags are a convenient mechanism to associate arbitrary key-value pairs with any variant object and support lookup for the object using the key (tag). When variants or other data types are written to the database, the persistence code identifies tags and adds them to a second table where the key is the tag plus variant ID and the value is the reference genomic table and location. This enables variants with certain tags to be identified without having to walk the entire contents of the main table.\n\n\n\n\n\n\n\n\n\n Figure 1. SeqWare Query Engine schema. The HBase database is a generic key-value, column oriented database that pairs well with the inherent sparse matrix nature of variant annotations. (a) The primary table stores multiple genomes worth of generic features, variants, coverages, and variant consequences using genomic location within a particular reference genome as the key. Each genome is represented by a particular column family label (such as \u201cvariant:genome7\u201d). For locations with more than one called variant the HBase timestamp is used to distinguish each. (b) Secondary indexing is accomplished using a secondary table per genome indexed. The key is the tag being indexed plus the ID of the object of interest, the value is the row key for the original table. This makes lookup by secondary indexes, \u201ctags\u201d for example, possible without having to iterate over all contents of the primary table.\n\n\n\nDatasets \nFourteen human genome datasets were chosen for loading into a common SeqWare Query Engine backend, see Table 1 for the complete list. Most datasets included just SNV, small indel, and a limited number of SV predictions. The U87MG human cancer cell line genome was used to test the load of large-scale, raw variant analysis data types. For this genome, SNVs, small indels, large deletions, translocation events, and base-by-base coverage were all loaded. For the SNV and small indels, any variant observed once or more in the underlying short read data were loaded, which resulted in large numbers of spurious variants (i.e. sequencing errors) being loaded in the database. This was done purposefully for two reasons: for this study, to facilitate stress testing the HBase backend, and for the U87MG sequencing project, to facilitate analysis algorithm development by giving practical access to the greatest potential universe of candidate variants. In particular, the fast querying abilities of the SeqWare Query Engine enabled rapid heuristic tuning of the variant calling pipeline parameters through many cycles of filtering and subsequent assessment.\n\n\n\n\n\n\n\nTable 1. Datasets\n\n\nDataset\n\nTechnology\n\nSNVs & Indels\n\nSV\n\nTranslocations\n\nReference\n\n\nEuropean-Venter\n\nSanger\n\nY\n\nY\n\nN\n\nLevy et al. 2007[3]\n\n\nEuropean-Watson\n\n454\n\nY\n\nY\n\nN\n\nWheeler et al. 2008[4]\n\n\nEuropean-Quake\n\nHelicos\n\nY\n\nY\n\nN\n\nPushkarev et al. 2009[5]\n\n\nAsian\n\nIllumina\n\nY\n\nY\n\nN\n\nWang et al. 2008[6]\n\n\nYoruban 18507\n\nIllumina\n\nY\n\nY\n\nN\n\nBentley et al. 2008[7]\n\n\nYoruban 18507\n\nSOLiD\n\nY\n\nY\n\nN\n\nMcKernan et al. 2009[8]\n\n\nKorean\n\nIllumina\n\nY\n\nY\n\nN\n\nAhn et al. 2009[9]\n\n\nKorean-AKI\n\nIllumina\n\nY\n\nY\n\nN\n\nKim et al. 2009[10]\n\n\n3 human genomes\n\nComplete Genomics\n\nY\n\nY\n\nN\n\nDrmanac et al. 2009[11]\n\n\nAML T\/N\n\nIllumina\n\nY\n\nY\n\nN\n\nLey et al. 2008[12]\n\n\nAML genome\n\nIllumina\n\nY\n\nY\n\nN\n\nMardis et al. 2009[13]\n\n\nMelanoma\n\nIllumina\n\nY\n\nY\n\nN\n\nPleasance et al. 2010[14]\n\n\nLung cancer\n\nSOLiD\n\nY\n\nY\n\nN\n\nPleasance et al. 2010[15]\n\n\nU87MG\n\nSOLiD\n\nY\n\nY\n\nY\n\nClark et al. 2010[16]\n\n\nFourteen whole genome datasets were loaded into the database, including the U87MG genome, with the March 2006 assembly of the human genome used as reference (NCBI36\/hg18). Variant types (SNVs, small\/large indels, SVs, etc) loaded and publication references are noted for each respective dataset. This table was adapted from Snyder et al. 2010.\n\n\n\nA secondary dataset generated in our lab, the \u201c1102 GBM\u201d tumor\/normal whole genome sequence pair, was used to compare the performance between the BerkeleyDB and HBase Query Engine backend types. This dataset, like the U87MG genome, included loading raw variant calls seen once or more in both backends in order to profile the load and query mechanisms.\n\nProgrammatic access \nThe SeqWare Query Engine provides a common database store interface that supports both the BerkleyDB and HBase backend types. This store object provides generic methods to read and write the full range of data types into and out of the underlying database. It handles the persistence and retrieval of keys and objects to and from the database using a flexible object mapping layer. Simple to write bindings are created when new data types are added to the database. The underlying schema-less nature of key\/value stores like BerkeleyDB and HBase make this process very straightforward. The store also supports queries that lookup all variants and filter by specific fields, such as coverage or variant call p-value, and it can also query based on secondary indexes, typically a tag lookup (key-value pair). The underlying implementation for each store type (BerkeleyDB or HBase) is quite different but the generic store interface masks the difference from the various import and export tools available in the project. The store interface was used whenever possible in order to maximize the portability of the query engine and to make it possible to switch backends in the future.\nTwo lower-level APIs are available for querying the HBase database directly. The first is the HBase API, which the store object uses for most of its operations including filtering variants by tags. This API is very similar to other database interfaces and lets the calling client iterate and filter result sets. HBase also support the use of a MapReduce source and sink, which allow for database traversal and load respectively. This was used to iterate over all variants in the database as quickly as possible and to perform basic analysis tasks such as variant type counting. The speed and flexibility of the MapReduce interface to HBase make it an attractive mechanism to implement future functionality.\n\nWeb service access \nThe web service is built on top of the programmatic, generic store object and uses the Restlet Java API (http:\/\/www.restlet.org). This provides a RESTful[24] web service interface for the query methods available through the store. When loaded from a web browser, the web service uses XSLT and CSS to display a navigatable web interface with user-friendly web forms for searching the database. Queries on three data types are supported: variants, coding consequence reports, and per-base coverage. Variants can be searched by tag and also a wide variety of fields such as their depth of coverage. The tag field is used to store a variety of annotations and, in the U87MG database, this includes dbSNP status, mutational consequence predictions, and names of overlapping genes, among others. For the variant reports, the standard BED file type (http:\/\/genome.ucsc.edu\/\u200bFAQ\/\u200bFAQformat) is supported and users can alternatively select to load the query result directly in the UCSC browser, the IGV browser, or generate a list of non-redundant tags associated with the variant query results. Coverage information can be queried only by location, and the result can be generated in WIG format (http:\/\/genome.ucsc.edu\/\u200bFAQ\/\u200bFAQformat), WIG with coverage averaged by each block, or loadable links for the UCSC and IGV browsers.\nWhen queried programmatically, the web service returns XML result documents. These contain enough metadata to construct URLs accessible from a wide variety of programming languages which can then be used to return query results in standardized formats (BED, WIG, etc). Since every query is just a URL, they can be created from within a web browser using the user-friendly form fields and cut-and-pasted into another tool or script. These URLs can then be shared over email, linked to in a publication, and bookmarked for later use, thereby providing a convenient, stateless, and universally interpretable reference to the results.\n\nData load tools \nMost of the genome datasets used in this project where limited to SNV, indel, and SV predictions. For those genomes, the variant information files that were available were loaded into the query engine using either standard file type loaders (GFF, key-value files, etc) or using custom annotation file parsers. The pileup file format (produced by SAMtools[25]) was used to load both the U87MG and \u201c1102 GBM\u201d tumor\/normal genome variants and coverage information. The variant loading tool supports a plugin interface so new annotation file types can be easily supported. This import tool is available in the query engine package and uses the generic store interface for loading information in database backend (see \u201cProgrammatic Access\u201d for more information). The HBase store currently uses an API very similar to other database connection APIs, and is therefore not inherently parallel, although multiple loads can occur simultaneously.\n\nAnalysis tools \nTwo prototype analysis tools were created for use with the U87MG and \u201c1102 GBM\u201d tumor\/normal databases. First, a MapReduce variant query tool was written to directly compare the performance of the retrieval of variants from HBase using the API versus using MapReduce. This simple tool used the HBase TableInputFormat object and a MapReduce job to traverse all database rows. The second analysis tool created was a simple somatic mutation detector for use with the \u201c1102 GBM\u201d tumor\/normal genome database. Again, the HBase TableInputFormat object was used to iterate over the variants from the tumor genome, evaluating each variant by user-specified quality criteria in the map step and identifying those that were present in the cancer but not in the normal. In the reduce phase the coverage at each putative somatic mutation location was checked and only those where the coverage was good and no normal sample variant was called where reported as putative somatic mutations.\n\nPerformance measurement \nBackend performance was measured for both the BerkeleyDB and HBase stores using both data import and export times as metrics. The \u201c1102 GBM\u201d tumor\/normal genome was used in this testing, which included enough variant calls (both true and spurious) to stress test both backends. A single threaded, API-based approach was taken for the load test with both the BerkeleyDB and HBase backends. Each chromosome was loaded in turn and the time taken to import these variants into the database was recorded. A similar approach was taken for the retrieval test except in this case the BerkeleyDB backend continued to use an API approach whereas the HBase backend used both an API and MapReduced approach. The export test was interleaved with the import test, wherein after a chromosome was loaded the variants where exported and both processes were timed. In that way we monitored both the import and export time as a function of overall database size.\nThe HBase tests were conducted on a 6 node HBase cluster where each node contained 8 2.4GHz Xeon CPUs, 24GB of RAM, and 6TB of hard drive space. Each node contained 16 map task slots and 4 reducer task slots. The BerkeleyDB tests were conducted on a single node, since it does not support server clustering, but with hardware identical to that used in the HBase tests.\n\nResults \nU87MG genome database \nFor our original U87MG human genome cell line sequencing project, we created the SeqWare Query Engine database, first built on BerkeleyDB (http:\/\/www.oracle.com\/\u200btechnetwork\/\u200bdatabase\/\u200bberkeleydb) and then later ported to HBase (http:\/\/hbase.apache.org). For the work presented here, the U87MG database was enhanced with the addition of the 13 other human genome datasets that were publicly available when this effort commenced. To further enhance the query engine, we also added new query strategies and utilities, such as a MapReduce-based variant search tool. Unlike the U87MG genome, which included all variant calls regardless of quality, the other genomes included only post quality filtered variants. Still, they offer a proof of concept that the HBase backend can represent multiple genomes worth of sequence variants and associated annotation data. This sample query engine is hosted at http:\/\/genome.ucla.edu\/\u200bU87 and can be used for both programmatic and interactive queries through the web service interface. A database snapshot is not available for download (due to its large size) but all the source datasets are publicly available and the database can be reconstituted in another location using either the BerkeleyDB or HBase backends along with the provided query engine load tools.\n\nPerformance \nFigure 2 shows a comparison between the BerkeleyDB and HBase backends for both variant load and variant export functions. BerkeleyDB was chosen as the original backend for multiple reasons: it did not require a database daemon to run, it provided a key-value store similar to the distributed key-value NoSQL stores we intended to move towards, it had a well-designed API, and it was known to be widely used and robust. In the load tests both BerkeleyDB and HBase performed comparably (Figure 2a), with both tests using a single thread with a similar client API (rather than a MapReduce loader which would be possible only with HBase). BerkeleyDB is slightly faster until about 6 million (6M) variants are loaded but HBase is faster after that point and eventually takes about 15 minutes less to load all 7M variants in this test. The test for variant export had a significantly different outcome (Figure 2b). Both MapReduce and standard single-thread API retrieval of variants from HBase were extremely efficient, with the MapReduce exporter completing in 45 seconds and the single-thread in 386 seconds. This is in sharp contrast with the BerkeleyDB backend which took 6,281 seconds to export the 7M variant records. When the BerkeleyDB database reached approximately 3.5M variants the run time to export ballooned quickly. This was likely due to memory limitations on the single node serving the BerkeleyDB, resulted in greatly degraded query performance once the index could no longer fit in memory. In contrast, the HBase cluster nodes were each responsible for storing and querying only a fraction of the data and this data sharding resulting in much more robust query performance. Rather than making a statement about the inherent merits of the database systems themselves, this results underscores the point that the distributed HBase\/Hadoop implementation (in this case spread across 6 nodes) has clear scalability advantages compared to daemons or processes like BerkeleyDB that are limited to a single server.\n\n\n\n\n\n\n\n\n\n Figure 2. Load and query performance. Comparisons of load and query times between the HBase and BerkeleyDB backend. (a) Load times for the \u201c1102 GBM\u201d tumor\/normal genomes where compared between HBase and BerkeleyDB. Both used a single-threaded approach to better compare relative performance. Both perform similarly but over time the load times for BerkeleyDB increase faster than with HBase. (b) Comparison of querying the 1102 genome database between BerkeleyDB, HBase single threaded, and HBase using MapReduce. Beyond 3M variants BerkeleyDB query times increase dramatically while both query types for HBase perform linearly, with MapReduce consistently exhibiting the best performance.\n\n\n\nSoftware availability \nThe SeqWare Query Engine is a sub-project of the SeqWare toolset for next generation sequence analysis. SeqWare includes a LIMS, pipeline based on Pegasus[26], and metadata schema in addition to the query engine. Like the rest of the SeqWare project, SeqWare Query Engine is fully open source and is distributed under the terms of the GNU General Public License v3 (http:\/\/www.gnu.org\/\u200blicenses\/\u200blicenses.\u200bhtml). The software can be downloaded from version control on the project\u2019s SourceForge.net site (http:\/\/seqware.sourceforge.net). Visitors to the site can also post questions on the mailing list, view documentation on wiki pages, and download a pre-configured SeqWare Pipeline and Query Engine as a virtual machine, suitable for running in a wide array of environments. The present authors are very interested in working with other developers on this project and welcome any contributions.\n\nDiscussion \nRecent innovations from search-oriented companies such as Google, Yahoo, Facebook, and LinkedIn provide compelling technologies that could potentially enable computation on petascale sequence data. Projects such as Hadoop and HBase, open source implementations of Google\u2019s MapReduce framework[27] and BigTable database[28] respectively, can be converted to powerful frameworks for analyzing next generation sequencing data. For example, MapReduce is based on a functional programming style where the basic methods available to perform analysis are a map phase, where data is transformed from one form to another, and a reduce phase, where data is sorted and condensed. This fits well to fundamental sequence analysis computations such as alignment and variant calling. Already there are versions of analysis algorithms such as Crossbow (alignment[29]) and GATK (variant calling and other tools[30]) that make use of the MapReduce paradigm. The approach is not fundamentally different from other functional programming languages that have come before, but in this case the approach is combined in Hadoop with both a distributed filesystem (HDFS) and tightly coupled execution engine. These tools, in turn, form the basis of the HBase database system. Unlike traditional grid technologies\u2014for example a sun grid engine computation cluster\u2014the Hadoop environment automatically partitions large data files across the underlying cluster, and computations can then be distributed across the individual data pieces without requiring the analysis program to know where the data resides, or manage such information. The HBase database system\u2014one of the most mature of the NoSQL database projects\u2014takes advantage of this sharding feature and allows a database to be broken into distinct pieces across the underlying computer cluster. This enables the creation of much larger databases than can be supported in traditional relational implementations which are constrained to run on a single database server, up to the scale of billions of rows (e.g. bases in a genome) and millions of columns (e.g. individual genomes). This is considerably larger than most database systems typically support, but is the correct scale needed to represent heavily annotated whole genome sequence data for future large scale biomedical research studies and clinical deployment to the broadest patient populations.\n\nConclusions \nHere we have introduced the open source SeqWare Query Engine that uses an HBase backend and the MapReduce\/Hadoop infrastructure to provide robust databasing of sequence data types for the SeqWare project. The results show the scaling benefits that result from these highly distributable technologies even when creating a database of genomic variants for just 14 human genome datasets. The basic database functions, such as importing and exporting, are one to two orders of magnitude faster with HBase compared to BerkelyDB. Moreover, the highly nonlinear improvement in scaling is readily demonstrated at the critical point where the standard database server becomes saturated, whereas the HBase server maintains a proper distributed load as the data burden is increased. This fully cloud-oriented database framework is ideal for the creation of whole genome sequence\/variant databases. It is capable of supporting large scale genome sequencing research projects involving hundreds to thousands of genomes as well as future large scale clinical deployments utilizing advanced sequencer technology that will soon involve tens to hundreds of thousands of genomes.\n\nList of abbreviations \nACID: atomicity, consistency, isolation, and durability\nAPI: Application Programming Interface\nBED: Browser Extensible Data (encodes variants)\nChIP-Seq: Chromatin Immunoprecipitation sequencing\nCSS: Cascading Style Sheets\nGATK: Genome Analysis Toolkit\nGBM: Glioblastoma Multiforme\nGFF: General Feature Format (encodes genomic features)\nHDFS: Hadoop Distributed File System\nIGV: Integrative Genomics Viewer\nindel: small (typically <10bp) insertion or deletion\nREST: Representational State Transfer\nRNA-Seq: RNA sequencing\nSNP: Single Nucleotide Polymorphism\nSNV: Single Nucleotide Variant\nSQL: Structured Query Language\nSV: Structural Variants\nTCGA: the Cancer Genome Atlas\nURL: Uniform Resource Locator\nVCF: Variant Call Format\nXML: Extensible Markup Language\nXSLT: XSL Transformations\n\nDeclarations \nAcknowledgments \nThis work was supported by grants from the NINDS (U24NS), the Dani Saleh Brain Tumor Fund, and the Henry Singleton Brain Tumor Fund. The authors would like to acknowledge Jordan Mendler for his contributions to the SeqWare Pipeline project and Hane Lee for her feedback on the SeqWare Query Engine tools.\nThis article has been published as part of BMC Bioinformatics Volume 11 Supplement 12, 2010: Proceedings of the 11th Annual Bioinformatics Open Source Conference (BOSC) 2010. The full contents of the supplement are available online at http:\/\/www.biomedcentral.com\/\u200b1471-2105\/\u200b11?\u200bissue=\u200bS12.\n\nAuthors\u2019 contributions \nBDO designed and implemented the SeqWare Query Engine. BM provided guidance on project goals, applications, selection of annotation databases, and choice of analysis algorithms used. SFN is the principal investigator for the U87MG genome sequencing project which supported the development of the SeqWare project including the query engine.\n\nCompeting interests \nThe authors have declared no competing interests.\n\nReferences \n\n\n\u2191 Snyder, M.; Du, J.; Gerstein, M. (2010). \"Personal genome sequencing: Current approaches and challenges\". Genes & Development 24 (5): 423\u2013431. doi:10.1101\/gad.1864110. PMC PMC2827837. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827837 .   \n\n\u2191 Lander, E.; Linton, L.; Birren, B. et al. (2001). \"Initial sequencing and analysis of the human genome\". Nature 409 (6822): 860-921. doi:10.1038\/35057062. PMID 11237011.   \n\n\u2191 3.0 3.1 Levy, S.; Sutton, G.; Ng, P. et al. (2007). \"The diploid genome sequence of an individual human\". PLOS Biology 5 (10): e254. doi:10.1371\/journal.pbio.0050254. PMC PMC1964779. PMID 17803354. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1964779 .   \n\n\u2191 4.0 4.1 Wheeler, D.A.; Srinivasan, M.; Egholm, M. et al. (2008). \"The complete genome of an individual by massively parallel DNA sequencing\". Nature 452 (7189): 872\u2013876. doi:10.1038\/nature06884. PMID 18421352.   \n\n\u2191 5.0 5.1 Pushkarev, D.; Neff, N.; Quake, S. (2009). \"Single-molecule sequencing of an individual human genome\". Nature Biotechnology 27 (9): 847-50. doi:10.1038\/nbt.1561. PMC PMC4117198. PMID 19668243. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4117198 .   \n\n\u2191 6.0 6.1 Wang, J.; Wang, W.; Li, R. et al. (2008). \"The diploid genome sequence of an Asian individual\". Nature 456 (7218): 60-5. doi:10.1038\/nature07484. PMC PMC2716080. PMID 18987735. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2716080 .   \n\n\u2191 7.0 7.1 Bentley, D.; Balasubramanian, S.; Swerdlow, H. et al. (2008). \"Accurate whole human genome sequencing using reversible terminator chemistry\". Nature 456 (7218): 53-9. doi:10.1038\/nature07517. PMC PMC2581791. PMID 18987734. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2581791 .   \n\n\u2191 8.0 8.1 McKernan, K.; Peckham, H.; Costa, G. et al. (2009). \"Sequence and structural variation in a human genome uncovered by short-read, massively parallel ligation sequencing using two-base encoding\". Genome Research 19 (9): 1527-41. doi:10.1101\/gr.091868.109. PMC PMC2752135. PMID 19546169. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752135 .   \n\n\u2191 9.0 9.1 Ahn, S.; Kim, T.; Lee, S. et al. (2009). \"The first Korean genome sequence and analysis: Full genome sequencing for a socio-ethnic group\". Genome Research 19 (9): 1622-9. doi:10.1101\/gr.092197.109. PMC PMC2752128. PMID 19470904. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752128 .   \n\n\u2191 10.0 10.1 Kim, J.; Ju, Y.; Park, H. et al. (2009). \"A highly annotated whole-genome sequence of a Korean individual\". Nature 460 (7258): 1011-5. doi:10.1038\/nature08211. PMC PMC2860965. PMID 19587683. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2860965 .   \n\n\u2191 11.0 11.1 Drmanac, R.; Sparks, A.; Callow, M. et al. (2010). \"Human genome sequencing using unchained base reads on self-assembling DNA nanoarrays\". Science 327 (5961): 78-81. doi:10.1126\/science.1181498. PMID 19892942.   \n\n\u2191 12.0 12.1 Ley, T.; Mardis, E.; Ding, L. et al. (2008). \"DNA sequencing of a cytogenetically normal acute myeloid leukaemia genome\". Nature 456 (7218): 66-72. doi:10.1038\/nature07485. PMC PMC2603574. PMID 18987736. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2603574 .   \n\n\u2191 13.0 13.1 Mardis, E.; Ding, L.; Dooling, D. et al. (2009). \"Recurring mutations found by sequencing an acute myeloid leukemia genome\". New England Journal of Medicine 361 (11): 1058-66. doi:10.1056\/NEJMoa0903840. PMC PMC3201812. PMID 19657110. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3201812 .   \n\n\u2191 14.0 14.1 Pleasance, E.; Stephens, P.; O\u2019Meara, S. et al. (2010). \"A small-cell lung cancer genome with complex signatures of tobacco exposure\". Nature 463 (7278): 184-90. doi:10.1038\/nature08629. PMC PMC2880489. PMID 20016488. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2880489 .   \n\n\u2191 15.0 15.1 Pleasance, E.; Cheetham, R.; Stephens, P. et al. (2010). \"A comprehensive catalogue of somatic mutations from a human cancer genome\". Nature 463 (7278): 191-6. doi:10.1038\/nature08658. PMC PMC3145108. PMID 20016485. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145108 .   \n\n\u2191 16.0 16.1 Clark, M.; Homer, N.; O\u2019Connor, B. et al. (2010). \"U87MG decoded: The genomic sequence of a cytogenetically aberrant human cancer cell line\". PLOS Genetics 6 (1): e1000832. doi:10.1371\/journal.pgen.1000832. PMC PMC2813426. PMID 20126413. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813426 .   \n\n\u2191 Rhead, B.; Karolchik, D.; Kuhn, R. et al. (2010). \"The UCSC genome browser database: update 2010\". Nucleic Acids Research 38 (Suppl 1): D613-D619. doi:10.1093\/nar\/gkp939. PMC PMC2808870. PMID 19906737. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2808870 .   \n\n\u2191 Mungall, C.; Emmert, D.; The FlyBase Consortium (2007). \"A Chado case study: an ontology-based modular schema for representing genome-associated biological information\". Bioinformatics 23 (13): i337-i346. doi:10.1093\/bioinformatics\/btm189. PMID 17646315.   \n\n\u2191 Hubbard, T.; Aken, B.; Beal, K. (2006). \"Ensembl 2007\". Nucleic Acids Research 35 (Suppl 1): D610-D617. doi:10.1093\/nar\/gkl996. PMC PMC1761443. PMID 17148474. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1761443 .   \n\n\u2191 Kent, W.; Sugnet, C.; Furey, T. et al. (2002). \"The human genome browser at UCSC\". Genome Research 12 (6): 996-1006. doi:10.1101\/gr.229102. PMC PMC186604. PMID 12045153. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604 .   \n\n\u2191 Stein, L.; Mungall, C.; Shu, S. et al. (2002). \"The generic genome browser: A building block for a model organism system database\". Genome Research 12 (10): 1599-610. doi:10.1101\/gr.403602. PMC PMC187535. PMID 12368253. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187535 .   \n\n\u2191 Karolchik, D.; Hinrichs, A.; Furey, T. et al. (2004). \"The UCSC Table Browser data retrieval tool\". Nucleic Acids Research 32 (Suppl 1): D493-D496. doi:10.1093\/nar\/gkh103. PMC PMC308837. PMID 14681465. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC308837 .   \n\n\u2191 Giardine, B.; Riemer, C.; Hardison, R.C.; Burhans, R.; Elnitski, L.; Shah, P.; Zhang, Y.; Blankenberg, D.; Albert, I.; Taylor, J.; Miller, W.; Kent, W.J.; Nekrutenko, A. (2005). \"Galaxy: A platform for interactive large-scale genome analysis\". Genome Research 15 (10): 1451\u20131455. doi:10.1101\/gr.4086505. PMC PMC1240089. PMID 16169926. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089 .   \n\n\u2191 Fielding, R.T. (2000). \"Architectural Styles and the Design of Network-based Software Architectures\". University of California, Irvine. https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm .   \n\n\u2191 Li, H.; Handsaker, B.; Wysoker, A. et al. (2009). \"The Sequence Alignment\/Map format and SAMtools\". Bioinformatics 25 (16): 2078-9. doi:10.1093\/bioinformatics\/btp352. PMC PMC2723002. PMID 19505943. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002 .   \n\n\u2191 Deelman, E.; Singh, G.; Su, M. et al. (2005). \"Pegasus: A framework for mapping complex scientific workflows onto distributed systems\". Scientific Programming 13 (3): 219\u2013237. doi:10.1155\/2005\/128026. http:\/\/dl.acm.org\/citation.cfm?id=1239653 .   \n\n\u2191 Dean, J.; Ghemawat, S. (2008). \"MapReduce: Simplified data processing on large clusters\". Communications of the ACM 51 (1): 107-113. doi:10.1145\/1327452.1327492. http:\/\/dl.acm.org\/citation.cfm?doid=1327452.1327492 .   \n\n\u2191 Chang, F.; Dean, J.; Ghemawat, S. et al. (2008). \"Bigtable: A distributed storage system for structured data\". ACM Transactions on Computer Systems (TOCS) 26 (2): 4. doi:10.1145\/1365815.1365816. http:\/\/dl.acm.org\/citation.cfm?doid=1365815.1365816 .   \n\n\u2191 Langmead, B.; Schatz, M.; Lin, J.; Pop, M; Salzberg, S. (2009). \"Searching for SNPs with cloud computing\". Genome Biology 10 (11): R134. doi:10.1186\/gb-2009-10-11-r134. PMC PMC3091327. PMID 19930550. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3091327 .   \n\n\u2191 McKenna, A.; Hanna, M.; Banks, E. et al. (2010). \"The Genome Analysis Toolkit: A MapReduce framework for analyzing next-generation DNA sequencing data\". Genome Research 20: 1297-1303. doi:10.1101\/gr.107524.110.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Additionally, URLs to the project page have been updated from the old, deprecated SourceForge location to the current GitHub site.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\">https:\/\/www.limswiki.org\/index.php\/Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:47.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,484 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","8d2093f9b49a2cde0a0f2b14e9a154a1_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_SeqWare_Query_Engine_Storing_and_searching_sequence_data_in_the_cloud skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:SeqWare Query Engine: Storing and searching sequence data in the cloud<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Since the introduction of next-generation DNA sequencers the rapid increase in <a href=\"https:\/\/www.limswiki.org\/index.php\/Sequencing\" title=\"Sequencing\" class=\"mw-disambig wiki-link\" target=\"_blank\" data-key=\"e36167a9eb152ca16a0c4c4e6d13f323\">sequencer<\/a> throughput, and associated drop in costs, has resulted in more than a dozen human <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" target=\"_blank\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomes<\/a> being resequenced over the last few years. These efforts are merely a prelude for a future in which genome resequencing will be commonplace for both biomedical research and clinical applications. The dramatic increase in sequencer output strains all facets of computational infrastructure, especially databases and query interfaces. The advent of cloud computing, and a variety of powerful tools designed to process petascale datasets, provide a compelling solution to these ever increasing demands.\n<\/p><p><b>Results:'<\/b> In this work, we present the <a href=\"https:\/\/www.limswiki.org\/index.php\/SeqWare\" title=\"SeqWare\" target=\"_blank\" class=\"wiki-link\" data-key=\"da4ce5a823f703b673f1ef4b38860d9a\">SeqWare<\/a> Query Engine which has been created using modern cloud computing technologies and designed to support databasing information from thousands of genomes. Our backend implementation was built using the highly scalable, NoSQL HBase database from the Hadoop project. We also created a web-based frontend that provides both a programmatic and interactive query interface and integrates with widely used genome browsers and tools. Using the query engine, users can load and query variants (SNVs, indels, translocations, etc) with a rich level of annotations including coverage and functional consequences. As a proof of concept we loaded several whole genome datasets including the U87MG cell line. We also used a glioblastoma multiforme tumor\/normal pair to both profile performance and provide an example of using the Hadoop MapReduce framework within the query engine. This software is open source and freely available from the SeqWare project (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/SeqWare\" target=\"_blank\">https:\/\/github.com\/SeqWare<\/a>).\n<\/p><p><b>Conclusions:<\/b> The SeqWare Query Engine provided an easy way to make the U87MG genome accessible to programmers and non-programmers alike. This enabled a faster and more open exploration of results, quicker tuning of parameters for heuristic variant calling filters, and a common data interface to simplify development of analytical tools. The range of data types supported, the ease of querying and integrating with existing tools, and the robust scalability of the underlying cloud-based technologies make SeqWare Query Engine a nature fit for storing and searching ever-growing genome sequence datasets.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Recent advances in sequencing technologies have led to a greatly reduced cost and increased throughput.<sup id=\"rdp-ebb-cite_ref-SnyderPers10_1-0\" class=\"reference\"><a href=\"#cite_note-SnyderPers10-1\" rel=\"external_link\">[1]<\/a><\/sup> The dramatic reductions in both time and financial costs have shaped the experiments scientists are able to perform and have opened up the possibility of whole human genome resequencing becoming commonplace. Currently over a dozen human genomes have been completed, most using one of the short read, high-throughput technologies that are responsible for this growth in sequencing.<sup id=\"rdp-ebb-cite_ref-LanderInit01_2-0\" class=\"reference\"><a href=\"#cite_note-LanderInit01-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LevyTheDip07_3-0\" class=\"reference\"><a href=\"#cite_note-LevyTheDip07-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WheelerTheCom08_4-0\" class=\"reference\"><a href=\"#cite_note-WheelerTheCom08-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PushkarevSingle09_5-0\" class=\"reference\"><a href=\"#cite_note-PushkarevSingle09-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WangTheDip08_6-0\" class=\"reference\"><a href=\"#cite_note-WangTheDip08-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BentleyAcc08_7-0\" class=\"reference\"><a href=\"#cite_note-BentleyAcc08-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-McKernanSeq09_8-0\" class=\"reference\"><a href=\"#cite_note-McKernanSeq09-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AhnTheFirst09_9-0\" class=\"reference\"><a href=\"#cite_note-AhnTheFirst09-9\" rel=\"external_link\">[9]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KimAHighly09_10-0\" class=\"reference\"><a href=\"#cite_note-KimAHighly09-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DrmanacHuman10_11-0\" class=\"reference\"><a href=\"#cite_note-DrmanacHuman10-11\" rel=\"external_link\">[11]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LeyDNA08_12-0\" class=\"reference\"><a href=\"#cite_note-LeyDNA08-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MardisRec09_13-0\" class=\"reference\"><a href=\"#cite_note-MardisRec09-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PleasanceASmall10_14-0\" class=\"reference\"><a href=\"#cite_note-PleasanceASmall10-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PleasanceAComp10_15-0\" class=\"reference\"><a href=\"#cite_note-PleasanceAComp10-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ClarkU87MG10_16-0\" class=\"reference\"><a href=\"#cite_note-ClarkU87MG10-16\" rel=\"external_link\">[16]<\/a><\/sup> The datatypes produced by these projects are varied, but most report single nucleotide variants (SNVs), small insertions\/deletions (indels, typically <10 bases), structural variants (SVs), and may include additional information such as haplotype phasing and novel sequence assemblies. Paired tumor\/normal samples can additionally be used to identify somatic mutation events by filtering for those variants present in the tumor but not the normal.\n<\/p><p>Full genome sequencing, while increasingly common, is just one of many experimental designs that are currently used with this generation of sequencing platforms. Targeted resequencing, whole-exome sequencing, RNA sequencing (RNA-Seq), Chromatin Immunoprecipitation sequencing (ChIP-Seq), and bisulfite sequencing for methylation detection are examples of other important analysis types that require large scale databasing capabilities. Efforts such as the 1000 Genomes project (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.1000genomes.org\" target=\"_blank\">http:\/\/www.1000genomes.org<\/a>), the Cancer Genome Atlas (TCGA, <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/cancergenome.nih.gov\" target=\"_blank\">http:\/\/cancergenome.nih.gov<\/a>), and the International Cancer Genome Consortium (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.icgc.org\" target=\"_blank\">http:\/\/www.icgc.org<\/a>) are each generating a wide variety of such data across hundreds to thousands of samples. The diversity and number of sequencing datasets already produced, in production, or being planned present huge infrastructure challenges for the research community.\n<\/p><p>Primary data, if available, are typically huge, difficult to transfer over public networks, and cumbersome to analyze without significant local computational infrastructure. These include large compute clusters, extensive data storage facilities, dedicated system administrators, and bioinformaticians adept at low-level programming. Highly annotated datasets, such as finished variant calls, are more commonly available, particularly for human datasets. These present a more compact representation of the most salient information, but are typically only available as flat text files in a variety of quasi-standard file formats that require reformatting and processing. This effort is substantial, particularly as the number of datasets grow, and, as a result, is typically undertaken by a small number of researchers that have a personal stake in the data rather than being more widely and easily accessible. In many cases, essential source information has been eliminated for the sake of data reduction, making recalculation impossible. These challenges, in terms of file sizes, diverse formats, limited data retention, and computational requirements, can make writing generic analysis tools complex and difficult. Efforts such as the Variant Call Format (VCF) from the 1000 Genomes Project provide a standard to exchange variant data. But to facilitate the integration of multiple experimental types and increase tool reuse, a common mechanism to both store and query variant calls and other key information from sequencing experiments is highly desirable. Properly databasing this information enables both a common underlying data structure and a search interface to support powerful data mining of sequence-derived information.\n<\/p><p>To date most biological database projects have focused on the storage of heavily annotated model organism reference sequences. For example, efforts such as the UCSC genome databases<sup id=\"rdp-ebb-cite_ref-RheadTheUCSC10_17-0\" class=\"reference\"><a href=\"#cite_note-RheadTheUCSC10-17\" rel=\"external_link\">[17]<\/a><\/sup>, the Generic Model Organism Database\u2019s Chado schema<sup id=\"rdp-ebb-cite_ref-MungallAChado07_18-0\" class=\"reference\"><a href=\"#cite_note-MungallAChado07-18\" rel=\"external_link\">[18]<\/a><\/sup>, and the Ensembl database<sup id=\"rdp-ebb-cite_ref-HubbardEnsembl06_19-0\" class=\"reference\"><a href=\"#cite_note-HubbardEnsembl06-19\" rel=\"external_link\">[19]<\/a><\/sup> all solve the problem of storing reference genome annotations in a complete and comprehensive way. The focus for these databases is the proper representation of biological data types and genome annotations, but not storing many thousands of genomes worth of variants relative to a given reference. While many biological database schemas currently in wide use could support tens or even hundreds of genomes worth of variant calls, ultimately these systems are limited by the resources of a single database instance. Since they focus on relatively modest amounts of annotation storage, loading hundreds of genomes worth of multi-terabyte sequencing coverage information, for example, would likely overwhelm these traditional database approaches. Yet the appeal of databasing next generation sequence data is clear since it would simplify tool development and allow for useful queries across samples and projects.\n<\/p><p>In this work we introduce the SeqWare Query Engine, a scalable database system intended to represent the full range of data types common to whole genome and other experimental designs for next generation sequence data. HBase was chosen as the underlying backend because of its robust querying abilities using the Hadoop MapReduce environment and its auto-sharding of data across a commodity cluster based on the Hadoop HDFS distributed filesystem (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hadoop.apache.org\" target=\"_blank\">http:\/\/hadoop.apache.org<\/a>). We also present a web service that wraps the use of MapReduce to allow for sophisticated queries of the database through a simple web interface. The web service can be used interactively or programmatically and makes it possible to easily integrate with genome browsers, such as the UCSC Browser<sup id=\"rdp-ebb-cite_ref-KentTheHuman02_20-0\" class=\"reference\"><a href=\"#cite_note-KentTheHuman02-20\" rel=\"external_link\">[20]<\/a><\/sup>, GBrowse<sup id=\"rdp-ebb-cite_ref-SteinTheGeneric02_21-0\" class=\"reference\"><a href=\"#cite_note-SteinTheGeneric02-21\" rel=\"external_link\">[21]<\/a><\/sup>, or IGV (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.broadinstitute.org\/\u200bigv\" target=\"_blank\">http:\/\/www.broadinstitute.org\/\u200bigv<\/a>), and with data analysis tools, such as the UCSC table browser<sup id=\"rdp-ebb-cite_ref-KarolchikTheUSCS04_22-0\" class=\"reference\"><a href=\"#cite_note-KarolchikTheUSCS04-22\" rel=\"external_link\">[22]<\/a><\/sup>, GALAXY<sup id=\"rdp-ebb-cite_ref-GiardineGalaxy05_23-0\" class=\"reference\"><a href=\"#cite_note-GiardineGalaxy05-23\" rel=\"external_link\">[23]<\/a><\/sup>, and others. The backend and web service can be used together to create databases containing varying levels of annotations, from raw variant calls and coverage to highly annotated and filtered SNV predictions. This flexibility allows the SeqWare Query Engine to scale from raw data analysis and algorithm tuning through highly annotated data dissemination and hosting. The design decision to move away from traditional relational databases in favor of the NoSQL-style of limited, but highly scalable, databases allowed us to support tens of genomes now and thousands of genomes in the future, limited only by the underlying cloud resources.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Design_approach\">Design approach<\/span><\/h3>\n<p>The HBase backend for SeqWare Query Engine is based on the increasingly popular design paradigm that focuses on scalability at the expense of full ACID compliance, relational database schemas, and the Structured Query Language (SQL, as reflected in the name \u201cNoSQL\u201d). The result is that, while scalable to thousands of compute nodes, the overall operations permitted on the database are limited. Each records consists of a key and the value, which consists of one or more \u201ccolumn families\u201d that are fixed at table creation time. Each column family can have many \u201clabels\u201d which can be added at any time, and each of these labels can have one or more \u201ctimestamps\u201d (versions). For the query engine database, the genomic start position of each feature was used as the key while four column families served to represent the core data types: variants (SNVs, indels, SVs, and translocations), coverage, features (any location-based annotations on the genome), and coding consequences which link back to the variants entries they report on. The coverage object stores individual base coverages in a hash and covers a user-defined range of bases to minimize storage requirements for this data type. New column families can be added to the database to support new data types beyond those described here. Additional column family labels are created as new genomes are loaded into the database, and timestamps are used to distinguish variants in the same genome at identical locations. Figure 1 shows an example row with two genomes\u2019 data loaded. This design was chosen because it meant identical variants in different genomes are stored within the same row, making comparisons between genomes extremely fast using MapReduce or similar simple, uniform operators (Figure 1a). The diagram also shows how secondary indexes are handled in the HBase backend (Figure 1b). Tags are a convenient mechanism to associate arbitrary key-value pairs with any variant object and support lookup for the object using the key (tag). When variants or other data types are written to the database, the persistence code identifies tags and adds them to a second table where the key is the tag plus variant ID and the value is the reference genomic table and location. This enables variants with certain tags to be identified without having to walk the entire contents of the main table.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_OConnor_BMCInformatics2010_11-12.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"c5bfd17b7fdff602879ce8f036f4b234\"><img alt=\"Fig1 OConnor BMCInformatics2010 11-12.jpg\" src=\"https:\/\/www.limswiki.org\/images\/0\/0f\/Fig1_OConnor_BMCInformatics2010_11-12.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1. SeqWare Query Engine schema.<\/b> The HBase database is a generic key-value, column oriented database that pairs well with the inherent sparse matrix nature of variant annotations. (a) The primary table stores multiple genomes worth of generic features, variants, coverages, and variant consequences using genomic location within a particular reference genome as the key. Each genome is represented by a particular column family label (such as \u201cvariant:genome7\u201d). For locations with more than one called variant the HBase timestamp is used to distinguish each. (b) Secondary indexing is accomplished using a secondary table per genome indexed. The key is the tag being indexed plus the ID of the object of interest, the value is the row key for the original table. This makes lookup by secondary indexes, \u201ctags\u201d for example, possible without having to iterate over all contents of the primary table.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Datasets\">Datasets<\/span><\/h3>\n<p>Fourteen human genome datasets were chosen for loading into a common SeqWare Query Engine backend, see Table 1 for the complete list. Most datasets included just SNV, small indel, and a limited number of SV predictions. The U87MG human cancer cell line genome was used to test the load of large-scale, raw variant analysis data types. For this genome, SNVs, small indels, large deletions, translocation events, and base-by-base coverage were all loaded. For the SNV and small indels, any variant observed once or more in the underlying short read data were loaded, which resulted in large numbers of spurious variants (i.e. sequencing errors) being loaded in the database. This was done purposefully for two reasons: for this study, to facilitate stress testing the HBase backend, and for the U87MG sequencing project, to facilitate analysis algorithm development by giving practical access to the greatest potential universe of candidate variants. In particular, the fast querying abilities of the SeqWare Query Engine enabled rapid heuristic tuning of the variant calling pipeline parameters through many cycles of filtering and subsequent assessment.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 1. Datasets<\/b>\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">Dataset\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Technology\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">SNVs & Indels\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">SV\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Translocations\n<\/th>\n<th style=\"padding-left:10px; padding-right:10px;\">Reference\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">European-Venter\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sanger\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Levy et al. 2007<sup id=\"rdp-ebb-cite_ref-LevyTheDip07_3-1\" class=\"reference\"><a href=\"#cite_note-LevyTheDip07-3\" rel=\"external_link\">[3]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">European-Watson\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">454\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wheeler et al. 2008<sup id=\"rdp-ebb-cite_ref-WheelerTheCom08_4-1\" class=\"reference\"><a href=\"#cite_note-WheelerTheCom08-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">European-Quake\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Helicos\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pushkarev et al. 2009<sup id=\"rdp-ebb-cite_ref-PushkarevSingle09_5-1\" class=\"reference\"><a href=\"#cite_note-PushkarevSingle09-5\" rel=\"external_link\">[5]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Asian\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Wang et al. 2008<sup id=\"rdp-ebb-cite_ref-WangTheDip08_6-1\" class=\"reference\"><a href=\"#cite_note-WangTheDip08-6\" rel=\"external_link\">[6]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yoruban 18507\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bentley et al. 2008<sup id=\"rdp-ebb-cite_ref-BentleyAcc08_7-1\" class=\"reference\"><a href=\"#cite_note-BentleyAcc08-7\" rel=\"external_link\">[7]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yoruban 18507\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SOLiD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">McKernan et al. 2009<sup id=\"rdp-ebb-cite_ref-McKernanSeq09_8-1\" class=\"reference\"><a href=\"#cite_note-McKernanSeq09-8\" rel=\"external_link\">[8]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Korean\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ahn et al. 2009<sup id=\"rdp-ebb-cite_ref-AhnTheFirst09_9-1\" class=\"reference\"><a href=\"#cite_note-AhnTheFirst09-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Korean-AKI\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Kim et al. 2009<sup id=\"rdp-ebb-cite_ref-KimAHighly09_10-1\" class=\"reference\"><a href=\"#cite_note-KimAHighly09-10\" rel=\"external_link\">[10]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3 human genomes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Complete Genomics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Drmanac et al. 2009<sup id=\"rdp-ebb-cite_ref-DrmanacHuman10_11-1\" class=\"reference\"><a href=\"#cite_note-DrmanacHuman10-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AML T\/N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Ley et al. 2008<sup id=\"rdp-ebb-cite_ref-LeyDNA08_12-1\" class=\"reference\"><a href=\"#cite_note-LeyDNA08-12\" rel=\"external_link\">[12]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">AML genome\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mardis et al. 2009<sup id=\"rdp-ebb-cite_ref-MardisRec09_13-1\" class=\"reference\"><a href=\"#cite_note-MardisRec09-13\" rel=\"external_link\">[13]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Melanoma\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pleasance et al. 2010<sup id=\"rdp-ebb-cite_ref-PleasanceASmall10_14-1\" class=\"reference\"><a href=\"#cite_note-PleasanceASmall10-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Lung cancer\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SOLiD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pleasance et al. 2010<sup id=\"rdp-ebb-cite_ref-PleasanceAComp10_15-1\" class=\"reference\"><a href=\"#cite_note-PleasanceAComp10-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">U87MG\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SOLiD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Clark et al. 2010<sup id=\"rdp-ebb-cite_ref-ClarkU87MG10_16-1\" class=\"reference\"><a href=\"#cite_note-ClarkU87MG10-16\" rel=\"external_link\">[16]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\">Fourteen whole genome datasets were loaded into the database, including the U87MG genome, with the March 2006 assembly of the human genome used as reference (NCBI36\/hg18). Variant types (SNVs, small\/large indels, SVs, etc) loaded and publication references are noted for each respective dataset. This table was adapted from Snyder et al. 2010.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>A secondary dataset generated in our lab, the \u201c1102 GBM\u201d tumor\/normal whole genome sequence pair, was used to compare the performance between the BerkeleyDB and HBase Query Engine backend types. This dataset, like the U87MG genome, included loading raw variant calls seen once or more in both backends in order to profile the load and query mechanisms.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Programmatic_access\">Programmatic access<\/span><\/h3>\n<p>The SeqWare Query Engine provides a common database store interface that supports both the BerkleyDB and HBase backend types. This store object provides generic methods to read and write the full range of data types into and out of the underlying database. It handles the persistence and retrieval of keys and objects to and from the database using a flexible object mapping layer. Simple to write bindings are created when new data types are added to the database. The underlying schema-less nature of key\/value stores like BerkeleyDB and HBase make this process very straightforward. The store also supports queries that lookup all variants and filter by specific fields, such as coverage or variant call p-value, and it can also query based on secondary indexes, typically a tag lookup (key-value pair). The underlying implementation for each store type (BerkeleyDB or HBase) is quite different but the generic store interface masks the difference from the various import and export tools available in the project. The store interface was used whenever possible in order to maximize the portability of the query engine and to make it possible to switch backends in the future.\n<\/p><p>Two lower-level APIs are available for querying the HBase database directly. The first is the HBase API, which the store object uses for most of its operations including filtering variants by tags. This API is very similar to other database interfaces and lets the calling client iterate and filter result sets. HBase also support the use of a MapReduce source and sink, which allow for database traversal and load respectively. This was used to iterate over all variants in the database as quickly as possible and to perform basic analysis tasks such as variant type counting. The speed and flexibility of the MapReduce interface to HBase make it an attractive mechanism to implement future functionality.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Web_service_access\">Web service access<\/span><\/h3>\n<p>The web service is built on top of the programmatic, generic store object and uses the Restlet Java API (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.restlet.org\" target=\"_blank\">http:\/\/www.restlet.org<\/a>). This provides a RESTful<sup id=\"rdp-ebb-cite_ref-FieldingArch00_24-0\" class=\"reference\"><a href=\"#cite_note-FieldingArch00-24\" rel=\"external_link\">[24]<\/a><\/sup> web service interface for the query methods available through the store. When loaded from a web browser, the web service uses XSLT and CSS to display a navigatable web interface with user-friendly web forms for searching the database. Queries on three data types are supported: variants, coding consequence reports, and per-base coverage. Variants can be searched by tag and also a wide variety of fields such as their depth of coverage. The tag field is used to store a variety of annotations and, in the U87MG database, this includes dbSNP status, mutational consequence predictions, and names of overlapping genes, among others. For the variant reports, the standard BED file type (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/genome.ucsc.edu\/\u200bFAQ\/\u200bFAQformat\" target=\"_blank\">http:\/\/genome.ucsc.edu\/\u200bFAQ\/\u200bFAQformat<\/a>) is supported and users can alternatively select to load the query result directly in the UCSC browser, the IGV browser, or generate a list of non-redundant tags associated with the variant query results. Coverage information can be queried only by location, and the result can be generated in WIG format (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/genome.ucsc.edu\/\u200bFAQ\/\u200bFAQformat\" target=\"_blank\">http:\/\/genome.ucsc.edu\/\u200bFAQ\/\u200bFAQformat<\/a>), WIG with coverage averaged by each block, or loadable links for the UCSC and IGV browsers.\n<\/p><p>When queried programmatically, the web service returns XML result documents. These contain enough metadata to construct URLs accessible from a wide variety of programming languages which can then be used to return query results in standardized formats (BED, WIG, etc). Since every query is just a URL, they can be created from within a web browser using the user-friendly form fields and cut-and-pasted into another tool or script. These URLs can then be shared over email, linked to in a publication, and bookmarked for later use, thereby providing a convenient, stateless, and universally interpretable reference to the results.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_load_tools\">Data load tools<\/span><\/h3>\n<p>Most of the genome datasets used in this project where limited to SNV, indel, and SV predictions. For those genomes, the variant information files that were available were loaded into the query engine using either standard file type loaders (GFF, key-value files, etc) or using custom annotation file parsers. The pileup file format (produced by SAMtools<sup id=\"rdp-ebb-cite_ref-LiTheSeq09_25-0\" class=\"reference\"><a href=\"#cite_note-LiTheSeq09-25\" rel=\"external_link\">[25]<\/a><\/sup>) was used to load both the U87MG and \u201c1102 GBM\u201d tumor\/normal genome variants and coverage information. The variant loading tool supports a plugin interface so new annotation file types can be easily supported. This import tool is available in the query engine package and uses the generic store interface for loading information in database backend (see \u201cProgrammatic Access\u201d for more information). The HBase store currently uses an API very similar to other database connection APIs, and is therefore not inherently parallel, although multiple loads can occur simultaneously.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Analysis_tools\">Analysis tools<\/span><\/h3>\n<p>Two prototype analysis tools were created for use with the U87MG and \u201c1102 GBM\u201d tumor\/normal databases. First, a MapReduce variant query tool was written to directly compare the performance of the retrieval of variants from HBase using the API versus using MapReduce. This simple tool used the HBase TableInputFormat object and a MapReduce job to traverse all database rows. The second analysis tool created was a simple somatic mutation detector for use with the \u201c1102 GBM\u201d tumor\/normal genome database. Again, the HBase TableInputFormat object was used to iterate over the variants from the tumor genome, evaluating each variant by user-specified quality criteria in the map step and identifying those that were present in the cancer but not in the normal. In the reduce phase the coverage at each putative somatic mutation location was checked and only those where the coverage was good and no normal sample variant was called where reported as putative somatic mutations.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Performance_measurement\">Performance measurement<\/span><\/h3>\n<p>Backend performance was measured for both the BerkeleyDB and HBase stores using both data import and export times as metrics. The \u201c1102 GBM\u201d tumor\/normal genome was used in this testing, which included enough variant calls (both true and spurious) to stress test both backends. A single threaded, API-based approach was taken for the load test with both the BerkeleyDB and HBase backends. Each chromosome was loaded in turn and the time taken to import these variants into the database was recorded. A similar approach was taken for the retrieval test except in this case the BerkeleyDB backend continued to use an API approach whereas the HBase backend used both an API and MapReduced approach. The export test was interleaved with the import test, wherein after a chromosome was loaded the variants where exported and both processes were timed. In that way we monitored both the import and export time as a function of overall database size.\n<\/p><p>The HBase tests were conducted on a 6 node HBase cluster where each node contained 8 2.4GHz Xeon CPUs, 24GB of RAM, and 6TB of hard drive space. Each node contained 16 map task slots and 4 reducer task slots. The BerkeleyDB tests were conducted on a single node, since it does not support server clustering, but with hardware identical to that used in the HBase tests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"U87MG_genome_database\">U87MG genome database<\/span><\/h3>\n<p>For our original U87MG human genome cell line sequencing project, we created the SeqWare Query Engine database, first built on BerkeleyDB (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.oracle.com\/\u200btechnetwork\/\u200bdatabase\/\u200bberkeleydb\" target=\"_blank\">http:\/\/www.oracle.com\/\u200btechnetwork\/\u200bdatabase\/\u200bberkeleydb<\/a>) and then later ported to HBase (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hbase.apache.org\" target=\"_blank\">http:\/\/hbase.apache.org<\/a>). For the work presented here, the U87MG database was enhanced with the addition of the 13 other human genome datasets that were publicly available when this effort commenced. To further enhance the query engine, we also added new query strategies and utilities, such as a MapReduce-based variant search tool. Unlike the U87MG genome, which included all variant calls regardless of quality, the other genomes included only post quality filtered variants. Still, they offer a proof of concept that the HBase backend can represent multiple genomes worth of sequence variants and associated annotation data. This sample query engine is hosted at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/genome.ucla.edu\/\u200bU87\" target=\"_blank\">http:\/\/genome.ucla.edu\/\u200bU87<\/a> and can be used for both programmatic and interactive queries through the web service interface. A database snapshot is not available for download (due to its large size) but all the source datasets are publicly available and the database can be reconstituted in another location using either the BerkeleyDB or HBase backends along with the provided query engine load tools.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Performance\">Performance<\/span><\/h3>\n<p>Figure 2 shows a comparison between the BerkeleyDB and HBase backends for both variant load and variant export functions. BerkeleyDB was chosen as the original backend for multiple reasons: it did not require a database daemon to run, it provided a key-value store similar to the distributed key-value NoSQL stores we intended to move towards, it had a well-designed API, and it was known to be widely used and robust. In the load tests both BerkeleyDB and HBase performed comparably (Figure 2a), with both tests using a single thread with a similar client API (rather than a MapReduce loader which would be possible only with HBase). BerkeleyDB is slightly faster until about 6 million (6M) variants are loaded but HBase is faster after that point and eventually takes about 15 minutes less to load all 7M variants in this test. The test for variant export had a significantly different outcome (Figure 2b). Both MapReduce and standard single-thread API retrieval of variants from HBase were extremely efficient, with the MapReduce exporter completing in 45 seconds and the single-thread in 386 seconds. This is in sharp contrast with the BerkeleyDB backend which took 6,281 seconds to export the 7M variant records. When the BerkeleyDB database reached approximately 3.5M variants the run time to export ballooned quickly. This was likely due to memory limitations on the single node serving the BerkeleyDB, resulted in greatly degraded query performance once the index could no longer fit in memory. In contrast, the HBase cluster nodes were each responsible for storing and querying only a fraction of the data and this data sharding resulting in much more robust query performance. Rather than making a statement about the inherent merits of the database systems themselves, this results underscores the point that the distributed HBase\/Hadoop implementation (in this case spread across 6 nodes) has clear scalability advantages compared to daemons or processes like BerkeleyDB that are limited to a single server.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_OConnor_BMCInformatics2010_11-12.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"54bcfad13a08de375f37f947470e6198\"><img alt=\"Fig2 OConnor BMCInformatics2010 11-12.jpg\" src=\"https:\/\/www.limswiki.org\/images\/a\/a5\/Fig2_OConnor_BMCInformatics2010_11-12.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2. Load and query performance.<\/b> Comparisons of load and query times between the HBase and BerkeleyDB backend. (a) Load times for the \u201c1102 GBM\u201d tumor\/normal genomes where compared between HBase and BerkeleyDB. Both used a single-threaded approach to better compare relative performance. Both perform similarly but over time the load times for BerkeleyDB increase faster than with HBase. (b) Comparison of querying the 1102 genome database between BerkeleyDB, HBase single threaded, and HBase using MapReduce. Beyond 3M variants BerkeleyDB query times increase dramatically while both query types for HBase perform linearly, with MapReduce consistently exhibiting the best performance.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Software_availability\">Software availability<\/span><\/h3>\n<p>The SeqWare Query Engine is a sub-project of the SeqWare toolset for next generation sequence analysis. SeqWare includes a LIMS, pipeline based on Pegasus<sup id=\"rdp-ebb-cite_ref-DeelmanPeg05_26-0\" class=\"reference\"><a href=\"#cite_note-DeelmanPeg05-26\" rel=\"external_link\">[26]<\/a><\/sup>, and metadata schema in addition to the query engine. Like the rest of the SeqWare project, SeqWare Query Engine is fully open source and is distributed under the terms of the GNU General Public License v3 (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.gnu.org\/\u200blicenses\/\u200blicenses.\u200bhtml\" target=\"_blank\">http:\/\/www.gnu.org\/\u200blicenses\/\u200blicenses.\u200bhtml<\/a>). The software can be downloaded from version control on the project\u2019s SourceForge.net site (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/seqware.sourceforge.net\" target=\"_blank\">http:\/\/seqware.sourceforge.net<\/a>). Visitors to the site can also post questions on the mailing list, view documentation on wiki pages, and download a pre-configured SeqWare Pipeline and Query Engine as a virtual machine, suitable for running in a wide array of environments. The present authors are very interested in working with other developers on this project and welcome any contributions.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>Recent innovations from search-oriented companies such as Google, Yahoo, Facebook, and LinkedIn provide compelling technologies that could potentially enable computation on petascale sequence data. Projects such as Hadoop and HBase, open source implementations of Google\u2019s MapReduce framework<sup id=\"rdp-ebb-cite_ref-DeanMap08_27-0\" class=\"reference\"><a href=\"#cite_note-DeanMap08-27\" rel=\"external_link\">[27]<\/a><\/sup> and BigTable database<sup id=\"rdp-ebb-cite_ref-ChangBig08_28-0\" class=\"reference\"><a href=\"#cite_note-ChangBig08-28\" rel=\"external_link\">[28]<\/a><\/sup> respectively, can be converted to powerful frameworks for analyzing next generation sequencing data. For example, MapReduce is based on a functional programming style where the basic methods available to perform analysis are a map phase, where data is transformed from one form to another, and a reduce phase, where data is sorted and condensed. This fits well to fundamental sequence analysis computations such as alignment and variant calling. Already there are versions of analysis algorithms such as Crossbow (alignment<sup id=\"rdp-ebb-cite_ref-LangmeadSearch09_29-0\" class=\"reference\"><a href=\"#cite_note-LangmeadSearch09-29\" rel=\"external_link\">[29]<\/a><\/sup>) and GATK (variant calling and other tools<sup id=\"rdp-ebb-cite_ref-McKennaTheGenome10_30-0\" class=\"reference\"><a href=\"#cite_note-McKennaTheGenome10-30\" rel=\"external_link\">[30]<\/a><\/sup>) that make use of the MapReduce paradigm. The approach is not fundamentally different from other functional programming languages that have come before, but in this case the approach is combined in Hadoop with both a distributed filesystem (HDFS) and tightly coupled execution engine. These tools, in turn, form the basis of the HBase database system. Unlike traditional grid technologies\u2014for example a sun grid engine computation cluster\u2014the Hadoop environment automatically partitions large data files across the underlying cluster, and computations can then be distributed across the individual data pieces without requiring the analysis program to know where the data resides, or manage such information. The HBase database system\u2014one of the most mature of the NoSQL database projects\u2014takes advantage of this sharding feature and allows a database to be broken into distinct pieces across the underlying computer cluster. This enables the creation of much larger databases than can be supported in traditional relational implementations which are constrained to run on a single database server, up to the scale of billions of rows (e.g. bases in a genome) and millions of columns (e.g. individual genomes). This is considerably larger than most database systems typically support, but is the correct scale needed to represent heavily annotated whole genome sequence data for future large scale biomedical research studies and clinical deployment to the broadest patient populations.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Here we have introduced the open source SeqWare Query Engine that uses an HBase backend and the MapReduce\/Hadoop infrastructure to provide robust databasing of sequence data types for the SeqWare project. The results show the scaling benefits that result from these highly distributable technologies even when creating a database of genomic variants for just 14 human genome datasets. The basic database functions, such as importing and exporting, are one to two orders of magnitude faster with HBase compared to BerkelyDB. Moreover, the highly nonlinear improvement in scaling is readily demonstrated at the critical point where the standard database server becomes saturated, whereas the HBase server maintains a proper distributed load as the data burden is increased. This fully cloud-oriented database framework is ideal for the creation of whole genome sequence\/variant databases. It is capable of supporting large scale genome sequencing research projects involving hundreds to thousands of genomes as well as future large scale clinical deployments utilizing advanced sequencer technology that will soon involve tens to hundreds of thousands of genomes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"List_of_abbreviations\">List of abbreviations<\/span><\/h2>\n<p><b>ACID:<\/b> atomicity, consistency, isolation, and durability\n<\/p><p><b>API:<\/b> Application Programming Interface\n<\/p><p><b>BED:<\/b> Browser Extensible Data (encodes variants)\n<\/p><p><b>ChIP-Seq:<\/b> Chromatin Immunoprecipitation sequencing\n<\/p><p><b>CSS:<\/b> Cascading Style Sheets\n<\/p><p><b>GATK:<\/b> Genome Analysis Toolkit\n<\/p><p><b>GBM:<\/b> Glioblastoma Multiforme\n<\/p><p><b>GFF:<\/b> General Feature Format (encodes genomic features)\n<\/p><p><b>HDFS:<\/b> Hadoop Distributed File System\n<\/p><p><b>IGV:<\/b> Integrative Genomics Viewer\n<\/p><p><b>indel:<\/b> small (typically <10bp) insertion or deletion\n<\/p><p><b>REST:<\/b> Representational State Transfer\n<\/p><p><b>RNA-Seq:<\/b> RNA sequencing\n<\/p><p><b>SNP:<\/b> Single Nucleotide Polymorphism\n<\/p><p><b>SNV:<\/b> Single Nucleotide Variant\n<\/p><p><b>SQL:<\/b> Structured Query Language\n<\/p><p><b>SV:<\/b> Structural Variants\n<\/p><p><b>TCGA:<\/b> the Cancer Genome Atlas\n<\/p><p><b>URL:<\/b> Uniform Resource Locator\n<\/p><p><b>VCF:<\/b> Variant Call Format\n<\/p><p><b>XML:<\/b> Extensible Markup Language\n<\/p><p><b>XSLT:<\/b> XSL Transformations\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Acknowledgments\">Acknowledgments<\/span><\/h3>\n<p>This work was supported by grants from the NINDS (U24NS), the Dani Saleh Brain Tumor Fund, and the Henry Singleton Brain Tumor Fund. The authors would like to acknowledge Jordan Mendler for his contributions to the SeqWare Pipeline project and Hane Lee for her feedback on the SeqWare Query Engine tools.\n<\/p><p>This article has been published as part of BMC Bioinformatics Volume 11 Supplement 12, 2010: Proceedings of the 11th Annual Bioinformatics Open Source Conference (BOSC) 2010. The full contents of the supplement are available online at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.biomedcentral.com\/\u200b1471-2105\/\u200b11?\u200bissue=\u200bS12\" target=\"_blank\">http:\/\/www.biomedcentral.com\/\u200b1471-2105\/\u200b11?\u200bissue=\u200bS12<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h3>\n<p>BDO designed and implemented the SeqWare Query Engine. BM provided guidance on project goals, applications, selection of annotation databases, and choice of analysis algorithms used. SFN is the principal investigator for the U87MG genome sequencing project which supported the development of the SeqWare project including the query engine.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors have declared no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-SnyderPers10-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SnyderPers10_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Snyder, M.; Du, J.; Gerstein, M. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827837\" target=\"_blank\">\"Personal genome sequencing: Current approaches and challenges\"<\/a>. <i>Genes & Development<\/i> <b>24<\/b> (5): 423\u2013431. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgad.1864110\" target=\"_blank\">10.1101\/gad.1864110<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2827837\/\" target=\"_blank\">PMC2827837<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827837\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2827837<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Personal+genome+sequencing%3A+Current+approaches+and+challenges&rft.jtitle=Genes+%26+Development&rft.aulast=Snyder%2C+M.%3B+Du%2C+J.%3B+Gerstein%2C+M.&rft.au=Snyder%2C+M.%3B+Du%2C+J.%3B+Gerstein%2C+M.&rft.date=2010&rft.volume=24&rft.issue=5&rft.pages=423%E2%80%93431&rft_id=info:doi\/10.1101%2Fgad.1864110&rft_id=info:pmc\/PMC2827837&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2827837&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LanderInit01-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LanderInit01_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lander, E.; Linton, L.; Birren, B. et al. (2001). \"Initial sequencing and analysis of the human genome\". <i>Nature<\/i> <b>409<\/b> (6822): 860-921. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F35057062\" target=\"_blank\">10.1038\/35057062<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11237011\" target=\"_blank\">11237011<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Initial+sequencing+and+analysis+of+the+human+genome&rft.jtitle=Nature&rft.aulast=Lander%2C+E.%3B+Linton%2C+L.%3B+Birren%2C+B.+et+al.&rft.au=Lander%2C+E.%3B+Linton%2C+L.%3B+Birren%2C+B.+et+al.&rft.date=2001&rft.volume=409&rft.issue=6822&rft.pages=860-921&rft_id=info:doi\/10.1038%2F35057062&rft_id=info:pmid\/11237011&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LevyTheDip07-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LevyTheDip07_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-LevyTheDip07_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Levy, S.; Sutton, G.; Ng, P. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1964779\" target=\"_blank\">\"The diploid genome sequence of an individual human\"<\/a>. <i>PLOS Biology<\/i> <b>5<\/b> (10): e254. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pbio.0050254\" target=\"_blank\">10.1371\/journal.pbio.0050254<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1964779\/\" target=\"_blank\">PMC1964779<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17803354\" target=\"_blank\">17803354<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1964779\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1964779<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+diploid+genome+sequence+of+an+individual+human&rft.jtitle=PLOS+Biology&rft.aulast=Levy%2C+S.%3B+Sutton%2C+G.%3B+Ng%2C+P.+et+al.&rft.au=Levy%2C+S.%3B+Sutton%2C+G.%3B+Ng%2C+P.+et+al.&rft.date=2007&rft.volume=5&rft.issue=10&rft.pages=e254&rft_id=info:doi\/10.1371%2Fjournal.pbio.0050254&rft_id=info:pmc\/PMC1964779&rft_id=info:pmid\/17803354&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1964779&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheelerTheCom08-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WheelerTheCom08_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-WheelerTheCom08_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wheeler, D.A.; Srinivasan, M.; Egholm, M. et al. (2008). \"The complete genome of an individual by massively parallel DNA sequencing\". <i>Nature<\/i> <b>452<\/b> (7189): 872\u2013876. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature06884\" target=\"_blank\">10.1038\/nature06884<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18421352\" target=\"_blank\">18421352<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+complete+genome+of+an+individual+by+massively+parallel+DNA+sequencing&rft.jtitle=Nature&rft.aulast=Wheeler%2C+D.A.%3B+Srinivasan%2C+M.%3B+Egholm%2C+M.+et+al.&rft.au=Wheeler%2C+D.A.%3B+Srinivasan%2C+M.%3B+Egholm%2C+M.+et+al.&rft.date=2008&rft.volume=452&rft.issue=7189&rft.pages=872%E2%80%93876&rft_id=info:doi\/10.1038%2Fnature06884&rft_id=info:pmid\/18421352&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PushkarevSingle09-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PushkarevSingle09_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-PushkarevSingle09_5-1\" rel=\"external_link\">5.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pushkarev, D.; Neff, N.; Quake, S. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4117198\" target=\"_blank\">\"Single-molecule sequencing of an individual human genome\"<\/a>. <i>Nature Biotechnology<\/i> <b>27<\/b> (9): 847-50. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt.1561\" target=\"_blank\">10.1038\/nbt.1561<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4117198\/\" target=\"_blank\">PMC4117198<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19668243\" target=\"_blank\">19668243<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4117198\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4117198<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Single-molecule+sequencing+of+an+individual+human+genome&rft.jtitle=Nature+Biotechnology&rft.aulast=Pushkarev%2C+D.%3B+Neff%2C+N.%3B+Quake%2C+S.&rft.au=Pushkarev%2C+D.%3B+Neff%2C+N.%3B+Quake%2C+S.&rft.date=2009&rft.volume=27&rft.issue=9&rft.pages=847-50&rft_id=info:doi\/10.1038%2Fnbt.1561&rft_id=info:pmc\/PMC4117198&rft_id=info:pmid\/19668243&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4117198&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangTheDip08-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WangTheDip08_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-WangTheDip08_6-1\" rel=\"external_link\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, J.; Wang, W.; Li, R. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2716080\" target=\"_blank\">\"The diploid genome sequence of an Asian individual\"<\/a>. <i>Nature<\/i> <b>456<\/b> (7218): 60-5. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature07484\" target=\"_blank\">10.1038\/nature07484<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2716080\/\" target=\"_blank\">PMC2716080<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18987735\" target=\"_blank\">18987735<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2716080\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2716080<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+diploid+genome+sequence+of+an+Asian+individual&rft.jtitle=Nature&rft.aulast=Wang%2C+J.%3B+Wang%2C+W.%3B+Li%2C+R.+et+al.&rft.au=Wang%2C+J.%3B+Wang%2C+W.%3B+Li%2C+R.+et+al.&rft.date=2008&rft.volume=456&rft.issue=7218&rft.pages=60-5&rft_id=info:doi\/10.1038%2Fnature07484&rft_id=info:pmc\/PMC2716080&rft_id=info:pmid\/18987735&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2716080&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BentleyAcc08-7\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-BentleyAcc08_7-0\" rel=\"external_link\">7.0<\/a><\/sup> <sup><a href=\"#cite_ref-BentleyAcc08_7-1\" rel=\"external_link\">7.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bentley, D.; Balasubramanian, S.; Swerdlow, H. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2581791\" target=\"_blank\">\"Accurate whole human genome sequencing using reversible terminator chemistry\"<\/a>. <i>Nature<\/i> <b>456<\/b> (7218): 53-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature07517\" target=\"_blank\">10.1038\/nature07517<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2581791\/\" target=\"_blank\">PMC2581791<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18987734\" target=\"_blank\">18987734<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2581791\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2581791<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Accurate+whole+human+genome+sequencing+using+reversible+terminator+chemistry&rft.jtitle=Nature&rft.aulast=Bentley%2C+D.%3B+Balasubramanian%2C+S.%3B+Swerdlow%2C+H.+et+al.&rft.au=Bentley%2C+D.%3B+Balasubramanian%2C+S.%3B+Swerdlow%2C+H.+et+al.&rft.date=2008&rft.volume=456&rft.issue=7218&rft.pages=53-9&rft_id=info:doi\/10.1038%2Fnature07517&rft_id=info:pmc\/PMC2581791&rft_id=info:pmid\/18987734&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2581791&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McKernanSeq09-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-McKernanSeq09_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-McKernanSeq09_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McKernan, K.; Peckham, H.; Costa, G. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752135\" target=\"_blank\">\"Sequence and structural variation in a human genome uncovered by short-read, massively parallel ligation sequencing using two-base encoding\"<\/a>. <i>Genome Research<\/i> <b>19<\/b> (9): 1527-41. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.091868.109\" target=\"_blank\">10.1101\/gr.091868.109<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2752135\/\" target=\"_blank\">PMC2752135<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19546169\" target=\"_blank\">19546169<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752135\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752135<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sequence+and+structural+variation+in+a+human+genome+uncovered+by+short-read%2C+massively+parallel+ligation+sequencing+using+two-base+encoding&rft.jtitle=Genome+Research&rft.aulast=McKernan%2C+K.%3B+Peckham%2C+H.%3B+Costa%2C+G.+et+al.&rft.au=McKernan%2C+K.%3B+Peckham%2C+H.%3B+Costa%2C+G.+et+al.&rft.date=2009&rft.volume=19&rft.issue=9&rft.pages=1527-41&rft_id=info:doi\/10.1101%2Fgr.091868.109&rft_id=info:pmc\/PMC2752135&rft_id=info:pmid\/19546169&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2752135&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AhnTheFirst09-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AhnTheFirst09_9-0\" rel=\"external_link\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-AhnTheFirst09_9-1\" rel=\"external_link\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ahn, S.; Kim, T.; Lee, S. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752128\" target=\"_blank\">\"The first Korean genome sequence and analysis: Full genome sequencing for a socio-ethnic group\"<\/a>. <i>Genome Research<\/i> <b>19<\/b> (9): 1622-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.092197.109\" target=\"_blank\">10.1101\/gr.092197.109<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2752128\/\" target=\"_blank\">PMC2752128<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19470904\" target=\"_blank\">19470904<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752128\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2752128<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+first+Korean+genome+sequence+and+analysis%3A+Full+genome+sequencing+for+a+socio-ethnic+group&rft.jtitle=Genome+Research&rft.aulast=Ahn%2C+S.%3B+Kim%2C+T.%3B+Lee%2C+S.+et+al.&rft.au=Ahn%2C+S.%3B+Kim%2C+T.%3B+Lee%2C+S.+et+al.&rft.date=2009&rft.volume=19&rft.issue=9&rft.pages=1622-9&rft_id=info:doi\/10.1101%2Fgr.092197.109&rft_id=info:pmc\/PMC2752128&rft_id=info:pmid\/19470904&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2752128&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KimAHighly09-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KimAHighly09_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-KimAHighly09_10-1\" rel=\"external_link\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kim, J.; Ju, Y.; Park, H. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2860965\" target=\"_blank\">\"A highly annotated whole-genome sequence of a Korean individual\"<\/a>. <i>Nature<\/i> <b>460<\/b> (7258): 1011-5. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature08211\" target=\"_blank\">10.1038\/nature08211<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2860965\/\" target=\"_blank\">PMC2860965<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19587683\" target=\"_blank\">19587683<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2860965\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2860965<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+highly+annotated+whole-genome+sequence+of+a+Korean+individual&rft.jtitle=Nature&rft.aulast=Kim%2C+J.%3B+Ju%2C+Y.%3B+Park%2C+H.+et+al.&rft.au=Kim%2C+J.%3B+Ju%2C+Y.%3B+Park%2C+H.+et+al.&rft.date=2009&rft.volume=460&rft.issue=7258&rft.pages=1011-5&rft_id=info:doi\/10.1038%2Fnature08211&rft_id=info:pmc\/PMC2860965&rft_id=info:pmid\/19587683&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2860965&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DrmanacHuman10-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-DrmanacHuman10_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-DrmanacHuman10_11-1\" rel=\"external_link\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Drmanac, R.; Sparks, A.; Callow, M. et al. (2010). \"Human genome sequencing using unchained base reads on self-assembling DNA nanoarrays\". <i>Science<\/i> <b>327<\/b> (5961): 78-81. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.1181498\" target=\"_blank\">10.1126\/science.1181498<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19892942\" target=\"_blank\">19892942<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Human+genome+sequencing+using+unchained+base+reads+on+self-assembling+DNA+nanoarrays&rft.jtitle=Science&rft.aulast=Drmanac%2C+R.%3B+Sparks%2C+A.%3B+Callow%2C+M.+et+al.&rft.au=Drmanac%2C+R.%3B+Sparks%2C+A.%3B+Callow%2C+M.+et+al.&rft.date=2010&rft.volume=327&rft.issue=5961&rft.pages=78-81&rft_id=info:doi\/10.1126%2Fscience.1181498&rft_id=info:pmid\/19892942&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LeyDNA08-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LeyDNA08_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-LeyDNA08_12-1\" rel=\"external_link\">12.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ley, T.; Mardis, E.; Ding, L. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2603574\" target=\"_blank\">\"DNA sequencing of a cytogenetically normal acute myeloid leukaemia genome\"<\/a>. <i>Nature<\/i> <b>456<\/b> (7218): 66-72. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature07485\" target=\"_blank\">10.1038\/nature07485<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2603574\/\" target=\"_blank\">PMC2603574<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18987736\" target=\"_blank\">18987736<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2603574\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2603574<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DNA+sequencing+of+a+cytogenetically+normal+acute+myeloid+leukaemia+genome&rft.jtitle=Nature&rft.aulast=Ley%2C+T.%3B+Mardis%2C+E.%3B+Ding%2C+L.+et+al.&rft.au=Ley%2C+T.%3B+Mardis%2C+E.%3B+Ding%2C+L.+et+al.&rft.date=2008&rft.volume=456&rft.issue=7218&rft.pages=66-72&rft_id=info:doi\/10.1038%2Fnature07485&rft_id=info:pmc\/PMC2603574&rft_id=info:pmid\/18987736&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2603574&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MardisRec09-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MardisRec09_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-MardisRec09_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mardis, E.; Ding, L.; Dooling, D. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3201812\" target=\"_blank\">\"Recurring mutations found by sequencing an acute myeloid leukemia genome\"<\/a>. <i>New England Journal of Medicine<\/i> <b>361<\/b> (11): 1058-66. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMoa0903840\" target=\"_blank\">10.1056\/NEJMoa0903840<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3201812\/\" target=\"_blank\">PMC3201812<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19657110\" target=\"_blank\">19657110<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3201812\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3201812<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Recurring+mutations+found+by+sequencing+an+acute+myeloid+leukemia+genome&rft.jtitle=New+England+Journal+of+Medicine&rft.aulast=Mardis%2C+E.%3B+Ding%2C+L.%3B+Dooling%2C+D.+et+al.&rft.au=Mardis%2C+E.%3B+Ding%2C+L.%3B+Dooling%2C+D.+et+al.&rft.date=2009&rft.volume=361&rft.issue=11&rft.pages=1058-66&rft_id=info:doi\/10.1056%2FNEJMoa0903840&rft_id=info:pmc\/PMC3201812&rft_id=info:pmid\/19657110&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3201812&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PleasanceASmall10-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PleasanceASmall10_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-PleasanceASmall10_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pleasance, E.; Stephens, P.; O\u2019Meara, S. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2880489\" target=\"_blank\">\"A small-cell lung cancer genome with complex signatures of tobacco exposure\"<\/a>. <i>Nature<\/i> <b>463<\/b> (7278): 184-90. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature08629\" target=\"_blank\">10.1038\/nature08629<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2880489\/\" target=\"_blank\">PMC2880489<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20016488\" target=\"_blank\">20016488<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2880489\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2880489<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+small-cell+lung+cancer+genome+with+complex+signatures+of+tobacco+exposure&rft.jtitle=Nature&rft.aulast=Pleasance%2C+E.%3B+Stephens%2C+P.%3B+O%E2%80%99Meara%2C+S.+et+al.&rft.au=Pleasance%2C+E.%3B+Stephens%2C+P.%3B+O%E2%80%99Meara%2C+S.+et+al.&rft.date=2010&rft.volume=463&rft.issue=7278&rft.pages=184-90&rft_id=info:doi\/10.1038%2Fnature08629&rft_id=info:pmc\/PMC2880489&rft_id=info:pmid\/20016488&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2880489&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PleasanceAComp10-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PleasanceAComp10_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-PleasanceAComp10_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pleasance, E.; Cheetham, R.; Stephens, P. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145108\" target=\"_blank\">\"A comprehensive catalogue of somatic mutations from a human cancer genome\"<\/a>. <i>Nature<\/i> <b>463<\/b> (7278): 191-6. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature08658\" target=\"_blank\">10.1038\/nature08658<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3145108\/\" target=\"_blank\">PMC3145108<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20016485\" target=\"_blank\">20016485<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145108\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145108<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+comprehensive+catalogue+of+somatic+mutations+from+a+human+cancer+genome&rft.jtitle=Nature&rft.aulast=Pleasance%2C+E.%3B+Cheetham%2C+R.%3B+Stephens%2C+P.+et+al.&rft.au=Pleasance%2C+E.%3B+Cheetham%2C+R.%3B+Stephens%2C+P.+et+al.&rft.date=2010&rft.volume=463&rft.issue=7278&rft.pages=191-6&rft_id=info:doi\/10.1038%2Fnature08658&rft_id=info:pmc\/PMC3145108&rft_id=info:pmid\/20016485&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3145108&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ClarkU87MG10-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ClarkU87MG10_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-ClarkU87MG10_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Clark, M.; Homer, N.; O\u2019Connor, B. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813426\" target=\"_blank\">\"U87MG decoded: The genomic sequence of a cytogenetically aberrant human cancer cell line\"<\/a>. <i>PLOS Genetics<\/i> <b>6<\/b> (1): e1000832. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pgen.1000832\" target=\"_blank\">10.1371\/journal.pgen.1000832<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2813426\/\" target=\"_blank\">PMC2813426<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20126413\" target=\"_blank\">20126413<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813426\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813426<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=U87MG+decoded%3A+The+genomic+sequence+of+a+cytogenetically+aberrant+human+cancer+cell+line&rft.jtitle=PLOS+Genetics&rft.aulast=Clark%2C+M.%3B+Homer%2C+N.%3B+O%E2%80%99Connor%2C+B.+et+al.&rft.au=Clark%2C+M.%3B+Homer%2C+N.%3B+O%E2%80%99Connor%2C+B.+et+al.&rft.date=2010&rft.volume=6&rft.issue=1&rft.pages=e1000832&rft_id=info:doi\/10.1371%2Fjournal.pgen.1000832&rft_id=info:pmc\/PMC2813426&rft_id=info:pmid\/20126413&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2813426&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RheadTheUCSC10-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RheadTheUCSC10_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rhead, B.; Karolchik, D.; Kuhn, R. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2808870\" target=\"_blank\">\"The UCSC genome browser database: update 2010\"<\/a>. <i>Nucleic Acids Research<\/i> <b>38<\/b> (Suppl 1): D613-D619. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkp939\" target=\"_blank\">10.1093\/nar\/gkp939<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2808870\/\" target=\"_blank\">PMC2808870<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19906737\" target=\"_blank\">19906737<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2808870\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2808870<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+UCSC+genome+browser+database%3A+update+2010&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Rhead%2C+B.%3B+Karolchik%2C+D.%3B+Kuhn%2C+R.+et+al.&rft.au=Rhead%2C+B.%3B+Karolchik%2C+D.%3B+Kuhn%2C+R.+et+al.&rft.date=2010&rft.volume=38&rft.issue=Suppl+1&rft.pages=D613-D619&rft_id=info:doi\/10.1093%2Fnar%2Fgkp939&rft_id=info:pmc\/PMC2808870&rft_id=info:pmid\/19906737&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2808870&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MungallAChado07-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MungallAChado07_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mungall, C.; Emmert, D.; The FlyBase Consortium (2007). \"A Chado case study: an ontology-based modular schema for representing genome-associated biological information\". <i>Bioinformatics<\/i> <b>23<\/b> (13): i337-i346. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtm189\" target=\"_blank\">10.1093\/bioinformatics\/btm189<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17646315\" target=\"_blank\">17646315<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+Chado+case+study%3A+an+ontology-based+modular+schema+for+representing+genome-associated+biological+information&rft.jtitle=Bioinformatics&rft.aulast=Mungall%2C+C.%3B+Emmert%2C+D.%3B+The+FlyBase+Consortium&rft.au=Mungall%2C+C.%3B+Emmert%2C+D.%3B+The+FlyBase+Consortium&rft.date=2007&rft.volume=23&rft.issue=13&rft.pages=i337-i346&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtm189&rft_id=info:pmid\/17646315&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HubbardEnsembl06-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HubbardEnsembl06_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hubbard, T.; Aken, B.; Beal, K. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1761443\" target=\"_blank\">\"Ensembl 2007\"<\/a>. <i>Nucleic Acids Research<\/i> <b>35<\/b> (Suppl 1): D610-D617. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkl996\" target=\"_blank\">10.1093\/nar\/gkl996<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1761443\/\" target=\"_blank\">PMC1761443<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17148474\" target=\"_blank\">17148474<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1761443\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1761443<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Ensembl+2007&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Hubbard%2C+T.%3B+Aken%2C+B.%3B+Beal%2C+K.&rft.au=Hubbard%2C+T.%3B+Aken%2C+B.%3B+Beal%2C+K.&rft.date=2006&rft.volume=35&rft.issue=Suppl+1&rft.pages=D610-D617&rft_id=info:doi\/10.1093%2Fnar%2Fgkl996&rft_id=info:pmc\/PMC1761443&rft_id=info:pmid\/17148474&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1761443&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KentTheHuman02-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KentTheHuman02_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kent, W.; Sugnet, C.; Furey, T. et al. (2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604\" target=\"_blank\">\"The human genome browser at UCSC\"<\/a>. <i>Genome Research<\/i> <b>12<\/b> (6): 996-1006. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.229102\" target=\"_blank\">10.1101\/gr.229102<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC186604\/\" target=\"_blank\">PMC186604<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12045153\" target=\"_blank\">12045153<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+human+genome+browser+at+UCSC&rft.jtitle=Genome+Research&rft.aulast=Kent%2C+W.%3B+Sugnet%2C+C.%3B+Furey%2C+T.+et+al.&rft.au=Kent%2C+W.%3B+Sugnet%2C+C.%3B+Furey%2C+T.+et+al.&rft.date=2002&rft.volume=12&rft.issue=6&rft.pages=996-1006&rft_id=info:doi\/10.1101%2Fgr.229102&rft_id=info:pmc\/PMC186604&rft_id=info:pmid\/12045153&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC186604&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SteinTheGeneric02-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SteinTheGeneric02_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stein, L.; Mungall, C.; Shu, S. et al. (2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187535\" target=\"_blank\">\"The generic genome browser: A building block for a model organism system database\"<\/a>. <i>Genome Research<\/i> <b>12<\/b> (10): 1599-610. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.403602\" target=\"_blank\">10.1101\/gr.403602<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC187535\/\" target=\"_blank\">PMC187535<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12368253\" target=\"_blank\">12368253<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187535\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC187535<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+generic+genome+browser%3A+A+building+block+for+a+model+organism+system+database&rft.jtitle=Genome+Research&rft.aulast=Stein%2C+L.%3B+Mungall%2C+C.%3B+Shu%2C+S.+et+al.&rft.au=Stein%2C+L.%3B+Mungall%2C+C.%3B+Shu%2C+S.+et+al.&rft.date=2002&rft.volume=12&rft.issue=10&rft.pages=1599-610&rft_id=info:doi\/10.1101%2Fgr.403602&rft_id=info:pmc\/PMC187535&rft_id=info:pmid\/12368253&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC187535&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KarolchikTheUSCS04-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KarolchikTheUSCS04_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Karolchik, D.; Hinrichs, A.; Furey, T. et al. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC308837\" target=\"_blank\">\"The UCSC Table Browser data retrieval tool\"<\/a>. <i>Nucleic Acids Research<\/i> <b>32<\/b> (Suppl 1): D493-D496. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkh103\" target=\"_blank\">10.1093\/nar\/gkh103<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC308837\/\" target=\"_blank\">PMC308837<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/14681465\" target=\"_blank\">14681465<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC308837\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC308837<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+UCSC+Table+Browser+data+retrieval+tool&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Karolchik%2C+D.%3B+Hinrichs%2C+A.%3B+Furey%2C+T.+et+al.&rft.au=Karolchik%2C+D.%3B+Hinrichs%2C+A.%3B+Furey%2C+T.+et+al.&rft.date=2004&rft.volume=32&rft.issue=Suppl+1&rft.pages=D493-D496&rft_id=info:doi\/10.1093%2Fnar%2Fgkh103&rft_id=info:pmc\/PMC308837&rft_id=info:pmid\/14681465&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC308837&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GiardineGalaxy05-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GiardineGalaxy05_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Giardine, B.; Riemer, C.; Hardison, R.C.; Burhans, R.; Elnitski, L.; Shah, P.; Zhang, Y.; Blankenberg, D.; Albert, I.; Taylor, J.; Miller, W.; Kent, W.J.; Nekrutenko, A. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089\" target=\"_blank\">\"Galaxy: A platform for interactive large-scale genome analysis\"<\/a>. <i>Genome Research<\/i> <b>15<\/b> (10): 1451\u20131455. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.4086505\" target=\"_blank\">10.1101\/gr.4086505<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1240089\/\" target=\"_blank\">PMC1240089<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16169926\" target=\"_blank\">16169926<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy%3A+A+platform+for+interactive+large-scale+genome+analysis&rft.jtitle=Genome+Research&rft.aulast=Giardine%2C+B.%3B+Riemer%2C+C.%3B+Hardison%2C+R.C.%3B+Burhans%2C+R.%3B+Elnitski%2C+L.%3B+Shah%2C+P.%3B+Zhang%2C+Y.%3B+Blankenberg%2C+D.%3B+Albert%2C+I.%3B+Taylor%2C+J.%3B+Miller%2C+W.%3B+Kent%2C+W.J.%3B+Nekrutenko%2C+A.&rft.au=Giardine%2C+B.%3B+Riemer%2C+C.%3B+Hardison%2C+R.C.%3B+Burhans%2C+R.%3B+Elnitski%2C+L.%3B+Shah%2C+P.%3B+Zhang%2C+Y.%3B+Blankenberg%2C+D.%3B+Albert%2C+I.%3B+Taylor%2C+J.%3B+Miller%2C+W.%3B+Kent%2C+W.J.%3B+Nekrutenko%2C+A.&rft.date=2005&rft.volume=15&rft.issue=10&rft.pages=1451%E2%80%931455&rft_id=info:doi\/10.1101%2Fgr.4086505&rft_id=info:pmc\/PMC1240089&rft_id=info:pmid\/16169926&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1240089&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FieldingArch00-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FieldingArch00_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Fielding, R.T. (2000). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm\" target=\"_blank\">\"Architectural Styles and the Design of Network-based Software Architectures\"<\/a>. University of California, Irvine<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm\" target=\"_blank\">https:\/\/www.ics.uci.edu\/~fielding\/pubs\/dissertation\/top.htm<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Architectural+Styles+and+the+Design+of+Network-based+Software+Architectures&rft.atitle=&rft.aulast=Fielding%2C+R.T.&rft.au=Fielding%2C+R.T.&rft.date=2000&rft.pub=University+of+California%2C+Irvine&rft_id=https%3A%2F%2Fwww.ics.uci.edu%2F%7Efielding%2Fpubs%2Fdissertation%2Ftop.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiTheSeq09-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiTheSeq09_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Handsaker, B.; Wysoker, A. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002\" target=\"_blank\">\"The Sequence Alignment\/Map format and SAMtools\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (16): 2078-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp352\" target=\"_blank\">10.1093\/bioinformatics\/btp352<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2723002\/\" target=\"_blank\">PMC2723002<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19505943\" target=\"_blank\">19505943<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Sequence+Alignment%2FMap+format+and+SAMtools&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.%3B+Handsaker%2C+B.%3B+Wysoker%2C+A.+et+al.&rft.au=Li%2C+H.%3B+Handsaker%2C+B.%3B+Wysoker%2C+A.+et+al.&rft.date=2009&rft.volume=25&rft.issue=16&rft.pages=2078-9&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp352&rft_id=info:pmc\/PMC2723002&rft_id=info:pmid\/19505943&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2723002&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeelmanPeg05-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeelmanPeg05_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Deelman, E.; Singh, G.; Su, M. et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1239653\" target=\"_blank\">\"Pegasus: A framework for mapping complex scientific workflows onto distributed systems\"<\/a>. <i>Scientific Programming<\/i> <b>13<\/b> (3): 219\u2013237. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1155%2F2005%2F128026\" target=\"_blank\">10.1155\/2005\/128026<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?id=1239653\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?id=1239653<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Pegasus%3A+A+framework+for+mapping+complex+scientific+workflows+onto+distributed+systems&rft.jtitle=Scientific+Programming&rft.aulast=Deelman%2C+E.%3B+Singh%2C+G.%3B+Su%2C+M.+et+al.&rft.au=Deelman%2C+E.%3B+Singh%2C+G.%3B+Su%2C+M.+et+al.&rft.date=2005&rft.volume=13&rft.issue=3&rft.pages=219%E2%80%93237&rft_id=info:doi\/10.1155%2F2005%2F128026&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fid%3D1239653&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeanMap08-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeanMap08_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dean, J.; Ghemawat, S. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?doid=1327452.1327492\" target=\"_blank\">\"MapReduce: Simplified data processing on large clusters\"<\/a>. <i>Communications of the ACM<\/i> <b>51<\/b> (1): 107-113. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1327452.1327492\" target=\"_blank\">10.1145\/1327452.1327492<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?doid=1327452.1327492\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?doid=1327452.1327492<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MapReduce%3A+Simplified+data+processing+on+large+clusters&rft.jtitle=Communications+of+the+ACM&rft.aulast=Dean%2C+J.%3B+Ghemawat%2C+S.&rft.au=Dean%2C+J.%3B+Ghemawat%2C+S.&rft.date=2008&rft.volume=51&rft.issue=1&rft.pages=107-113&rft_id=info:doi\/10.1145%2F1327452.1327492&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fdoid%3D1327452.1327492&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChangBig08-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChangBig08_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Chang, F.; Dean, J.; Ghemawat, S. et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dl.acm.org\/citation.cfm?doid=1365815.1365816\" target=\"_blank\">\"Bigtable: A distributed storage system for structured data\"<\/a>. <i>ACM Transactions on Computer Systems (TOCS)<\/i> <b>26<\/b> (2): 4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F1365815.1365816\" target=\"_blank\">10.1145\/1365815.1365816<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dl.acm.org\/citation.cfm?doid=1365815.1365816\" target=\"_blank\">http:\/\/dl.acm.org\/citation.cfm?doid=1365815.1365816<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bigtable%3A+A+distributed+storage+system+for+structured+data&rft.jtitle=ACM+Transactions+on+Computer+Systems+%28TOCS%29&rft.aulast=Chang%2C+F.%3B+Dean%2C+J.%3B+Ghemawat%2C+S.+et+al.&rft.au=Chang%2C+F.%3B+Dean%2C+J.%3B+Ghemawat%2C+S.+et+al.&rft.date=2008&rft.volume=26&rft.issue=2&rft.pages=4&rft_id=info:doi\/10.1145%2F1365815.1365816&rft_id=http%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fdoid%3D1365815.1365816&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LangmeadSearch09-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LangmeadSearch09_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Langmead, B.; Schatz, M.; Lin, J.; Pop, M; Salzberg, S. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3091327\" target=\"_blank\">\"Searching for SNPs with cloud computing\"<\/a>. <i>Genome Biology<\/i> <b>10<\/b> (11): R134. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2009-10-11-r134\" target=\"_blank\">10.1186\/gb-2009-10-11-r134<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3091327\/\" target=\"_blank\">PMC3091327<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19930550\" target=\"_blank\">19930550<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3091327\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3091327<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Searching+for+SNPs+with+cloud+computing&rft.jtitle=Genome+Biology&rft.aulast=Langmead%2C+B.%3B+Schatz%2C+M.%3B+Lin%2C+J.%3B+Pop%2C+M%3B+Salzberg%2C+S.&rft.au=Langmead%2C+B.%3B+Schatz%2C+M.%3B+Lin%2C+J.%3B+Pop%2C+M%3B+Salzberg%2C+S.&rft.date=2009&rft.volume=10&rft.issue=11&rft.pages=R134&rft_id=info:doi\/10.1186%2Fgb-2009-10-11-r134&rft_id=info:pmc\/PMC3091327&rft_id=info:pmid\/19930550&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3091327&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McKennaTheGenome10-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McKennaTheGenome10_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McKenna, A.; Hanna, M.; Banks, E. et al. (2010). \"The Genome Analysis Toolkit: A MapReduce framework for analyzing next-generation DNA sequencing data\". <i>Genome Research<\/i> <b>20<\/b>: 1297-1303. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.107524.110\" target=\"_blank\">10.1101\/gr.107524.110<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Genome+Analysis+Toolkit%3A+A+MapReduce+framework+for+analyzing+next-generation+DNA+sequencing+data&rft.jtitle=Genome+Research&rft.aulast=McKenna%2C+A.%3B+Hanna%2C+M.%3B+Banks%2C+E.+et+al.&rft.au=McKenna%2C+A.%3B+Hanna%2C+M.%3B+Banks%2C+E.+et+al.&rft.date=2010&rft.volume=20&rft.pages=1297-1303&rft_id=info:doi\/10.1101%2Fgr.107524.110&rfr_id=info:sid\/en.wikipedia.org:Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Additionally, URLs to the project page have been updated from the old, deprecated SourceForge location to the current GitHub site.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210222\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.771 seconds\nReal time usage: 0.802 seconds\nPreprocessor visited node count: 26074\/1000000\nPreprocessor generated node count: 35143\/1000000\nPost\u2010expand include size: 238967\/2097152 bytes\nTemplate argument size: 73167\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 762.122 1 - -total\n 85.89% 654.564 1 - Template:Reflist\n 77.30% 589.125 29 - Template:Cite_journal\n 75.82% 577.859 30 - Template:Citation\/core\n 12.11% 92.319 74 - Template:Citation\/identifier\n 8.26% 62.951 1 - Template:Infobox_journal_article\n 7.85% 59.863 1 - Template:Infobox\n 4.68% 35.674 30 - Template:Citation\/make_link\n 4.60% 35.041 80 - Template:Infobox\/row\n 4.52% 34.442 169 - Template:Hide_in_print\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7912-0!*!0!!en!5!* and timestamp 20181213210221 and revision id 23399\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud\">https:\/\/www.limswiki.org\/index.php\/Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","8d2093f9b49a2cde0a0f2b14e9a154a1_images":["https:\/\/www.limswiki.org\/images\/0\/0f\/Fig1_OConnor_BMCInformatics2010_11-12.jpg","https:\/\/www.limswiki.org\/images\/a\/a5\/Fig2_OConnor_BMCInformatics2010_11-12.jpg"],"8d2093f9b49a2cde0a0f2b14e9a154a1_timestamp":1544734941,"9847a6beea0bba9383335c5a41e65647_type":"article","9847a6beea0bba9383335c5a41e65647_title":"SaDA: From sampling to data analysis\u2014An extensible open source infrastructure for rapid, robust and automated management and analysis of modern ecological high-throughput microarray data (Singh et al. 2015)","9847a6beea0bba9383335c5a41e65647_url":"https:\/\/www.limswiki.org\/index.php\/Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data","9847a6beea0bba9383335c5a41e65647_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:SaDA: From sampling to data analysis\u2014An extensible open source infrastructure for rapid, robust and automated management and analysis of modern ecological high-throughput microarray data\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nSaDA: From sampling to data analysis\u2014An extensible open source infrastructure for rapid, robust\r\nand automated management and analysis of modern ecological high-throughput microarray dataJournal\n \nInternational Journal of Environmental Research and Public HealthAuthor(s)\n \nSingh, Kumar Saurabh; Thual, Dominique; Spurio, Roberto; Cannata, NicolaAuthor affiliation(s)\n \nUniversity of Camerino; Next Generation Bioinformatics s.r.lPrimary contact\n \nTel.: +44-1582-763133 (ext. 5230); Fax: +44-1582-762595Editors\n \nTchounwou, Paul B.Year published\n \n2015Volume and issue\n \n12 (6)Page(s)\n \n6352-6366DOI\n \n10.3390\/ijerph120606352ISSN\n \n1660-4601Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/htmDownload\n \nhttp:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n\n2.1 State-of-art technology \n\n\n3 Material and methods \n\n3.1 Implementation \n3.2 Experimental design and data acquisition \n3.3 The features of SaDA \n\n3.3.1 Samples and filter tracking \n3.3.2 Management of project specific experimental data \n3.3.3 Microarray data management and analysis \n\n\n3.4 Database schema \n3.5 Batch uploads and downloads of data \n\n\n4 Results and discussion \n\n4.1 Utility of the SaDA infrastructure \n4.2 Limitations of SaDA \n4.3 Availability and future directions \n\n\n5 Conclusions \n6 Supplementary materials \n7 Acknowledgments \n8 Author contributions \n9 Conflicts of interest \n10 References \n11 Notes \n\n\n\nAbstract \nOne of the most crucial characteristics of day-to-day laboratory information management is the collection, storage and retrieval of information about research subjects and environmental or biomedical samples. An efficient link between sample data and experimental results is absolutely important for the successful outcome of a collaborative project. Currently available software solutions are largely limited to large scale, expensive commercial Laboratory Information Management Systems (LIMS). Acquiring such LIMS indeed can bring laboratory information management to a higher level, but most of the times this requires a sufficient investment of money, time and technical efforts. There is a clear need for a light weighted open source system which can easily be managed on local servers and handled by individual researchers. Here we present a software named SaDA for storing, retrieving and analyzing data originated from microorganism monitoring experiments. SaDA is fully integrated in the management of environmental samples, oligonucleotide sequences, microarray data and the subsequent downstream analysis procedures. It is simple and generic software, and can be extended and customized for various environmental and biomedical studies.\nKeywords: software; data management; microarrays; ecological assessment; environmental studies; LIMS; open source system\n\nIntroduction \nTo gain an understanding of complex microbial environments, researchers have to assemble different type of information originating from heterogeneous sources. Every bit of information is valuable and should be made available whenever it is required. An exhaustive and clear protocol for data collection, processing and analysis is critical to achieve the projects end goals. The rapid increase in the discovery and application of molecular techniques constitutes important steps towards the detection and identification of microorganisms present in complex environmental samples. Identification and characterization of microorganisms is a key part of the management of environment and quality of life, tracing contaminants and troubleshooting problems such as pathogenesis or pollution indication.[1] Identification of an unknown species can help to assess whether it poses a safety concern or not. Generally, the process of bio-monitoring plays a crucial role in the execution of large scale projects which focus either on the protection of people\u2019s health, safety of materials or dealing with environment quality. Big ecology related projects address very complicated issues.[2] They require generation and collection of data from a wide spectrum of sources, collaboration of people from different disciplines and the application of highly complex analytical approaches.[3] Moreover, information management is important especially when a large volume of complex data is collected. It becomes even more important when multiple organizations, at different geographical locations, are involved in the data collection procedures. It is absolutely essential that data are collected and managed using the methods recognized by scientific community. Existing software systems do not typically overlap the full pipeline as far as ecological collaborative projects are concerned. They are limited by data types, provide limited extensibility or require additional skills to handle data which are beyond the scope of experimental researchers. We developed Sampling to Data Analysis (SaDA) keeping in mind the requirements of individual laboratory involved in big collaborative projects. The system is a complete portal for ecology related high-throughput data management and analyses. Extra care was taken in designing database schema, information retrieval and in data presentation. SaDA is customizable and extensible to meet the needs of diverse research activities as the source code is freely available. The core of SaDA is based on model dependent development where abstract models of software systems are created and systematically transformed to its implementation.[4] The current version of SaDA supports data management and analysis protocols involved in fresh water monitoring systems utilizing microarray technology as a state-of-the-art technology. \n\nState-of-art technology \nTo the best of our knowledge no freely available software is focused on microarray data management and analysis in terms of its ecological application, as in environmental monitoring. The aim of SaDA is to provide a streamlined work-flow from environmental sample collection and its downstream analysis. Microarray technology, however, is used here as a state-of-the-art technology to detect and quantify pathogens and other microorganisms in environmental freshwater samples. SaDA currently supports Agilent\u2019s one channel oligonucleotide hybridization microarray technology. In addition, the downstream data analysis detects the presence or absence of signals, normalization and estimating cell counts from the micro-array signal intensities. SaDA is not a tool for gene expression analysis. Although existing software tools could meet some of the requirements of SaDA, none meet all of them in the form of a comprehensive, end-to-end platform available as open source and extensible system. Some tools have experienced only limited use. The state of art in Laboratory Information Management System (LIMS) provides all aspect of microarray data management and analysis, using advanced automation and precise robotic control. This type of integrated solutions ensures high quality and reliable data output and enables multiple projects to be managed in parallel. LIMS are mainly commercial and expensive software products. Few commercial systems lack clarity of open source systems, come at considerable costs, and requires high level of technical expertise to install and configure hardware, software and database and is not discussed in the present work. \nMARS[5] provides a comprehensive MIAME supportive suite for storing, retrieving, and analyzing multi-color microarray data. iLAP[6], based on a workflow driven modular architecture, was developed specifically to create and manage experimental protocols and to analyze and share laboratory data. Base[7] has been proved to be a comprehensive Microarray data repository, providing researchers an efficient data management and analysis tool. Labkey Server[8] is a complete system with easy to use interfaces for specimen request submission across collaborators, allowing users to graphically define new data-types to fit in diverse datasets. It interacts dynamically with external data sources and supports developing custom interfaces using client libraries. GnomEx[9] provides comprehensive automated protocols for sample submission, sample tracking, billing, data management and analysis workflows for both Next Generation Sequencing (NGS) and Microarrays. LIMS Light System[10] has been developed to meet the demands of high-throughput primary data storage and data upload\/retrieval. This system provides an efficient storage system with a Hierarchical Storage Management System and an Oracle relational database. OpenBis[11] is a flexible framework that has been adapted to be used for proteomics, high content screening and NGS projects. The SMITH[12] and WASP[13] system has been developed to meet the demands of NGS experiments and clinical tests. They provide embedded pipelines for the analysis of ChIP-Seq, RNA-Seq, miRNA-Seq and Exome-Seq experiments. There is also an extension with LIMS functionality to the popular Galaxy[14] workflow engine called Galaxy LIMS.[15] The system supports request submission, offers assistance during flow cell layout, and automatically launches Illumina\u2019s CASAVA software[16] to perform de-multiplexing and delivery of the user-specific NGS raw files. Being integrated with Galaxy, the data are automatically available to be processed by analysis pipelines stored in Galaxy. SLIMS[17] is a sample management tool for genotyping laboratories. \nAn open-source LIMS must be utilized to the specific infrastructure and the project needs of a research institution, which are in constant evolution. Therefore, the LIMS system must be modified accordingly. The effort required to tweak an existing LIMS and gain significant insight into its code base in order to be able to adjust it in a productive way must be compared with the effort of developing a new system. Large number of available LIMS software and tools substantiate the notion of creating a new system against tweaking an existing one. From this perspective, it seems that a simple LIMS has more chances of being shared than a more complex one. In any case, there is no LIMS that serves all needs in a simple turn-key solution. SaDA is not properly a LIMS based software solution but it provides a platform for model driven data management and analysis, and is flexible enough to include new features quickly and efficiently.\n\nMaterial and methods \nSaDA is a web application implemented in Ruby on Rails that runs on Phusion passenger application server and stores its data in a directory structure with in SaDA for quick file manipulations and also in a relational database engine for efficient and quick data querying and retrieval. The current version of SaDA is supported on Linux, UNIX, Mac OS and Microsoft Windows.\n\nImplementation \nSaDA has been developed using the Ruby on Rails web application development framework, which is based on the Model-View-Controller (MVC) architectural pattern[18] of software development. MVC partition the application into multiple components. As described schematically in Figure 1, application input, output and processing (application server and application layer) are compartmentalized in two layers. Application\u2019s core functionalities and database are encapsulated in layer 2, also described as \u201cdatabase layer\u201d in Figure 1. The modular architecture of SaDA supports different features to be encapsulated as individual models, shown as ellipses Figure 1. Sharp rectangles are individual sections encompassing models with associations and relationships. For example, Sampling sites can have different features like land mapping, geographical positions, altitude type, water type & kind of water in use (for fresh water environments). These features are individual relations\/tables associated with each other and also with features of different sections in the SaDA database. Database layer of SaDA is independent of input behavior and output data representation and forms the Model of MVC pattern. User interfaces and application server are components of layer 1. This part handles most of the input, output and query processing operations. It sends query to database and receives the data which in turn is displayed on SaDA user interfaces. The user interacts with the system solely through layer 1. The separation of the software layers allows freedom from the one-table-one-view paradigm. Therefore SaDA can display multiple interfaces of the same model allowing display of same data in multiple ways. Layer 1, in Figure 1, also includes an application server layer. The main job of this application layer is to support the construction of dynamic pages by handling the application\u2019s program logic. Additionally, it also handles database connection and data transfers between client and application in a smooth and efficient way. External tools and libraries are used in SaDA to enhance some of the SaDA\u2019s core functionalities described in later sections of the paper.\n\r\n\n\n\n\n\n\n\n\n\n\n Figure 1. SaDA architecture: A multilayer architecture, comprising of layer 1 for data upload, processing and retrieval and a RDBMS storage layer with data access control, metadata storage for fast data retrieval. SaDA, in addition, has an external component for third party tools and libraries.\n\n\n\nExperimental design and data acquisition \nArchiving detailed information on collection and analysis of samples is imperative for both efficient reporting of any study and for successive data analysis.[19] Collecting and storing these information is of utmost importance in terms of its application in high-throughput technologies. SaDA currently supports MySQL as a back end relational database management system (RDBMS). It can also support any other available RDBMS like SQLite and PostgreSQL. This kind of architecture permits multilevel security, as the application can be made secured at both application level and database level. SaDA provides a secure solution by implementing proper authentication control to each research participant. The database level security can be achieved by providing user level privileges. The system is web-based, which means that non-identifiable information is kept on a server and can be securely accessed on-line for queries or new submissions by authorized users via a web-browser. SaDA is simple and generic, and thus can be customized for various types of ecological as well as biological studies.\n\n\n\n\n\n\n\n\n\n Figure 2. Display of Sampling Sites related data within SaDA.\n\n\n\nThe features of SaDA \nSaDA was originally developed on the guidelines provided by the MicroAQUA consortium.[20] The source code is now available to the public. Due to its modular framework (Figure 1), SaDA can be easily modified and applied to any environmental related projects. The current implementation of SaDA includes different modules storing data from sample collection, DNA\/RNA extraction, oligonucleotides annotation, and microarrays raw data storage and validation results.\n\nSamples and filter tracking \nUsers can track sample related information like quantity of samples and instrument utilized in sampling. From this point, the application assigns a unique identifier to the sample and provides the ability to be cross referenced later in other modules. Sampling sites, in addition, acts as one of the attributes of sampling procedure. Therefore SaDA integrates a Google Maps plug-in[21] for displaying sample collection sites (Figure 2) dynamically. Filter sample module saves information about the filter used to concentrate the collected water sample. A unique identifier specifies the filter associated with sampling and partner code. It carries information like pore size of the filter, number of filter tubes utilized and volume of water sample filtered.\n\nManagement of project specific experimental data \nSaDA supports oligonucleotide microarrays as a fresh water environment monitoring technology. SaDA includes data containers specific to the microarray experimental procedure and data analysis pipeline. Users can store information related to the procedures used to extract and quantify total RNA or DNA. Management of designed oligonucleotides is also possible in SaDA. The oligonucleotides module has an inbuilt taxonomy browser to assign taxonomy classification to the oligonucleotides. This particular feature helps in maintaining the classification of oligonucleotides which can be further categorized into hierarchical oligonucleotides.\n\nMicroarray data management and analysis \nA collection of microarray data can be viewed abstractly as a table with rows representing genes and columns representing various samples. Each position in the table describes the detection of a particular gene in a particular sample. We call this table a hybridization matrix. In SaDA, the information required to describe a microarray experiment can be divided conceptually into five logical parts which corresponds to raw and analyzed files. Raw data are images derived from scans of microarray following hybridization experiments. These images are analyzed to identify and quantify each spot of the array. There are at least three levels of data which are relevant to a microarray experiment:\n\nThe scanned images (raw data) available in images sub-module.\nThe quantitative outputs from the image analysis procedure (microarray quantitation matrices), available as GPR files that can be uploaded via Microarray .GPR section.\nThe derived measurements (data matrices) that could be viewed from Microarray Analysis section.\n\n\n\n\n\n\n\n\n\n Figure 3. (A) An image showing Microarray Experiments module. The display includes a unique microarray experiment and a list of oligonucleotides used in this particular microarray experiment; (B) Display of saved microarray experiments.\n\n\n\nSaDA summarizes multiple microarray experiments under Microarray experiments section (Figure 3). In this section, user can keep track of the sample utilized in each microarray experiment. The Microarray experiment section further contains other sections encapsulating data associated with GAL (array data) and GPR (hybridization data) files. Users can manually upload and download files, check the summary and perform downstream data analysis.\nIn addition to the sampling and experimental modules, SaDA has other data containers for storing users and project partners\u2019 related information. Access records are stored securely in encrypted form. SaDA provides access rights in three forms: view-only, access for editing and full edition and deletion rights.\n\nDatabase schema \nSaDA currently consists of 53 MySQL relations covering different scientific procedure and different analytical data types. Structuring and managing database schema in Ruby on Rails is easy and efficient.\n\nBatch uploads and downloads of data \nTo relieve researchers from the tedious task of entering data one by one, a set of batch file parsers were created. The upload of data files and images can be done asynchronously. The information generated during sampling procedures and oligonucleotides design can be uploaded in SaDA as plain comma-separated or tab-separated files. Files uploaded to SaDA are stored in a directory structure. The information within files is simultaneously stored in the database together with the meta-data for their quick and easy retrieval using SQL. Download of files is a straightforward approach via SaDA web interface. Download format is currently plain comma-separated or excel files.\n\nResults and discussion \nIn the present study we describe SaDA, a software to manage and analyze microarray-based datasets. There is no scarcity of freely available open source LIMS software solutions. Some of the LIMS-based solutions, for instance Base, iLAP, MARS, LabKey Server and PASSIM[22] are meaningful in big laboratories which, besides data analysis, are involved in protocol development, data acquisition and data sharing. Freely available tools for microarray analysis like TM4[23], ArrayNorm[24], ArrayPipe[25] and GPR-Analyzer[26] have either become obsolete or lack key features in terms of extensibility and integration with analysis tools, like R and modern graphing libraries. In the following sections we have described the key features of SaDA compared to any other light weighted domain specific LIMS. SaDA can be easily handled by a novice user. A working knowledge of system (Linux\/UNIX\/Mac OS\/Microsoft Windows) tools installation is required to install, set up and use SaDA. However, if a user wants to extend SaDA\u2019s functionalities and integrate new modules, knowledge of Ruby on Rails and R software programming is essential.\n\nUtility of the SaDA infrastructure \nSaDA is a collection of well-integrated set of models with a very flexible and intuitive user interface: SaDA is developed to address the needs of many laboratories which produce their own batch of microarray data. The database backed software architecture aids in managing and representing oligonucleotide hybridization based microarray data. The flexible and generic database design facilitates mapping of the steadily changing laboratories work-flow. To allow the import of generic file formats, we have implemented a user definable parser that allows reading comma\/tab delimited text files. Additionally, links to these files are maintained in a relational database to prevent the deletion of already imported, linked, or used files. The Microarray Gene Expression Data Society (MGED) develops and maintains standards for data acquisition, representation, and interchange such as the MIAME guidelines[27], the MAGE-TAB interchange format[28], and the MGED Ontology for microarray experiments.[29] SaDA does not enforce the use of the MGED standards. However it supports storage and retrieval of information required by MIAME. \nSaDA encapsulates open source software and graphing libraries: The usability of SaDA and the functionality of the interfaces are accounted by the inclusion of different models. These models encapsulate data from analytical pipelines of microarray analysis. To aid in fulfilling these functionalities, SaDA is supported by R programming software and jQWidgets[30] graphing libraries. SaDA utilizes RinRuby[31] gem to call R expressions\/functions\/script from Ruby on Rails interfaces. SaDA makes use of a jqGrid plugin[32] for a tabular data representation. \nSaDA facilitates comparative analysis of microarray results: SaDA implements an algorithm, running in the S2C server[33], for correlating microarray signal intensities with cell numbers from standard calibration plot to estimate cell counts. This way SaDA simplifies the prediction of cell counts from microarray data. With the implementation of flexible normalization procedures, it aids in the quantitative analysis and monitoring of microorganisms. In addition, the use of SaDA assists in making decisions entailing whether a cluster of species is below a set threshold and also in checking if poor hybridization quality or large proportion of nonspecific probe-targets interactions are responsible for negative results.\nSaDA is extensible: Scientists have a natural inclination for software tools that can be customized to the particular needs of their labs. Tools for rapid modification and customization have proven particularly important in both the adoption of SaDA and the dissemination of MicroAQUA project. The concise and expressive nature of Ruby programming language provides API that is easy to change. There is a less boilerplate and a huge library of gems, including BioRuby project[34], designed for rails applications covering everything from integrating with networking tools to authorization products. The modular nature of Rails framework also supports integration of new features quickly without touching the application\u2019s core components.\n\nLimitations of SaDA \nMost biomedical research applications are based on project specific guidelines. This limits the scope of any application to serve diverse purposes and creates boundaries of the platform that originates from its scientific approach. The main limitation encapsulating SaDA\u2019s core functionality is its scientific focus. Currently it only implements microarray data management and analysis modules targeting Agilent\u2019s single channel oligonucleotide hybridization technology. The current work does not aim to replace robust LIMS software solutions as it only targets a specific research study.\n\nAvailability and future directions \nSaDA is open source software and is freely available as a GitHub repository for download from https:\/\/github.com\/kumarsaurabh20\/bioaqua1 under the terms of the Apache License 2.0.[35] This site also provides documentation, tutorials and demos for users and developers, plus instructions for developers who wish to contribute code to the project through the SaDA GitHub repository. For its complete installation from scratch, we have developed a shell script which will install (for Linux based systems) all the components required for SaDA and clone SaDA automatically from the GitHub repositories. Since SaDA is a server application, it is platform independent and it can be run on any operating system. With some modifications, SaDA can also be made to run under other database environments. The other main requirement of SaDA is the working installation of R programming software. The shell script can detect the presence or absence of R software and in case it is not available on the system, the shell script will install it with the user permission. The future version will include handling and analysis of metagenomic data. The first implementation of SaDA was developed in the context of the EU FP7 project MicroAQUA which aimed at developing a universal microarray chip for the detection of fresh water pathogens. The data analysis steps in SaDA comply with the guidelines provided by MicroAQUA consortium.\nThere are many data-transformation and analysis algorithms that are feasible for integration in SaDA as an extended features and external modules. SaDA is developed keeping ecological studies and environmental monitoring in mind. Active development is underway to support other datasets which prove useful in this particular field of interest such as flow-cytometry datasets and their analytical representation, for the detection of toxins in the water based environmental systems. In addition, we are planning to extend the core functionality of SaDA to include metagenomic based data. Use of these standard technologies and datasets will allow interoperability of SaDA with other software. A major goal for SaDA development is the addition of support for other types of ecological high-throughput data. Development will be required to support the emergence of new arrays and platforms, particularly with respect to integrating results across different generations, and possibly different technologies. Support for experiments based on metagenomic data is the next immediate priority. These datasets enable high-throughput genome-scale investigations of interactions between genome and its surrounding environment. This extension will require the incorporation of new data analysis methods. An interface for combining microarray, metagenomics, proteomics and interaction based data within SaDA is already planned.\n\nConclusions \nWe have developed an integrated system consisting of a database that has been tailored to serve specific needs of microarray based environmental monitoring projects. Due to unique fusion of open source tools and libraries, the system can provide a model of building up a similar platform for other emerging genomics technologies. This utility enables users to collect, manipulate and analyze diverse and rapidly changing environmental data-set and their related biological data-types. The system allows easy customization of interfaces and visualization, thus providing a useful platform for collaborative research activities. Implementation of SaDA by the members of MicroAQUA consortium over a period of about three years where water sampling data, molecular analysis and high-throughput microarray experiments have been carried out, has helped to fine-tune the features of this software to suit the needs of a wide range of researchers. The successful application of SaDA suggests the platform has the potential to become an appropriate tool for a wide range of users focused not only in water based environmental research but also in other studies aimed at exploring and analyzing complex ecological habitats. \n\nSupplementary materials \nGraphical workflows for SaDA\u2019s core modules are available as supplementary materials and can be accessed at: http:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/.\n\nAcknowledgments \nThe authors would like to acknowledge the contributions of partners of the MicroAQUA project for their valuable inputs on the design of user interfaces. The work was financially supported by funds from the European Commission 7th Framework to the project MicroAQUA (KBBE-2010-3-2-04: Innovative aquatic biosensors\u2014Medium Collaborative Project).\n\nAuthor contributions \nKumar Saurabh Singh was the main responsible for the scientific outcome of the article and development of SaDA. Dominique Thual contributed software code in initial sections of SaDA in Rails version 2.3.11 which was later upgraded to Rails version 3.2.13 by Kumar Saurabh Singh. Nicola Cannata was involved in general design guidelines of SaDA. Roberto Spurio was the general coordinator of the work. Kumar Saurabh Singh drafted the manuscript. All authors have read and approved this manuscript.\n\nConflicts of interest \nThe authors declare no conflict of interest.\n\nReferences \n\n\n\u2191 R\u00fcegg, J.; Gries, C.; Bond-Lamberty, B.; Bowen, G.J.; Felzer, B.S.; McIntyre, N.E.; Soranno, P.; Vanderbilt, K.L.; Weathers, K.C. (2014). \"Completing the data life cycle: Using information management in macrosystems ecology research\". Frontiers in Ecology and the Environment 12 (1): 24\u201330. doi:10.1890\/120375. http:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/htm .   \n\n\u2191 Heffernan, J.B.; Soranno, P.; Angilletta, M.J.; Buckley, L.B.; Gruner, D.S.; Keitt, T.H.; Kellner, J.R.; Kominoski, J.S.; Rocha, A.V.; Xiao, J. et al. (2014). \"Macrosystems ecology: Understanding ecological patterns and processes at continental scales\". Frontiers in Ecology and the Environment 12 (1): 5\u201314. doi:10.1890\/130017. http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/130017 .   \n\n\u2191 Goring, S.J.; Weathers, K.C.; Dodds, W.K.; Soranno, P.; Sweet, L.C.; Cheruvelil, K.S.; Kominoski, J.S.; R\u00fcegg, J.; Thorn, A.M.; Utz, R.M. (2014). \"Improving the culture of interdisciplinary collaboration in ecology by expanding measures of success\". Frontiers in Ecology and the Environment 12 (1): 39\u201347. doi:10.1890\/120370. http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/120370 .   \n\n\u2191 Bushmann, F.; Meunier, R.; Rohnert, H.; Sommerlad, P.; Stal, M. (1996). Pattern-Oriented Software Architecture: A System of Patterns. 1. West Sussex, England: John Wiley & Sons Ltd. pp. 476. ISBN 9780471958697. http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0471958697.html .   \n\n\u2191 Maurer, M.; Molidor, R.; Sturn, A.; Hartler, J.; Hackl, H.; Stocker, G.; Prokesch, A.; Scheideler, M.; Trajanoski, Z. (2005). \"MARS: Microarray analysis, retrieval, and storage system\". BMC Bioinformatics 6: 101. doi:10.1186\/1471-2105-6-101. PMC PMC1090551. PMID 15836795. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551 .   \n\n\u2191 Stocker, G.; Fischer, M.; Rieder, D.; Bindea, G.; Kainz, S.; Oberstolz, M.; McNally, J.G.; Trajanoski, Z. (2009). \"iLAP: A workflow-driven software for experimental protocol development, data acquisition and analysis\". BMC Bioinformatics 10: 390. doi:10.1186\/1471-2105-10-390. PMC PMC2789074. PMID 19941647. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2789074 .   \n\n\u2191 Vallon-Christersson, J.; Nordborg, N.; Svensson, M.; H\u00e4kkinen, J. (2009). \"BASE - 2nd generation software for microarray data management and analysis\". BMC Bioinformatics 10: 330. doi:10.1186\/1471-2105-10-330. PMC PMC2768720. PMID 19822003. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2768720 .   \n\n\u2191 Nelson, E.K.; Piehler, B.; Eckels, J.; Rauch, A.; Bellew, M.; Hussey, P.; Ramsay, S.; Nathe, C.; Lum, K.; Krouse, K; Stearns, D.; Connolly, B.; Skillman, T.; Igra, M. (2011). \"LabKey Server: An open source platform for scientific data integration, analysis and collaboration\". BMC Bioinformatics 12 (71). doi:10.1186\/1471-2105-12-71. PMC PMC3062597. PMID 21385461. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597 .   \n\n\u2191 Nix, D.; di Sera, T.L.; Dalley, B.K.; Milash, B.; Cundick, R.M.; Quinn, K.S.; Courdy, S.J. (2010). \"Next generation tools for genomic data generation, distribution, and visualization\". BMC Bioinformatics 11 (455). doi:10.1186\/1471-2105-11-455. PMC PMC2944281. PMID 20828407. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281 .   \n\n\u2191 Colmsee, C.; Flemming, S.; Klapperst\u00fcck, M.; Lange, M.; Scholz, U. (2011). \"A case study for efficient management of high throughput primary lab data\". BMC Research Notes 4: 413. doi:10.1186\/1756-0500-4-413. PMC PMC3217054. PMID 22005096. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3217054 .   \n\n\u2191 Bauch, A.; Adamczyk, I.; Buczek, P.; Elmer, F.-J.; Enimanev, K.; Glyzewski, P.; Kohler, M.; Pylak, T.; Quandt, A.; Ramakrishnan, C.; Beisel, C.; Malmstr\u00f6m, L.; Aebersold, R.; Rinn, B. (2011). \"openBIS: a flexible framework for managing and analyzing complex data in biology research\". BMC Bioinformatics 12: 468. doi:10.1186\/1471-2105-12-468. PMC PMC3275639. PMID 22151573. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639 .   \n\n\u2191 Venco, F.; Ceol, A.; Muller, H. (2013). \"SLIMS: A LIMS for handling next-generation sequencing workflows\". EMBnet.journal 19 (B): 85\u201387. doi:10.14806\/ej.19.B.739. http:\/\/journal.embnet.org\/index.php\/embnetjournal\/article\/view\/739 .   \n\n\u2191 McLellan, A.S.; Dubin, R.; Jing, Q.; Broin, P.\u00d3.; Moskowitz, D.; Suzuki, M.; Calder, R.B.; Hargitai, J.; Golden, A.; Greally, J.M. (2012). \"The Wasp System: An open source environment for managing and analyzing genomic data\". Genomics 100 (6): 345-51. doi:10.1016\/j.ygeno.2012.08.005. PMID 22944616.   \n\n\u2191 Giardine, B.; Riemer, C.; Hardison, R.C.; Burhans, R.; Elnitski, L.; Shah, P.; Zhang, Y.; Blankenberg, D.; Albert, I.; Taylor, J.; Miller, W.; Kent, W.J.; Nekrutenko, A. (2005). \"Galaxy: A platform for interactive large-scale genome analysis\". Genome Research 15 (10): 1451\u20131455. doi:10.1101\/gr.4086505. PMC PMC1240089. PMID 16169926. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089 .   \n\n\u2191 Scholtalbers, J.; R\u00f6ssler, J.; Sorn, P.; de Graaf, J.; Boisgu\u00e9rin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". Bioinformatics 29 (9): 1233-1234. doi:10.1093\/bioinformatics\/btt115. PMID 23479349.   \n\n\u2191 \"Sequencing Documentation\". CASAVA User Guide. Illumina, Inc. http:\/\/support.illumina.com\/sequencing\/documentation.html . Retrieved 02 June 2015 .   \n\n\u2191 Van Rossum, T.; Tripp, B.; Daley, D. (2010). \"SLIMS\u2014A user-friendly sample operations and inventory management system for genotyping labs\". Bioinformatics 26 (14): 1808\u201310. doi:10.1093\/bioinformatics\/btq271. PMC PMC2894515. PMID 20513665. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515 .   \n\n\u2191 Protopsaltou, A. (2007) (PDF). Model Driven Development with Ruby on Rails Ruby on Rails and Reverse Engineering. G\u00f6teborg, Sweden: IT University of G\u00f6teborg. pp. 1\u201334. https:\/\/gupea.ub.gu.se\/bitstream\/2077\/4647\/1\/ISOFM_ITU870_V07_Protopsaltou,Antonios_Model%20Driven%20Development%20with%20Ruby%20on%20Rails.pdf .   \n\n\u2191 Rosen, L.S.R. (2009). Web Application Architecture: Principles, Protocol and Practices. West Sussex, England: John Wiley & Sons Ltd. pp. 440. https:\/\/books.google.com\/books?id=gcHJQwAACAAJ .   \n\n\u2191 \"Welcome to MicroAqua\". MicroAqua Project Consortium. Archived from the original on 14 February 2015. http:\/\/web.archive.org\/web\/20150214182305\/http:\/\/www.microaqua.eu\/ . Retrieved 02 June 2015 .   \n\n\u2191 \"A port of the YM4R-GM plugin for rails, to github\". GitHub. https:\/\/github.com\/queso\/ym4r-gm . Retrieved 02 June 2015 .   \n\n\u2191 Viksna, J.; Celms, E.; Opmanis, M.; Podnieks, K.; Rucevskis, P.; Zarins, A.; Barrett, A.; Neogi, S.G.; Krestyaninova, M.; McCarthy, M.I. et al. (2007). \"PASSIM\u2014An open source software system for managing information in biomedical studies\". BMC Bioinformatics 8: 52. doi:10.1186\/1471-2105-8-52. PMC PMC1803798. PMID 17291344. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798 .   \n\n\u2191 Saeed, A.I.; Sharov, V.; White, J.; Li, J.; Liang, W.; Bhagabati, N.; Braisted, J.; Klapa, M.; Currier, T.; Thia-garajan, M. et al. (2003). \"TM4: A free, open-source system for microarray data management and analysis\". BioTechniques 34 (2): 374-8. PMID 12613259.   \n\n\u2191 Pieler, R.; Sanchez-Cabo, F.; Hackl, H.; Thallinger, G.G.; Trajanoski, Z. (2004). \"ArrayNorm: Comprehensive normalization and analysis of microarray data\". Bioinformatics 20 (12): 1971-3. doi:10.1093\/bioinformatics\/bth174. PMID 15073026.   \n\n\u2191 Hokamp, K.; Roche, F.M.; Acab, M.; Rousseau, M.E.; Kuo, B.; Goode, D.; Aeschliman, D.; Bryan, J.; Babiuk, L.; Hancock, R.E.W. et al. (2004). \"ArrayPipe: A flexible processing pipeline for microarray data\". Nucleic Acids Research 32 (Suppl 2): W457-W459. doi:10.1093\/nar\/gkh446. PMC PMC441584. PMID 15215429. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC441584 .   \n\n\u2191 Dittami, S.M.; Edvardsen, B. (2013). \"GPR-Analyzer: A simple tool for quantitative analysis of hierarchical multispecies microarrays\". Environmental Science and Pollution Research 20 (10): 6808-15. doi:10.1007\/s11356-012-1051-5. PMID 22767354.   \n\n\u2191 Yilmaz, P.; Kottmann, R.; Field, D.; Knight, R.; Cole, J.R.; Amaral-Zettler, L.; Gilbert, J.; Karsch-Mizrachi, I.; Johnston, A.; Cochrane, G. et al. (2011). \"Minimum information about a marker gene sequence (MIMARKS) and minimum information about any (x) sequence (MIxS) specifications\". Nature Biotechnology 29 (5): 415-20. doi:10.1038\/nbt.1823. PMC PMC3367316. PMID 21552244. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3367316 .   \n\n\u2191 Rayner, T.F.; Rocca-Serra, P.; Spellman, P.T.; Causton, H.C.; Farne, A.; Holloway, E.; Irizarry, R.; Liu, J.; Maier, D.S.; Miller, M. et al. (2006). \"A simple spreadsheet-based, MIAME-supportive format for microarray data: MAGE-TAB\". BMC Bioinformatics 7: 489. doi:10.1186\/1471-2105-7-489. PMC PMC1687205. PMID 17087822. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1687205 .   \n\n\u2191 \"MAGE-TAB Specification Version 1.1\" (PDF). Functional Genomics Data Society. 31 May 2011. http:\/\/fged.org\/site_media\/pdf\/MAGE-TABv1.1_2011_07_28.pdf . Retrieved 02 June 2015 .   \n\n\u2191 \"jQWidgets: Advanced jQuery and HTML5 UI framework\". jQWidgets Ltd. http:\/\/www.jqwidgets.com\/ . Retrieved 02 June 2015 .   \n\n\u2191 Dahl, D.B.; Crawford, S. (2009). \"RinRuby: Accessing the R interpreter from pure ruby\". Journal of Statistical Software 29 (4). doi:10.18637\/jss.v029.i04. http:\/\/www.jstatsoft.org\/article\/view\/v029i04 .   \n\n\u2191 \"jQuery Grid Plugin - jqGrid\". TriRand Ltd. http:\/\/www.trirand.com\/blog\/ . Retrieved 02 June 2015 .   \n\n\u2191 \"PlotApp - S2C implementation on rails using RinRuby ruby script\". GitHub. https:\/\/github.com\/kumarsaurabh20\/PlotApp . Retrieved 02 June 2015 .   \n\n\u2191 Mitsuteru, N.G.; Mitsuteru, N.G.; Goto, N.; Goto, N.; Nakao, M.C.; Nakao, M.C.; Kawashima, S.; Kawashima, S.; Katayama, T.; Katayama, T. et al. (2003). \"BioRuby: Open-source bioinformatics library\" (PDF). Genome Informatics 14: 629-630. https:\/\/www.jsbi.org\/pdfs\/journal1\/GIW03\/GIW03P191.pdf .   \n\n\u2191 \"Apache License\". Apache Software Foundation. January 2004. http:\/\/www.apache.org\/licenses\/LICENSE-2.0 . Retrieved 02 June 2015 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\">https:\/\/www.limswiki.org\/index.php\/Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on environmental informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 21:00.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,202 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","9847a6beea0bba9383335c5a41e65647_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_SaDA_From_sampling_to_data_analysis\u2014An_extensible_open_source_infrastructure_for_rapid_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:SaDA: From sampling to data analysis\u2014An extensible open source infrastructure for rapid, robust and automated management and analysis of modern ecological high-throughput microarray data<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>One of the most crucial characteristics of day-to-day <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> information management is the collection, storage and retrieval of <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> about research subjects and environmental or biomedical samples. An efficient link between sample data and experimental results is absolutely important for the successful outcome of a collaborative project. Currently available software solutions are largely limited to large scale, expensive commercial <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">Laboratory Information Management Systems<\/a> (LIMS). Acquiring such LIMS indeed can bring laboratory information management to a higher level, but most of the times this requires a sufficient investment of money, time and technical efforts. There is a clear need for a light weighted open source system which can easily be managed on local servers and handled by individual researchers. Here we present a software named SaDA for storing, retrieving and analyzing data originated from microorganism monitoring experiments. SaDA is fully integrated in the management of environmental samples, oligonucleotide sequences, microarray data and the subsequent downstream analysis procedures. It is simple and generic software, and can be extended and customized for various environmental and biomedical studies.\n<\/p><p><b>Keywords:<\/b> software; data management; microarrays; ecological assessment; environmental studies; LIMS; open source system\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>To gain an understanding of complex microbial environments, researchers have to assemble different type of information originating from heterogeneous sources. Every bit of information is valuable and should be made available whenever it is required. An exhaustive and clear protocol for data collection, processing and analysis is critical to achieve the projects end goals. The rapid increase in the discovery and application of molecular techniques constitutes important steps towards the detection and identification of microorganisms present in complex environmental samples. Identification and characterization of microorganisms is a key part of the management of environment and quality of life, tracing contaminants and troubleshooting problems such as pathogenesis or pollution indication.<sup id=\"rdp-ebb-cite_ref-RueggComp14_1-0\" class=\"reference\"><a href=\"#cite_note-RueggComp14-1\" rel=\"external_link\">[1]<\/a><\/sup> Identification of an unknown species can help to assess whether it poses a safety concern or not. Generally, the process of bio-monitoring plays a crucial role in the execution of large scale projects which focus either on the protection of people\u2019s health, safety of materials or dealing with environment quality. Big ecology related projects address very complicated issues.<sup id=\"rdp-ebb-cite_ref-HeffernanMacro14_2-0\" class=\"reference\"><a href=\"#cite_note-HeffernanMacro14-2\" rel=\"external_link\">[2]<\/a><\/sup> They require generation and collection of data from a wide spectrum of sources, collaboration of people from different disciplines and the application of highly complex analytical approaches.<sup id=\"rdp-ebb-cite_ref-GoringImp14_3-0\" class=\"reference\"><a href=\"#cite_note-GoringImp14-3\" rel=\"external_link\">[3]<\/a><\/sup> Moreover, information management is important especially when a large volume of complex data is collected. It becomes even more important when multiple organizations, at different geographical locations, are involved in the data collection procedures. It is absolutely essential that data are collected and managed using the methods recognized by scientific community. Existing software systems do not typically overlap the full pipeline as far as ecological collaborative projects are concerned. They are limited by data types, provide limited extensibility or require additional skills to handle data which are beyond the scope of experimental researchers. We developed Sampling to Data Analysis (SaDA) keeping in mind the requirements of individual laboratory involved in big collaborative projects. The system is a complete portal for ecology related high-throughput data management and analyses. Extra care was taken in designing database schema, information retrieval and in data presentation. SaDA is customizable and extensible to meet the needs of diverse research activities as the source code is freely available. The core of SaDA is based on model dependent development where abstract models of software systems are created and systematically transformed to its implementation.<sup id=\"rdp-ebb-cite_ref-BushmannPatern96_4-0\" class=\"reference\"><a href=\"#cite_note-BushmannPatern96-4\" rel=\"external_link\">[4]<\/a><\/sup> The current version of SaDA supports data management and analysis protocols involved in fresh water monitoring systems utilizing microarray technology as a state-of-the-art technology. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"State-of-art_technology\">State-of-art technology<\/span><\/h3>\n<p>To the best of our knowledge no freely available software is focused on microarray data management and analysis in terms of its ecological application, as in environmental monitoring. The aim of SaDA is to provide a streamlined work-flow from environmental sample collection and its downstream analysis. Microarray technology, however, is used here as a state-of-the-art technology to detect and quantify pathogens and other microorganisms in environmental freshwater samples. SaDA currently supports Agilent\u2019s one channel oligonucleotide hybridization microarray technology. In addition, the downstream data analysis detects the presence or absence of signals, normalization and estimating cell counts from the micro-array signal intensities. SaDA is not a tool for gene expression analysis. Although existing software tools could meet some of the requirements of SaDA, none meet all of them in the form of a comprehensive, end-to-end platform available as open source and extensible system. Some tools have experienced only limited use. The state of art in Laboratory Information Management System (LIMS) provides all aspect of microarray data management and analysis, using advanced automation and precise robotic control. This type of integrated solutions ensures high quality and reliable data output and enables multiple projects to be managed in parallel. LIMS are mainly commercial and expensive software products. Few commercial systems lack clarity of open source systems, come at considerable costs, and requires high level of technical expertise to install and configure hardware, software and database and is not discussed in the present work. \n<\/p><p>MARS<sup id=\"rdp-ebb-cite_ref-MaurerMARS05_5-0\" class=\"reference\"><a href=\"#cite_note-MaurerMARS05-5\" rel=\"external_link\">[5]<\/a><\/sup> provides a comprehensive MIAME supportive suite for storing, retrieving, and analyzing multi-color microarray data. <a href=\"https:\/\/www.limswiki.org\/index.php\/Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\" title=\"Journal:ILAP: A workflow-driven software for experimental protocol development, data acquisition and analysis\" target=\"_blank\" class=\"wiki-link\" data-key=\"59816fe6940deb7e43cf53893a6ea71a\">iLAP<\/a><sup id=\"rdp-ebb-cite_ref-Stocker_iLAP09_6-0\" class=\"reference\"><a href=\"#cite_note-Stocker_iLAP09-6\" rel=\"external_link\">[6]<\/a><\/sup>, based on a workflow driven modular architecture, was developed specifically to create and manage experimental protocols and to analyze and share laboratory data. <a href=\"https:\/\/www.limswiki.org\/index.php\/BASE\" title=\"BASE\" target=\"_blank\" class=\"wiki-link\" data-key=\"36231d20d7206b3d9d0ca18dff0aac02\">Base<\/a><sup id=\"rdp-ebb-cite_ref-VallonBASE09_7-0\" class=\"reference\"><a href=\"#cite_note-VallonBASE09-7\" rel=\"external_link\">[7]<\/a><\/sup> has been proved to be a comprehensive Microarray data repository, providing researchers an efficient data management and analysis tool. <a href=\"https:\/\/www.limswiki.org\/index.php\/LabKey_Server\" title=\"LabKey Server\" target=\"_blank\" class=\"wiki-link\" data-key=\"856c6fe009646beaaed7ecd21dda4932\">Labkey Server<\/a><sup id=\"rdp-ebb-cite_ref-NelsonLabkey11_8-0\" class=\"reference\"><a href=\"#cite_note-NelsonLabkey11-8\" rel=\"external_link\">[8]<\/a><\/sup> is a complete system with easy to use interfaces for specimen request submission across collaborators, allowing users to graphically define new data-types to fit in diverse datasets. It interacts dynamically with external data sources and supports developing custom interfaces using client libraries. <a href=\"https:\/\/www.limswiki.org\/index.php\/GNomEx\" title=\"GNomEx\" target=\"_blank\" class=\"wiki-link\" data-key=\"ffe2672873462be87660d574029bf8b2\">GnomEx<\/a><sup id=\"rdp-ebb-cite_ref-NixNext10_9-0\" class=\"reference\"><a href=\"#cite_note-NixNext10-9\" rel=\"external_link\">[9]<\/a><\/sup> provides comprehensive automated protocols for sample submission, sample tracking, billing, data management and analysis workflows for both Next Generation Sequencing (NGS) and Microarrays. LIMS Light System<sup id=\"rdp-ebb-cite_ref-ColmseeACase11_10-0\" class=\"reference\"><a href=\"#cite_note-ColmseeACase11-10\" rel=\"external_link\">[10]<\/a><\/sup> has been developed to meet the demands of high-throughput primary data storage and data upload\/retrieval. This system provides an efficient storage system with a Hierarchical Storage Management System and an Oracle relational database. <a href=\"https:\/\/www.limswiki.org\/index.php\/OpenBIS\" title=\"OpenBIS\" target=\"_blank\" class=\"wiki-link\" data-key=\"3bbca2b6be0c273677fc99a9474e49ec\">OpenBis<\/a><sup id=\"rdp-ebb-cite_ref-Bauch_openBIS11_11-0\" class=\"reference\"><a href=\"#cite_note-Bauch_openBIS11-11\" rel=\"external_link\">[11]<\/a><\/sup> is a flexible framework that has been adapted to be used for proteomics, high content screening and NGS projects. The <a href=\"https:\/\/www.limswiki.org\/index.php\/SMITH\" title=\"SMITH\" target=\"_blank\" class=\"wiki-link\" data-key=\"de4e2a9e63484e867d5d844c95afdedf\">SMITH<\/a><sup id=\"rdp-ebb-cite_ref-VencoSLIMS13_12-0\" class=\"reference\"><a href=\"#cite_note-VencoSLIMS13-12\" rel=\"external_link\">[12]<\/a><\/sup> and WASP<sup id=\"rdp-ebb-cite_ref-McLellanTheWasp12_13-0\" class=\"reference\"><a href=\"#cite_note-McLellanTheWasp12-13\" rel=\"external_link\">[13]<\/a><\/sup> system has been developed to meet the demands of NGS experiments and clinical tests. They provide embedded pipelines for the analysis of ChIP-Seq, RNA-Seq, miRNA-Seq and Exome-Seq experiments. There is also an extension with LIMS functionality to the popular <a href=\"https:\/\/www.limswiki.org\/index.php\/Galaxy_(biomedical_software)\" title=\"Galaxy (biomedical software)\" target=\"_blank\" class=\"wiki-link\" data-key=\"ead5d6ebaa8d67744d2f68d454d89ce6\">Galaxy<\/a><sup id=\"rdp-ebb-cite_ref-GiardineGalaxy05_14-0\" class=\"reference\"><a href=\"#cite_note-GiardineGalaxy05-14\" rel=\"external_link\">[14]<\/a><\/sup> workflow engine called Galaxy LIMS.<sup id=\"rdp-ebb-cite_ref-ScholtalbersGal13_15-0\" class=\"reference\"><a href=\"#cite_note-ScholtalbersGal13-15\" rel=\"external_link\">[15]<\/a><\/sup> The system supports request submission, offers assistance during flow cell layout, and automatically launches Illumina\u2019s CASAVA software<sup id=\"rdp-ebb-cite_ref-CASAVA_16-0\" class=\"reference\"><a href=\"#cite_note-CASAVA-16\" rel=\"external_link\">[16]<\/a><\/sup> to perform de-multiplexing and delivery of the user-specific NGS raw files. Being integrated with Galaxy, the data are automatically available to be processed by analysis pipelines stored in Galaxy. SLIMS<sup id=\"rdp-ebb-cite_ref-VanRossumSLIMS10_17-0\" class=\"reference\"><a href=\"#cite_note-VanRossumSLIMS10-17\" rel=\"external_link\">[17]<\/a><\/sup> is a sample management tool for genotyping laboratories. \n<\/p><p>An open-source LIMS must be utilized to the specific infrastructure and the project needs of a research institution, which are in constant evolution. Therefore, the LIMS system must be modified accordingly. The effort required to tweak an existing LIMS and gain significant insight into its code base in order to be able to adjust it in a productive way must be compared with the effort of developing a new system. Large number of available LIMS software and tools substantiate the notion of creating a new system against tweaking an existing one. From this perspective, it seems that a simple LIMS has more chances of being shared than a more complex one. In any case, there is no LIMS that serves all needs in a simple turn-key solution. SaDA is not properly a LIMS based software solution but it provides a platform for model driven data management and analysis, and is flexible enough to include new features quickly and efficiently.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Material_and_methods\">Material and methods<\/span><\/h2>\n<p>SaDA is a web application implemented in <a href=\"https:\/\/www.limswiki.org\/index.php\/Ruby_on_Rails\" title=\"Ruby on Rails\" target=\"_blank\" class=\"wiki-link\" data-key=\"bec38424b2640687620b42397f4e7c52\">Ruby on Rails<\/a> that runs on Phusion passenger application server and stores its data in a directory structure with in SaDA for quick file manipulations and also in a relational database engine for efficient and quick data querying and retrieval. The current version of SaDA is supported on Linux, UNIX, Mac OS and Microsoft Windows.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h3>\n<p>SaDA has been developed using the Ruby on Rails web application development framework, which is based on the Model-View-Controller (MVC) architectural pattern<sup id=\"rdp-ebb-cite_ref-ProtopsaltouModel07_18-0\" class=\"reference\"><a href=\"#cite_note-ProtopsaltouModel07-18\" rel=\"external_link\">[18]<\/a><\/sup> of software development. MVC partition the application into multiple components. As described schematically in Figure 1, application input, output and processing (application server and application layer) are compartmentalized in two layers. Application\u2019s core functionalities and database are encapsulated in layer 2, also described as \u201cdatabase layer\u201d in Figure 1. The modular architecture of SaDA supports different features to be encapsulated as individual models, shown as ellipses Figure 1. Sharp rectangles are individual sections encompassing models with associations and relationships. For example, Sampling sites can have different features like land mapping, geographical positions, altitude type, water type & kind of water in use (for fresh water environments). These features are individual relations\/tables associated with each other and also with features of different sections in the SaDA database. Database layer of SaDA is independent of input behavior and output data representation and forms the Model of MVC pattern. User interfaces and application server are components of layer 1. This part handles most of the input, output and query processing operations. It sends query to database and receives the data which in turn is displayed on SaDA user interfaces. The user interacts with the system solely through layer 1. The separation of the software layers allows freedom from the one-table-one-view paradigm. Therefore SaDA can display multiple interfaces of the same model allowing display of same data in multiple ways. Layer 1, in Figure 1, also includes an application server layer. The main job of this application layer is to support the construction of dynamic pages by handling the application\u2019s program logic. Additionally, it also handles database connection and data transfers between client and application in a smooth and efficient way. External tools and libraries are used in SaDA to enhance some of the SaDA\u2019s core functionalities described in later sections of the paper.\n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_SinghBMCBioinformatics2015_12-6.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"df577a157d00a02ee8a2d910e7d5065d\"><img alt=\"Fig1 SinghBMCBioinformatics2015 12-6.png\" src=\"https:\/\/www.limswiki.org\/images\/8\/88\/Fig1_SinghBMCBioinformatics2015_12-6.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1.<\/b> SaDA architecture: A multilayer architecture, comprising of layer 1 for data upload, processing and retrieval and a RDBMS storage layer with data access control, metadata storage for fast data retrieval. SaDA, in addition, has an external component for third party tools and libraries.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Experimental_design_and_data_acquisition\">Experimental design and data acquisition<\/span><\/h3>\n<p>Archiving detailed information on collection and analysis of samples is imperative for both efficient reporting of any study and for successive data analysis.<sup id=\"rdp-ebb-cite_ref-RosenWeb09_19-0\" class=\"reference\"><a href=\"#cite_note-RosenWeb09-19\" rel=\"external_link\">[19]<\/a><\/sup> Collecting and storing these information is of utmost importance in terms of its application in high-throughput technologies. SaDA currently supports MySQL as a back end relational database management system (RDBMS). It can also support any other available RDBMS like SQLite and <a href=\"https:\/\/www.limswiki.org\/index.php\/PostgreSQL\" title=\"PostgreSQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"a5dd945cdcb63e2d8f7a5edb3a896d82\">PostgreSQL<\/a>. This kind of architecture permits multilevel security, as the application can be made secured at both application level and database level. SaDA provides a secure solution by implementing proper authentication control to each research participant. The database level security can be achieved by providing user level privileges. The system is web-based, which means that non-identifiable information is kept on a server and can be securely accessed on-line for queries or new submissions by authorized users via a web-browser. SaDA is simple and generic, and thus can be customized for various types of ecological as well as biological studies.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_SinghBMCBioinformatics2015_12-6.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"d61feb798e8f36dbb6e8ad4cdd632671\"><img alt=\"Fig2 SinghBMCBioinformatics2015 12-6.png\" src=\"https:\/\/www.limswiki.org\/images\/a\/a1\/Fig2_SinghBMCBioinformatics2015_12-6.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2.<\/b> Display of Sampling Sites related data within SaDA.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"The_features_of_SaDA\">The features of SaDA<\/span><\/h3>\n<p>SaDA was originally developed on the guidelines provided by the MicroAQUA consortium.<sup id=\"rdp-ebb-cite_ref-MicroAqua_20-0\" class=\"reference\"><a href=\"#cite_note-MicroAqua-20\" rel=\"external_link\">[20]<\/a><\/sup> The source code is now available to the public. Due to its modular framework (Figure 1), SaDA can be easily modified and applied to any environmental related projects. The current implementation of SaDA includes different modules storing data from sample collection, DNA\/RNA extraction, oligonucleotides annotation, and microarrays raw data storage and validation results.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Samples_and_filter_tracking\">Samples and filter tracking<\/span><\/h4>\n<p>Users can track sample related information like quantity of samples and instrument utilized in sampling. From this point, the application assigns a unique identifier to the sample and provides the ability to be cross referenced later in other modules. Sampling sites, in addition, acts as one of the attributes of sampling procedure. Therefore SaDA integrates a Google Maps plug-in<sup id=\"rdp-ebb-cite_ref-YM4R_21-0\" class=\"reference\"><a href=\"#cite_note-YM4R-21\" rel=\"external_link\">[21]<\/a><\/sup> for displaying sample collection sites (Figure 2) dynamically. Filter sample module saves information about the filter used to concentrate the collected water sample. A unique identifier specifies the filter associated with sampling and partner code. It carries information like pore size of the filter, number of filter tubes utilized and volume of water sample filtered.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Management_of_project_specific_experimental_data\">Management of project specific experimental data<\/span><\/h4>\n<p>SaDA supports oligonucleotide microarrays as a fresh water environment monitoring technology. SaDA includes data containers specific to the microarray experimental procedure and data analysis pipeline. Users can store information related to the procedures used to extract and quantify total RNA or DNA. Management of designed oligonucleotides is also possible in SaDA. The oligonucleotides module has an inbuilt taxonomy browser to assign taxonomy classification to the oligonucleotides. This particular feature helps in maintaining the classification of oligonucleotides which can be further categorized into hierarchical oligonucleotides.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Microarray_data_management_and_analysis\">Microarray data management and analysis<\/span><\/h4>\n<p>A collection of microarray data can be viewed abstractly as a table with rows representing genes and columns representing various samples. Each position in the table describes the detection of a particular gene in a particular sample. We call this table a hybridization matrix. In SaDA, the information required to describe a microarray experiment can be divided conceptually into five logical parts which corresponds to raw and analyzed files. Raw data are images derived from scans of microarray following hybridization experiments. These images are analyzed to identify and quantify each spot of the array. There are at least three levels of data which are relevant to a microarray experiment:\n<\/p>\n<ol type=\"I\"><li>The scanned images (raw data) available in images sub-module.<\/li>\n<li>The quantitative outputs from the image analysis procedure (microarray quantitation matrices), available as GPR files that can be uploaded via Microarray .GPR section.<\/li>\n<li>The derived measurements (data matrices) that could be viewed from Microarray Analysis section.<\/li><\/ol>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_SinghBMCBioinformatics2015_12-6.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"8b6981cbb5a2e73f1587b9925b012a8e\"><img alt=\"Fig3 SinghBMCBioinformatics2015 12-6.png\" src=\"https:\/\/www.limswiki.org\/images\/c\/c0\/Fig3_SinghBMCBioinformatics2015_12-6.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3.<\/b> <b>(A)<\/b> An image showing Microarray Experiments module. The display includes a unique microarray experiment and a list of oligonucleotides used in this particular microarray experiment; <b>(B)<\/b> Display of saved microarray experiments.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>SaDA summarizes multiple microarray experiments under <i>Microarray experiments<\/i> section (Figure 3). In this section, user can keep track of the sample utilized in each microarray experiment. The <i>Microarray experiment<\/i> section further contains other sections encapsulating data associated with GAL (array data) and GPR (hybridization data) files. Users can manually upload and download files, check the summary and perform downstream data analysis.\n<\/p><p>In addition to the sampling and experimental modules, SaDA has other data containers for storing users and project partners\u2019 related information. Access records are stored securely in encrypted form. SaDA provides access rights in three forms: view-only, access for editing and full edition and deletion rights.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Database_schema\">Database schema<\/span><\/h3>\n<p>SaDA currently consists of 53 <a href=\"https:\/\/www.limswiki.org\/index.php\/MySQL\" title=\"MySQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"35005451bfcd508bce47c58e72260128\">MySQL<\/a> relations covering different scientific procedure and different analytical data types. Structuring and managing database schema in Ruby on Rails is easy and efficient.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Batch_uploads_and_downloads_of_data\">Batch uploads and downloads of data<\/span><\/h3>\n<p>To relieve researchers from the tedious task of entering data one by one, a set of batch file parsers were created. The upload of data files and images can be done asynchronously. The information generated during sampling procedures and oligonucleotides design can be uploaded in SaDA as plain comma-separated or tab-separated files. Files uploaded to SaDA are stored in a directory structure. The information within files is simultaneously stored in the database together with the meta-data for their quick and easy retrieval using SQL. Download of files is a straightforward approach via SaDA web interface. Download format is currently plain comma-separated or excel files.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<p>In the present study we describe SaDA, a software to manage and analyze microarray-based datasets. There is no scarcity of freely available open source LIMS software solutions. Some of the LIMS-based solutions, for instance Base, iLAP, MARS, LabKey Server and PASSIM<sup id=\"rdp-ebb-cite_ref-ViksnaPASSIM07_22-0\" class=\"reference\"><a href=\"#cite_note-ViksnaPASSIM07-22\" rel=\"external_link\">[22]<\/a><\/sup> are meaningful in big laboratories which, besides data analysis, are involved in protocol development, data acquisition and data sharing. Freely available tools for microarray analysis like TM4<sup id=\"rdp-ebb-cite_ref-SaeedTM403_23-0\" class=\"reference\"><a href=\"#cite_note-SaeedTM403-23\" rel=\"external_link\">[23]<\/a><\/sup>, ArrayNorm<sup id=\"rdp-ebb-cite_ref-PielerArray04_24-0\" class=\"reference\"><a href=\"#cite_note-PielerArray04-24\" rel=\"external_link\">[24]<\/a><\/sup>, ArrayPipe<sup id=\"rdp-ebb-cite_ref-HokampArray04_25-0\" class=\"reference\"><a href=\"#cite_note-HokampArray04-25\" rel=\"external_link\">[25]<\/a><\/sup> and GPR-Analyzer<sup id=\"rdp-ebb-cite_ref-DittamiGPR13_26-0\" class=\"reference\"><a href=\"#cite_note-DittamiGPR13-26\" rel=\"external_link\">[26]<\/a><\/sup> have either become obsolete or lack key features in terms of extensibility and integration with analysis tools, like R and modern graphing libraries. In the following sections we have described the key features of SaDA compared to any other light weighted domain specific LIMS. SaDA can be easily handled by a novice user. A working knowledge of system (Linux\/UNIX\/Mac OS\/Microsoft Windows) tools installation is required to install, set up and use SaDA. However, if a user wants to extend SaDA\u2019s functionalities and integrate new modules, knowledge of Ruby on Rails and R software programming is essential.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Utility_of_the_SaDA_infrastructure\">Utility of the SaDA infrastructure<\/span><\/h3>\n<p>SaDA is a collection of well-integrated set of models with a very flexible and intuitive user interface: SaDA is developed to address the needs of many laboratories which produce their own batch of microarray data. The database backed software architecture aids in managing and representing oligonucleotide hybridization based microarray data. The flexible and generic database design facilitates mapping of the steadily changing laboratories work-flow. To allow the import of generic file formats, we have implemented a user definable parser that allows reading comma\/tab delimited text files. Additionally, links to these files are maintained in a relational database to prevent the deletion of already imported, linked, or used files. The Microarray Gene Expression Data Society (MGED) develops and maintains standards for data acquisition, representation, and interchange such as the MIAME guidelines<sup id=\"rdp-ebb-cite_ref-YilmazMin11_27-0\" class=\"reference\"><a href=\"#cite_note-YilmazMin11-27\" rel=\"external_link\">[27]<\/a><\/sup>, the MAGE-TAB interchange format<sup id=\"rdp-ebb-cite_ref-RaynerASimp06_28-0\" class=\"reference\"><a href=\"#cite_note-RaynerASimp06-28\" rel=\"external_link\">[28]<\/a><\/sup>, and the MGED Ontology for microarray experiments.<sup id=\"rdp-ebb-cite_ref-_29-0\" class=\"reference\"><a href=\"#cite_note--29\" rel=\"external_link\">[29]<\/a><\/sup> SaDA does not enforce the use of the MGED standards. However it supports storage and retrieval of information required by MIAME. \n<\/p><p>SaDA encapsulates open source software and graphing libraries: The usability of SaDA and the functionality of the interfaces are accounted by the inclusion of different models. These models encapsulate data from analytical pipelines of microarray analysis. To aid in fulfilling these functionalities, SaDA is supported by R programming software and jQWidgets<sup id=\"rdp-ebb-cite_ref-jQWidgets_30-0\" class=\"reference\"><a href=\"#cite_note-jQWidgets-30\" rel=\"external_link\">[30]<\/a><\/sup> graphing libraries. SaDA utilizes RinRuby<sup id=\"rdp-ebb-cite_ref-CrawfordRinRuby09_31-0\" class=\"reference\"><a href=\"#cite_note-CrawfordRinRuby09-31\" rel=\"external_link\">[31]<\/a><\/sup> gem to call R expressions\/functions\/script from Ruby on Rails interfaces. SaDA makes use of a jqGrid plugin<sup id=\"rdp-ebb-cite_ref-jqGrid_32-0\" class=\"reference\"><a href=\"#cite_note-jqGrid-32\" rel=\"external_link\">[32]<\/a><\/sup> for a tabular data representation. \n<\/p><p>SaDA facilitates comparative analysis of microarray results: SaDA implements an algorithm, running in the S2C server<sup id=\"rdp-ebb-cite_ref-S2C_33-0\" class=\"reference\"><a href=\"#cite_note-S2C-33\" rel=\"external_link\">[33]<\/a><\/sup>, for correlating microarray signal intensities with cell numbers from standard calibration plot to estimate cell counts. This way SaDA simplifies the prediction of cell counts from microarray data. With the implementation of flexible normalization procedures, it aids in the quantitative analysis and monitoring of microorganisms. In addition, the use of SaDA assists in making decisions entailing whether a cluster of species is below a set threshold and also in checking if poor hybridization quality or large proportion of nonspecific probe-targets interactions are responsible for negative results.\n<\/p><p>SaDA is extensible: Scientists have a natural inclination for software tools that can be customized to the particular needs of their labs. Tools for rapid modification and customization have proven particularly important in both the adoption of SaDA and the dissemination of MicroAQUA project. The concise and expressive nature of Ruby programming language provides API that is easy to change. There is a less boilerplate and a huge library of gems, including BioRuby project<sup id=\"rdp-ebb-cite_ref-MitsuteroBioRuby03_34-0\" class=\"reference\"><a href=\"#cite_note-MitsuteroBioRuby03-34\" rel=\"external_link\">[34]<\/a><\/sup>, designed for rails applications covering everything from integrating with networking tools to authorization products. The modular nature of Rails framework also supports integration of new features quickly without touching the application\u2019s core components.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Limitations_of_SaDA\">Limitations of SaDA<\/span><\/h3>\n<p>Most biomedical research applications are based on project specific guidelines. This limits the scope of any application to serve diverse purposes and creates boundaries of the platform that originates from its scientific approach. The main limitation encapsulating SaDA\u2019s core functionality is its scientific focus. Currently it only implements microarray data management and analysis modules targeting Agilent\u2019s single channel oligonucleotide hybridization technology. The current work does not aim to replace robust LIMS software solutions as it only targets a specific research study.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Availability_and_future_directions\">Availability and future directions<\/span><\/h3>\n<p>SaDA is open source software and is freely available as a GitHub repository for download from <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/kumarsaurabh20\/bioaqua1\" target=\"_blank\">https:\/\/github.com\/kumarsaurabh20\/bioaqua1<\/a> under the terms of the Apache License 2.0.<sup id=\"rdp-ebb-cite_ref-Apache20_35-0\" class=\"reference\"><a href=\"#cite_note-Apache20-35\" rel=\"external_link\">[35]<\/a><\/sup> This site also provides documentation, tutorials and demos for users and developers, plus instructions for developers who wish to contribute code to the project through the SaDA GitHub repository. For its complete installation from scratch, we have developed a shell script which will install (for Linux based systems) all the components required for SaDA and clone SaDA automatically from the GitHub repositories. Since SaDA is a server application, it is platform independent and it can be run on any operating system. With some modifications, SaDA can also be made to run under other database environments. The other main requirement of SaDA is the working installation of R programming software. The shell script can detect the presence or absence of R software and in case it is not available on the system, the shell script will install it with the user permission. The future version will include handling and analysis of metagenomic data. The first implementation of SaDA was developed in the context of the EU FP7 project MicroAQUA which aimed at developing a universal microarray chip for the detection of fresh water pathogens. The data analysis steps in SaDA comply with the guidelines provided by MicroAQUA consortium.\n<\/p><p>There are many data-transformation and analysis algorithms that are feasible for integration in SaDA as an extended features and external modules. SaDA is developed keeping ecological studies and environmental monitoring in mind. Active development is underway to support other datasets which prove useful in this particular field of interest such as flow-cytometry datasets and their analytical representation, for the detection of toxins in the water based environmental systems. In addition, we are planning to extend the core functionality of SaDA to include metagenomic based data. Use of these standard technologies and datasets will allow interoperability of SaDA with other software. A major goal for SaDA development is the addition of support for other types of ecological high-throughput data. Development will be required to support the emergence of new arrays and platforms, particularly with respect to integrating results across different generations, and possibly different technologies. Support for experiments based on metagenomic data is the next immediate priority. These datasets enable high-throughput genome-scale investigations of interactions between genome and its surrounding environment. This extension will require the incorporation of new data analysis methods. An interface for combining microarray, metagenomics, proteomics and interaction based data within SaDA is already planned.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>We have developed an integrated system consisting of a database that has been tailored to serve specific needs of microarray based environmental monitoring projects. Due to unique fusion of open source tools and libraries, the system can provide a model of building up a similar platform for other emerging genomics technologies. This utility enables users to collect, manipulate and analyze diverse and rapidly changing environmental data-set and their related biological data-types. The system allows easy customization of interfaces and visualization, thus providing a useful platform for collaborative research activities. Implementation of SaDA by the members of MicroAQUA consortium over a period of about three years where water sampling data, molecular analysis and high-throughput microarray experiments have been carried out, has helped to fine-tune the features of this software to suit the needs of a wide range of researchers. The successful application of SaDA suggests the platform has the potential to become an appropriate tool for a wide range of users focused not only in water based environmental research but also in other studies aimed at exploring and analyzing complex ecological habitats. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplementary_materials\">Supplementary materials<\/span><\/h2>\n<p>Graphical workflows for SaDA\u2019s core modules are available as supplementary materials and can be accessed at: <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/\" target=\"_blank\">http:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgments\">Acknowledgments<\/span><\/h2>\n<p>The authors would like to acknowledge the contributions of partners of the MicroAQUA project for their valuable inputs on the design of user interfaces. The work was financially supported by funds from the European Commission 7th Framework to the project MicroAQUA (KBBE-2010-3-2-04: Innovative aquatic biosensors\u2014Medium Collaborative Project).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h2>\n<p>Kumar Saurabh Singh was the main responsible for the scientific outcome of the article and development of SaDA. Dominique Thual contributed software code in initial sections of SaDA in Rails version 2.3.11 which was later upgraded to Rails version 3.2.13 by Kumar Saurabh Singh. Nicola Cannata was involved in general design guidelines of SaDA. Roberto Spurio was the general coordinator of the work. Kumar Saurabh Singh drafted the manuscript. All authors have read and approved this manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflicts_of_interest\">Conflicts of interest<\/span><\/h2>\n<p>The authors declare no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-RueggComp14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RueggComp14_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">R\u00fcegg, J.; Gries, C.; Bond-Lamberty, B.; Bowen, G.J.; Felzer, B.S.; McIntyre, N.E.; Soranno, P.; Vanderbilt, K.L.; Weathers, K.C. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/htm\" target=\"_blank\">\"Completing the data life cycle: Using information management in macrosystems ecology research\"<\/a>. <i>Frontiers in Ecology and the Environment<\/i> <b>12<\/b> (1): 24\u201330. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1890%2F120375\" target=\"_blank\">10.1890\/120375<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/htm\" target=\"_blank\">http:\/\/www.mdpi.com\/1660-4601\/12\/6\/6352\/htm<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Completing+the+data+life+cycle%3A+Using+information+management+in+macrosystems+ecology+research&rft.jtitle=Frontiers+in+Ecology+and+the+Environment&rft.aulast=R%C3%BCegg%2C+J.%3B+Gries%2C+C.%3B+Bond-Lamberty%2C+B.%3B+Bowen%2C+G.J.%3B+Felzer%2C+B.S.%3B+McIntyre%2C+N.E.%3B+Soranno%2C+P.%3B+Vanderbilt%2C+K.L.%3B+Weathers%2C+K.C.&rft.au=R%C3%BCegg%2C+J.%3B+Gries%2C+C.%3B+Bond-Lamberty%2C+B.%3B+Bowen%2C+G.J.%3B+Felzer%2C+B.S.%3B+McIntyre%2C+N.E.%3B+Soranno%2C+P.%3B+Vanderbilt%2C+K.L.%3B+Weathers%2C+K.C.&rft.date=2014&rft.volume=12&rft.issue=1&rft.pages=24%E2%80%9330&rft_id=info:doi\/10.1890%2F120375&rft_id=http%3A%2F%2Fwww.mdpi.com%2F1660-4601%2F12%2F6%2F6352%2Fhtm&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeffernanMacro14-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeffernanMacro14_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Heffernan, J.B.; Soranno, P.; Angilletta, M.J.; Buckley, L.B.; Gruner, D.S.; Keitt, T.H.; Kellner, J.R.; Kominoski, J.S.; Rocha, A.V.; Xiao, J. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/130017\" target=\"_blank\">\"Macrosystems ecology: Understanding ecological patterns and processes at continental scales\"<\/a>. <i>Frontiers in Ecology and the Environment<\/i> <b>12<\/b> (1): 5\u201314. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1890%2F130017\" target=\"_blank\">10.1890\/130017<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/130017\" target=\"_blank\">http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/130017<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Macrosystems+ecology%3A+Understanding+ecological+patterns+and+processes+at+continental+scales&rft.jtitle=Frontiers+in+Ecology+and+the+Environment&rft.aulast=Heffernan%2C+J.B.%3B+Soranno%2C+P.%3B+Angilletta%2C+M.J.%3B+Buckley%2C+L.B.%3B+Gruner%2C+D.S.%3B+Keitt%2C+T.H.%3B+Kellner%2C+J.R.%3B+Kominoski%2C+J.S.%3B+Rocha%2C+A.V.%3B+Xiao%2C+J.+et+al.&rft.au=Heffernan%2C+J.B.%3B+Soranno%2C+P.%3B+Angilletta%2C+M.J.%3B+Buckley%2C+L.B.%3B+Gruner%2C+D.S.%3B+Keitt%2C+T.H.%3B+Kellner%2C+J.R.%3B+Kominoski%2C+J.S.%3B+Rocha%2C+A.V.%3B+Xiao%2C+J.+et+al.&rft.date=2014&rft.volume=12&rft.issue=1&rft.pages=5%E2%80%9314&rft_id=info:doi\/10.1890%2F130017&rft_id=http%3A%2F%2Fwww.esajournals.org%2Fdoi%2Fabs%2F10.1890%2F130017&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoringImp14-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoringImp14_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goring, S.J.; Weathers, K.C.; Dodds, W.K.; Soranno, P.; Sweet, L.C.; Cheruvelil, K.S.; Kominoski, J.S.; R\u00fcegg, J.; Thorn, A.M.; Utz, R.M. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/120370\" target=\"_blank\">\"Improving the culture of interdisciplinary collaboration in ecology by expanding measures of success\"<\/a>. <i>Frontiers in Ecology and the Environment<\/i> <b>12<\/b> (1): 39\u201347. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1890%2F120370\" target=\"_blank\">10.1890\/120370<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/120370\" target=\"_blank\">http:\/\/www.esajournals.org\/doi\/abs\/10.1890\/120370<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Improving+the+culture+of+interdisciplinary+collaboration+in+ecology+by+expanding+measures+of+success&rft.jtitle=Frontiers+in+Ecology+and+the+Environment&rft.aulast=Goring%2C+S.J.%3B+Weathers%2C+K.C.%3B+Dodds%2C+W.K.%3B+Soranno%2C+P.%3B+Sweet%2C+L.C.%3B+Cheruvelil%2C+K.S.%3B+Kominoski%2C+J.S.%3B+R%C3%BCegg%2C+J.%3B+Thorn%2C+A.M.%3B+Utz%2C+R.M.&rft.au=Goring%2C+S.J.%3B+Weathers%2C+K.C.%3B+Dodds%2C+W.K.%3B+Soranno%2C+P.%3B+Sweet%2C+L.C.%3B+Cheruvelil%2C+K.S.%3B+Kominoski%2C+J.S.%3B+R%C3%BCegg%2C+J.%3B+Thorn%2C+A.M.%3B+Utz%2C+R.M.&rft.date=2014&rft.volume=12&rft.issue=1&rft.pages=39%E2%80%9347&rft_id=info:doi\/10.1890%2F120370&rft_id=http%3A%2F%2Fwww.esajournals.org%2Fdoi%2Fabs%2F10.1890%2F120370&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BushmannPatern96-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BushmannPatern96_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Bushmann, F.; Meunier, R.; Rohnert, H.; Sommerlad, P.; Stal, M. (1996). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0471958697.html\" target=\"_blank\"><i>Pattern-Oriented Software Architecture: A System of Patterns<\/i><\/a>. <b>1<\/b>. West Sussex, England: John Wiley & Sons Ltd. pp. 476. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780471958697<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0471958697.html\" target=\"_blank\">http:\/\/www.wiley.com\/WileyCDA\/WileyTitle\/productCd-0471958697.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Pattern-Oriented+Software+Architecture%3A+A+System+of+Patterns&rft.aulast=Bushmann%2C+F.%3B+Meunier%2C+R.%3B+Rohnert%2C+H.%3B+Sommerlad%2C+P.%3B+Stal%2C+M.&rft.au=Bushmann%2C+F.%3B+Meunier%2C+R.%3B+Rohnert%2C+H.%3B+Sommerlad%2C+P.%3B+Stal%2C+M.&rft.date=1996&rft.volume=1&rft.pages=pp.%26nbsp%3B476&rft.place=West+Sussex%2C+England&rft.pub=John+Wiley+%26+Sons+Ltd&rft.isbn=9780471958697&rft_id=http%3A%2F%2Fwww.wiley.com%2FWileyCDA%2FWileyTitle%2FproductCd-0471958697.html&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaurerMARS05-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaurerMARS05_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maurer, M.; Molidor, R.; Sturn, A.; Hartler, J.; Hackl, H.; Stocker, G.; Prokesch, A.; Scheideler, M.; Trajanoski, Z. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551\" target=\"_blank\">\"MARS: Microarray analysis, retrieval, and storage system\"<\/a>. <i>BMC Bioinformatics<\/i> <b>6<\/b>: 101. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-6-101\" target=\"_blank\">10.1186\/1471-2105-6-101<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1090551\/\" target=\"_blank\">PMC1090551<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15836795\" target=\"_blank\">15836795<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MARS%3A+Microarray+analysis%2C+retrieval%2C+and+storage+system&rft.jtitle=BMC+Bioinformatics&rft.aulast=Maurer%2C+M.%3B+Molidor%2C+R.%3B+Sturn%2C+A.%3B+Hartler%2C+J.%3B+Hackl%2C+H.%3B+Stocker%2C+G.%3B+Prokesch%2C+A.%3B+Scheideler%2C+M.%3B+Trajanoski%2C+Z.&rft.au=Maurer%2C+M.%3B+Molidor%2C+R.%3B+Sturn%2C+A.%3B+Hartler%2C+J.%3B+Hackl%2C+H.%3B+Stocker%2C+G.%3B+Prokesch%2C+A.%3B+Scheideler%2C+M.%3B+Trajanoski%2C+Z.&rft.date=2005&rft.volume=6&rft.pages=101&rft_id=info:doi\/10.1186%2F1471-2105-6-101&rft_id=info:pmc\/PMC1090551&rft_id=info:pmid\/15836795&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1090551&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Stocker_iLAP09-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Stocker_iLAP09_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stocker, G.; Fischer, M.; Rieder, D.; Bindea, G.; Kainz, S.; Oberstolz, M.; McNally, J.G.; Trajanoski, Z. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2789074\" target=\"_blank\">\"iLAP: A workflow-driven software for experimental protocol development, data acquisition and analysis\"<\/a>. <i>BMC Bioinformatics<\/i> <b>10<\/b>: 390. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-10-390\" target=\"_blank\">10.1186\/1471-2105-10-390<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2789074\/\" target=\"_blank\">PMC2789074<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19941647\" target=\"_blank\">19941647<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2789074\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2789074<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=iLAP%3A+A+workflow-driven+software+for+experimental+protocol+development%2C+data+acquisition+and+analysis&rft.jtitle=BMC+Bioinformatics&rft.aulast=Stocker%2C+G.%3B+Fischer%2C+M.%3B+Rieder%2C+D.%3B+Bindea%2C+G.%3B+Kainz%2C+S.%3B+Oberstolz%2C+M.%3B+McNally%2C+J.G.%3B+Trajanoski%2C+Z.&rft.au=Stocker%2C+G.%3B+Fischer%2C+M.%3B+Rieder%2C+D.%3B+Bindea%2C+G.%3B+Kainz%2C+S.%3B+Oberstolz%2C+M.%3B+McNally%2C+J.G.%3B+Trajanoski%2C+Z.&rft.date=2009&rft.volume=10&rft.pages=390&rft_id=info:doi\/10.1186%2F1471-2105-10-390&rft_id=info:pmc\/PMC2789074&rft_id=info:pmid\/19941647&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2789074&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VallonBASE09-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VallonBASE09_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Vallon-Christersson, J.; Nordborg, N.; Svensson, M.; H\u00e4kkinen, J. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2768720\" target=\"_blank\">\"BASE - 2nd generation software for microarray data management and analysis\"<\/a>. <i>BMC Bioinformatics<\/i> <b>10<\/b>: 330. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-10-330\" target=\"_blank\">10.1186\/1471-2105-10-330<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2768720\/\" target=\"_blank\">PMC2768720<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19822003\" target=\"_blank\">19822003<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2768720\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2768720<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BASE+-+2nd+generation+software+for+microarray+data+management+and+analysis&rft.jtitle=BMC+Bioinformatics&rft.aulast=Vallon-Christersson%2C+J.%3B+Nordborg%2C+N.%3B+Svensson%2C+M.%3B+H%C3%A4kkinen%2C+J.&rft.au=Vallon-Christersson%2C+J.%3B+Nordborg%2C+N.%3B+Svensson%2C+M.%3B+H%C3%A4kkinen%2C+J.&rft.date=2009&rft.volume=10&rft.pages=330&rft_id=info:doi\/10.1186%2F1471-2105-10-330&rft_id=info:pmc\/PMC2768720&rft_id=info:pmid\/19822003&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2768720&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NelsonLabkey11-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NelsonLabkey11_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nelson, E.K.; Piehler, B.; Eckels, J.; Rauch, A.; Bellew, M.; Hussey, P.; Ramsay, S.; Nathe, C.; Lum, K.; Krouse, K; Stearns, D.; Connolly, B.; Skillman, T.; Igra, M. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597\" target=\"_blank\">\"LabKey Server: An open source platform for scientific data integration, analysis and collaboration\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b> (71). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-71\" target=\"_blank\">10.1186\/1471-2105-12-71<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3062597\/\" target=\"_blank\">PMC3062597<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21385461\" target=\"_blank\">21385461<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3062597<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LabKey+Server%3A+An+open+source+platform+for+scientific+data+integration%2C+analysis+and+collaboration&rft.jtitle=BMC+Bioinformatics&rft.aulast=Nelson%2C+E.K.%3B+Piehler%2C+B.%3B+Eckels%2C+J.%3B+Rauch%2C+A.%3B+Bellew%2C+M.%3B+Hussey%2C+P.%3B+Ramsay%2C+S.%3B+Nathe%2C+C.%3B+Lum%2C+K.%3B+Krouse%2C+K%3B+Stearns%2C+D.%3B+Connolly%2C+B.%3B+Skillman%2C+T.%3B+Igra%2C+M.&rft.au=Nelson%2C+E.K.%3B+Piehler%2C+B.%3B+Eckels%2C+J.%3B+Rauch%2C+A.%3B+Bellew%2C+M.%3B+Hussey%2C+P.%3B+Ramsay%2C+S.%3B+Nathe%2C+C.%3B+Lum%2C+K.%3B+Krouse%2C+K%3B+Stearns%2C+D.%3B+Connolly%2C+B.%3B+Skillman%2C+T.%3B+Igra%2C+M.&rft.date=2011&rft.volume=12&rft.issue=71&rft_id=info:doi\/10.1186%2F1471-2105-12-71&rft_id=info:pmc\/PMC3062597&rft_id=info:pmid\/21385461&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3062597&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NixNext10-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NixNext10_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nix, D.; di Sera, T.L.; Dalley, B.K.; Milash, B.; Cundick, R.M.; Quinn, K.S.; Courdy, S.J. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281\" target=\"_blank\">\"Next generation tools for genomic data generation, distribution, and visualization\"<\/a>. <i>BMC Bioinformatics<\/i> <b>11<\/b> (455). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-11-455\" target=\"_blank\">10.1186\/1471-2105-11-455<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2944281\/\" target=\"_blank\">PMC2944281<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20828407\" target=\"_blank\">20828407<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Next+generation+tools+for+genomic+data+generation%2C+distribution%2C+and+visualization&rft.jtitle=BMC+Bioinformatics&rft.aulast=Nix%2C+D.%3B+di+Sera%2C+T.L.%3B+Dalley%2C+B.K.%3B+Milash%2C+B.%3B+Cundick%2C+R.M.%3B+Quinn%2C+K.S.%3B+Courdy%2C+S.J.&rft.au=Nix%2C+D.%3B+di+Sera%2C+T.L.%3B+Dalley%2C+B.K.%3B+Milash%2C+B.%3B+Cundick%2C+R.M.%3B+Quinn%2C+K.S.%3B+Courdy%2C+S.J.&rft.date=2010&rft.volume=11&rft.issue=455&rft_id=info:doi\/10.1186%2F1471-2105-11-455&rft_id=info:pmc\/PMC2944281&rft_id=info:pmid\/20828407&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2944281&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ColmseeACase11-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ColmseeACase11_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Colmsee, C.; Flemming, S.; Klapperst\u00fcck, M.; Lange, M.; Scholz, U. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3217054\" target=\"_blank\">\"A case study for efficient management of high throughput primary lab data\"<\/a>. <i>BMC Research Notes<\/i> <b>4<\/b>: 413. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1756-0500-4-413\" target=\"_blank\">10.1186\/1756-0500-4-413<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3217054\/\" target=\"_blank\">PMC3217054<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22005096\" target=\"_blank\">22005096<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3217054\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3217054<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+case+study+for+efficient+management+of+high+throughput+primary+lab+data&rft.jtitle=BMC+Research+Notes&rft.aulast=Colmsee%2C+C.%3B+Flemming%2C+S.%3B+Klapperst%C3%BCck%2C+M.%3B+Lange%2C+M.%3B+Scholz%2C+U.&rft.au=Colmsee%2C+C.%3B+Flemming%2C+S.%3B+Klapperst%C3%BCck%2C+M.%3B+Lange%2C+M.%3B+Scholz%2C+U.&rft.date=2011&rft.volume=4&rft.pages=413&rft_id=info:doi\/10.1186%2F1756-0500-4-413&rft_id=info:pmc\/PMC3217054&rft_id=info:pmid\/22005096&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3217054&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Bauch_openBIS11-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Bauch_openBIS11_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bauch, A.; Adamczyk, I.; Buczek, P.; Elmer, F.-J.; Enimanev, K.; Glyzewski, P.; Kohler, M.; Pylak, T.; Quandt, A.; Ramakrishnan, C.; Beisel, C.; Malmstr\u00f6m, L.; Aebersold, R.; Rinn, B. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639\" target=\"_blank\">\"openBIS: a flexible framework for managing and analyzing complex data in biology research\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>: 468. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-468\" target=\"_blank\">10.1186\/1471-2105-12-468<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3275639\/\" target=\"_blank\">PMC3275639<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22151573\" target=\"_blank\">22151573<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=openBIS%3A+a+flexible+framework+for+managing+and+analyzing+complex+data+in+biology+research&rft.jtitle=BMC+Bioinformatics&rft.aulast=Bauch%2C+A.%3B+Adamczyk%2C+I.%3B+Buczek%2C+P.%3B+Elmer%2C+F.-J.%3B+Enimanev%2C+K.%3B+Glyzewski%2C+P.%3B+Kohler%2C+M.%3B+Pylak%2C+T.%3B+Quandt%2C+A.%3B+Ramakrishnan%2C+C.%3B+Beisel%2C+C.%3B+Malmstr%C3%B6m%2C+L.%3B+Aebersold%2C+R.%3B+Rinn%2C+B.&rft.au=Bauch%2C+A.%3B+Adamczyk%2C+I.%3B+Buczek%2C+P.%3B+Elmer%2C+F.-J.%3B+Enimanev%2C+K.%3B+Glyzewski%2C+P.%3B+Kohler%2C+M.%3B+Pylak%2C+T.%3B+Quandt%2C+A.%3B+Ramakrishnan%2C+C.%3B+Beisel%2C+C.%3B+Malmstr%C3%B6m%2C+L.%3B+Aebersold%2C+R.%3B+Rinn%2C+B.&rft.date=2011&rft.volume=12&rft.pages=468&rft_id=info:doi\/10.1186%2F1471-2105-12-468&rft_id=info:pmc\/PMC3275639&rft_id=info:pmid\/22151573&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3275639&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VencoSLIMS13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VencoSLIMS13_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Venco, F.; Ceol, A.; Muller, H. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/journal.embnet.org\/index.php\/embnetjournal\/article\/view\/739\" target=\"_blank\">\"SLIMS: A LIMS for handling next-generation sequencing workflows\"<\/a>. <i>EMBnet.journal<\/i> <b>19<\/b> (B): 85\u201387. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.14806%2Fej.19.B.739\" target=\"_blank\">10.14806\/ej.19.B.739<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/journal.embnet.org\/index.php\/embnetjournal\/article\/view\/739\" target=\"_blank\">http:\/\/journal.embnet.org\/index.php\/embnetjournal\/article\/view\/739<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SLIMS%3A+A+LIMS+for+handling+next-generation+sequencing+workflows&rft.jtitle=EMBnet.journal&rft.aulast=Venco%2C+F.%3B+Ceol%2C+A.%3B+Muller%2C+H.&rft.au=Venco%2C+F.%3B+Ceol%2C+A.%3B+Muller%2C+H.&rft.date=2013&rft.volume=19&rft.issue=B&rft.pages=85%E2%80%9387&rft_id=info:doi\/10.14806%2Fej.19.B.739&rft_id=http%3A%2F%2Fjournal.embnet.org%2Findex.php%2Fembnetjournal%2Farticle%2Fview%2F739&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-McLellanTheWasp12-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-McLellanTheWasp12_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">McLellan, A.S.; Dubin, R.; Jing, Q.; Broin, P.\u00d3.; Moskowitz, D.; Suzuki, M.; Calder, R.B.; Hargitai, J.; Golden, A.; Greally, J.M. (2012). \"The Wasp System: An open source environment for managing and analyzing genomic data\". <i>Genomics<\/i> <b>100<\/b> (6): 345-51. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.ygeno.2012.08.005\" target=\"_blank\">10.1016\/j.ygeno.2012.08.005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22944616\" target=\"_blank\">22944616<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Wasp+System%3A+An+open+source+environment+for+managing+and+analyzing+genomic+data&rft.jtitle=Genomics&rft.aulast=McLellan%2C+A.S.%3B+Dubin%2C+R.%3B+Jing%2C+Q.%3B+Broin%2C+P.%C3%93.%3B+Moskowitz%2C+D.%3B+Suzuki%2C+M.%3B+Calder%2C+R.B.%3B+Hargitai%2C+J.%3B+Golden%2C+A.%3B+Greally%2C+J.M.&rft.au=McLellan%2C+A.S.%3B+Dubin%2C+R.%3B+Jing%2C+Q.%3B+Broin%2C+P.%C3%93.%3B+Moskowitz%2C+D.%3B+Suzuki%2C+M.%3B+Calder%2C+R.B.%3B+Hargitai%2C+J.%3B+Golden%2C+A.%3B+Greally%2C+J.M.&rft.date=2012&rft.volume=100&rft.issue=6&rft.pages=345-51&rft_id=info:doi\/10.1016%2Fj.ygeno.2012.08.005&rft_id=info:pmid\/22944616&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GiardineGalaxy05-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GiardineGalaxy05_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Giardine, B.; Riemer, C.; Hardison, R.C.; Burhans, R.; Elnitski, L.; Shah, P.; Zhang, Y.; Blankenberg, D.; Albert, I.; Taylor, J.; Miller, W.; Kent, W.J.; Nekrutenko, A. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089\" target=\"_blank\">\"Galaxy: A platform for interactive large-scale genome analysis\"<\/a>. <i>Genome Research<\/i> <b>15<\/b> (10): 1451\u20131455. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.4086505\" target=\"_blank\">10.1101\/gr.4086505<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1240089\/\" target=\"_blank\">PMC1240089<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16169926\" target=\"_blank\">16169926<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy%3A+A+platform+for+interactive+large-scale+genome+analysis&rft.jtitle=Genome+Research&rft.aulast=Giardine%2C+B.%3B+Riemer%2C+C.%3B+Hardison%2C+R.C.%3B+Burhans%2C+R.%3B+Elnitski%2C+L.%3B+Shah%2C+P.%3B+Zhang%2C+Y.%3B+Blankenberg%2C+D.%3B+Albert%2C+I.%3B+Taylor%2C+J.%3B+Miller%2C+W.%3B+Kent%2C+W.J.%3B+Nekrutenko%2C+A.&rft.au=Giardine%2C+B.%3B+Riemer%2C+C.%3B+Hardison%2C+R.C.%3B+Burhans%2C+R.%3B+Elnitski%2C+L.%3B+Shah%2C+P.%3B+Zhang%2C+Y.%3B+Blankenberg%2C+D.%3B+Albert%2C+I.%3B+Taylor%2C+J.%3B+Miller%2C+W.%3B+Kent%2C+W.J.%3B+Nekrutenko%2C+A.&rft.date=2005&rft.volume=15&rft.issue=10&rft.pages=1451%E2%80%931455&rft_id=info:doi\/10.1101%2Fgr.4086505&rft_id=info:pmc\/PMC1240089&rft_id=info:pmid\/16169926&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1240089&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScholtalbersGal13-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScholtalbersGal13_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scholtalbers, J.; R\u00f6ssler, J.; Sorn, P.; de Graaf, J.; Boisgu\u00e9rin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". <i>Bioinformatics<\/i> <b>29<\/b> (9): 1233-1234. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt115\" target=\"_blank\">10.1093\/bioinformatics\/btt115<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23479349\" target=\"_blank\">23479349<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy+LIMS+for+next-generation+sequencing&rft.jtitle=Bioinformatics&rft.aulast=Scholtalbers%2C+J.%3B+R%C3%B6ssler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisgu%C3%A9rin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.au=Scholtalbers%2C+J.%3B+R%C3%B6ssler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisgu%C3%A9rin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.date=2013&rft.volume=29&rft.issue=9&rft.pages=1233-1234&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt115&rft_id=info:pmid\/23479349&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CASAVA-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CASAVA_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/support.illumina.com\/sequencing\/documentation.html\" target=\"_blank\">\"Sequencing Documentation\"<\/a>. <i>CASAVA User Guide<\/i>. Illumina, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/support.illumina.com\/sequencing\/documentation.html\" target=\"_blank\">http:\/\/support.illumina.com\/sequencing\/documentation.html<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Sequencing+Documentation&rft.atitle=CASAVA+User+Guide&rft.pub=Illumina%2C+Inc&rft_id=http%3A%2F%2Fsupport.illumina.com%2Fsequencing%2Fdocumentation.html&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VanRossumSLIMS10-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VanRossumSLIMS10_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Van Rossum, T.; Tripp, B.; Daley, D. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515\" target=\"_blank\">\"SLIMS\u2014A user-friendly sample operations and inventory management system for genotyping labs\"<\/a>. <i>Bioinformatics<\/i> <b>26<\/b> (14): 1808\u201310. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtq271\" target=\"_blank\">10.1093\/bioinformatics\/btq271<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2894515\/\" target=\"_blank\">PMC2894515<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20513665\" target=\"_blank\">20513665<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SLIMS%E2%80%94A+user-friendly+sample+operations+and+inventory+management+system+for+genotyping+labs&rft.jtitle=Bioinformatics&rft.aulast=Van+Rossum%2C+T.%3B+Tripp%2C+B.%3B+Daley%2C+D.&rft.au=Van+Rossum%2C+T.%3B+Tripp%2C+B.%3B+Daley%2C+D.&rft.date=2010&rft.volume=26&rft.issue=14&rft.pages=1808%E2%80%9310&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtq271&rft_id=info:pmc\/PMC2894515&rft_id=info:pmid\/20513665&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2894515&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ProtopsaltouModel07-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ProtopsaltouModel07_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Protopsaltou, A. (2007) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/gupea.ub.gu.se\/bitstream\/2077\/4647\/1\/ISOFM_ITU870_V07_Protopsaltou,Antonios_Model%20Driven%20Development%20with%20Ruby%20on%20Rails.pdf\" target=\"_blank\"><i>Model Driven Development with Ruby on Rails Ruby on Rails and Reverse Engineering<\/i><\/a>. G\u00f6teborg, Sweden: IT University of G\u00f6teborg. pp. 1\u201334<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/gupea.ub.gu.se\/bitstream\/2077\/4647\/1\/ISOFM_ITU870_V07_Protopsaltou,Antonios_Model%20Driven%20Development%20with%20Ruby%20on%20Rails.pdf\" target=\"_blank\">https:\/\/gupea.ub.gu.se\/bitstream\/2077\/4647\/1\/ISOFM_ITU870_V07_Protopsaltou,Antonios_Model%20Driven%20Development%20with%20Ruby%20on%20Rails.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Model+Driven+Development+with+Ruby+on+Rails+Ruby+on+Rails+and+Reverse+Engineering&rft.aulast=Protopsaltou%2C+A.&rft.au=Protopsaltou%2C+A.&rft.date=2007&rft.pages=pp.%26nbsp%3B1%E2%80%9334&rft.place=G%C3%B6teborg%2C+Sweden&rft.pub=IT+University+of+G%C3%B6teborg&rft_id=https%3A%2F%2Fgupea.ub.gu.se%2Fbitstream%2F2077%2F4647%2F1%2FISOFM_ITU870_V07_Protopsaltou%2CAntonios_Model%2520Driven%2520Development%2520with%2520Ruby%2520on%2520Rails.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RosenWeb09-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RosenWeb09_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Rosen, L.S.R. (2009). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/books.google.com\/books?id=gcHJQwAACAAJ\" target=\"_blank\"><i>Web Application Architecture: Principles, Protocol and Practices<\/i><\/a>. West Sussex, England: John Wiley & Sons Ltd. pp. 440<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/books.google.com\/books?id=gcHJQwAACAAJ\" target=\"_blank\">https:\/\/books.google.com\/books?id=gcHJQwAACAAJ<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Web+Application+Architecture%3A+Principles%2C+Protocol+and+Practices&rft.aulast=Rosen%2C+L.S.R.&rft.au=Rosen%2C+L.S.R.&rft.date=2009&rft.pages=pp.%26nbsp%3B440&rft.place=West+Sussex%2C+England&rft.pub=John+Wiley+%26+Sons+Ltd&rft_id=https%3A%2F%2Fbooks.google.com%2Fbooks%3Fid%3DgcHJQwAACAAJ&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MicroAqua-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MicroAqua_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/web.archive.org\/web\/20150214182305\/http:\/\/www.microaqua.eu\/\" target=\"_blank\">\"Welcome to MicroAqua\"<\/a>. MicroAqua Project Consortium. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.microaqua.eu\/\" target=\"_blank\">the original<\/a> on 14 February 2015<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/web.archive.org\/web\/20150214182305\/http:\/\/www.microaqua.eu\/\" target=\"_blank\">http:\/\/web.archive.org\/web\/20150214182305\/http:\/\/www.microaqua.eu\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Welcome+to+MicroAqua&rft.atitle=&rft.pub=MicroAqua+Project+Consortium&rft_id=http%3A%2F%2Fweb.archive.org%2Fweb%2F20150214182305%2Fhttp%3A%2F%2Fwww.microaqua.eu%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YM4R-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YM4R_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/queso\/ym4r-gm\" target=\"_blank\">\"A port of the YM4R-GM plugin for rails, to github\"<\/a>. GitHub<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/queso\/ym4r-gm\" target=\"_blank\">https:\/\/github.com\/queso\/ym4r-gm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+port+of+the+YM4R-GM+plugin+for+rails%2C+to+github&rft.atitle=&rft.pub=GitHub&rft_id=https%3A%2F%2Fgithub.com%2Fqueso%2Fym4r-gm&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ViksnaPASSIM07-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ViksnaPASSIM07_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Viksna, J.; Celms, E.; Opmanis, M.; Podnieks, K.; Rucevskis, P.; Zarins, A.; Barrett, A.; Neogi, S.G.; Krestyaninova, M.; McCarthy, M.I. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798\" target=\"_blank\">\"PASSIM\u2014An open source software system for managing information in biomedical studies\"<\/a>. <i>BMC Bioinformatics<\/i> <b>8<\/b>: 52. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-8-52\" target=\"_blank\">10.1186\/1471-2105-8-52<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1803798\/\" target=\"_blank\">PMC1803798<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17291344\" target=\"_blank\">17291344<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PASSIM%E2%80%94An+open+source+software+system+for+managing+information+in+biomedical+studies&rft.jtitle=BMC+Bioinformatics&rft.aulast=Viksna%2C+J.%3B+Celms%2C+E.%3B+Opmanis%2C+M.%3B+Podnieks%2C+K.%3B+Rucevskis%2C+P.%3B+Zarins%2C+A.%3B+Barrett%2C+A.%3B+Neogi%2C+S.G.%3B+Krestyaninova%2C+M.%3B+McCarthy%2C+M.I.+et+al.&rft.au=Viksna%2C+J.%3B+Celms%2C+E.%3B+Opmanis%2C+M.%3B+Podnieks%2C+K.%3B+Rucevskis%2C+P.%3B+Zarins%2C+A.%3B+Barrett%2C+A.%3B+Neogi%2C+S.G.%3B+Krestyaninova%2C+M.%3B+McCarthy%2C+M.I.+et+al.&rft.date=2007&rft.volume=8&rft.pages=52&rft_id=info:doi\/10.1186%2F1471-2105-8-52&rft_id=info:pmc\/PMC1803798&rft_id=info:pmid\/17291344&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1803798&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaeedTM403-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SaeedTM403_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saeed, A.I.; Sharov, V.; White, J.; Li, J.; Liang, W.; Bhagabati, N.; Braisted, J.; Klapa, M.; Currier, T.; Thia-garajan, M. et al. (2003). \"TM4: A free, open-source system for microarray data management and analysis\". <i>BioTechniques<\/i> <b>34<\/b> (2): 374-8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12613259\" target=\"_blank\">12613259<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=TM4%3A+A+free%2C+open-source+system+for+microarray+data+management+and+analysis&rft.jtitle=BioTechniques&rft.aulast=Saeed%2C+A.I.%3B+Sharov%2C+V.%3B+White%2C+J.%3B+Li%2C+J.%3B+Liang%2C+W.%3B+Bhagabati%2C+N.%3B+Braisted%2C+J.%3B+Klapa%2C+M.%3B+Currier%2C+T.%3B+Thia-garajan%2C+M.+et+al.&rft.au=Saeed%2C+A.I.%3B+Sharov%2C+V.%3B+White%2C+J.%3B+Li%2C+J.%3B+Liang%2C+W.%3B+Bhagabati%2C+N.%3B+Braisted%2C+J.%3B+Klapa%2C+M.%3B+Currier%2C+T.%3B+Thia-garajan%2C+M.+et+al.&rft.date=2003&rft.volume=34&rft.issue=2&rft.pages=374-8&rft_id=info:pmid\/12613259&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PielerArray04-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PielerArray04_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pieler, R.; Sanchez-Cabo, F.; Hackl, H.; Thallinger, G.G.; Trajanoski, Z. (2004). \"ArrayNorm: Comprehensive normalization and analysis of microarray data\". <i>Bioinformatics<\/i> <b>20<\/b> (12): 1971-3. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbth174\" target=\"_blank\">10.1093\/bioinformatics\/bth174<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15073026\" target=\"_blank\">15073026<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ArrayNorm%3A+Comprehensive+normalization+and+analysis+of+microarray+data&rft.jtitle=Bioinformatics&rft.aulast=Pieler%2C+R.%3B+Sanchez-Cabo%2C+F.%3B+Hackl%2C+H.%3B+Thallinger%2C+G.G.%3B+Trajanoski%2C+Z.&rft.au=Pieler%2C+R.%3B+Sanchez-Cabo%2C+F.%3B+Hackl%2C+H.%3B+Thallinger%2C+G.G.%3B+Trajanoski%2C+Z.&rft.date=2004&rft.volume=20&rft.issue=12&rft.pages=1971-3&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbth174&rft_id=info:pmid\/15073026&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HokampArray04-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HokampArray04_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hokamp, K.; Roche, F.M.; Acab, M.; Rousseau, M.E.; Kuo, B.; Goode, D.; Aeschliman, D.; Bryan, J.; Babiuk, L.; Hancock, R.E.W. et al. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC441584\" target=\"_blank\">\"ArrayPipe: A flexible processing pipeline for microarray data\"<\/a>. <i>Nucleic Acids Research<\/i> <b>32<\/b> (Suppl 2): W457-W459. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkh446\" target=\"_blank\">10.1093\/nar\/gkh446<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC441584\/\" target=\"_blank\">PMC441584<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15215429\" target=\"_blank\">15215429<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC441584\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC441584<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ArrayPipe%3A+A+flexible+processing+pipeline+for+microarray+data&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Hokamp%2C+K.%3B+Roche%2C+F.M.%3B+Acab%2C+M.%3B+Rousseau%2C+M.E.%3B+Kuo%2C+B.%3B+Goode%2C+D.%3B+Aeschliman%2C+D.%3B+Bryan%2C+J.%3B+Babiuk%2C+L.%3B+Hancock%2C+R.E.W.+et+al.&rft.au=Hokamp%2C+K.%3B+Roche%2C+F.M.%3B+Acab%2C+M.%3B+Rousseau%2C+M.E.%3B+Kuo%2C+B.%3B+Goode%2C+D.%3B+Aeschliman%2C+D.%3B+Bryan%2C+J.%3B+Babiuk%2C+L.%3B+Hancock%2C+R.E.W.+et+al.&rft.date=2004&rft.volume=32&rft.issue=Suppl+2&rft.pages=W457-W459&rft_id=info:doi\/10.1093%2Fnar%2Fgkh446&rft_id=info:pmc\/PMC441584&rft_id=info:pmid\/15215429&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC441584&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DittamiGPR13-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DittamiGPR13_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dittami, S.M.; Edvardsen, B. (2013). \"GPR-Analyzer: A simple tool for quantitative analysis of hierarchical multispecies microarrays\". <i>Environmental Science and Pollution Research<\/i> <b>20<\/b> (10): 6808-15. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs11356-012-1051-5\" target=\"_blank\">10.1007\/s11356-012-1051-5<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22767354\" target=\"_blank\">22767354<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GPR-Analyzer%3A+A+simple+tool+for+quantitative+analysis+of+hierarchical+multispecies+microarrays&rft.jtitle=Environmental+Science+and+Pollution+Research&rft.aulast=Dittami%2C+S.M.%3B+Edvardsen%2C+B.&rft.au=Dittami%2C+S.M.%3B+Edvardsen%2C+B.&rft.date=2013&rft.volume=20&rft.issue=10&rft.pages=6808-15&rft_id=info:doi\/10.1007%2Fs11356-012-1051-5&rft_id=info:pmid\/22767354&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YilmazMin11-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YilmazMin11_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yilmaz, P.; Kottmann, R.; Field, D.; Knight, R.; Cole, J.R.; Amaral-Zettler, L.; Gilbert, J.; Karsch-Mizrachi, I.; Johnston, A.; Cochrane, G. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3367316\" target=\"_blank\">\"Minimum information about a marker gene sequence (MIMARKS) and minimum information about any (x) sequence (MIxS) specifications\"<\/a>. <i>Nature Biotechnology<\/i> <b>29<\/b> (5): 415-20. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt.1823\" target=\"_blank\">10.1038\/nbt.1823<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3367316\/\" target=\"_blank\">PMC3367316<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21552244\" target=\"_blank\">21552244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3367316\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3367316<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Minimum+information+about+a+marker+gene+sequence+%28MIMARKS%29+and+minimum+information+about+any+%28x%29+sequence+%28MIxS%29+specifications&rft.jtitle=Nature+Biotechnology&rft.aulast=Yilmaz%2C+P.%3B+Kottmann%2C+R.%3B+Field%2C+D.%3B+Knight%2C+R.%3B+Cole%2C+J.R.%3B+Amaral-Zettler%2C+L.%3B+Gilbert%2C+J.%3B+Karsch-Mizrachi%2C+I.%3B+Johnston%2C+A.%3B+Cochrane%2C+G.+et+al.&rft.au=Yilmaz%2C+P.%3B+Kottmann%2C+R.%3B+Field%2C+D.%3B+Knight%2C+R.%3B+Cole%2C+J.R.%3B+Amaral-Zettler%2C+L.%3B+Gilbert%2C+J.%3B+Karsch-Mizrachi%2C+I.%3B+Johnston%2C+A.%3B+Cochrane%2C+G.+et+al.&rft.date=2011&rft.volume=29&rft.issue=5&rft.pages=415-20&rft_id=info:doi\/10.1038%2Fnbt.1823&rft_id=info:pmc\/PMC3367316&rft_id=info:pmid\/21552244&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3367316&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RaynerASimp06-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RaynerASimp06_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rayner, T.F.; Rocca-Serra, P.; Spellman, P.T.; Causton, H.C.; Farne, A.; Holloway, E.; Irizarry, R.; Liu, J.; Maier, D.S.; Miller, M. et al. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1687205\" target=\"_blank\">\"A simple spreadsheet-based, MIAME-supportive format for microarray data: MAGE-TAB\"<\/a>. <i>BMC Bioinformatics<\/i> <b>7<\/b>: 489. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-7-489\" target=\"_blank\">10.1186\/1471-2105-7-489<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1687205\/\" target=\"_blank\">PMC1687205<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17087822\" target=\"_blank\">17087822<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1687205\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1687205<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+simple+spreadsheet-based%2C+MIAME-supportive+format+for+microarray+data%3A+MAGE-TAB&rft.jtitle=BMC+Bioinformatics&rft.aulast=Rayner%2C+T.F.%3B+Rocca-Serra%2C+P.%3B+Spellman%2C+P.T.%3B+Causton%2C+H.C.%3B+Farne%2C+A.%3B+Holloway%2C+E.%3B+Irizarry%2C+R.%3B+Liu%2C+J.%3B+Maier%2C+D.S.%3B+Miller%2C+M.+et+al.&rft.au=Rayner%2C+T.F.%3B+Rocca-Serra%2C+P.%3B+Spellman%2C+P.T.%3B+Causton%2C+H.C.%3B+Farne%2C+A.%3B+Holloway%2C+E.%3B+Irizarry%2C+R.%3B+Liu%2C+J.%3B+Maier%2C+D.S.%3B+Miller%2C+M.+et+al.&rft.date=2006&rft.volume=7&rft.pages=489&rft_id=info:doi\/10.1186%2F1471-2105-7-489&rft_id=info:pmc\/PMC1687205&rft_id=info:pmid\/17087822&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1687205&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note--29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/fged.org\/site_media\/pdf\/MAGE-TABv1.1_2011_07_28.pdf\" target=\"_blank\">\"MAGE-TAB Specification Version 1.1\"<\/a> (PDF). Functional Genomics Data Society. 31 May 2011<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/fged.org\/site_media\/pdf\/MAGE-TABv1.1_2011_07_28.pdf\" target=\"_blank\">http:\/\/fged.org\/site_media\/pdf\/MAGE-TABv1.1_2011_07_28.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MAGE-TAB+Specification+Version+1.1&rft.atitle=&rft.date=31+May+2011&rft.pub=Functional+Genomics+Data+Society&rft_id=http%3A%2F%2Ffged.org%2Fsite_media%2Fpdf%2FMAGE-TABv1.1_2011_07_28.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-jQWidgets-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-jQWidgets_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.jqwidgets.com\/\" target=\"_blank\">\"jQWidgets: Advanced jQuery and HTML5 UI framework\"<\/a>. jQWidgets Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.jqwidgets.com\/\" target=\"_blank\">http:\/\/www.jqwidgets.com\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=jQWidgets%3A+Advanced+jQuery+and+HTML5+UI+framework&rft.atitle=&rft.pub=jQWidgets+Ltd&rft_id=http%3A%2F%2Fwww.jqwidgets.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CrawfordRinRuby09-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CrawfordRinRuby09_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Dahl, D.B.; Crawford, S. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.jstatsoft.org\/article\/view\/v029i04\" target=\"_blank\">\"RinRuby: Accessing the R interpreter from pure ruby\"<\/a>. <i>Journal of Statistical Software<\/i> <b>29<\/b> (4). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.18637%2Fjss.v029.i04\" target=\"_blank\">10.18637\/jss.v029.i04<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.jstatsoft.org\/article\/view\/v029i04\" target=\"_blank\">http:\/\/www.jstatsoft.org\/article\/view\/v029i04<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=RinRuby%3A+Accessing+the+R+interpreter+from+pure+ruby&rft.jtitle=Journal+of+Statistical+Software&rft.aulast=Dahl%2C+D.B.%3B+Crawford%2C+S.&rft.au=Dahl%2C+D.B.%3B+Crawford%2C+S.&rft.date=2009&rft.volume=29&rft.issue=4&rft_id=info:doi\/10.18637%2Fjss.v029.i04&rft_id=http%3A%2F%2Fwww.jstatsoft.org%2Farticle%2Fview%2Fv029i04&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-jqGrid-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-jqGrid_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.trirand.com\/blog\/\" target=\"_blank\">\"jQuery Grid Plugin - jqGrid\"<\/a>. TriRand Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.trirand.com\/blog\/\" target=\"_blank\">http:\/\/www.trirand.com\/blog\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=jQuery+Grid+Plugin+-+jqGrid&rft.atitle=&rft.pub=TriRand+Ltd&rft_id=http%3A%2F%2Fwww.trirand.com%2Fblog%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-S2C-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-S2C_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/github.com\/kumarsaurabh20\/PlotApp\" target=\"_blank\">\"PlotApp - S2C implementation on rails using RinRuby ruby script\"<\/a>. GitHub<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/kumarsaurabh20\/PlotApp\" target=\"_blank\">https:\/\/github.com\/kumarsaurabh20\/PlotApp<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=PlotApp+-+S2C+implementation+on+rails+using+RinRuby+ruby+script&rft.atitle=&rft.pub=GitHub&rft_id=https%3A%2F%2Fgithub.com%2Fkumarsaurabh20%2FPlotApp&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MitsuteroBioRuby03-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MitsuteroBioRuby03_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mitsuteru, N.G.; Mitsuteru, N.G.; Goto, N.; Goto, N.; Nakao, M.C.; Nakao, M.C.; Kawashima, S.; Kawashima, S.; Katayama, T.; Katayama, T. et al. (2003). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.jsbi.org\/pdfs\/journal1\/GIW03\/GIW03P191.pdf\" target=\"_blank\">\"BioRuby: Open-source bioinformatics library\"<\/a> (PDF). <i>Genome Informatics<\/i> <b>14<\/b>: 629-630<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.jsbi.org\/pdfs\/journal1\/GIW03\/GIW03P191.pdf\" target=\"_blank\">https:\/\/www.jsbi.org\/pdfs\/journal1\/GIW03\/GIW03P191.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioRuby%3A+Open-source+bioinformatics+library&rft.jtitle=Genome+Informatics&rft.aulast=Mitsuteru%2C+N.G.%3B+Mitsuteru%2C+N.G.%3B+Goto%2C+N.%3B+Goto%2C+N.%3B+Nakao%2C+M.C.%3B+Nakao%2C+M.C.%3B+Kawashima%2C+S.%3B+Kawashima%2C+S.%3B+Katayama%2C+T.%3B+Katayama%2C+T.+et+al.&rft.au=Mitsuteru%2C+N.G.%3B+Mitsuteru%2C+N.G.%3B+Goto%2C+N.%3B+Goto%2C+N.%3B+Nakao%2C+M.C.%3B+Nakao%2C+M.C.%3B+Kawashima%2C+S.%3B+Kawashima%2C+S.%3B+Katayama%2C+T.%3B+Katayama%2C+T.+et+al.&rft.date=2003&rft.volume=14&rft.pages=629-630&rft_id=https%3A%2F%2Fwww.jsbi.org%2Fpdfs%2Fjournal1%2FGIW03%2FGIW03P191.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Apache20-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Apache20_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.apache.org\/licenses\/LICENSE-2.0\" target=\"_blank\">\"Apache License\"<\/a>. Apache Software Foundation. January 2004<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.apache.org\/licenses\/LICENSE-2.0\" target=\"_blank\">http:\/\/www.apache.org\/licenses\/LICENSE-2.0<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 June 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Apache+License&rft.atitle=&rft.date=January+2004&rft.pub=Apache+Software+Foundation&rft_id=http%3A%2F%2Fwww.apache.org%2Flicenses%2FLICENSE-2.0&rfr_id=info:sid\/en.wikipedia.org:Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210221\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.834 seconds\nReal time usage: 0.873 seconds\nPreprocessor visited node count: 27449\/1000000\nPreprocessor generated node count: 37586\/1000000\nPost\u2010expand include size: 265132\/2097152 bytes\nTemplate argument size: 88265\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 827.259 1 - -total\n 87.88% 726.988 1 - Template:Reflist\n 76.39% 631.923 35 - Template:Citation\/core\n 60.10% 497.151 24 - Template:Cite_journal\n 14.00% 115.840 8 - Template:Cite_web\n 8.68% 71.779 54 - Template:Citation\/identifier\n 7.97% 65.951 1 - Template:Infobox_journal_article\n 7.61% 62.972 1 - Template:Infobox\n 7.48% 61.896 3 - Template:Cite_book\n 4.43% 36.660 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7907-0!*!0!!en!5!* and timestamp 20181213210220 and revision id 23420\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data\">https:\/\/www.limswiki.org\/index.php\/Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","9847a6beea0bba9383335c5a41e65647_images":["https:\/\/www.limswiki.org\/images\/8\/88\/Fig1_SinghBMCBioinformatics2015_12-6.png","https:\/\/www.limswiki.org\/images\/a\/a1\/Fig2_SinghBMCBioinformatics2015_12-6.png","https:\/\/www.limswiki.org\/images\/c\/c0\/Fig3_SinghBMCBioinformatics2015_12-6.png"],"9847a6beea0bba9383335c5a41e65647_timestamp":1544734940,"0a86ce45c610776ca0038eb4e754f3c5_type":"article","0a86ce45c610776ca0038eb4e754f3c5_title":"Requirements for data integration platforms in biomedical research networks: A reference model (Ganzinger et al. 2015)","0a86ce45c610776ca0038eb4e754f3c5_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model","0a86ce45c610776ca0038eb4e754f3c5_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Requirements for data integration platforms in biomedical research networks: A reference model\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nRequirements for data integration platforms in biomedical research networks: A reference modelJournal\n \nPeerJ Computer ScienceAuthor(s)\n \nGanzinger\u200b, Matthias; Knaup, PetraAuthor affiliation(s)\n \nHeidelberg UniversityPrimary contact\n \nEmail: matthias.ganzinger@med.uni-heidelberg.deEditors\n \nJuan, Hsueh-FenYear published\n \n2015Volume and issue\n \n3Page(s)\n \ne755DOI\n \n10.7717\/peerj.755ISSN\n \n2376-5992Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttps:\/\/peerj.com\/articles\/755\/Download\n \nhttps:\/\/peerj.com\/articles\/755.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Methods \n4 Results \n\n4.1 Reference model \n4.2 Sample application: pelican \n\n4.2.1 Specific model for requirements \n4.2.2 System architecture \n\n\n\n\n5 Discussion \n6 Supplemental information \n7 Additional information and declarations \n\n7.1 Competing interests \n7.2 Author contributions \n7.3 Human ethics \n7.4 Funding \n\n\n8 References \n9 Notes \n\n\n\nAbstract \nBiomedical research networks need to integrate research data among their members and with external partners. To support such data sharing activities, an adequate information technology infrastructure is necessary. To facilitate the establishment of such an infrastructure, we developed a reference model for the requirements. The reference model consists of five reference goals and 15 reference requirements. Using the Unified Modeling Language, the goals and requirements are set into relation to each other. In addition, all goals and requirements are described textually in tables. This reference model can be used by research networks as a basis for a resource efficient acquisition of their project specific requirements. Furthermore, a concrete instance of the reference model is described for a research network on liver cancer. The reference model is transferred into a requirements model of the specific network. Based on this concrete requirements model, a service-oriented information technology architecture is derived and also described in this paper.\nKeywords: Research network, Reference model, Data integration, Biomedical informatics, Service-oriented architecture \n\nIntroduction \nCurrent biomedical research is supported by modern biotechnological methods producing vast amounts of data (Frey, Maojo & Mitchell, 2007[1]; Baker, 2010[2]). In order to get a comprehensive picture of the physiology and pathogenic processes of diseases, many facets of biological mechanisms need to be examined. Contemporary research, e.g., investigating cancer, is a complex endeavor that can be conducted most successfully when researchers of multiple disciplines cooperate and draw conclusions from comprehensive scientific data sets (Welsh, Jirotka & Gavaghan, 2006[3]; Mathew et al., 2007[4]). As a frequent measure to support cooperation, research networks sharing common resources are established.\nTo generate added value from such a network, all available scientific and clinical data should be combined to facilitate a new, comprehensive perspective. This requires provision of adequate information technology (IT) which is a challenge on all levels of biomedical research. For example, it is inevitable for research networks to use an IT infrastructure for sharing data and findings in order to leverage joint analyses. Data generated by biotechnological devices can only be evaluated thoroughly by applying biostatistical methods with IT tools.\nHowever, data structures are often heterogeneous, resulting in the need for a data integration process. This process involves the harmonization of data structures by defining appropriate metadata (Cimino, 1998[5]). Depending on the specific needs and data structures of the research network, often a non-standard IT platform needs to be developed to meet the specific requirements. An important requirement might be the protection of data in terms of security and privacy, especially when patient data are involved.\nIn the German research network SFB\/TRR77 \u2014 Liver Cancer: From Molecular Pathogenesis to Targeted Therapies it was our task to explore the most appropriate IT-architecture for supporting networked research (Woll, Manns & Schirmacher, 2013[6]). The research network consists of 22 projects sharing common resources and research data. To provide this network with a data integration platform we implemented a service-oriented architecture (SOA) (Taylor et al., 2004[7]; Papazoglou et al., 2008[8]; Wei & Blake, 2010[9]; Bosin, Dess\u00ec & Pes, 2011[10]). The IT system is based on the cancer Common Ontologic Representation Environment Software Development Kit (caCORE SDK) components of the cancer Biomedical Informatics Grid (caBIG) (Komatsoulis et al., 2008[11]; Kunz, Lin & Frey, 2009[12]). The resulting system is called pelican (platform enabling liver cancer networked research) (Ganzinger et al., 2011[13]). Transfer of these data sharing concepts to other networks investigating different disease areas is possible.\nWe consider our research network as a typical example for a whole class of biomedical research networks. To support this kind of projects, we provide a framework for the development of data integration platforms for such projects. Specifically, we strive for the following two objectives:\nObjective 1: Provide a reference model of requirements of biomedical research networks regarding an IT platform for sharing and analyzing data.\nObjective 2: Design a SOA of an IT platform for our research network on liver cancer. It should implement the reference model for requirements. While this SOA is specific to this project, parts can be reused for similar projects.\n\nMethods \nFor the design of a data integration platform it is important to first assess the requirements of the system\u2019s intended users. To support this task, we developed a reference model for requirements. A reference model is a generic model which is valid not only for a specific research network but for a class of such organizations. For the development of the reference model, we used the research network on liver cancer as primary source. These requirements were consolidated and abstracted to get a generic model that can be applied to other research networks.\nIn the first step, a general understanding of the network\u2019s aims and tasks was acquired by analyzing written descriptions of the participating projects. In addition, questionnaires were sent to the principal investigators to capture the data types and data formats used within the projects. In the second step, projects were visited and their research subjects and processes were captured by interviewing project members.\nFor the reference model, we use the term goal to describe the highest level of requirements. This is in accordance with ISO\/IEC\/IEEE 24765, where goal is defined as \u201can intended outcome\u201d (ISO\/IEC\/IEEE, 2011[14]). In contrast, requirement is defined as \u201ca condition or capability needed by a user to solve a problem or achieve an objective\u201d (ISO\/IEC\/IEEE, 2011[14]). In our reference model, each requirement was related to a goal, either directly or indirectly. Requirements were then mapped to concrete functions in the resulting data integration system. On the other hand, goals were used to structure requirements and usually do not lead to a specific function of the system.\nTo provide a more detailed characterization of the goals, we provide a standardized table for each of them which covers the reference number, name, description and weight of the goal. Table 1 shows the structure of such a table. The complete set of tables for all the goals is available in Table S1.\n\n\n\n\n\n\n\n Feature\n\n Explanation\n\n\nNumber\n\nNumber for uniquely identifying requirements\n\n\nName\n\nName of the requirement\n\n\nDescription\n\nVerbal description of the requirement and its properties\n\n\nWeighting\n\nImportance of the requirement for fulfilling the goals of the project (low, medium, high)\n\n\n\n\nTable 1. Schema for documenting reference requirements and goals;\r\nCite this object using this DOI: 10.7717\/peerj.755\/table-1\nThe requirements are documented in the same way as the goals. Figure 1 shows a Unified Modeling Language (UML) diagram with all elements used for describing of both goals and their subordinated requirements (Object Management Group, 2012[15]). As for the goals, we provide a set of tables with more detailed descriptions for all requirements in Table S1 In total, we identified 15 requirements for the reference model.\n\n\n\nFigure 1: Overview of the UML elements used in requirements diagrams;\r\nCite this object using this DOI: 10.7717\/peerj.755\/fig-1\nThe instantiation of the reference model for requirements to meet the needs of a specific research network provides the basis for the architecture of the desired data integration and its subsequent implementation. We provide a concrete instance of a reference model as well as the resulting IT-architecture in this manuscript.\nFor the research described in this paper, ethics approval was not deemed necessary. This work involved no human subjects in the sense of medical research, as e.g., covered by the Declaration of Helsinki (World Medical Association, 2013[16]). At no time were patients included for survey or interview. Data was only acquired from scientists regarding their work and data, but no personal or patient related data were gathered. Participants were not required to participate in this study; they consented by returning the questionnaire. No research was conducted outside Germany, the authors\u2019 country of residence. However, in other countries the approval of an institutional review board or other authority might be necessary to apply the reference model.\n\nResults \nIn this section we describe the reference model for requirements. Then, we show how a concrete model for requirements and an IT-architecture is derived from this reference model. The reference model for requirements is an abstract model and thus a universally usable artefact. It is mapped in several steps to the network specific system architecture.\n\nReference model \nThe reference model for requirements covers five reference goals (RG). An overview of the goals and their relations is shown in Fig. 2 by means of a UML requirements diagram. The reference goals are:\n\n Conduct research project (reference goal RG1): The ultimate goal of a research network is to fulfill the intended research tasks. This usually corresponds to the project specification of the funding organization.\n Answer research questions (reference goal RG2): Each research network has specific research questions it pursues to answer. These questions frame the core of the network and led to its establishment in the first place.\n Create, store, and retrieve data (reference goal RG3): Research networks need data to conduct the project. Thus, it is necessary to generate and handle them.\n Analyze data (reference goal RG4): To generate knowledge out of the data it is necessary to analyze them.\n Control data access and usage (reference goal RG5): Research networks need to protect their data. This includes the prevention of unauthorized access to protected data like patient data as well as aspects of intellectual property rights that need to be respected by authorized users as well.\nThese goals are ordered in a hierarchical structure: Goal RG1 acts as the root node, which has the two sub goals RG2 and RG3. Goal RG4 is subordinated to Goal RG2, whereas Goal RG5 is a sub goal of Goal RG3.\n\n\n\nFigure 2: Reference model for goals of a research network;\r\nCite this object using this DOI: 10.7717\/peerj.755\/fig-2\nEach goal has several requirements. In total, the reference model contains 15 reference requirements (RR). A UML diagram with all reference goals and reference requirements is shown in Fig. 3. Reference requirements are associated with the reference goals as follows:\n\n\n\nFigure 3: Reference model for goals and requirements of a research network;\r\nCite this object using this DOI: 10.7717\/peerj.755\/fig-3\nGoal 3 is associated with the reference requirements to create data (RR1) and to retrieve external data (RR2). These two requirements respect possible sources of data necessary for the research network. Reference requirement RR3, represent data, is further defined by its subordinate requirements define syntax (RR4), define data model (RR5), identify data (RR6), and define semantics (RR7).\nGoal 5, control data access and usage, has two aspects, which are represented by reference requirements RR8 and RR9. RR8 requires the creator\u2019s contribution in the generation of data for the research network to be recognized when data are used by others. As a consequence, even users with legitimate access to the system have to adhere to usage regulations. These regulations should be checked and enforced by the system as far as possible. In contrast, RR9 covers the requirement to protect data from unauthorized access.\nThe second group of reference requirements covers data analysis. At the highest level we identify goal 2, answer research questions. It is associated downstream with goal RG4, analyze data. Goal RG4 is composed of two reference requirements: RR11 integrate data and RR13 define analytical process. RR11 is associated with RR3, since the technical provision of data within the research network is of great relevance for the integration of data. RR13 has subordinate requirement RR12, define analytical methods, which covers the low-level data analysis methods. RR14 and RR15 cover two distinct instances of RR13: RR14 describes static workflows with all process steps being fixed. In this case, the order of analytical steps and data sources used cannot be changed by the users of the system. In contrast, RR15 considers dynamic workflows, allowing users to compose analytical steps and data sources as needed. Since the type of data involved in a dynamic workflow is not known upfront, this reference requirement is more demanding in terms of the semantic description of data sources. Precise annotation of data sources is necessary in order to perform automated transformations for matching different data fields.\nRR13 is further associated with RR10, show results. RR10 covers the requirement to present the results of the analysis adequately. Thus, it partially fulfills goal 2, answer research questions.\nThe reference model for requirements is the basis for a network specific requirements model. We present an example for creating such a model and all the following steps in the next section. All goals and requirements from the reference model are mapped to network specific instances. In this process, elements of the reference model are checked for their applicability to the specific research network. Further special requirements of the network are considered at this point as well.\nThe network specific requirements model is then mapped to system properties. These are qualities contributed to the system by different components. At first, we consider abstract components instead of specific products. For example, in a research network, reference requirement RR1 create data might be mapped to a system property automated creation of data services. This property is then mapped to the specific component responsible for the implementation of this property.\nIn a second step, the abstract components are mapped to specific components in accordance with the research network\u2019s requirements. Specific components can be preexisting modules with a product character, software development frameworks providing specific functionality, or newly developed components.\nIn a final modelling step a distribution model of the components is created. All components need to be mapped to system resources down to the hardware level. Among others, the following aspects have to be considered is this step:\n\n Security: Components with high security requirements should be isolated against other, less sensitive components and thus be run on a separate system node.\n Performance: All components must be distributed in a way that availability of sufficient system resources is ensured.\n Maintainability: To ensure that the possibly complex distributed system can be managed efficiently, components should be grouped together in a sensible way.\nSample application: pelican \nWe now describe a sample application of our reference model within the research network SFB\/TRR77 on liver cancer. Further we describe two specifications for metadata we developed for the research network.\n\nSpecific model for requirements \nIn this section we summarize key requirements specific to our research network. The complete list of requirements is shown in Table S2.\nThe first goal of the research network, an instance of reference goal RG1, is defined by its research assignment of gaining a deeper understanding of the molecular basis of liver cancer development. This spans research on the chronic liver disease to progression of metastatic cancer. Further, the research network aims to identify novel preventive, diagnostic and therapeutic approaches on liver cancer. Subordinated to goal G1 is G3, the instance of reference goal RG3 regarding the data necessary for the network. Since molecular processes play a major role within the research network, genomic microarray data are of central importance. They are complemented by imaging data like tissue microarray (TMA) data and clinical data.\nGoal G2, answering research questions, is characterized by the following two questions:\n\n Which generic or specific mechanisms of chronic liver diseases, especially of chronic virus infections and inflammation mediated processes predispose or initiate liver cancer?\n Which molecular key events promoting or keeping up liver cancer could act as tumor markers or are promising targets for future therapeutic interventions?\nGoal G5 requires making the data available for cross project analysis within the network, but to protect data against unauthorized access at the same time. Especially important to the members of the project is the requirement R8, subordinated to goal G5: The projects contributing data to the network require keeping control over the data in order to ensure proper crediting of their intellectual property. Thus, they require fine-grained rules for data access control. Depending on the type of data, they should be available only to specific members of the network, to all members of the project or the general public.\n\nSystem architecture \nTo acknowledge the project\u2019s requirement R8, to keep the ownership over their data, a federation was used as the underlying concept of the system architecture. Technically, pelican implements a SOA. All data sources of the projects are transformed into data services and made available to the research network. The data services stay under the control of the contributing project. This can even go as far as running the service on computer hardware on the projects\u2019 premises. Data services are complemented by analytical services. All services are described by standardized metadata to help finding appropriate services and allow for automated access to the services\u2019 interfaces. Using a web-based user interface, researchers can chain data services and analytical services to answer specific research questions.\nComponent model\nThe requirements are mapped to system properties first. In the next step, components are identified to provide these properties as module of the new system. In Table 2 we show the complete chain of mappings from requirements over system features to components. Each component is realized either by a readily available product or by a newly developed module. In Table 3 we give an overview of our components.\n\n\n\n\n\n\n\n Requirement\n\n System feature\n\n Component\n\n\nRequirements associated with G2\n\n\nR1 Create data\n\nAutomated creation of data services\n\nData service framework\n\n\nR2 Retrieve external data\n\nIntegration of external data services\n\nData service framework, portal\n\n\nR3 Represent data\n\nData service, document service\n\nData service framework, document management system\n\n\nR4 Define syntax\n\nService description\n\nData service framework\n\n\nR5 Define data model\n\nDefined data model\n\nData service framework\n\n\nR6 Identify data\n\nProvisioning of information on service location\n\nMeta data directory\n\n\nR7 Define semantics\n\nDefinition of controlled vocabulary and ontologies\n\nTerminology server\n\n\nRequirements associated with G4\n\n\nR8 Administrate intellectual property\n\nLog data usage\n\nPortal, data service framework\n\n\nR9 Protect data\n\nUser authentication \/ User authorization\n\nPortal, security service Portal, data service framework, security service\n\n\nRequirements associated with G5\n\n\nR10 Show results\n\nData specific portlets\n\nPortal\n\n\nR11 Integrate data\n\nAnalytical services\n\nPortal, statistics service, data service framework\n\n\nR12 Define analytical methods\n\nStatistical methods\n\nStatistics service\n\n\nR13 Define analytical process\n\nDocumentation service\n\nDocument management system\n\n\nR14 Static workflow\n\nWorkflow in portal application\n\nPortal\n\n\nR15 Dynamic workflow\n\nFlexible pipeline\n\nPipeline management\n\n\n\n\nTable 2. Mapping of requirements to corresponding system features and components;\r\nCite this object using this DOI: 10.7717\/peerj.755\/table-2\n\n\n\n\n\n\n Abstract component\n\n Implementing component\n\n\nPortal\n\nLiferay\n\n\nData service framework\n\ncaCORE SDK\n\n\nMeta data directory\n\nInternal development (based on caCORE SDK)\n\n\nTerminology server\n\nTemaTres\n\n\nSecurity service\n\ncaCORE SDK, LDAP\n\n\nStatistics service\n\nR\n\n\nDocument management system\n\nAlfresco\n\n\nPipeline management\n\nGalaxy (planned)\n\n\n\n\nTable 3. Specification of concrete implementation components for the elements of the component model;\r\nCite this object using this DOI: 10.7717\/peerj.755\/table-3\nThe portal component provides the user interface to the system. It is implemented using the open source software Liferay (http:\/\/www.liferay.com, accessed: 2014-07-03) (Sezov, Jr, 2011[17]). Liferay provides a number of functions affecting several components of our model. Thus, we provide a decomposition of the portal components in Fig. 4. One important subcomponent of the portal is the document management system. It is realized by the Alfresco component (http:\/\/www.alfresco.com, accessed: 2014-07-03) (Berman, Barnett & Mooney, 2012[18]). The user interface of Alfresco can be integrated into the Liferay portal or be accessed with a separate unified resource locator (URL). The portal provides user management functionality to control access to portal pages and components like portlets (Java Community Process, 2008[19]). However, the user account information including username, passwords, and others is stored in a separate component using the Lightweight Directory Access Protocol (LDAP). Thus, it is possible for all components of the SOA-network to commonly access the users\u2019 identity information.\n\n\n\nFigure 4: Structure of the component portal;\r\nCite this object using this DOI: 10.7717\/peerj.755\/fig-4\nData services are generated by using caCORE SDK (Wiley & Gagne, 2012[20]). With caCORE SDK it is not necessary to program the software for the service in a traditional way. Instead, a UML data model in Extensible Markup Language Metadata Interchange (XMI) notation has to be prepared (Object Management Group, 2002[15]; Bray et al., 2006[21]). From this model, caCORE SDK generates several artefacts resulting in a deployment packages for Java application servers like Apache Tomcat (The Apache Software Foundation, 2014[22]). To simplify this process for spreadsheet based microarray data, we developed a software tool to generate the XMI file as well. As a result, a service conforming to the web services specification ready for deployment is generated. For the provision of network specific metadata we chose TemaTres to serve our controlled vocabulary in standard formats like SKOS or Dublin Core (Weibel, 1997[23]; Miles & Bechhofer, 2009[24]; Gonzales-Aguilar, Ram\u00edrez-Posada & Ferreyra, 2012[25]). Our analytical services are backed by the open source language and environment for statistical computing called R (R Core Team, 2014[26]). R is integrated into the services using the Rserve component (Urbanek, 2003[27]).\nDeployment model\nIn a final modelling step the components are distributed to the physical resources available for the system. In our case we used two servers with a common virtualization layer based on VMware VSphere server. Thus, all nodes in our deployment model represent virtual machines (VM). Using virtual switches, routers, and firewall appliances we were able to implement our Internet Protocol (IP) network infrastructure. To enhance security, we implement a network zoning model comprised of an internet zone, a demilitarized zone (DMZ) and an internal zone. Figure 5 shows the deployment model in UML notation. The services shown in the model (service 1 to service n) are to be considered as examples, since the concrete number of services is permanently changing. The deployment model also reflects the different levels of control that can be executed by the owners of the data. They range from shared nodes on the common servers over a dedicated VM to deployment on external hardware controlled by the respective projects.\n\n\n\nFigure 5: Deployment diagram of the components of the architecture in UML notation;\r\nCite this object using this DOI: 10.7717\/peerj.755\/fig-5\nDiscussion \nIn this manuscript, we describe a reference model for the requirements of research networks towards an IT platform. For many funding programs, including research grants of the European Commission, the collaboration of several research organizations at different sites is mandatory. This leads to a structural similarity to our research network on liver cancer. Even though other research networks will have different research aims, there are still requirements that are common to most networks. Since the reference model already covers a basic set of requirements, it allows future research networks to focus on defining specific requirements distinguishing them from other networks.\nUsers of the reference model are responsible for assessing the reference model\u2019s applicability to their project-specific needs. The reference model is based on data of a real research network that were generalized. To avoid bias in the model that might hinder transferability, we incorporated different views in the process of constructing the model. However, the transferability of the model to another context is, as for any model, limited. As a consequence, future research networks will have to derive a project specific instance of the reference model to reflect the corresponding characteristics of the project. The reference model is a tool intended to help its users to create a concrete model covering the requirements of a research network with a high degree of completeness. The reference model provides guidance for this task. We expect that it will help in reducing the effort to acquire all requirements.\nA common technique used in requirements engineering for software is use case modelling (Jacobson, 1992[28]; Bittner & Spence, 2002[29]). A use case is a compact scenario describing certain aspects of how a technical system behaves and how it interacts with other actors like its users. A use case model is a way of capturing requirements in an interactive way. Thus, a use case model can be developed further into a requirements model. In our context we found it hard to apply the use case approach since the researchers in our network mostly have biomedical backgrounds and thus are not familiar with software development. Nevertheless, use case modelling might be a helpful tool for other research networks when applying our reference model.\nWe applied the reference model successfully to a research network on liver cancer. Some specific requirements in this network led to the decision to set up a federated system allowing for a maximum of control of the individual projects over their respective data. The system was implemented as a service-oriented architecture using, among others, components of the caBIG project. A public version of our system with limited functionality is available at https:\/\/livercancer.imbi.uni-heidelberg.de\/data. There, a gene symbol like BRCA1 can be entered into the search field and services are invoked for data retrieval as discussed in this paper. Other projects can benefit from this architecture as well, but the architecture is tailored to research networks with the requirements of federating data as data services. With this architecture, we try to acknowledge the data protection requirements of the participating projects. Still, further research regarding the use of data and crediting creatorship of data is necessary. First steps were made as part of this project (He et al., 2013[30]).\nIn case the requirements regarding data control are more relaxed, an alternative would be to keep the data in a central data warehouse instead of the federation. In that case, i2b2 might be a suitable component to provide the data warehouse component (Murphy et al., 2007[31]). Such a centralized approach also affects how and when data are harmonized: In a central research data warehouse data are harmonized at the time of loading the database, which ideally leads to a completely and consistently harmonized database. In a service-oriented approach data are provided by means of data services as they are. All services are described by corresponding metadata, enabling automated transformation of the data at time of access.\nOur sample research network concentrates more on basic research than clinical application. In the future, we plan to apply our reference model to further projects with a stronger translational component. By doing so, we will be able to reevaluate the framework in a more clinical context.\n\nSupplemental information \n\n\n\n\n\n\n Table S1. Reference goals and requirements\n\n\nCite this object using this DOI: 10.7717\/peerj.755\/supp-1\nFormat: PDF Download file\n\n\n\n\n\n\n Table S2. Goals and requirements for pelican\n\n\nCite this object using this DOI: 10.7717\/peerj.755\/supp-2\nFormat: PDF Download file\n\n\n\n\nAdditional information and declarations \nCompeting interests \nThe authors declare there are no competing interests.\n\nAuthor contributions \nMatthias Ganzinger conceived and designed the experiments, performed the experiments, analyzed the data, wrote the paper, prepared figures and\/or tables, reviewed drafts of the paper.\nPetra Knaup wrote the paper, reviewed drafts of the paper.\n\nHuman ethics \nThe following information was supplied relating to ethical approvals (i.e., approving body and any reference numbers):\nFor the research described in this paper, ethics approval was not deemed necessary. This work involved no human subjects in the sense of medical research, as e.g., covered by the Declaration of Helsinki (World Medical Association, 2013[16]). At no time patients were included for survey or interview. Data was only acquired from scientists regarding their work and data, but no personal or patient related data were gathered. Participants were not required to participate in this study; they consented by returning the questionnaire. No research was conducted outside Germany, the authors\u2019 country of residence. However, in other countries the approval of an institutional review board or other authority might be necessary to apply the reference model.\n\nFunding \nThis work was funded by the SFB\/TRR 77 \u201cLiver Cancer. From Molecular Pathogenesis to Targeted Therapies\u201d of the Deutsche Forschungsgemeinschaft (DFG, http:\/\/www.dfg.de). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\n\nReferences \n\n\n\u2191 Frey, L.J.; Maojo, V.; Mitchell, J.A. (2007). \"Bioinformatics linkage of heterogeneous clinical and genomic information in support of personalized medicine\". IMIA Yearbook of Medical Informatics 2007: 98-105. ISSN 0943-4747. PMID 17700912.   \n\n\u2191 Baker, M. (2010). \"Next-generation sequencing: adjusting to data overload\". Nature Methods 7 (7): 495-499. doi:10.1038\/nmeth0710-495.   \n\n\u2191 Welsh, E.; Jirotka, M.; Gavaghan, D. (2006). \"Post-genomic science: cross-disciplinary and large-scale collaborative research and its organizational and technological challenges for the scientific research process\". Philosophical Transactions of the Royal Society A 364 (1843): 1533-1549. doi:10.1098\/rsta.2006.1785. PMID 16766359.   \n\n\u2191 Mathew, J.P.; Taylor, B.S.; Bader, G.D.; Pyarajan, S.; Antoniotti, M.; Chinnaiyan, A.M.; Sander, C.; Burakoff, S.J.; Mishra, B. (2007). \"From bytes to bedside: data integration and computational biology for translational cancer research\". PLOS Computational Biology 3 (2): e12. doi:10.1371\/journal.pcbi.0030012. PMC PMC1808026. PMID 17319736. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1808026 .   \n\n\u2191 Cimino, J.J. (1998). \"Desiderata for controlled medical vocabularies in the twenty-first century\". Methods of Information in Medicine 37 (4\u20135): 394-403. PMC PMC3415631. PMID 9865037. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3415631 .   \n\n\u2191 Woll, K.; Manns, M.; Schirmacher, P. (2013). \"Sonderforschungsbereich SFB\/TRR77: Leberkrebs: Von der molekularen Pathogenese zur zielgerichteten Therapie\". Der Pathologe 34 (2): 232-234. doi:10.1007\/s00292-013-1820-z.   \n\n\u2191 Taylor, K.L.; O\u2019Keefe, C.M.; Colton, J.; Baxter, R.; Sparks, R.; Srinivasan, U.; Cameron, M.A.; Lefort, L. (2004). \"A service oriented architecture for a health research data network\". Proceedings of the 16th International Conference on Scientific and Statistical Database Management 2004: 443-444. doi:10.1109\/SSDM.2004.1311251.   \n\n\u2191 Papazoglou, M.P.; Traverso, P.; Dustdar, S.; Leymann, F. (2008). \"Service-oriented computing: A research roadmap\". International Journal of Cooperative Information Systems 17: 223. doi:10.1142\/S0218843008001816.   \n\n\u2191 Wei, Y.; Blake, M.B. (2010). \"Service-oriented computing and cloud computing: Challenges and opportunities\". IEEE Internet Computing 14 (6): 72\u201375. doi:10.1109\/MIC.2010.147.   \n\n\u2191 Bosin, A.; Dess\u00ec, N.; Pes, B. (2011). \"Extending the SOA paradigm to e-Science environments\". Future Generation Computer Systems 27 (1): 20\u201331. doi:10.1016\/j.future.2010.07.003.   \n\n\u2191 Komatsoulis, G.A.; Warzel, D.B.; Hartel, F.W.; Shanbhag, K.; Chilukuri, R.; Fragoso, G.; de Coronado, S.; Reeves, D.M.; Hadfield, J.B.; Ludet, C.; Covitz, P.A. (2008). \"caCORE version 3: Implementation of a model driven, service-oriented architecture for semantic interoperability\". Journal of Biomedical Informatics 41 (1): 106\u2013123. doi:10.1016\/j.jbi.2007.03.009. PMC PMC2254758. PMID 17512259. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2254758 .   \n\n\u2191 Kunz, I.; Lin, M.; Frey, L. (2009). \"Metadata mapping and reuse in caBIG\". BMC Bioinformatics 10 (Suppl 2): S4. doi:10.1186\/1471-2105-10-S2-S4. PMC PMC2646244. PMID 19208192. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2646244 .   \n\n\u2191 Ganzinger, M.; Noack, T.; Diederichs, S.; Longerich, T. Knaup, P. (2011). \"Service oriented data integration for a biomedical research network\". Studies in Health Technology and Informatics 169: 867-71. doi:10.3233\/978-1-60750-806-9-867. PMID 21893870.   \n\n\u2191 14.0 14.1 ISO\/IEC\/IEEE (2011). ISO\/IEC\/IEEE 24765: Systems and software engineering \u2014 Vocabulary. Geneva, Switzerland: ISO.   \n\n\u2191 15.0 15.1 \"OMG XML Metadata Interchange (SMI) Specification (Version 1.2)\" (PDF). Object Management Group, Inc. January 2002. http:\/\/www.omg.org\/cgi-bin\/doc?formal\/02-01-01.pdf . Retrieved 09 December 2014 .   \n\n\u2191 16.0 16.1 World Medical Association (2013). \"World Medical Association Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects\". JAMA 310 (20): 2191-2194. doi:10.1001\/jama.2013.281053. PMID 12432198.   \n\n\u2191 Sezov, Rich (2011). Liferay in Action: The Official Guide to Liferay Portal Development. Shelter Island, NY: Manning Publications. ISBN 9781935182825.   \n\n\u2191 Berman, A.E.; Barnett, W.K.; Mooney, S.D. (2012). \"Collaborative software for traditional and translational research\". Human Genomics 6: 21. doi:10.1186\/1479-7364-6-21. PMC PMC3500217. PMID 23157911. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3500217 .   \n\n\u2191 \"JSR 286: Portlet Specification 2.0\". Oracle Corporation. 2008. https:\/\/www.jcp.org\/en\/jsr\/detail?id=286 . Retrieved 09 December 2014 .   \n\n\u2191 Wiley, A.; Gagne, B. (18 April 2012). \"caCORE SDK Version 4.3 Object Relational Mapping Guide\". National Cancer Institute. https:\/\/wiki.nci.nih.gov\/display\/caCORE\/caCORE+SDK+Version+4.3+Object+Relational+Mapping+Guide . Retrieved 09 December 2014 .   \n\n\u2191 Bray, T.; Paoli, J.; Sperberg-McQueen, C.M.; Maler, E.; Yergeau, F.; Cowan, J. (29 September 2006). \"Extensible Markup Language (XML) 1.1 (Second Edition)\". World Wide Web Consortium. http:\/\/www.w3.org\/TR\/xml11\/ . Retrieved 09 December 2014 .   \n\n\u2191 \"Apache Tomcat\". The Apache Software Foundation. 2014. http:\/\/tomcat.apache.org\/ . Retrieved 02 December 2014 .   \n\n\u2191 Weibel, S. (October\/November 1997). \"The Dublin Core: A Simple Content Description Model for Electronic Resources\". Bulletin of the American Society for Information Science and Technology 24 (1): 9\u201311. doi:10.1002\/bult.70.   \n\n\u2191 Miles, A.; Bechhofer, S. (18 August 2009). \"SKOS Simple Knowledge Organization System Reference\". World Wide Web Consortium. http:\/\/www.w3.org\/TR\/skos-reference\/ . Retrieved 09 December 2014 .   \n\n\u2191 Gonzales-Aguilar, A.; Ram\u00edrez-Posada, M.; Ferreyra, D. (2012). \"Tematres: software para gestionar tesauros\". El Profesional de la Informaci\u00f3n 21 (3): 319\u2013325. doi:10.3145\/epi.2012.may.14.   \n\n\u2191 R Development Core Team (2015). \"R: A language and environment for statistical computing\". The R Foundation for Statistical Computing. https:\/\/www.r-project.org\/ . Retrieved 09 December 2014 .   \n\n\u2191 Urbanek, S. (2003). \"Rserve: A Fast Way to Provide R Functionality to Applications\". In Hornik, K.; Leisch, F.; Zeileis, A.. Proceedings of the 3rd international workshop on distributed statistical computing (DSC 2003).   \n\n\u2191 Jacobson, I. (1992). Object Oriented Software Engineering: A Use Case Driven Approach. Wokingham, England: Addison-Wesley Professional.   \n\n\u2191 Bittner, K.; Spence, I. (2002). Use Case Modeling. Boston, MA: Addison-Wesley Professional.   \n\n\u2191 He, S.; Ganzinger, M.; Hurdle, J.F.; Knaup, P. (2013). \"Proposal for a data publication and citation framework when sharing biomedical research resources\". Studies in Health Technology and Informatics 192: 1201. doi:10.3233\/978-1-61499-289-9-1201. PMID 23920975.   \n\n\u2191 Murphy, S.N.; Mendis, M.; Hackett, K.; Kuttan, R.; Pan, W.; Phillips, L.C.; Gainer, V.; Berkowicz, D.; Glaser, J.P.; Kohane, I.; Chueh, H.C. (2007). \"Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside\". AMIA Annual Symposium Proceedings 2007: 548\u2013552. PMC PMC2655844. PMID 18693896. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655844 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. References were in alphabetical on the original, but references are listed in order of appearance on this wiki. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\">https:\/\/www.limswiki.org\/index.php\/Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on information technology\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:59.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,234 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","0a86ce45c610776ca0038eb4e754f3c5_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Requirements_for_data_integration_platforms_in_biomedical_research_networks_A_reference_model skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Requirements for data integration platforms in biomedical research networks: A reference model<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Biomedical research networks need to integrate research data among their members and with external partners. To support such data sharing activities, an adequate information technology infrastructure is necessary. To facilitate the establishment of such an infrastructure, we developed a reference model for the requirements. The reference model consists of five reference goals and 15 reference requirements. Using the Unified Modeling Language, the goals and requirements are set into relation to each other. In addition, all goals and requirements are described textually in tables. This reference model can be used by research networks as a basis for a resource efficient acquisition of their project specific requirements. Furthermore, a concrete instance of the reference model is described for a research network on liver cancer. The reference model is transferred into a requirements model of the specific network. Based on this concrete requirements model, a service-oriented information technology architecture is derived and also described in this paper.\n<\/p><p><b>Keywords:<\/b> Research network, Reference model, Data integration, Biomedical informatics, Service-oriented architecture \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>Current biomedical research is supported by modern biotechnological methods producing vast amounts of data (Frey, Maojo & Mitchell, 2007<sup id=\"rdp-ebb-cite_ref-FreyBio07_1-0\" class=\"reference\"><a href=\"#cite_note-FreyBio07-1\" rel=\"external_link\">[1]<\/a><\/sup>; Baker, 2010<sup id=\"rdp-ebb-cite_ref-BakerNext10_2-0\" class=\"reference\"><a href=\"#cite_note-BakerNext10-2\" rel=\"external_link\">[2]<\/a><\/sup>). In order to get a comprehensive picture of the physiology and pathogenic processes of diseases, many facets of biological mechanisms need to be examined. Contemporary research, e.g., investigating cancer, is a complex endeavor that can be conducted most successfully when researchers of multiple disciplines cooperate and draw conclusions from comprehensive scientific data sets (Welsh, Jirotka & Gavaghan, 2006<sup id=\"rdp-ebb-cite_ref-WelshPost06_3-0\" class=\"reference\"><a href=\"#cite_note-WelshPost06-3\" rel=\"external_link\">[3]<\/a><\/sup>; Mathew et al., 2007<sup id=\"rdp-ebb-cite_ref-MathewFrom07_4-0\" class=\"reference\"><a href=\"#cite_note-MathewFrom07-4\" rel=\"external_link\">[4]<\/a><\/sup>). As a frequent measure to support cooperation, research networks sharing common resources are established.\n<\/p><p>To generate added value from such a network, all available scientific and clinical data should be combined to facilitate a new, comprehensive perspective. This requires provision of adequate information technology (IT) which is a challenge on all levels of biomedical research. For example, it is inevitable for research networks to use an IT infrastructure for sharing data and findings in order to leverage joint analyses. Data generated by biotechnological devices can only be evaluated thoroughly by applying biostatistical methods with IT tools.\n<\/p><p>However, data structures are often heterogeneous, resulting in the need for a data integration process. This process involves the harmonization of data structures by defining appropriate metadata (Cimino, 1998<sup id=\"rdp-ebb-cite_ref-CiminoDes98_5-0\" class=\"reference\"><a href=\"#cite_note-CiminoDes98-5\" rel=\"external_link\">[5]<\/a><\/sup>). Depending on the specific needs and data structures of the research network, often a non-standard IT platform needs to be developed to meet the specific requirements. An important requirement might be the protection of data in terms of security and privacy, especially when patient data are involved.\n<\/p><p>In the German research network <i>SFB\/TRR77 \u2014 Liver Cancer: From Molecular Pathogenesis to Targeted Therapies<\/i> it was our task to explore the most appropriate IT-architecture for supporting networked research (Woll, Manns & Schirmacher, 2013<sup id=\"rdp-ebb-cite_ref-WollSFB13_6-0\" class=\"reference\"><a href=\"#cite_note-WollSFB13-6\" rel=\"external_link\">[6]<\/a><\/sup>). The research network consists of 22 projects sharing common resources and research data. To provide this network with a data integration platform we implemented a service-oriented architecture (SOA) (Taylor et al., 2004<sup id=\"rdp-ebb-cite_ref-TaylorASer04_7-0\" class=\"reference\"><a href=\"#cite_note-TaylorASer04-7\" rel=\"external_link\">[7]<\/a><\/sup>; Papazoglou et al., 2008<sup id=\"rdp-ebb-cite_ref-PapazoglouServ08_8-0\" class=\"reference\"><a href=\"#cite_note-PapazoglouServ08-8\" rel=\"external_link\">[8]<\/a><\/sup>; Wei & Blake, 2010<sup id=\"rdp-ebb-cite_ref-WeiServ10_9-0\" class=\"reference\"><a href=\"#cite_note-WeiServ10-9\" rel=\"external_link\">[9]<\/a><\/sup>; Bosin, Dess\u00ec & Pes, 2011<sup id=\"rdp-ebb-cite_ref-BosinExt11_10-0\" class=\"reference\"><a href=\"#cite_note-BosinExt11-10\" rel=\"external_link\">[10]<\/a><\/sup>). The IT system is based on the cancer Common Ontologic Representation Environment Software Development Kit (caCORE SDK) components of the cancer Biomedical Informatics Grid (caBIG) (Komatsoulis et al., 2008<sup id=\"rdp-ebb-cite_ref-Komatsoulis_caCore08_11-0\" class=\"reference\"><a href=\"#cite_note-Komatsoulis_caCore08-11\" rel=\"external_link\">[11]<\/a><\/sup>; Kunz, Lin & Frey, 2009<sup id=\"rdp-ebb-cite_ref-KunzMeta09_12-0\" class=\"reference\"><a href=\"#cite_note-KunzMeta09-12\" rel=\"external_link\">[12]<\/a><\/sup>). The resulting system is called pelican (platform enabling liver cancer networked research) (Ganzinger et al., 2011<sup id=\"rdp-ebb-cite_ref-GanzingerServ11_13-0\" class=\"reference\"><a href=\"#cite_note-GanzingerServ11-13\" rel=\"external_link\">[13]<\/a><\/sup>). Transfer of these data sharing concepts to other networks investigating different disease areas is possible.\n<\/p><p>We consider our research network as a typical example for a whole class of biomedical research networks. To support this kind of projects, we provide a framework for the development of data integration platforms for such projects. Specifically, we strive for the following two objectives:\n<\/p><p>Objective 1: Provide a reference model of requirements of biomedical research networks regarding an IT platform for sharing and analyzing data.\n<\/p><p>Objective 2: Design a SOA of an IT platform for our research network on liver cancer. It should implement the reference model for requirements. While this SOA is specific to this project, parts can be reused for similar projects.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>For the design of a data integration platform it is important to first assess the requirements of the system\u2019s intended users. To support this task, we developed a reference model for requirements. A reference model is a generic model which is valid not only for a specific research network but for a class of such organizations. For the development of the reference model, we used the research network on liver cancer as primary source. These requirements were consolidated and abstracted to get a generic model that can be applied to other research networks.\n<\/p><p>In the first step, a general understanding of the network\u2019s aims and tasks was acquired by analyzing written descriptions of the participating projects. In addition, questionnaires were sent to the principal investigators to capture the data types and data formats used within the projects. In the second step, projects were visited and their research subjects and processes were captured by interviewing project members.\n<\/p><p>For the reference model, we use the term goal to describe the highest level of requirements. This is in accordance with ISO\/IEC\/IEEE 24765, where goal is defined as \u201can intended outcome\u201d (ISO\/IEC\/IEEE, 2011<sup id=\"rdp-ebb-cite_ref-ISO24765_14-0\" class=\"reference\"><a href=\"#cite_note-ISO24765-14\" rel=\"external_link\">[14]<\/a><\/sup>). In contrast, requirement is defined as \u201ca condition or capability needed by a user to solve a problem or achieve an objective\u201d (ISO\/IEC\/IEEE, 2011<sup id=\"rdp-ebb-cite_ref-ISO24765_14-1\" class=\"reference\"><a href=\"#cite_note-ISO24765-14\" rel=\"external_link\">[14]<\/a><\/sup>). In our reference model, each requirement was related to a goal, either directly or indirectly. Requirements were then mapped to concrete functions in the resulting data integration system. On the other hand, goals were used to structure requirements and usually do not lead to a specific function of the system.\n<\/p><p>To provide a more detailed characterization of the goals, we provide a standardized table for each of them which covers the reference number, name, description and weight of the goal. Table 1 shows the structure of such a table. The complete set of tables for all the goals is available in <a href=\"#Supplemental_information\" rel=\"external_link\">Table S1<\/a>.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Feature\n<\/th>\n<th> Explanation\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Number\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Number for uniquely identifying requirements\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Name\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Name of the requirement\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Description\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Verbal description of the requirement and its properties\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Weighting\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Importance of the requirement for fulfilling the goals of the project (low, medium, high)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 1.<\/b> Schema for documenting reference requirements and goals;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717\/peerj.755\/table-1\" target=\"_blank\">10.7717\/peerj.755\/table-1<\/a><\/blockquote>\n<p>The requirements are documented in the same way as the goals. Figure 1 shows a Unified Modeling Language (UML) diagram with all elements used for describing of both goals and their subordinated requirements (Object Management Group, 2012<sup id=\"rdp-ebb-cite_ref-OMGXML02_15-0\" class=\"reference\"><a href=\"#cite_note-OMGXML02-15\" rel=\"external_link\">[15]<\/a><\/sup>). As for the goals, we provide a set of tables with more detailed descriptions for all requirements in <a href=\"#Supplemental_information\" rel=\"external_link\">Table S1<\/a> In total, we identified 15 requirements for the reference model.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_GanzingerPeerJCS2015_3.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"136dab964d78b87d8c76b8ca79dee049\"><img alt=\"Fig1 GanzingerPeerJCS2015 3.png\" src=\"https:\/\/www.limswiki.org\/images\/4\/42\/Fig1_GanzingerPeerJCS2015_3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1:<\/b> Overview of the UML elements used in requirements diagrams;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dx.doi.org\/10.7717\/peerj.755\/fig-1\" target=\"_blank\">10.7717\/peerj.755\/fig-1<\/a><\/blockquote>\n<p>The instantiation of the reference model for requirements to meet the needs of a specific research network provides the basis for the architecture of the desired data integration and its subsequent implementation. We provide a concrete instance of a reference model as well as the resulting IT-architecture in this manuscript.\n<\/p><p>For the research described in this paper, ethics approval was not deemed necessary. This work involved no human subjects in the sense of medical research, as e.g., covered by the Declaration of Helsinki (World Medical Association, 2013<sup id=\"rdp-ebb-cite_ref-WMOEth13_16-0\" class=\"reference\"><a href=\"#cite_note-WMOEth13-16\" rel=\"external_link\">[16]<\/a><\/sup>). At no time were patients included for survey or interview. Data was only acquired from scientists regarding their work and data, but no personal or patient related data were gathered. Participants were not required to participate in this study; they consented by returning the questionnaire. No research was conducted outside Germany, the authors\u2019 country of residence. However, in other countries the approval of an institutional review board or other authority might be necessary to apply the reference model.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h2>\n<p>In this section we describe the reference model for requirements. Then, we show how a concrete model for requirements and an IT-architecture is derived from this reference model. The reference model for requirements is an abstract model and thus a universally usable artefact. It is mapped in several steps to the network specific system architecture.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Reference_model\">Reference model<\/span><\/h3>\n<p>The reference model for requirements covers five reference goals (RG). An overview of the goals and their relations is shown in Fig. 2 by means of a UML requirements diagram. The reference goals are:\n<\/p>\n<ul><li> Conduct research project (reference goal RG1): The ultimate goal of a research network is to fulfill the intended research tasks. This usually corresponds to the project specification of the funding organization.<\/li>\n<li> Answer research questions (reference goal RG2): Each research network has specific research questions it pursues to answer. These questions frame the core of the network and led to its establishment in the first place.<\/li>\n<li> Create, store, and retrieve data (reference goal RG3): Research networks need data to conduct the project. Thus, it is necessary to generate and handle them.<\/li>\n<li> Analyze data (reference goal RG4): To generate knowledge out of the data it is necessary to analyze them.<\/li>\n<li> Control data access and usage (reference goal RG5): Research networks need to protect their data. This includes the prevention of unauthorized access to protected data like patient data as well as aspects of intellectual property rights that need to be respected by authorized users as well.<\/li><\/ul>\n<p>These goals are ordered in a hierarchical structure: Goal RG1 acts as the root node, which has the two sub goals RG2 and RG3. Goal RG4 is subordinated to Goal RG2, whereas Goal RG5 is a sub goal of Goal RG3.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_GanzingerPeerJCS2015_3.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"fa21547a41c89ba9c6addd3dfd53f145\"><img alt=\"Fig2 GanzingerPeerJCS2015 3.png\" src=\"https:\/\/www.limswiki.org\/images\/a\/a7\/Fig2_GanzingerPeerJCS2015_3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2:<\/b> Reference model for goals of a research network;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dx.doi.org\/10.7717\/peerj.755\/fig-2\" target=\"_blank\">10.7717\/peerj.755\/fig-2<\/a><\/blockquote>\n<p>Each goal has several requirements. In total, the reference model contains 15 reference requirements (RR). A UML diagram with all reference goals and reference requirements is shown in Fig. 3. Reference requirements are associated with the reference goals as follows:\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_GanzingerPeerJCS2015_3.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a2351547eefd49af0de3d5e0abfe2764\"><img alt=\"Fig3 GanzingerPeerJCS2015 3.png\" src=\"https:\/\/www.limswiki.org\/images\/f\/fd\/Fig3_GanzingerPeerJCS2015_3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3:<\/b> Reference model for goals and requirements of a research network;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dx.doi.org\/10.7717\/peerj.755\/fig-3\" target=\"_blank\">10.7717\/peerj.755\/fig-3<\/a><\/blockquote>\n<p>Goal 3 is associated with the reference requirements to <i>create data<\/i> (RR1) and to <i>retrieve external data<\/i> (RR2). These two requirements respect possible sources of data necessary for the research network. Reference requirement RR3, <i>represent data<\/i>, is further defined by its subordinate requirements <i>define syntax<\/i> (RR4), <i>define data model<\/i> (RR5), <i>identify data<\/i> (RR6), and <i>define semantics<\/i> (RR7).\n<\/p><p>Goal 5, <i>control data access<\/i> and usage, has two aspects, which are represented by reference requirements RR8 and RR9. RR8 requires the creator\u2019s contribution in the generation of data for the research network to be recognized when data are used by others. As a consequence, even users with legitimate access to the system have to adhere to usage regulations. These regulations should be checked and enforced by the system as far as possible. In contrast, RR9 covers the requirement to <i>protect data from unauthorized access<\/i>.\n<\/p><p>The second group of reference requirements covers data analysis. At the highest level we identify goal 2, <i>answer research questions<\/i>. It is associated downstream with goal RG4, <i>analyze data<\/i>. Goal RG4 is composed of two reference requirements: RR11 <i>integrate data<\/i> and RR13 <i>define analytical process<\/i>. RR11 is associated with RR3, since the technical provision of data within the research network is of great relevance for the integration of data. RR13 has subordinate requirement RR12, <i>define analytical methods<\/i>, which covers the low-level data analysis methods. RR14 and RR15 cover two distinct instances of RR13: RR14 describes <i>static workflows<\/i> with all process steps being fixed. In this case, the order of analytical steps and data sources used cannot be changed by the users of the system. In contrast, RR15 considers <i>dynamic workflows<\/i>, allowing users to compose analytical steps and data sources as needed. Since the type of data involved in a dynamic workflow is not known upfront, this reference requirement is more demanding in terms of the semantic description of data sources. Precise annotation of data sources is necessary in order to perform automated transformations for matching different data fields.\n<\/p><p>RR13 is further associated with RR10, <i>show results<\/i>. RR10 covers the requirement to present the results of the analysis adequately. Thus, it partially fulfills goal 2, <i>answer research questions<\/i>.\n<\/p><p>The reference model for requirements is the basis for a network specific requirements model. We present an example for creating such a model and all the following steps in the next section. All goals and requirements from the reference model are mapped to network specific instances. In this process, elements of the reference model are checked for their applicability to the specific research network. Further special requirements of the network are considered at this point as well.\n<\/p><p>The network specific requirements model is then mapped to system properties. These are qualities contributed to the system by different components. At first, we consider abstract components instead of specific products. For example, in a research network, reference requirement RR1 <i>create data<\/i> might be mapped to a system property <i>automated creation of data services<\/i>. This property is then mapped to the specific component responsible for the implementation of this property.\n<\/p><p>In a second step, the abstract components are mapped to specific components in accordance with the research network\u2019s requirements. Specific components can be preexisting modules with a product character, software development frameworks providing specific functionality, or newly developed components.\n<\/p><p>In a final modelling step a distribution model of the components is created. All components need to be mapped to system resources down to the hardware level. Among others, the following aspects have to be considered is this step:\n<\/p>\n<ul><li> Security: Components with high security requirements should be isolated against other, less sensitive components and thus be run on a separate system node.<\/li>\n<li> Performance: All components must be distributed in a way that availability of sufficient system resources is ensured.<\/li>\n<li> Maintainability: To ensure that the possibly complex distributed system can be managed efficiently, components should be grouped together in a sensible way.<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Sample_application:_pelican\">Sample application: pelican<\/span><\/h3>\n<p>We now describe a sample application of our reference model within the research network SFB\/TRR77 on liver cancer. Further we describe two specifications for metadata we developed for the research network.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"Specific_model_for_requirements\">Specific model for requirements<\/span><\/h4>\n<p>In this section we summarize key requirements specific to our research network. The complete list of requirements is shown in <a href=\"#Supplemental_information\" rel=\"external_link\">Table S2<\/a>.\n<\/p><p>The first goal of the research network, an instance of reference goal RG1, is defined by its research assignment of gaining a deeper understanding of the molecular basis of liver cancer development. This spans research on the chronic liver disease to progression of metastatic cancer. Further, the research network aims to identify novel preventive, diagnostic and therapeutic approaches on liver cancer. Subordinated to goal G1 is G3, the instance of reference goal RG3 regarding the data necessary for the network. Since molecular processes play a major role within the research network, genomic microarray data are of central importance. They are complemented by imaging data like tissue microarray (TMA) data and clinical data.\n<\/p><p>Goal G2, answering research questions, is characterized by the following two questions:\n<\/p>\n<ul><li> Which generic or specific mechanisms of chronic liver diseases, especially of chronic virus infections and inflammation mediated processes predispose or initiate liver cancer?<\/li>\n<li> Which molecular key events promoting or keeping up liver cancer could act as tumor markers or are promising targets for future therapeutic interventions?<\/li><\/ul>\n<p>Goal G5 requires making the data available for cross project analysis within the network, but to protect data against unauthorized access at the same time. Especially important to the members of the project is the requirement R8, subordinated to goal G5: The projects contributing data to the network require keeping control over the data in order to ensure proper crediting of their intellectual property. Thus, they require fine-grained rules for data access control. Depending on the type of data, they should be available only to specific members of the network, to all members of the project or the general public.\n<\/p>\n<h4><span class=\"mw-headline\" id=\"System_architecture\">System architecture<\/span><\/h4>\n<p>To acknowledge the project\u2019s requirement R8, to keep the ownership over their data, a federation was used as the underlying concept of the system architecture. Technically, pelican implements a SOA. All data sources of the projects are transformed into data services and made available to the research network. The data services stay under the control of the contributing project. This can even go as far as running the service on computer hardware on the projects\u2019 premises. Data services are complemented by analytical services. All services are described by standardized metadata to help finding appropriate services and allow for automated access to the services\u2019 interfaces. Using a web-based user interface, researchers can chain data services and analytical services to answer specific research questions.\n<\/p><p><b>Component model<\/b>\n<\/p><p>The requirements are mapped to system properties first. In the next step, components are identified to provide these properties as module of the new system. In Table 2 we show the complete chain of mappings from requirements over system features to components. Each component is realized either by a readily available product or by a newly developed module. In Table 3 we give an overview of our components.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Requirement\n<\/th>\n<th> System feature\n<\/th>\n<th> Component\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Requirements associated with G2<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R1 Create data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Automated creation of data services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data service framework\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R2 Retrieve external data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Integration of external data services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data service framework, portal\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R3 Represent data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data service, document service\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data service framework, document management system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R4 Define syntax\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Service description\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data service framework\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R5 Define data model\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Defined data model\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data service framework\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R6 Identify data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Provisioning of information on service location\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Meta data directory\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R7 Define semantics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Definition of controlled vocabulary and ontologies\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Terminology server\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Requirements associated with G4<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R8 Administrate intellectual property\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Log data usage\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Portal, data service framework\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R9 Protect data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">User authentication \/ User authorization\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Portal, security service Portal, data service framework, security service\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Requirements associated with G5<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R10 Show results\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data specific portlets\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Portal\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R11 Integrate data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Analytical services\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Portal, statistics service, data service framework\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R12 Define analytical methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Statistical methods\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Statistics service\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R13 Define analytical process\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Documentation service\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Document management system\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R14 Static workflow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Workflow in portal application\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Portal\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R15 Dynamic workflow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Flexible pipeline\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pipeline management\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 2.<\/b> Mapping of requirements to corresponding system features and components;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717\/peerj.755\/table-2\" target=\"_blank\">10.7717\/peerj.755\/table-2<\/a><\/blockquote>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Abstract component\n<\/th>\n<th> Implementing component\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Portal\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Liferay\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data service framework\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">caCORE SDK\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Meta data directory\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Internal development (based on caCORE SDK)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Terminology server\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">TemaTres\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Security service\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">caCORE SDK, LDAP\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Statistics service\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Document management system\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Alfresco\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Pipeline management\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Galaxy (planned)\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Table 3.<\/b> Specification of concrete implementation components for the elements of the component model;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717\/peerj.755\/table-3\" target=\"_blank\">10.7717\/peerj.755\/table-3<\/a><\/blockquote>\n<p>The portal component provides the user interface to the system. It is implemented using the open source software Liferay (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.liferay.com\" target=\"_blank\">http:\/\/www.liferay.com<\/a>, accessed: 2014-07-03) (Sezov, Jr, 2011<sup id=\"rdp-ebb-cite_ref-SezovLife11_17-0\" class=\"reference\"><a href=\"#cite_note-SezovLife11-17\" rel=\"external_link\">[17]<\/a><\/sup>). Liferay provides a number of functions affecting several components of our model. Thus, we provide a decomposition of the portal components in Fig. 4. One important subcomponent of the portal is the document management system. It is realized by the Alfresco component (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.alfresco.com\" target=\"_blank\">http:\/\/www.alfresco.com<\/a>, accessed: 2014-07-03) (Berman, Barnett & Mooney, 2012<sup id=\"rdp-ebb-cite_ref-BermanColl12_18-0\" class=\"reference\"><a href=\"#cite_note-BermanColl12-18\" rel=\"external_link\">[18]<\/a><\/sup>). The user interface of Alfresco can be integrated into the Liferay portal or be accessed with a separate unified resource locator (URL). The portal provides user management functionality to control access to portal pages and components like portlets (Java Community Process, 2008<sup id=\"rdp-ebb-cite_ref-JavaPort08_19-0\" class=\"reference\"><a href=\"#cite_note-JavaPort08-19\" rel=\"external_link\">[19]<\/a><\/sup>). However, the user account information including username, passwords, and others is stored in a separate component using the Lightweight Directory Access Protocol (LDAP). Thus, it is possible for all components of the SOA-network to commonly access the users\u2019 identity information.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_GanzingerPeerJCS2015_3.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"c9f759f6390e8f7097772b6e5469514e\"><img alt=\"Fig4 GanzingerPeerJCS2015 3.png\" src=\"https:\/\/www.limswiki.org\/images\/d\/d9\/Fig4_GanzingerPeerJCS2015_3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4:<\/b> Structure of the component <i>portal<\/i>;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dx.doi.org\/10.7717\/peerj.755\/fig-4\" target=\"_blank\">10.7717\/peerj.755\/fig-4<\/a><\/blockquote>\n<p>Data services are generated by using caCORE SDK (Wiley & Gagne, 2012<sup id=\"rdp-ebb-cite_ref-Wiley_caCORE12_20-0\" class=\"reference\"><a href=\"#cite_note-Wiley_caCORE12-20\" rel=\"external_link\">[20]<\/a><\/sup>). With caCORE SDK it is not necessary to program the software for the service in a traditional way. Instead, a UML data model in Extensible Markup Language Metadata Interchange (XMI) notation has to be prepared (Object Management Group, 2002<sup id=\"rdp-ebb-cite_ref-OMGXML02_15-1\" class=\"reference\"><a href=\"#cite_note-OMGXML02-15\" rel=\"external_link\">[15]<\/a><\/sup>; Bray et al., 2006<sup id=\"rdp-ebb-cite_ref-BrayExt06_21-0\" class=\"reference\"><a href=\"#cite_note-BrayExt06-21\" rel=\"external_link\">[21]<\/a><\/sup>). From this model, caCORE SDK generates several artefacts resulting in a deployment packages for Java application servers like Apache Tomcat (The Apache Software Foundation, 2014<sup id=\"rdp-ebb-cite_ref-ApacheTom_22-0\" class=\"reference\"><a href=\"#cite_note-ApacheTom-22\" rel=\"external_link\">[22]<\/a><\/sup>). To simplify this process for spreadsheet based microarray data, we developed a software tool to generate the XMI file as well. As a result, a service conforming to the web services specification ready for deployment is generated. For the provision of network specific metadata we chose TemaTres to serve our controlled vocabulary in standard formats like SKOS or Dublin Core (Weibel, 1997<sup id=\"rdp-ebb-cite_ref-WeibelTheDub97_23-0\" class=\"reference\"><a href=\"#cite_note-WeibelTheDub97-23\" rel=\"external_link\">[23]<\/a><\/sup>; Miles & Bechhofer, 2009<sup id=\"rdp-ebb-cite_ref-MilesSKOS09_24-0\" class=\"reference\"><a href=\"#cite_note-MilesSKOS09-24\" rel=\"external_link\">[24]<\/a><\/sup>; Gonzales-Aguilar, Ram\u00edrez-Posada & Ferreyra, 2012<sup id=\"rdp-ebb-cite_ref-GonzalesTem12_25-0\" class=\"reference\"><a href=\"#cite_note-GonzalesTem12-25\" rel=\"external_link\">[25]<\/a><\/sup>). Our analytical services are backed by the open source language and environment for statistical computing called R (R Core Team, 2014<sup id=\"rdp-ebb-cite_ref-RTheR_26-0\" class=\"reference\"><a href=\"#cite_note-RTheR-26\" rel=\"external_link\">[26]<\/a><\/sup>). R is integrated into the services using the Rserve component (Urbanek, 2003<sup id=\"rdp-ebb-cite_ref-UrbanekRserve03_27-0\" class=\"reference\"><a href=\"#cite_note-UrbanekRserve03-27\" rel=\"external_link\">[27]<\/a><\/sup>).\n<\/p><p><b>Deployment model<\/b>\n<\/p><p>In a final modelling step the components are distributed to the physical resources available for the system. In our case we used two servers with a common virtualization layer based on VMware VSphere server. Thus, all nodes in our deployment model represent virtual machines (VM). Using virtual switches, routers, and firewall appliances we were able to implement our Internet Protocol (IP) network infrastructure. To enhance security, we implement a network zoning model comprised of an internet zone, a demilitarized zone (DMZ) and an internal zone. Figure 5 shows the deployment model in UML notation. The services shown in the model (service 1 to service <i>n<\/i>) are to be considered as examples, since the concrete number of services is permanently changing. The deployment model also reflects the different levels of control that can be executed by the owners of the data. They range from shared nodes on the common servers over a dedicated VM to deployment on external hardware controlled by the respective projects.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_GanzingerPeerJCS2015_3.png\" class=\"image wiki-link\" target=\"_blank\" data-key=\"580d422e816281799bc77b079627cabe\"><img alt=\"Fig5 GanzingerPeerJCS2015 3.png\" src=\"https:\/\/www.limswiki.org\/images\/5\/52\/Fig5_GanzingerPeerJCS2015_3.png\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 5:<\/b> Deployment diagram of the components of the architecture in UML notation;<br \/>Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dx.doi.org\/10.7717\/peerj.755\/fig-5\" target=\"_blank\">10.7717\/peerj.755\/fig-5<\/a><\/blockquote>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>In this manuscript, we describe a reference model for the requirements of research networks towards an IT platform. For many funding programs, including research grants of the European Commission, the collaboration of several research organizations at different sites is mandatory. This leads to a structural similarity to our research network on liver cancer. Even though other research networks will have different research aims, there are still requirements that are common to most networks. Since the reference model already covers a basic set of requirements, it allows future research networks to focus on defining specific requirements distinguishing them from other networks.\n<\/p><p>Users of the reference model are responsible for assessing the reference model\u2019s applicability to their project-specific needs. The reference model is based on data of a real research network that were generalized. To avoid bias in the model that might hinder transferability, we incorporated different views in the process of constructing the model. However, the transferability of the model to another context is, as for any model, limited. As a consequence, future research networks will have to derive a project specific instance of the reference model to reflect the corresponding characteristics of the project. The reference model is a tool intended to help its users to create a concrete model covering the requirements of a research network with a high degree of completeness. The reference model provides guidance for this task. We expect that it will help in reducing the effort to acquire all requirements.\n<\/p><p>A common technique used in requirements engineering for software is <i>use case modelling<\/i> (Jacobson, 1992<sup id=\"rdp-ebb-cite_ref-JacobsonObj92_28-0\" class=\"reference\"><a href=\"#cite_note-JacobsonObj92-28\" rel=\"external_link\">[28]<\/a><\/sup>; Bittner & Spence, 2002<sup id=\"rdp-ebb-cite_ref-BittnerUse02_29-0\" class=\"reference\"><a href=\"#cite_note-BittnerUse02-29\" rel=\"external_link\">[29]<\/a><\/sup>). A use case is a compact scenario describing certain aspects of how a technical system behaves and how it interacts with other actors like its users. A use case model is a way of capturing requirements in an interactive way. Thus, a use case model can be developed further into a requirements model. In our context we found it hard to apply the use case approach since the researchers in our network mostly have biomedical backgrounds and thus are not familiar with software development. Nevertheless, use case modelling might be a helpful tool for other research networks when applying our reference model.\n<\/p><p>We applied the reference model successfully to a research network on liver cancer. Some specific requirements in this network led to the decision to set up a federated system allowing for a maximum of control of the individual projects over their respective data. The system was implemented as a service-oriented architecture using, among others, components of the caBIG project. A public version of our system with limited functionality is available at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/livercancer.imbi.uni-heidelberg.de\/data\" target=\"_blank\">https:\/\/livercancer.imbi.uni-heidelberg.de\/data<\/a>. There, a gene symbol like BRCA1 can be entered into the search field and services are invoked for data retrieval as discussed in this paper. Other projects can benefit from this architecture as well, but the architecture is tailored to research networks with the requirements of federating data as data services. With this architecture, we try to acknowledge the data protection requirements of the participating projects. Still, further research regarding the use of data and crediting creatorship of data is necessary. First steps were made as part of this project (He et al., 2013<sup id=\"rdp-ebb-cite_ref-HeProp13_30-0\" class=\"reference\"><a href=\"#cite_note-HeProp13-30\" rel=\"external_link\">[30]<\/a><\/sup>).\n<\/p><p>In case the requirements regarding data control are more relaxed, an alternative would be to keep the data in a central data warehouse instead of the federation. In that case, i2b2 might be a suitable component to provide the data warehouse component (Murphy et al., 2007<sup id=\"rdp-ebb-cite_ref-MurphyArch07_31-0\" class=\"reference\"><a href=\"#cite_note-MurphyArch07-31\" rel=\"external_link\">[31]<\/a><\/sup>). Such a centralized approach also affects how and when data are harmonized: In a central research data warehouse data are harmonized at the time of loading the database, which ideally leads to a completely and consistently harmonized database. In a service-oriented approach data are provided by means of data services as they are. All services are described by corresponding metadata, enabling automated transformation of the data at time of access.\n<\/p><p>Our sample research network concentrates more on basic research than clinical application. In the future, we plan to apply our reference model to further projects with a stronger translational component. By doing so, we will be able to reevaluate the framework in a more clinical context.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Supplemental_information\">Supplemental information<\/span><\/h2>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Table S1. Reference goals and requirements\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717\/peerj.755\/supp-1\" target=\"_blank\">10.7717\/peerj.755\/supp-1<\/a>\n<p>Format: PDF <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dfzljdn9uc3pi.cloudfront.net\/2015\/755\/1\/SI1_Table_Reference_model.pdf\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Table S2. Goals and requirements for pelican\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cite this object using this DOI: <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.7717\/peerj.755\/supp-2\" target=\"_blank\">10.7717\/peerj.755\/supp-2<\/a>\n<p>Format: PDF <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/dfzljdn9uc3pi.cloudfront.net\/2015\/755\/1\/SI2_Table_pelican_model.pdf\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Additional_information_and_declarations\">Additional information and declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>The authors declare there are no competing interests.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Author_contributions\">Author contributions<\/span><\/h3>\n<p>Matthias Ganzinger conceived and designed the experiments, performed the experiments, analyzed the data, wrote the paper, prepared figures and\/or tables, reviewed drafts of the paper.\n<\/p><p>Petra Knaup wrote the paper, reviewed drafts of the paper.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Human_ethics\">Human ethics<\/span><\/h3>\n<p>The following information was supplied relating to ethical approvals (i.e., approving body and any reference numbers):\n<\/p><p>For the research described in this paper, ethics approval was not deemed necessary. This work involved no human subjects in the sense of medical research, as e.g., covered by the Declaration of Helsinki (World Medical Association, 2013<sup id=\"rdp-ebb-cite_ref-WMOEth13_16-1\" class=\"reference\"><a href=\"#cite_note-WMOEth13-16\" rel=\"external_link\">[16]<\/a><\/sup>). At no time patients were included for survey or interview. Data was only acquired from scientists regarding their work and data, but no personal or patient related data were gathered. Participants were not required to participate in this study; they consented by returning the questionnaire. No research was conducted outside Germany, the authors\u2019 country of residence. However, in other countries the approval of an institutional review board or other authority might be necessary to apply the reference model.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Funding\">Funding<\/span><\/h3>\n<p>This work was funded by the SFB\/TRR 77 \u201cLiver Cancer. From Molecular Pathogenesis to Targeted Therapies\u201d of the Deutsche Forschungsgemeinschaft (DFG, <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dfg.de\" target=\"_blank\">http:\/\/www.dfg.de<\/a>). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-FreyBio07-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FreyBio07_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Frey, L.J.; Maojo, V.; Mitchell, J.A. (2007). \"Bioinformatics linkage of heterogeneous clinical and genomic information in support of personalized medicine\". <i>IMIA Yearbook of Medical Informatics<\/i> <b>2007<\/b>: 98-105. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Serial_Number\" target=\"_blank\">ISSN<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.worldcat.org\/issn\/0943-4747\" target=\"_blank\">0943-4747<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17700912\" target=\"_blank\">17700912<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bioinformatics+linkage+of+heterogeneous+clinical+and+genomic+information+in+support+of+personalized+medicine&rft.jtitle=IMIA+Yearbook+of+Medical+Informatics&rft.aulast=Frey%2C+L.J.%3B+Maojo%2C+V.%3B+Mitchell%2C+J.A.&rft.au=Frey%2C+L.J.%3B+Maojo%2C+V.%3B+Mitchell%2C+J.A.&rft.date=2007&rft.volume=2007&rft.pages=98-105&rft.issn=0943-4747&rft_id=info:pmid\/17700912&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BakerNext10-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BakerNext10_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Baker, M. (2010). \"Next-generation sequencing: adjusting to data overload\". <i>Nature Methods<\/i> <b>7<\/b> (7): 495-499. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth0710-495\" target=\"_blank\">10.1038\/nmeth0710-495<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Next-generation+sequencing%3A+adjusting+to+data+overload&rft.jtitle=Nature+Methods&rft.aulast=Baker%2C+M.&rft.au=Baker%2C+M.&rft.date=2010&rft.volume=7&rft.issue=7&rft.pages=495-499&rft_id=info:doi\/10.1038%2Fnmeth0710-495&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WelshPost06-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WelshPost06_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Welsh, E.; Jirotka, M.; Gavaghan, D. (2006). \"Post-genomic science: cross-disciplinary and large-scale collaborative research and its organizational and technological challenges for the scientific research process\". <i>Philosophical Transactions of the Royal Society A<\/i> <b>364<\/b> (1843): 1533-1549. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1098%2Frsta.2006.1785\" target=\"_blank\">10.1098\/rsta.2006.1785<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16766359\" target=\"_blank\">16766359<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Post-genomic+science%3A+cross-disciplinary+and+large-scale+collaborative+research+and+its+organizational+and+technological+challenges+for+the+scientific+research+process&rft.jtitle=Philosophical+Transactions+of+the+Royal+Society+A&rft.aulast=Welsh%2C+E.%3B+Jirotka%2C+M.%3B+Gavaghan%2C+D.&rft.au=Welsh%2C+E.%3B+Jirotka%2C+M.%3B+Gavaghan%2C+D.&rft.date=2006&rft.volume=364&rft.issue=1843&rft.pages=1533-1549&rft_id=info:doi\/10.1098%2Frsta.2006.1785&rft_id=info:pmid\/16766359&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MathewFrom07-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MathewFrom07_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mathew, J.P.; Taylor, B.S.; Bader, G.D.; Pyarajan, S.; Antoniotti, M.; Chinnaiyan, A.M.; Sander, C.; Burakoff, S.J.; Mishra, B. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1808026\" target=\"_blank\">\"From bytes to bedside: data integration and computational biology for translational cancer research\"<\/a>. <i>PLOS Computational Biology<\/i> <b>3<\/b> (2): e12. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1371%2Fjournal.pcbi.0030012\" target=\"_blank\">10.1371\/journal.pcbi.0030012<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1808026\/\" target=\"_blank\">PMC1808026<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17319736\" target=\"_blank\">17319736<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1808026\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1808026<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=From+bytes+to+bedside%3A+data+integration+and+computational+biology+for+translational+cancer+research&rft.jtitle=PLOS+Computational+Biology&rft.aulast=Mathew%2C+J.P.%3B+Taylor%2C+B.S.%3B+Bader%2C+G.D.%3B+Pyarajan%2C+S.%3B+Antoniotti%2C+M.%3B+Chinnaiyan%2C+A.M.%3B+Sander%2C+C.%3B+Burakoff%2C+S.J.%3B+Mishra%2C+B.&rft.au=Mathew%2C+J.P.%3B+Taylor%2C+B.S.%3B+Bader%2C+G.D.%3B+Pyarajan%2C+S.%3B+Antoniotti%2C+M.%3B+Chinnaiyan%2C+A.M.%3B+Sander%2C+C.%3B+Burakoff%2C+S.J.%3B+Mishra%2C+B.&rft.date=2007&rft.volume=3&rft.issue=2&rft.pages=e12&rft_id=info:doi\/10.1371%2Fjournal.pcbi.0030012&rft_id=info:pmc\/PMC1808026&rft_id=info:pmid\/17319736&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1808026&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CiminoDes98-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CiminoDes98_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cimino, J.J. (1998). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3415631\" target=\"_blank\">\"Desiderata for controlled medical vocabularies in the twenty-first century\"<\/a>. <i>Methods of Information in Medicine<\/i> <b>37<\/b> (4\u20135): 394-403. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3415631\/\" target=\"_blank\">PMC3415631<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/9865037\" target=\"_blank\">9865037<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3415631\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3415631<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Desiderata+for+controlled+medical+vocabularies+in+the+twenty-first+century&rft.jtitle=Methods+of+Information+in+Medicine&rft.aulast=Cimino%2C+J.J.&rft.au=Cimino%2C+J.J.&rft.date=1998&rft.volume=37&rft.issue=4%E2%80%935&rft.pages=394-403&rft_id=info:pmc\/PMC3415631&rft_id=info:pmid\/9865037&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3415631&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WollSFB13-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WollSFB13_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Woll, K.; Manns, M.; Schirmacher, P. (2013). \"Sonderforschungsbereich SFB\/TRR77: Leberkrebs: Von der molekularen Pathogenese zur zielgerichteten Therapie\". <i>Der Pathologe<\/i> <b>34<\/b> (2): 232-234. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2Fs00292-013-1820-z\" target=\"_blank\">10.1007\/s00292-013-1820-z<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sonderforschungsbereich+SFB%2FTRR77%3A+Leberkrebs%3A+Von+der+molekularen+Pathogenese+zur+zielgerichteten+Therapie&rft.jtitle=Der+Pathologe&rft.aulast=Woll%2C+K.%3B+Manns%2C+M.%3B+Schirmacher%2C+P.&rft.au=Woll%2C+K.%3B+Manns%2C+M.%3B+Schirmacher%2C+P.&rft.date=2013&rft.volume=34&rft.issue=2&rft.pages=232-234&rft_id=info:doi\/10.1007%2Fs00292-013-1820-z&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorASer04-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorASer04_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, K.L.; O\u2019Keefe, C.M.; Colton, J.; Baxter, R.; Sparks, R.; Srinivasan, U.; Cameron, M.A.; Lefort, L. (2004). \"A service oriented architecture for a health research data network\". <i>Proceedings of the 16th International Conference on Scientific and Statistical Database Management<\/i> <b>2004<\/b>: 443-444. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FSSDM.2004.1311251\" target=\"_blank\">10.1109\/SSDM.2004.1311251<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+service+oriented+architecture+for+a+health+research+data+network&rft.jtitle=Proceedings+of+the+16th+International+Conference+on+Scientific+and+Statistical+Database+Management&rft.aulast=Taylor%2C+K.L.%3B+O%E2%80%99Keefe%2C+C.M.%3B+Colton%2C+J.%3B+Baxter%2C+R.%3B+Sparks%2C+R.%3B+Srinivasan%2C+U.%3B+Cameron%2C+M.A.%3B+Lefort%2C+L.&rft.au=Taylor%2C+K.L.%3B+O%E2%80%99Keefe%2C+C.M.%3B+Colton%2C+J.%3B+Baxter%2C+R.%3B+Sparks%2C+R.%3B+Srinivasan%2C+U.%3B+Cameron%2C+M.A.%3B+Lefort%2C+L.&rft.date=2004&rft.volume=2004&rft.pages=443-444&rft_id=info:doi\/10.1109%2FSSDM.2004.1311251&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PapazoglouServ08-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PapazoglouServ08_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Papazoglou, M.P.; Traverso, P.; Dustdar, S.; Leymann, F. (2008). \"Service-oriented computing: A research roadmap\". <i>International Journal of Cooperative Information Systems<\/i> <b>17<\/b>: 223. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1142%2FS0218843008001816\" target=\"_blank\">10.1142\/S0218843008001816<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Service-oriented+computing%3A+A+research+roadmap&rft.jtitle=International+Journal+of+Cooperative+Information+Systems&rft.aulast=Papazoglou%2C+M.P.%3B+Traverso%2C+P.%3B+Dustdar%2C+S.%3B+Leymann%2C+F.&rft.au=Papazoglou%2C+M.P.%3B+Traverso%2C+P.%3B+Dustdar%2C+S.%3B+Leymann%2C+F.&rft.date=2008&rft.volume=17&rft.pages=223&rft_id=info:doi\/10.1142%2FS0218843008001816&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeiServ10-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WeiServ10_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wei, Y.; Blake, M.B. (2010). \"Service-oriented computing and cloud computing: Challenges and opportunities\". <i>IEEE Internet Computing<\/i> <b>14<\/b> (6): 72\u201375. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FMIC.2010.147\" target=\"_blank\">10.1109\/MIC.2010.147<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Service-oriented+computing+and+cloud+computing%3A+Challenges+and+opportunities&rft.jtitle=IEEE+Internet+Computing&rft.aulast=Wei%2C+Y.%3B+Blake%2C+M.B.&rft.au=Wei%2C+Y.%3B+Blake%2C+M.B.&rft.date=2010&rft.volume=14&rft.issue=6&rft.pages=72%E2%80%9375&rft_id=info:doi\/10.1109%2FMIC.2010.147&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BosinExt11-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BosinExt11_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bosin, A.; Dess\u00ec, N.; Pes, B. (2011). \"Extending the SOA paradigm to e-Science environments\". <i>Future Generation Computer Systems<\/i> <b>27<\/b> (1): 20\u201331. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.future.2010.07.003\" target=\"_blank\">10.1016\/j.future.2010.07.003<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Extending+the+SOA+paradigm+to+e-Science+environments&rft.jtitle=Future+Generation+Computer+Systems&rft.aulast=Bosin%2C+A.%3B+Dess%C3%AC%2C+N.%3B+Pes%2C+B.&rft.au=Bosin%2C+A.%3B+Dess%C3%AC%2C+N.%3B+Pes%2C+B.&rft.date=2011&rft.volume=27&rft.issue=1&rft.pages=20%E2%80%9331&rft_id=info:doi\/10.1016%2Fj.future.2010.07.003&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Komatsoulis_caCore08-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Komatsoulis_caCore08_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Komatsoulis, G.A.; Warzel, D.B.; Hartel, F.W.; Shanbhag, K.; Chilukuri, R.; Fragoso, G.; de Coronado, S.; Reeves, D.M.; Hadfield, J.B.; Ludet, C.; Covitz, P.A. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2254758\" target=\"_blank\">\"caCORE version 3: Implementation of a model driven, service-oriented architecture for semantic interoperability\"<\/a>. <i>Journal of Biomedical Informatics<\/i> <b>41<\/b> (1): 106\u2013123. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2007.03.009\" target=\"_blank\">10.1016\/j.jbi.2007.03.009<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2254758\/\" target=\"_blank\">PMC2254758<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17512259\" target=\"_blank\">17512259<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2254758\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2254758<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=caCORE+version+3%3A+Implementation+of+a+model+driven%2C+service-oriented+architecture+for+semantic+interoperability&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=Komatsoulis%2C+G.A.%3B+Warzel%2C+D.B.%3B+Hartel%2C+F.W.%3B+Shanbhag%2C+K.%3B+Chilukuri%2C+R.%3B+Fragoso%2C+G.%3B+de+Coronado%2C+S.%3B+Reeves%2C+D.M.%3B+Hadfield%2C+J.B.%3B+Ludet%2C+C.%3B+Covitz%2C+P.A.&rft.au=Komatsoulis%2C+G.A.%3B+Warzel%2C+D.B.%3B+Hartel%2C+F.W.%3B+Shanbhag%2C+K.%3B+Chilukuri%2C+R.%3B+Fragoso%2C+G.%3B+de+Coronado%2C+S.%3B+Reeves%2C+D.M.%3B+Hadfield%2C+J.B.%3B+Ludet%2C+C.%3B+Covitz%2C+P.A.&rft.date=2008&rft.volume=41&rft.issue=1&rft.pages=106%E2%80%93123&rft_id=info:doi\/10.1016%2Fj.jbi.2007.03.009&rft_id=info:pmc\/PMC2254758&rft_id=info:pmid\/17512259&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2254758&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KunzMeta09-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KunzMeta09_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kunz, I.; Lin, M.; Frey, L. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2646244\" target=\"_blank\">\"Metadata mapping and reuse in caBIG\"<\/a>. <i>BMC Bioinformatics<\/i> <b>10<\/b> (Suppl 2): S4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-10-S2-S4\" target=\"_blank\">10.1186\/1471-2105-10-S2-S4<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2646244\/\" target=\"_blank\">PMC2646244<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19208192\" target=\"_blank\">19208192<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2646244\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2646244<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Metadata+mapping+and+reuse+in+caBIG&rft.jtitle=BMC+Bioinformatics&rft.aulast=Kunz%2C+I.%3B+Lin%2C+M.%3B+Frey%2C+L.&rft.au=Kunz%2C+I.%3B+Lin%2C+M.%3B+Frey%2C+L.&rft.date=2009&rft.volume=10&rft.issue=Suppl+2&rft.pages=S4&rft_id=info:doi\/10.1186%2F1471-2105-10-S2-S4&rft_id=info:pmc\/PMC2646244&rft_id=info:pmid\/19208192&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2646244&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GanzingerServ11-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GanzingerServ11_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ganzinger, M.; Noack, T.; Diederichs, S.; Longerich, T. Knaup, P. (2011). \"Service oriented data integration for a biomedical research network\". <i>Studies in Health Technology and Informatics<\/i> <b>169<\/b>: 867-71. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2F978-1-60750-806-9-867\" target=\"_blank\">10.3233\/978-1-60750-806-9-867<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21893870\" target=\"_blank\">21893870<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Service+oriented+data+integration+for+a+biomedical+research+network&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=Ganzinger%2C+M.%3B+Noack%2C+T.%3B+Diederichs%2C+S.%3B+Longerich%2C+T.+Knaup%2C+P.&rft.au=Ganzinger%2C+M.%3B+Noack%2C+T.%3B+Diederichs%2C+S.%3B+Longerich%2C+T.+Knaup%2C+P.&rft.date=2011&rft.volume=169&rft.pages=867-71&rft_id=info:doi\/10.3233%2F978-1-60750-806-9-867&rft_id=info:pmid\/21893870&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ISO24765-14\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ISO24765_14-0\" rel=\"external_link\">14.0<\/a><\/sup> <sup><a href=\"#cite_ref-ISO24765_14-1\" rel=\"external_link\">14.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">ISO\/IEC\/IEEE (2011). <i>ISO\/IEC\/IEEE 24765: Systems and software engineering \u2014 Vocabulary<\/i>. Geneva, Switzerland: ISO.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=ISO%2FIEC%2FIEEE+24765%3A+Systems+and+software+engineering+%E2%80%94+Vocabulary&rft.aulast=ISO%2FIEC%2FIEEE&rft.au=ISO%2FIEC%2FIEEE&rft.date=2011&rft.place=Geneva%2C+Switzerland&rft.pub=ISO&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OMGXML02-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-OMGXML02_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-OMGXML02_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.omg.org\/cgi-bin\/doc?formal\/02-01-01.pdf\" target=\"_blank\">\"OMG XML Metadata Interchange (SMI) Specification (Version 1.2)\"<\/a> (PDF). Object Management Group, Inc. January 2002<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.omg.org\/cgi-bin\/doc?formal\/02-01-01.pdf\" target=\"_blank\">http:\/\/www.omg.org\/cgi-bin\/doc?formal\/02-01-01.pdf<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=OMG+XML+Metadata+Interchange+%28SMI%29+Specification+%28Version+1.2%29&rft.atitle=&rft.date=January+2002&rft.pub=Object+Management+Group%2C+Inc&rft_id=http%3A%2F%2Fwww.omg.org%2Fcgi-bin%2Fdoc%3Fformal%2F02-01-01.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WMOEth13-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WMOEth13_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-WMOEth13_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">World Medical Association (2013). \"World Medical Association Declaration of Helsinki: Ethical Principles for Medical Research Involving Human Subjects\". <i>JAMA<\/i> <b>310<\/b> (20): 2191-2194. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1001%2Fjama.2013.281053\" target=\"_blank\">10.1001\/jama.2013.281053<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12432198\" target=\"_blank\">12432198<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=World+Medical+Association+Declaration+of+Helsinki%3A+Ethical+Principles+for+Medical+Research+Involving+Human+Subjects&rft.jtitle=JAMA&rft.aulast=World+Medical+Association&rft.au=World+Medical+Association&rft.date=2013&rft.volume=310&rft.issue=20&rft.pages=2191-2194&rft_id=info:doi\/10.1001%2Fjama.2013.281053&rft_id=info:pmid\/12432198&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SezovLife11-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SezovLife11_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Sezov, Rich (2011). <i>Liferay in Action: The Official Guide to Liferay Portal Development<\/i>. Shelter Island, NY: Manning Publications. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9781935182825.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Liferay+in+Action%3A+The+Official+Guide+to+Liferay+Portal+Development&rft.aulast=Sezov%2C+Rich&rft.au=Sezov%2C+Rich&rft.date=2011&rft.place=Shelter+Island%2C+NY&rft.pub=Manning+Publications&rft.isbn=9781935182825&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BermanColl12-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BermanColl12_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Berman, A.E.; Barnett, W.K.; Mooney, S.D. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3500217\" target=\"_blank\">\"Collaborative software for traditional and translational research\"<\/a>. <i>Human Genomics<\/i> <b>6<\/b>: 21. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1479-7364-6-21\" target=\"_blank\">10.1186\/1479-7364-6-21<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3500217\/\" target=\"_blank\">PMC3500217<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23157911\" target=\"_blank\">23157911<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3500217\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3500217<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Collaborative+software+for+traditional+and+translational+research&rft.jtitle=Human+Genomics&rft.aulast=Berman%2C+A.E.%3B+Barnett%2C+W.K.%3B+Mooney%2C+S.D.&rft.au=Berman%2C+A.E.%3B+Barnett%2C+W.K.%3B+Mooney%2C+S.D.&rft.date=2012&rft.volume=6&rft.pages=21&rft_id=info:doi\/10.1186%2F1479-7364-6-21&rft_id=info:pmc\/PMC3500217&rft_id=info:pmid\/23157911&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3500217&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JavaPort08-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JavaPort08_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.jcp.org\/en\/jsr\/detail?id=286\" target=\"_blank\">\"JSR 286: Portlet Specification 2.0\"<\/a>. Oracle Corporation. 2008<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.jcp.org\/en\/jsr\/detail?id=286\" target=\"_blank\">https:\/\/www.jcp.org\/en\/jsr\/detail?id=286<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=JSR+286%3A+Portlet+Specification+2.0&rft.atitle=&rft.date=2008&rft.pub=Oracle+Corporation&rft_id=https%3A%2F%2Fwww.jcp.org%2Fen%2Fjsr%2Fdetail%3Fid%3D286&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Wiley_caCORE12-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Wiley_caCORE12_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Wiley, A.; Gagne, B. (18 April 2012). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/wiki.nci.nih.gov\/display\/caCORE\/caCORE+SDK+Version+4.3+Object+Relational+Mapping+Guide\" target=\"_blank\">\"caCORE SDK Version 4.3 Object Relational Mapping Guide\"<\/a>. National Cancer Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/wiki.nci.nih.gov\/display\/caCORE\/caCORE+SDK+Version+4.3+Object+Relational+Mapping+Guide\" target=\"_blank\">https:\/\/wiki.nci.nih.gov\/display\/caCORE\/caCORE+SDK+Version+4.3+Object+Relational+Mapping+Guide<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=caCORE+SDK+Version+4.3+Object+Relational+Mapping+Guide&rft.atitle=&rft.aulast=Wiley%2C+A.%3B+Gagne%2C+B.&rft.au=Wiley%2C+A.%3B+Gagne%2C+B.&rft.date=18+April+2012&rft.pub=National+Cancer+Institute&rft_id=https%3A%2F%2Fwiki.nci.nih.gov%2Fdisplay%2FcaCORE%2FcaCORE%2BSDK%2BVersion%2B4.3%2BObject%2BRelational%2BMapping%2BGuide&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrayExt06-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrayExt06_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bray, T.; Paoli, J.; Sperberg-McQueen, C.M.; Maler, E.; Yergeau, F.; Cowan, J. (29 September 2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.w3.org\/TR\/xml11\/\" target=\"_blank\">\"Extensible Markup Language (XML) 1.1 (Second Edition)\"<\/a>. World Wide Web Consortium<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.w3.org\/TR\/xml11\/\" target=\"_blank\">http:\/\/www.w3.org\/TR\/xml11\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Extensible+Markup+Language+%28XML%29+1.1+%28Second+Edition%29&rft.atitle=&rft.aulast=Bray%2C+T.%3B+Paoli%2C+J.%3B+Sperberg-McQueen%2C+C.M.%3B+Maler%2C+E.%3B+Yergeau%2C+F.%3B+Cowan%2C+J.&rft.au=Bray%2C+T.%3B+Paoli%2C+J.%3B+Sperberg-McQueen%2C+C.M.%3B+Maler%2C+E.%3B+Yergeau%2C+F.%3B+Cowan%2C+J.&rft.date=29+September+2006&rft.pub=World+Wide+Web+Consortium&rft_id=http%3A%2F%2Fwww.w3.org%2FTR%2Fxml11%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ApacheTom-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ApacheTom_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/tomcat.apache.org\/\" target=\"_blank\">\"Apache Tomcat\"<\/a>. The Apache Software Foundation. 2014<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/tomcat.apache.org\/\" target=\"_blank\">http:\/\/tomcat.apache.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 02 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Apache+Tomcat&rft.atitle=&rft.date=2014&rft.pub=The+Apache+Software+Foundation&rft_id=http%3A%2F%2Ftomcat.apache.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WeibelTheDub97-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WeibelTheDub97_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Weibel, S. (October\/November 1997). \"The Dublin Core: A Simple Content Description Model for Electronic Resources\". <i>Bulletin of the American Society for Information Science and Technology<\/i> <b>24<\/b> (1): 9\u201311. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fbult.70\" target=\"_blank\">10.1002\/bult.70<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Dublin+Core%3A+A+Simple+Content+Description+Model+for+Electronic+Resources&rft.jtitle=Bulletin+of+the+American+Society+for+Information+Science+and+Technology&rft.aulast=Weibel%2C+S.&rft.au=Weibel%2C+S.&rft.date=October%2FNovember+1997&rft.volume=24&rft.issue=1&rft.pages=9%E2%80%9311&rft_id=info:doi\/10.1002%2Fbult.70&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MilesSKOS09-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MilesSKOS09_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Miles, A.; Bechhofer, S. (18 August 2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.w3.org\/TR\/skos-reference\/\" target=\"_blank\">\"SKOS Simple Knowledge Organization System Reference\"<\/a>. World Wide Web Consortium<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.w3.org\/TR\/skos-reference\/\" target=\"_blank\">http:\/\/www.w3.org\/TR\/skos-reference\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=SKOS+Simple+Knowledge+Organization+System+Reference&rft.atitle=&rft.aulast=Miles%2C+A.%3B+Bechhofer%2C+S.&rft.au=Miles%2C+A.%3B+Bechhofer%2C+S.&rft.date=18+August+2009&rft.pub=World+Wide+Web+Consortium&rft_id=http%3A%2F%2Fwww.w3.org%2FTR%2Fskos-reference%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GonzalesTem12-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GonzalesTem12_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gonzales-Aguilar, A.; Ram\u00edrez-Posada, M.; Ferreyra, D. (2012). \"Tematres: software para gestionar tesauros\". <i>El Profesional de la Informaci\u00f3n<\/i> <b>21<\/b> (3): 319\u2013325. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3145%2Fepi.2012.may.14\" target=\"_blank\">10.3145\/epi.2012.may.14<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Tematres%3A+software+para+gestionar+tesauros&rft.jtitle=El+Profesional+de+la+Informaci%C3%B3n&rft.aulast=Gonzales-Aguilar%2C+A.%3B+Ram%C3%ADrez-Posada%2C+M.%3B+Ferreyra%2C+D.&rft.au=Gonzales-Aguilar%2C+A.%3B+Ram%C3%ADrez-Posada%2C+M.%3B+Ferreyra%2C+D.&rft.date=2012&rft.volume=21&rft.issue=3&rft.pages=319%E2%80%93325&rft_id=info:doi\/10.3145%2Fepi.2012.may.14&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RTheR-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RTheR_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">R Development Core Team (2015). <a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.r-project.org\/\" target=\"_blank\">\"R: A language and environment for statistical computing\"<\/a>. The R Foundation for Statistical Computing<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.r-project.org\/\" target=\"_blank\">https:\/\/www.r-project.org\/<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 09 December 2014<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=R%3A+A+language+and+environment+for+statistical+computing&rft.atitle=&rft.aulast=R+Development+Core+Team&rft.au=R+Development+Core+Team&rft.date=2015&rft.pub=The+R+Foundation+for+Statistical+Computing&rft_id=https%3A%2F%2Fwww.r-project.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UrbanekRserve03-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UrbanekRserve03_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Urbanek, S. (2003). \"Rserve: A Fast Way to Provide R Functionality to Applications\". In Hornik, K.; Leisch, F.; Zeileis, A.. <i>Proceedings of the 3rd international workshop on distributed statistical computing (DSC 2003)<\/i>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Rserve%3A+A+Fast+Way+to+Provide+R+Functionality+to+Applications&rft.atitle=Proceedings+of+the+3rd+international+workshop+on+distributed+statistical+computing+%28DSC+2003%29&rft.aulast=Urbanek%2C+S.&rft.au=Urbanek%2C+S.&rft.date=2003&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JacobsonObj92-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JacobsonObj92_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Jacobson, I. (1992). <i>Object Oriented Software Engineering: A Use Case Driven Approach<\/i>. Wokingham, England: Addison-Wesley Professional.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Object+Oriented+Software+Engineering%3A+A+Use+Case+Driven+Approach&rft.aulast=Jacobson%2C+I.&rft.au=Jacobson%2C+I.&rft.date=1992&rft.place=Wokingham%2C+England&rft.pub=Addison-Wesley+Professional&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BittnerUse02-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BittnerUse02_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Bittner, K.; Spence, I. (2002). <i>Use Case Modeling<\/i>. Boston, MA: Addison-Wesley Professional.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Use+Case+Modeling&rft.aulast=Bittner%2C+K.%3B+Spence%2C+I.&rft.au=Bittner%2C+K.%3B+Spence%2C+I.&rft.date=2002&rft.place=Boston%2C+MA&rft.pub=Addison-Wesley+Professional&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HeProp13-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HeProp13_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">He, S.; Ganzinger, M.; Hurdle, J.F.; Knaup, P. (2013). \"Proposal for a data publication and citation framework when sharing biomedical research resources\". <i>Studies in Health Technology and Informatics<\/i> <b>192<\/b>: 1201. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.3233%2F978-1-61499-289-9-1201\" target=\"_blank\">10.3233\/978-1-61499-289-9-1201<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23920975\" target=\"_blank\">23920975<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Proposal+for+a+data+publication+and+citation+framework+when+sharing+biomedical+research+resources&rft.jtitle=Studies+in+Health+Technology+and+Informatics&rft.aulast=He%2C+S.%3B+Ganzinger%2C+M.%3B+Hurdle%2C+J.F.%3B+Knaup%2C+P.&rft.au=He%2C+S.%3B+Ganzinger%2C+M.%3B+Hurdle%2C+J.F.%3B+Knaup%2C+P.&rft.date=2013&rft.volume=192&rft.pages=1201&rft_id=info:doi\/10.3233%2F978-1-61499-289-9-1201&rft_id=info:pmid\/23920975&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MurphyArch07-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MurphyArch07_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Murphy, S.N.; Mendis, M.; Hackett, K.; Kuttan, R.; Pan, W.; Phillips, L.C.; Gainer, V.; Berkowicz, D.; Glaser, J.P.; Kohane, I.; Chueh, H.C. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655844\" target=\"_blank\">\"Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside\"<\/a>. <i>AMIA Annual Symposium Proceedings<\/i> <b>2007<\/b>: 548\u2013552. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2655844\/\" target=\"_blank\">PMC2655844<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18693896\" target=\"_blank\">18693896<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655844\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2655844<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Architecture+of+the+open-source+clinical+research+chart+from+Informatics+for+Integrating+Biology+and+the+Bedside&rft.jtitle=AMIA+Annual+Symposium+Proceedings&rft.aulast=Murphy%2C+S.N.%3B+Mendis%2C+M.%3B+Hackett%2C+K.%3B+Kuttan%2C+R.%3B+Pan%2C+W.%3B+Phillips%2C+L.C.%3B+Gainer%2C+V.%3B+Berkowicz%2C+D.%3B+Glaser%2C+J.P.%3B+Kohane%2C+I.%3B+Chueh%2C+H.C.&rft.au=Murphy%2C+S.N.%3B+Mendis%2C+M.%3B+Hackett%2C+K.%3B+Kuttan%2C+R.%3B+Pan%2C+W.%3B+Phillips%2C+L.C.%3B+Gainer%2C+V.%3B+Berkowicz%2C+D.%3B+Glaser%2C+J.P.%3B+Kohane%2C+I.%3B+Chueh%2C+H.C.&rft.date=2007&rft.volume=2007&rft.pages=548%E2%80%93552&rft_id=info:pmc\/PMC2655844&rft_id=info:pmid\/18693896&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2655844&rfr_id=info:sid\/en.wikipedia.org:Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. References were in alphabetical on the original, but references are listed in order of appearance on this wiki. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210220\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.700 seconds\nReal time usage: 0.737 seconds\nPreprocessor visited node count: 23717\/1000000\nPreprocessor generated node count: 36782\/1000000\nPost\u2010expand include size: 176525\/2097152 bytes\nTemplate argument size: 57930\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 686.345 1 - -total\n 85.23% 584.993 1 - Template:Reflist\n 74.11% 508.666 31 - Template:Citation\/core\n 52.51% 360.383 19 - Template:Cite_journal\n 14.57% 100.000 7 - Template:Cite_web\n 11.87% 81.459 5 - Template:Cite_book\n 9.90% 67.949 1 - Template:Infobox_journal_article\n 9.55% 65.520 1 - Template:Infobox\n 6.81% 46.752 35 - Template:Citation\/identifier\n 5.47% 37.534 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7847-0!*!*!!en!5!* and timestamp 20181213210219 and revision id 23418\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model\">https:\/\/www.limswiki.org\/index.php\/Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","0a86ce45c610776ca0038eb4e754f3c5_images":["https:\/\/www.limswiki.org\/images\/4\/42\/Fig1_GanzingerPeerJCS2015_3.png","https:\/\/www.limswiki.org\/images\/a\/a7\/Fig2_GanzingerPeerJCS2015_3.png","https:\/\/www.limswiki.org\/images\/f\/fd\/Fig3_GanzingerPeerJCS2015_3.png","https:\/\/www.limswiki.org\/images\/d\/d9\/Fig4_GanzingerPeerJCS2015_3.png","https:\/\/www.limswiki.org\/images\/5\/52\/Fig5_GanzingerPeerJCS2015_3.png"],"0a86ce45c610776ca0038eb4e754f3c5_timestamp":1544734939,"03ae650a31def9630dc93ec76a1927f6_type":"article","03ae650a31def9630dc93ec76a1927f6_title":"Personalized Oncology Suite: Integrating next-generation sequencing data and whole-slide bioimages (Dander et al. 2014)","03ae650a31def9630dc93ec76a1927f6_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages","03ae650a31def9630dc93ec76a1927f6_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Personalized Oncology Suite: Integrating next-generation sequencing data and whole-slide bioimages\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nPersonalized Oncology Suite: Integrating next-generation sequencing data and whole-slide bioimagesJournal\n \nBMC BioinformaticsAuthor(s)\n \nDander, Andreas; Baldauf, Matthias; Sperk, Michael; Pabinger, Stephan; Hiltpolt, Benjamin; Trajanoski, ZlatkoAuthor affiliation(s)\n \nInnsbruck Medical University, AIT-Austrian Institute of Technology, Oncotyrol GmbHPrimary contact\n \nEmail: zlatko.trajanoski@i-med.ac.atYear published\n \n2014Volume and issue\n \n15Page(s)\n \n306DOI\n \n10.1186\/1471-2105-15-306ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.biomedcentral.com\/1471-2105\/15\/306Download\n \nhttp:\/\/www.biomedcentral.com\/content\/pdf\/1471-2105-15-306.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Implementation \n4 Results and discussion \n\n4.1 Clinical data and tumor staging \n4.2 Tumor staging \n4.3 Next-generation sequencing data \n4.4 Genome browser \n4.5 Bioimages \n4.6 Image upload \n4.7 Bisque connection \n4.8 Flexible batch upload module \n4.9 Authorization and authentication system \n4.10 Installation \n\n\n5 Conclusions \n6 Availability and requirements \n7 Abbreviations \n8 Competing interests \n9 Authors\u2019 contributions \n10 Acknowledgements \n11 References \n12 Notes \n\n\n\nAbstract \nBackground: Cancer immunotherapy has recently entered a remarkable renaissance phase with the approval of several agents for treatment. Cancer treatment platforms have demonstrated profound tumor regressions including complete cure in patients with metastatic cancer. Moreover, technological advances in next-generation sequencing (NGS) as well as the development of devices for scanning whole-slide bioimages from tissue sections and image analysis software for quantitation of tumor-infiltrating lymphocytes (TILs) allow, for the first time, the development of personalized cancer immunotherapies that target patient specific mutations. However, there is currently no bioinformatics solution that supports the integration of these heterogeneous datasets.\nResults: We have developed a bioinformatics platform \u2013 Personalized Oncology Suite (POS) \u2013 that integrates clinical data, NGS data and whole-slide bioimages from tissue sections. POS is a web-based platform that is scalable, flexible and expandable. The underlying database is based on a data warehouse schema, which is used to integrate information from different sources. POS stores clinical data, genomic data (SNPs and INDELs identified from NGS analysis), and scanned whole-slide images. It features a genome browser as well as access to several instances of the bioimage management application Bisque. POS provides different visualization techniques and offers sophisticated upload and download possibilities. The modular architecture of POS allows the community to easily modify and extend the application.\nConclusions: The web-based integration of clinical, NGS, and imaging data represents a valuable resource for clinical researchers and future application in medical oncology. POS can be used not only in the context of cancer immunology but also in other studies in which NGS data and images of tissue sections are generated. The application is open-source and can be downloaded at http:\/\/www.icbi.at\/POS.\nKeywords: Personalized oncology; Data integration; Next-generation sequencing; Whole-slide bioimaging; Application; Open-source\n\nBackground \nCancer immunotherapy has recently entered a remarkable renaissance phase with the approval of several agents for treatment.[1] Cancer immunotherapies that involve the use of the adaptive immune system, such as anti-checkpoint antibodies and adoptive T-cell therapies, have demonstrated profound tumor regressions including complete cure in patients with metastatic cancer. Technological advances in next-generation sequencing (NGS) as well as the development of devices for scanning whole-slide images from tissue sections and image analysis software for quantitation of tumor-infiltrating lymphocytes (TILs) allow, for the first time, the development of personalized cancer immunotherapies that target patient specific mutations. The use of NGS technologies to characterize tumor samples enables one not only to comprehensively study the interactions between human cancers and the immune system, but also to identify targets for patient stratification. Moreover, the quantitation of TILs will improve therapeutic efficacy, even in the absence of immunotherapy. It will enable a precise characterization of the immune infiltrates in the tumor and will help to identify mechanisms of tumor regression and disentangle the complex tumor-immune cell interactions. For example, understanding the molecular basis of the interactions between cytotoxic chemotherapeutics or targeted anti-cancer agents and the immune system is essential for the development of optimal therapeutic schemes and in the long run will result in clinical benefit for the patients.\nHowever, the real value of the disparate datasets can be truly exploited only when the data are integrated. In our experience it is of utmost importance to establish a local database hosting only the necessary data. Only pre-processed and normalized data will be stored in a dedicated relational database whereas primary data are archived at separate locations including public repositories.[2] To this end, a database that integrates clinical, NGS, and bioimaging data would be extremely helpful for clinical cancer research and in near future also for routine applications in medical oncology. However, to the best of our knowledge there is currently no application that supports this integration. As of today there are different applications integrating either clinical data and NGS data or bioimages (Table 1) but no integrated solution has been created. We therefore developed the bioinformatics platform Personalized Oncology Suite (POS) to overcome this bottleneck and support the researchers working in this exciting field.\n\n\n\n\n\n\n\nTable 1. This table compares POS with other applications in the context of data integration\n\n\n Application\n\n Edit\n\n Clinical\n\n TNM\n\n Sequencing\n\n Vis.\n\n Bioimages\n\n Public\n\n Ref.\n\n\nBioIMAX\n\nY\n\nN\n\nN\n\nN\n\nN\n\nWhole-slide\n\nN\n\n[3]\n\n\nBisque\n\nY\n\nN\n\nN\n\nN\n\nN\n\nWhole-slide\n\nN\n\n[4]\n\n\nGalaxy LIMS\n\nY\n\nN\n\nN\n\nY\n\nN\n\nN\n\nN\n\n[5]\n\n\nNG6\n\nN\n\nN\n\nN\n\nRaw 454 and HiSeq\n\nN\n\nN\n\nN\n\n[6]\n\n\nNGS tools\n\nY\n\nN\n\nN\n\nRaw Illumina\n\nN\n\nN\n\nDAS[7]\n\n[8]\n\n\nOMERO\n\nY\n\nN\n\nN\n\nN\n\nN\n\nN\n\nN\n\n[9]\n\n\nONCO-i2b2\n\nN\n\nY\n\nY\n\nN\n\nN\n\nN\n\nSNOMED\n\n[10]\n\n\nopenBIS\n\nY\n\nN\n\nN\n\nRaw Illumina\n\nN\n\nN\n\nN\n\n[11]\n\n\nTaverna\n\nN\n\nN\n\nN\n\nY\n\nN\n\nN\n\nY\n\n[12]\n\n\nPOS\n\nY\n\nY\n\nY\n\nvcf Files\n\nY\n\nWhole-slide\n\nCOSMIC[13]\n\n-\n\n\nThe columns Clinical and TNM show if these data types are available. Sequencing depicts which type of next-generation sequencing data can be uploaded, and the column Vis. shows if mutations can be visualized. The column Bioimages shows which type of images can be used and the final column Public states available public annotations.\n\n\n\nImplementation \nIn order to be scalable, flexible, and expandable, POS makes use of state of the art software engineering techniques and architectures like the Java Enterprise Edition 6 (J2EE 6) technology stack. It is a web-based platform relying on the JBoss Application Server in version 7.1.1. The modular three-tier architecture (web frontend, application core, database backend) and the release under the open-source license GNU AGPL enables the community to easily modify and extend the application with further functionalities. Figure 1 outlines the software architecture of POS and depicts the main used libraries. PrimeFaces and PrimeFaces Extensions are used for the creation of JSF components, whereas Hibernate Validator provides input validation of user entries. As access scopes of Java Beans are crucial within JavaEE applications, Apache CODI is used to include additional scopes. Due to the fact that POS deals with different types of collections, the Guava libraries were chosen to support POS with a set of helpful functionalities regarding collections. Used Bisque instances are shown at the top of Figure 1. Furthermore, the standalone application POS Image Uploader allows batch uploading of numerous images at once. \n\n\n\n\n\n\n\n\n\n Figure 1: Software architecture of POS. The JBoss Application Server of POS is shown as the central rectangle containing different JSF libraries. On the right hand side the attached Authorization and Authentication System (AAS) is depicted. POS uses PostgreSQL as database management system and applies EclipseLink for the object-relational mapping. The Bioconductor package Gviz handles rendering of the genome browser\r\ntracks, and the R package Rserver provides an R server available through a network connection. On top, distributed Bisque instances are shown. The POS Image Uploader, a JavaFX based standalone application, is outlined on the top left in the figure. This application enables users to upload several images at once.\n\n\n\n\r\n\nThe underlying database is based on a data warehouse schema. This schema was chosen as it is widely used for integrating information from different sources. All defined entities are outlined in Additional file 1. POS uses the Java library EclipseLink for object-relational mapping. In the default configuration POS runs with PostgreSQL, but can be easily exchanged with another relational database. \n\n\n\n\n\n\n\n Additional file 1. Database schema of the Personalized Oncology Suite.\n\n\nThe database schema of POS is based on a data warehouse schema. Therefore, the central entity represents patients. Each patient belongs to an institute, which stores the externalid referencing an institute within the attached Authorization and Authentication System. It can be seen that each institute can be connected to an imagerepository containing information about the connection to a Bisque instance. The entity institutesharing manages information about shared data. Clinicaldata as well as tnm and immunoscore for staging of cancer are related to the entity patient. For the integration of somatic mutations within POS the entities snp and indel are used. The entity analysis contains metadata about the next-generation sequencing itself. Image manages the attributes externalimageid and externalresourceid which are IDs used for accessing the image within Bisque. The attached imagetype contains information about the staining of the image. All entities with a name like <name>_audittrail hold information about documented changes made to the attached entity <name>. It is shown that the timestamp, the name of the author and the performed changes are recorded within these entities. Several entities comprise a deleted flag. If such an entity gets deleted it will not be removed within the database, but will not be shown in the frontend. This has the advantage that deleted entities can be restored by a database administrator.\nFormat: PNG; Size: 1.5MB Download file\n\n\n\n\nResults and discussion \nThe Personalized Oncology Suite is an application combining biological and clinical data into one integrated solution. In this context, clinical data comprises information about cancer patients, TNM staging, and density values of TILs used for immune score estimation. Biological data describes mutations found via next-generation sequencing and whole-slide bioimages, which can be uploaded to the application. POS features different types of visualization techniques for all integrated data types. Furthermore, publicly available data from the COSMIC database[13] is integrated into POS. As data types are stored in different file formats, POS includes several data import and export possibilities for the most important formats. In addition, different filters on the data can be applied either individually or in combination (e.g. age at diagnosis and TNM stage). In the current implementation queries can be done only for single modalities (e.g. genetic features, images, or clinical parameters). Furthermore, patients can be selected based on their UICC stage or Immunoscore by using a range slider. POS features different user interface languages using internationalization and is fault tolerant as all inputs are validated. Furthermore, POS implements exception handling as well as an intelligent logging functionality. \nPOS has been released under the open-source license GNU AGPL to allow integration of new components from the scientific community. The source code can be accessed via the project website http:\/\/www.icbi.at\/POS. Figure 2 outlines the different layers between the JSF based frontend and the relational database. \n\n\n\n\n\n\n\n\n\n Figure 2: Software layers of POS. a) The JSF based presentation layer on top (blue) access the relational database (green) via the Java backend. This backend can be accessed via the classes Controller or LazyDataModel. These classes make use of the DataAccessObject which provides a single access point to the relational database by using object-relational mapping. b) The Controller is responsible for secure CRUD (create, read, update and delete) operations. c) The LazyDataModel is used for providing immutable, filtered, sorted, and paginated collections to the presentation layer.\n\n\n\nClinical data and tumor staging \nPOS integrates clinical data of cancer patients including various attributes such as gender, date of diagnosis, disease duration, adjuvant therapy, and relapse. We have included compliant measures in the design of the software ensuring that patient-identifying data such as name, academic title, address, telephone number, e-mail address, date and place of birth, as well as date and place of death[14] are not collected. Patients are uniquely defined by alphanumerical identifiers. Thus, data from multiple clinical visits or any other information can be unambiguously assigned to an individual patient. \n\nTumor staging \nIn addition to clinical data, POS integrates the TNM cancer staging system. The T, N, and M categories are stored separately within the database and the resulting AJCC\/UICC stage (0-IV) is determined from this information. POS facilitates the creation of descriptive plots for comparing patients in different TNM stages. These plots can also be used for comparison of data among different participating institutes, provided the user has permission to access the information. In addition to input data manually, clinical and staging data can be uploaded to POS in CSV format. Furthermore, these data types can be exported as XLS, CSV and PDF files.\n\nNext-generation sequencing data \nWith the use of next-generation sequencing, biologists are able to determine the order of nucleotide bases composing the DNA. The identification of somatic mutations can be performed by sequencing tumor\/normal pairs and subsequently comparing cancerous to healthy tissue. Several different applications exist which are able to analyze tumor\/normal pairs regarding their somatic SNPs and INDELs.[15] Since there are a plethora of methods for analyses of NGS data, the design of the software focused on the integration of disparate data types rather than the analyses of raw data. Furthermore, as the methods and available tools improve at fast pace, integration of processed data makes the system more flexible and versatile. POS is able to integrate somatic SNPs and INDELs identified by such experiments. For uploading this type of data, users first need to define an analysis, which can be used to attach called mutations. POS supports data upload via VCF (Variant Call Format) files or by manual input of mutations. \n\nGenome browser \nAs visualization of identified mutations tremendously supports the interpretation of these results, genome browsers have been developed to display mutations in the context of a reference genome.[15] POS includes a genome browser that features a combined view of different tracks, each containing a dedicated plot. Figure 3 shows a screenshot of the genome browser view within POS. On top of the genome browser panel, the region of interest can be specified by defining chromosome number, start and end position or by choosing a gene of interest. The user can select several patients at once and mutations for each patient will be displayed in separate tracks. \n\n\n\n\n\n\n\n\n\n Figure 3: View of the genome browser. A genome browser visualizing mutations in the context of the reference genome including publicly available annotations is shown. The first track depicts the ideogram of the chosen chromosome, followed by an axis showing its genomic coordinates. Next, publicly available annotations derived from BioMart[16] are depicted. The bottom track holds information about uploaded mutations. The patient name and the shown mutations were randomly generated.\n\n\n\nBioimages \nWhole-slide bioimages can be used to estimate the density of TILs. However, due to the size of the files the required disk space can be very large. Therefore, the raw image files are not stored in POS. In order to integrate whole-slide images within POS, the application connects to several instances of the bioimage management application Bisque[4] as outlined in Figure 1. POS supports the following formats: 1) SVS format from Aperio, 2) VSI format from Olympus, 3) Hamamatsu formats NDPI and VMS, 4) TIFF format from Trestle, 5) Leica\u2019s SCN format, 6) formats ZVI and CZI from Zeiss, and 7) OME.TIFF format from the Open Microscopy Environment OME.[17][18]\n\nImage upload \nPOS supports different ways for uploading images. The first option is a direct upload module, where a patient needs to be selected before the upload starts. The second possibility allows access to already uploaded images within a connected Bisque instance and their assignment to selected patients in POS. The third option is an external upload application supporting the batch upload of multiple images at once. This application is also able to assign the correct relations between uploaded images and patients within POS.\n\nBisque connection \nThe connection to Bisque instances is administered by POS, which handles correct URL, port, username and password settings. In addition, tailored Java servlets for managing the encrypted communication between POS and Bisque have been developed. Downscaling and tiling of the images is performed by the connected Bisque systems. POS is able to display the created tiles in a Google maps like manner, using an adapted version of the interactive JavaScript widget PanoJS3. Hence, only tiles of the currently displayed region are fetched from Bisque. A screenshot of the POS image viewer is displayed in Figure 4.\n\n\n\n\n\n\n\n\n\n Figure 4: Image viewer. This Figure depicts a screenshot of an example of a whole-slide bioimage. The image is stored at an external Bisque instance and only tiles that correspond to the current image section are transferred to POS. General information about the patient and the image are depicted on the right hand side. Various attributes like width, height, resolution, and information about the scanner are directly fetched from Bisque. Shown clinical data were randomly generated.\n\n\n\nFlexible batch upload module \nAs the JSF based image upload form of POS does not support batch upload of several images at once, the standalone application POS Image Uploader has been developed. This application is based on JavaFX and can be started on all systems where Java is installed. Figure 1 depicts this standalone application at the top. The POS Image Uploader is able to acquire the URL and corresponding login information of Bisque instances directly from POS. During the batch upload, each image is uploaded to a selected Bisque instance. Bisque returns identifiers for each uploaded image, which are stored within the POS database and linked to related patients. The communication between the standalone tool and the web-based application is managed by tailored web services.\n\nAuthorization and authentication system \nAs POS holds confidential and patient related data, the application needs to be secured. POS is secured by an authorization and authentication system (AAS) where participating users and institutes can be configured within this easy to use user management system.[19] The backend of the used AAS can either be a simple XML file or a web-based application running on a dedicated JBoss Application Server (see Figure 1). For tailored user access, POS applies different user roles, provided by the AAS. Access roles are defined for clinical, NGS, and imaging data as well as for administration. In order to provide custom access profiles, users can be assigned to several roles at once. Furthermore, all web services, used by the POS Image Uploader, authenticate each single connection via the AAS.\nFinally, POS users can also share their data with other participating users, with the possibility to restrict access to certain data types. It should be noted that as access to patient related data needs to be secured, the implemented sharing concept and the AAS do not support public access to any data type. Rather, users can specify the type of data that can be shared with other users including images, genetic data, or clinical data. If an external user wants access to this integrated data a request must be performed and all participating institutes will be informed about this request. \n\nInstallation \nAs POS is a web-based application, it can be accessed from any operating system. The only requirements are an up to date web browser and a network connection to the POS server. The server itself is able to run on any operating system with installed Java and R, and has been extensively tested on Linux and Windows machines. The database backend is interchangeable - POS has been tested with the database systems PostgreSQL and MySQL. A detailed installation guide for POS is available at the project home page, accessible at http:\/\/www.icbi.at\/POS. As POS makes use of several Bisque installations, an installation guide for Bisque is provided at the same location.\n\nConclusions \nPOS is a web-based application combining clinical and biomolecular data including NGS data and bioimages. Due to its modular and flexible architecture it can be easily extended and adapted to different requirements. POS provides an intuitive user interface supporting data upload, manipulation and visualization of integrated data types. In summary, POS represents an effective solution for current challenges in clinical cancer research. The suite can be used not only in the context of personalized cancer immunotherapy but also in other studies where NGS data and images of tissue sections are generated.\n\nAvailability and requirements \nProject name: Personalized Oncology Suite (POS)\nProject home page: http:\/\/www.icbi.at\/POS\nOperating system: Platform independent\nProgramming language: Java, R\/Bioconductor\nLicense: GNU AGPL\nAny restrictions to use by non-academics: No\n\nAbbreviations \nAAS: Authorization and Authentication System \nAGPL: Affero General Public License\nCOSMIC: Catalogue of Somatic Mutations in Cancer\nCRUD: Create, Read, Update and Delete\nINDEL: Small insertion or deletion\nJSF: JavaServer Faces\nPOS: Personalized Oncology Suite\nSNP: Single-nucleotide polymorphism\nTIL: Tumor-infiltrating lymphocyte\nTNM: Classification of Malignant Tumors\nURL: Uniform Resource Locator\nVCF: Variant Call Format\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors\u2019 contributions \nAD is the principle developer of POS. MB, MS and BH developed parts of the application. SP as well as ZT reviewed and tested POS and provided suggestions for additional features. All authors participated in writing of the manuscript, read and approved the final manuscript.\n\nAcknowledgements \nThis work was supported by ONCOTYROL (FFG, Austrian Federal Ministries BMVIT\/BMWFJ) and a grant from the Standortagentur Tirol.\n\nReferences \n\n\n\u2191 Couzin-Frankel, J. (2013). \"Cancer Immunotherapy\". Science 342 (6165): 1432-1433. doi:10.1126\/science.342.6165.1432. PMID 24357284.   \n\n\u2191 Hackl, H.; Stocker, G.; Charoentong, P.; Mlecnik, B.; Bindea, G.; Galon, J.; Trajanoski, Z. (2010). \"Information technology solutions for integration of biomolecular and clinical data in the identification of new cancer biomarkers and targets for therapy\". Pharmacology & Therapeutics 128 (3): 488\u2013498. doi:10.1016\/j.pharmthera.2010.08.012. PMID 20832425.   \n\n\u2191 Loyek, C.; Rajpoot, N.M.; Khan, M.; Nattkemper, T.W. (2011). \"BioIMAX: A Web 2.0 approach for easy exploratory and collaborative access to multivariate bioimage data\". BMC Bioinformatics 12: 297. doi:10.1186\/1471-2105-12-297. PMC PMC3161928. PMID 21777450. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3161928 .   \n\n\u2191 4.0 4.1 Kvilekval, K.; Fedorov, D.; Obara, B.; Singh, A.; Manjunath, B.S. (2010). \"Bisque: A platform for bioimage analysis and management\". Bioinformatics 26 (4): 544-52. doi:10.1093\/bioinformatics\/btp699. PMID 20031971.   \n\n\u2191 Scholtalbers, J.; R\u00f6ssler, J.; Sorn, P.; de Graaf, J.; Boisgu\u00e9rin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". Bioinformatics 29 (9): 1233-1234. doi:10.1093\/bioinformatics\/btt115. PMID 23479349.   \n\n\u2191 Mariette, J.; Escudi\u00e9, F.; Allias, N.; Salin, G.; Noirot, C.; Thomas, S.; Klopp, C. (2012). \"NG6: Integrated next generation sequencing storage and processing environment\". BMC Genomics 13: 462. doi:10.1186\/1471-2164-13-462. PMC PMC3444930. PMID 22958229. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3444930 .   \n\n\u2191 Jenkinson, A.M.; Albrecht, M.; Birney, E.; Blankenburg, H.; Down, T.; Finn, R.D.; Hermjakob, H.; Hubbard, T.J.P.; Jimenez, R.C.; Jones, P.; K\u00e4h\u00e4ri, A.; Kulesha, E.; Mac\u00edas, J.R.; Reeves, G.A.; Prli\u0107, A. (2008). \"Integrating biological data-the Distributed Annotation System\". BMC Bioinformatics 9 (Suppl 8): S3. doi:10.1186\/1471-2105-9-S8-S3.   \n\n\u2191 Nix, D.A.; Di Sera, T.L.; Dalley, B.K.; Milash, B.A.; Cundick, R.M.; Quinn, K.S.; Courdy, S.J. (2010). \"Next generation tools for genomic data generation, distribution, and visualization\". BMC Bioinformatics 11: 455. doi:10.1186\/1471-2105-11-455. PMC PMC2944281. PMID 20828407. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281 .   \n\n\u2191 Allan, C.; Burel, J-M.; Moore, J.; Blackburn, C.; Linkert, M.; Loynton, S.; MacDonald, D.; Moore, W.J.; Neves, C.; Patterson, A.; Porter, M.; Tarkowska, A.; Loranger, B.; Avondo, J.; Lagerstedt, I.; Lianas, L.; Leo, S.; Hands, K.; Hay, R.T.; Patwardhan, A.; Best, C.; Kleywegt, G.J.; Zanetti, G.; Swedlow, J.R. (2012). \"OMERO: flexible, model-driven data management for experimental biology\". Nature Methods 9 (3): 245-53. doi:10.1038\/nmeth.1896. PMC PMC3437820. PMID 22373911. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3437820 .   \n\n\u2191 Segagni, D.; Tibollo, V.; Dagliati, A.; Zambelli, A.; Priori, S.G.; Bellazzi, R. (2012). \"An ICT infrastructure to integrate clinical and molecular data in oncology research\". BMC Bioinformatics 13: 4. doi:10.1186\/1471-2105-13-4. PMC PMC3280155. PMID 22226192. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3280155 .   \n\n\u2191 Bauch, A.; Adamczyk, I.; Buczek, P.; Elmer, F-J.; Enimanev, K.; Glyzewski, P.; Kohler, M.; Pylak, T.; Quandt, A.; Ramakrishnan, C.; Beisel, C.; Malmstr\u00f6m, L.; Aebersold, R.; Rinn, B. (2011). \"openBIS: A flexible framework for managing and analyzing complex data in biology research\". BMC Bioinformatics 12: 468. doi:10.1186\/1471-2105-12-468. PMC PMC3275639. PMID 22151573. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639 .   \n\n\u2191 Wolstencroft, K.; Haines, R.; Fellows, D.; Williams, A.; Withers, D.; Owen, S.; Soiland-Reyes, S.; Dunlop, I.; Nenadic, A.; Fisher, P.; Bhagat, J.; Belhajjame, K.; Bacall, F.; Hardisty, A.; Nieva de la Hidalga, A.; Balcazar Vargas, M.P.; Sufi, S.; Goble, C. (2013). \"The Taverna workflow suite: Designing and executing workflows of Web Services on the desktop, web or in the cloud\". Nucleic Acids Research 41 (W1): W557-W561. doi:10.1093\/nar\/gkt328. PMC PMC3692062. PMID 23640334. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3692062 .   \n\n\u2191 13.0 13.1 Forbes, S.A.; Bindal, N.; Bamford, S.; Cole, C.; Kok, C.Y.; Beare, D.; Jia, M.; Shepherd, R.; Leung, K.; Menzies, A.; Teague, J.W.; Campbell, P.J.; Stratton, M.R.; Futreal, P.A. (2011). \"COSMIC: mining complete cancer genomes in the Catalogue of Somatic Mutations in Cancer\". Nucleic Acids Research 39 (Suppl 1): D945-D950. doi:10.1093\/nar\/gkq929. PMC PMC3013785. PMID 20952405. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013785 .   \n\n\u2191 Sch\u00fctze, B. (2012). \"Use of medical treatment data outside of the patient supply \u2013 best way pseudonymisation\". Deutsche Medizinische Wochenschrift 137 (16): 844-50. doi:10.1055\/s-0031-1299040. PMID 22495919.   \n\n\u2191 15.0 15.1 Pabinger, S.; Dander, A.; Fischer, M.; Snajder, R.; Sperk, M.; Efremova, M.; Krabichler, B.; Speicher, M.R.; Zschocke, J.; Trajanoski, Z. (2014). \"A survey of tools for variant analysis of next-generation genome sequencing data\". Briefings in Bioinformatics 15 (2): 256-78. doi:10.1093\/bib\/bbs086. PMC PMC3956068. PMID 23341494. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3956068 .   \n\n\u2191 Kasprzyk, A. (2011). \"BioMart: Driving a paradigm change in biological data management\". Database 2011: bar049. doi:10.1093\/database\/bar049. PMC PMC3215098. PMID 22083790. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215098 .   \n\n\u2191 Goode, A.; Gilbert, B.; Harkes, J.; Jukic, D.; Satyanarayanan, M. (2013). \"OpenSlide: A vendor-neutral software foundation for digital pathology\". Journal of Pathology Informatics 4: 27. doi:10.4103\/2153-3539.119005. PMC PMC3815078. PMID 24244884. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3815078 .   \n\n\u2191 Linkert, M.; Rueden, C.T.; Allan, C.; Burel, J-M.; Moore, W.; Patterson, A.; Loranger, B.; Moore, J.; Neves, C.; MacDonald, D.; Tarkowska, A.; Sticco, C.; Hill, E.; Rossner, M.; Eliceiri, K.W.; Swedlow, J.R. (2010). \"Metadata matters: access to image data in the real world\". Journal of Cell Biology 189 (5): 777-82. doi:10.1083\/jcb.201004104. PMC PMC2878938. PMID 20513764. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938 .   \n\n\u2191 Maurer, M.; Molidor, R.; Sturn, A.; Hartler, J.; Hackl, H.; Stocker, G.; Prokesch, A.; Scheideler, M.; Trajanoski, Z. (2005). \"MARS: microarray analysis, retrieval, and storage system\". BMC Bioinformatics 6: 101. doi:10.1186\/1471-2105-6-101. PMC PMC1090551. PMID 15836795. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Table 1 has also been modified to a Y\/N format from its original check mark\/X format.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\">https:\/\/www.limswiki.org\/index.php\/Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on educationLIMSwiki journal articles on imaging informatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 12 January 2016, at 16:38.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,486 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","03ae650a31def9630dc93ec76a1927f6_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Personalized_Oncology_Suite_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Personalized Oncology Suite: Integrating next-generation sequencing data and whole-slide bioimages<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Cancer immunotherapy has recently entered a remarkable renaissance phase with the approval of several agents for treatment. <a href=\"https:\/\/www.limswiki.org\/index.php\/Cancer_informatics\" title=\"Cancer informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"b665bf61e7903152714531827699ad0d\">Cancer treatment platforms<\/a> have demonstrated profound tumor regressions including complete cure in patients with metastatic cancer. Moreover, technological advances in next-generation sequencing (NGS) as well as the development of devices for scanning whole-slide bioimages from tissue sections and <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioimage_informatics\" title=\"Bioimage informatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"f800be798ff5d084163ea63bae18ce79\">image analysis software<\/a> for quantitation of tumor-infiltrating lymphocytes (TILs) allow, for the first time, the development of personalized cancer immunotherapies that target patient specific mutations. However, there is currently no <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> solution that supports the integration of these heterogeneous datasets.\n<\/p><p><b>Results:<\/b> We have developed a bioinformatics platform \u2013 Personalized Oncology Suite (POS) \u2013 that integrates clinical data, NGS data and whole-slide bioimages from tissue sections. POS is a web-based platform that is scalable, flexible and expandable. The underlying database is based on a data warehouse schema, which is used to integrate <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> from different sources. POS stores clinical data, <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" target=\"_blank\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomic<\/a> data (SNPs and INDELs identified from NGS analysis), and scanned whole-slide images. It features a genome browser as well as access to several instances of the bioimage management application Bisque. POS provides different visualization techniques and offers sophisticated upload and download possibilities. The modular architecture of POS allows the community to easily modify and extend the application.\n<\/p><p><b>Conclusions:<\/b> The web-based integration of clinical, NGS, and imaging data represents a valuable resource for clinical researchers and future application in medical oncology. POS can be used not only in the context of cancer immunology but also in other studies in which NGS data and images of tissue sections are generated. The application is open-source and can be downloaded at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.icbi.at\/POS\" target=\"_blank\">http:\/\/www.icbi.at\/POS<\/a>.\n<\/p><p><b>Keywords:<\/b> Personalized oncology; Data integration; Next-generation sequencing; Whole-slide bioimaging; Application; Open-source\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Cancer immunotherapy has recently entered a remarkable renaissance phase with the approval of several agents for treatment.<sup id=\"rdp-ebb-cite_ref-CouzinCancer13_1-0\" class=\"reference\"><a href=\"#cite_note-CouzinCancer13-1\" rel=\"external_link\">[1]<\/a><\/sup> Cancer immunotherapies that involve the use of the adaptive immune system, such as anti-checkpoint antibodies and adoptive T-cell therapies, have demonstrated profound tumor regressions including complete cure in patients with metastatic cancer. Technological advances in next-generation sequencing (NGS) as well as the development of devices for scanning whole-slide images from tissue sections and image analysis software for quantitation of tumor-infiltrating lymphocytes (TILs) allow, for the first time, the development of personalized cancer immunotherapies that target patient specific mutations. The use of NGS technologies to characterize tumor samples enables one not only to comprehensively study the interactions between human cancers and the immune system, but also to identify targets for patient stratification. Moreover, the quantitation of TILs will improve therapeutic efficacy, even in the absence of immunotherapy. It will enable a precise characterization of the immune infiltrates in the tumor and will help to identify mechanisms of tumor regression and disentangle the complex tumor-immune cell interactions. For example, understanding the molecular basis of the interactions between cytotoxic chemotherapeutics or targeted anti-cancer agents and the immune system is essential for the development of optimal therapeutic schemes and in the long run will result in clinical benefit for the patients.\n<\/p><p>However, the real value of the disparate datasets can be truly exploited only when the data are integrated. In our experience it is of utmost importance to establish a local database hosting only the necessary data. Only pre-processed and normalized data will be stored in a dedicated relational database whereas primary data are archived at separate locations including public repositories.<sup id=\"rdp-ebb-cite_ref-HacklInfo10_2-0\" class=\"reference\"><a href=\"#cite_note-HacklInfo10-2\" rel=\"external_link\">[2]<\/a><\/sup> To this end, a database that integrates clinical, NGS, and bioimaging data would be extremely helpful for clinical cancer research and in near future also for routine applications in medical oncology. However, to the best of our knowledge there is currently no application that supports this integration. As of today there are different applications integrating either clinical data and NGS data or bioimages (Table 1) but no integrated solution has been created. We therefore developed the bioinformatics platform Personalized Oncology Suite (POS) to overcome this bottleneck and support the researchers working in this exciting field.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"9\"><b>Table 1.<\/b> This table compares POS with other applications in the context of data integration\n<\/td><\/tr>\n<tr>\n<th> Application\n<\/th>\n<th> Edit\n<\/th>\n<th> Clinical\n<\/th>\n<th> TNM\n<\/th>\n<th> Sequencing\n<\/th>\n<th> Vis.\n<\/th>\n<th> Bioimages\n<\/th>\n<th> Public\n<\/th>\n<th> Ref.\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BioIMAX\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Whole-slide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-LoyekBio11_3-0\" class=\"reference\"><a href=\"#cite_note-LoyekBio11-3\" rel=\"external_link\">[3]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bisque\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Whole-slide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-KvilekvalBis10_4-0\" class=\"reference\"><a href=\"#cite_note-KvilekvalBis10-4\" rel=\"external_link\">[4]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Galaxy LIMS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-ScholtalbersGal13_5-0\" class=\"reference\"><a href=\"#cite_note-ScholtalbersGal13-5\" rel=\"external_link\">[5]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NG6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Raw 454 and HiSeq\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-MarietteNG612_6-0\" class=\"reference\"><a href=\"#cite_note-MarietteNG612-6\" rel=\"external_link\">[6]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">NGS tools\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Raw Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DAS<sup id=\"rdp-ebb-cite_ref-JenkinsonInt08_7-0\" class=\"reference\"><a href=\"#cite_note-JenkinsonInt08-7\" rel=\"external_link\">[7]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-NixNext10_8-0\" class=\"reference\"><a href=\"#cite_note-NixNext10-8\" rel=\"external_link\">[8]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">OMERO\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-AllenOMERO12_9-0\" class=\"reference\"><a href=\"#cite_note-AllenOMERO12-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ONCO-i2b2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SNOMED\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-SegagniAn12_10-0\" class=\"reference\"><a href=\"#cite_note-SegagniAn12-10\" rel=\"external_link\">[10]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">openBIS\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Raw Illumina\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-Bauch_open11_11-0\" class=\"reference\"><a href=\"#cite_note-Bauch_open11-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Taverna\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">N\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><sup id=\"rdp-ebb-cite_ref-WolstencroftTheTav13_12-0\" class=\"reference\"><a href=\"#cite_note-WolstencroftTheTav13-12\" rel=\"external_link\">[12]<\/a><\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><b>POS<\/b>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">vcf Files\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Y\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Whole-slide\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">COSMIC<sup id=\"rdp-ebb-cite_ref-ForbesCOSMIC11_13-0\" class=\"reference\"><a href=\"#cite_note-ForbesCOSMIC11-13\" rel=\"external_link\">[13]<\/a><\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"9\">The columns <i>Clinical<\/i> and <i>TNM<\/i> show if these data types are available. <i>Sequencing<\/i> depicts which type of next-generation sequencing data can be uploaded, and the column <i>Vis.<\/i> shows if mutations can be visualized. The column <i>Bioimages<\/i> shows which type of images can be used and the final column <i>Public<\/i> states available public annotations.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<p>In order to be scalable, flexible, and expandable, POS makes use of state of the art software engineering techniques and architectures like the Java Enterprise Edition 6 (J2EE 6) technology stack. It is a web-based platform relying on the JBoss Application Server in version 7.1.1. The modular three-tier architecture (web frontend, application core, database backend) and the release under the open-source license GNU AGPL enables the community to easily modify and extend the application with further functionalities. Figure 1 outlines the software architecture of POS and depicts the main used libraries. PrimeFaces and PrimeFaces Extensions are used for the creation of JSF components, whereas Hibernate Validator provides input validation of user entries. As access scopes of Java Beans are crucial within JavaEE applications, Apache CODI is used to include additional scopes. Due to the fact that POS deals with different types of collections, the Guava libraries were chosen to support POS with a set of helpful functionalities regarding collections. Used Bisque instances are shown at the top of Figure 1. Furthermore, the standalone application POS Image Uploader allows batch uploading of numerous images at once. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Dander_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"746dfcf6864f8d5012e74f3b2cb159d9\"><img alt=\"Fig1 Dander BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/bf\/Fig1_Dander_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1: Software architecture of POS.<\/b> The JBoss Application Server of POS is shown as the central rectangle containing different JSF libraries. On the right hand side the attached Authorization and Authentication System (AAS) is depicted. POS uses PostgreSQL as database management system and applies EclipseLink for the object-relational mapping. The Bioconductor package Gviz handles rendering of the genome browser<br \/>tracks, and the R package Rserver provides an R server available through a network connection. On top, distributed Bisque instances are shown. The POS Image Uploader, a JavaFX based standalone application, is outlined on the top left in the figure. This application enables users to upload several images at once.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><br \/>\nThe underlying database is based on a data warehouse schema. This schema was chosen as it is widely used for integrating information from different sources. All defined entities are outlined in Additional file 1. POS uses the Java library EclipseLink for object-relational mapping. In the default configuration POS runs with <a href=\"https:\/\/www.limswiki.org\/index.php\/PostgreSQL\" title=\"PostgreSQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"a5dd945cdcb63e2d8f7a5edb3a896d82\">PostgreSQL<\/a>, but can be easily exchanged with another relational database. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 1. <i>Database schema of the Personalized Oncology Suite.<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The database schema of POS is based on a data warehouse schema. Therefore, the central entity represents patients. Each <i>patient<\/i> belongs to an <i>institute<\/i>, which stores the <i>externalid<\/i> referencing an institute within the attached Authorization and Authentication System. It can be seen that each institute can be connected to an <i>imagerepository<\/i> containing information about the connection to a Bisque instance. The entity <i>institutesharing<\/i> manages information about shared data. <i>Clinicaldata<\/i> as well as <i>tnm<\/i> and <i>immunoscore<\/i> for staging of cancer are related to the entity <i>patient<\/i>. For the integration of somatic mutations within POS the entities <i>snp<\/i> and <i>indel<\/i> are used. The entity <i>analysis<\/i> contains metadata about the next-generation sequencing itself. <i>Image<\/i> manages the attributes <i>externalimageid<\/i> and <i>externalresourceid<\/i> which are IDs used for accessing the image within Bisque. The attached <i>imagetype<\/i> contains information about the staining of the image. All entities with a name like <i><name>_audittrail<\/i> hold information about documented changes made to the attached entity <i><name><\/i>. It is shown that the <i>timestamp<\/i>, the name of the <i>author<\/i> and the performed changes are recorded within these entities. Several entities comprise a <i>deleted<\/i> flag. If such an entity gets deleted it will not be removed within the database, but will not be shown in the frontend. This has the advantage that deleted entities can be restored by a database administrator.\n<p>Format: PNG; Size: 1.5MB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-15-306-s1.png\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<p>The Personalized Oncology Suite is an application combining biological and clinical data into one integrated solution. In this context, clinical data comprises information about cancer patients, TNM staging, and density values of TILs used for immune score estimation. Biological data describes mutations found via next-generation sequencing and whole-slide bioimages, which can be uploaded to the application. POS features different types of visualization techniques for all integrated data types. Furthermore, publicly available data from the COSMIC database<sup id=\"rdp-ebb-cite_ref-ForbesCOSMIC11_13-1\" class=\"reference\"><a href=\"#cite_note-ForbesCOSMIC11-13\" rel=\"external_link\">[13]<\/a><\/sup> is integrated into POS. As data types are stored in different file formats, POS includes several data import and export possibilities for the most important formats. In addition, different filters on the data can be applied either individually or in combination (e.g. age at diagnosis and TNM stage). In the current implementation queries can be done only for single modalities (e.g. genetic features, images, or clinical parameters). Furthermore, patients can be selected based on their UICC stage or Immunoscore by using a range slider. POS features different user interface languages using internationalization and is fault tolerant as all inputs are validated. Furthermore, POS implements exception handling as well as an intelligent logging functionality. \n<\/p><p>POS has been released under the open-source license GNU AGPL to allow integration of new components from the scientific community. The source code can be accessed via the project website <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.icbi.at\/POS\" target=\"_blank\">http:\/\/www.icbi.at\/POS<\/a>. Figure 2 outlines the different layers between the JSF based frontend and the relational database. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Dander_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2771179198eca680110ce19903eca9ec\"><img alt=\"Fig2 Dander BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/6\/65\/Fig2_Dander_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2: Software layers of POS.<\/b> <b>a)<\/b> The JSF based presentation layer on top (blue) access the relational database (green) via the Java backend. This backend can be accessed via the classes Controller or LazyDataModel. These classes make use of the DataAccessObject which provides a single access point to the relational database by using object-relational mapping. <b>b)<\/b> The Controller is responsible for secure CRUD (create, read, update and delete) operations. <b>c)<\/b> The LazyDataModel is used for providing immutable, filtered, sorted, and paginated collections to the presentation layer.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Clinical_data_and_tumor_staging\">Clinical data and tumor staging<\/span><\/h3>\n<p>POS integrates clinical data of cancer patients including various attributes such as gender, date of diagnosis, disease duration, adjuvant therapy, and relapse. We have included compliant measures in the design of the software ensuring that patient-identifying data such as name, academic title, address, telephone number, e-mail address, date and place of birth, as well as date and place of death<sup id=\"rdp-ebb-cite_ref-SchutzeUse12_14-0\" class=\"reference\"><a href=\"#cite_note-SchutzeUse12-14\" rel=\"external_link\">[14]<\/a><\/sup> are not collected. Patients are uniquely defined by alphanumerical identifiers. Thus, data from multiple clinical visits or any other information can be unambiguously assigned to an individual patient. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Tumor_staging\">Tumor staging<\/span><\/h3>\n<p>In addition to clinical data, POS integrates the TNM cancer staging system. The T, N, and M categories are stored separately within the database and the resulting AJCC\/UICC stage (0-IV) is determined from this information. POS facilitates the creation of descriptive plots for comparing patients in different TNM stages. These plots can also be used for comparison of data among different participating institutes, provided the user has permission to access the information. In addition to input data manually, clinical and staging data can be uploaded to POS in CSV format. Furthermore, these data types can be exported as XLS, CSV and PDF files.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Next-generation_sequencing_data\">Next-generation sequencing data<\/span><\/h3>\n<p>With the use of next-generation sequencing, biologists are able to determine the order of nucleotide bases composing the DNA. The identification of somatic mutations can be performed by sequencing tumor\/normal pairs and subsequently comparing cancerous to healthy tissue. Several different applications exist which are able to analyze tumor\/normal pairs regarding their somatic SNPs and INDELs.<sup id=\"rdp-ebb-cite_ref-PabingerASurv14_15-0\" class=\"reference\"><a href=\"#cite_note-PabingerASurv14-15\" rel=\"external_link\">[15]<\/a><\/sup> Since there are a plethora of methods for analyses of NGS data, the design of the software focused on the integration of disparate data types rather than the analyses of raw data. Furthermore, as the methods and available tools improve at fast pace, integration of processed data makes the system more flexible and versatile. POS is able to integrate somatic SNPs and INDELs identified by such experiments. For uploading this type of data, users first need to define an analysis, which can be used to attach called mutations. POS supports data upload via VCF (Variant Call Format) files or by manual input of mutations. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Genome_browser\">Genome browser<\/span><\/h3>\n<p>As visualization of identified mutations tremendously supports the interpretation of these results, genome browsers have been developed to display mutations in the context of a reference genome.<sup id=\"rdp-ebb-cite_ref-PabingerASurv14_15-1\" class=\"reference\"><a href=\"#cite_note-PabingerASurv14-15\" rel=\"external_link\">[15]<\/a><\/sup> POS includes a genome browser that features a combined view of different tracks, each containing a dedicated plot. Figure 3 shows a screenshot of the genome browser view within POS. On top of the genome browser panel, the region of interest can be specified by defining chromosome number, start and end position or by choosing a gene of interest. The user can select several patients at once and mutations for each patient will be displayed in separate tracks. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Dander_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a5229875de2af30e4fec1e4a3ddcbb24\"><img alt=\"Fig3 Dander BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/6\/64\/Fig3_Dander_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 3: View of the genome browser.<\/b> A genome browser visualizing mutations in the context of the reference genome including publicly available annotations is shown. The first track depicts the ideogram of the chosen chromosome, followed by an axis showing its genomic coordinates. Next, publicly available annotations derived from BioMart<sup id=\"rdp-ebb-cite_ref-KasprzykBio11_16-0\" class=\"reference\"><a href=\"#cite_note-KasprzykBio11-16\" rel=\"external_link\">[16]<\/a><\/sup> are depicted. The bottom track holds information about uploaded mutations. The patient name and the shown mutations were randomly generated.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Bioimages\">Bioimages<\/span><\/h3>\n<p>Whole-slide bioimages can be used to estimate the density of TILs. However, due to the size of the files the required disk space can be very large. Therefore, the raw image files are not stored in POS. In order to integrate whole-slide images within POS, the application connects to several instances of the bioimage management application Bisque<sup id=\"rdp-ebb-cite_ref-KvilekvalBis10_4-1\" class=\"reference\"><a href=\"#cite_note-KvilekvalBis10-4\" rel=\"external_link\">[4]<\/a><\/sup> as outlined in Figure 1. POS supports the following formats: 1) SVS format from Aperio, 2) VSI format from Olympus, 3) Hamamatsu formats NDPI and VMS, 4) TIFF format from Trestle, 5) Leica\u2019s SCN format, 6) formats ZVI and CZI from Zeiss, and 7) OME.TIFF format from the Open Microscopy Environment OME.<sup id=\"rdp-ebb-cite_ref-GoodeOpen13_17-0\" class=\"reference\"><a href=\"#cite_note-GoodeOpen13-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LinkertMeta10_18-0\" class=\"reference\"><a href=\"#cite_note-LinkertMeta10-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Image_upload\">Image upload<\/span><\/h3>\n<p>POS supports different ways for uploading images. The first option is a direct upload module, where a patient needs to be selected before the upload starts. The second possibility allows access to already uploaded images within a connected Bisque instance and their assignment to selected patients in POS. The third option is an external upload application supporting the batch upload of multiple images at once. This application is also able to assign the correct relations between uploaded images and patients within POS.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Bisque_connection\">Bisque connection<\/span><\/h3>\n<p>The connection to Bisque instances is administered by POS, which handles correct URL, port, username and password settings. In addition, tailored Java servlets for managing the encrypted communication between POS and Bisque have been developed. Downscaling and tiling of the images is performed by the connected Bisque systems. POS is able to display the created tiles in a Google maps like manner, using an adapted version of the interactive JavaScript widget PanoJS3. Hence, only tiles of the currently displayed region are fetched from Bisque. A screenshot of the POS image viewer is displayed in Figure 4.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Dander_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"e00a5a7af2c761c341998071502d9397\"><img alt=\"Fig4 Dander BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/9a\/Fig4_Dander_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 4: Image viewer.<\/b> This Figure depicts a screenshot of an example of a whole-slide bioimage. The image is stored at an external Bisque instance and only tiles that correspond to the current image section are transferred to POS. General information about the patient and the image are depicted on the right hand side. Various attributes like width, height, resolution, and information about the scanner are directly fetched from Bisque. Shown clinical data were randomly generated.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Flexible_batch_upload_module\">Flexible batch upload module<\/span><\/h3>\n<p>As the JSF based image upload form of POS does not support batch upload of several images at once, the standalone application POS Image Uploader has been developed. This application is based on JavaFX and can be started on all systems where Java is installed. Figure 1 depicts this standalone application at the top. The POS Image Uploader is able to acquire the URL and corresponding login information of Bisque instances directly from POS. During the batch upload, each image is uploaded to a selected Bisque instance. Bisque returns identifiers for each uploaded image, which are stored within the POS database and linked to related patients. The communication between the standalone tool and the web-based application is managed by tailored web services.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authorization_and_authentication_system\">Authorization and authentication system<\/span><\/h3>\n<p>As POS holds confidential and patient related data, the application needs to be secured. POS is secured by an authorization and authentication system (AAS) where participating users and institutes can be configured within this easy to use user management system.<sup id=\"rdp-ebb-cite_ref-MaurerMARS05_19-0\" class=\"reference\"><a href=\"#cite_note-MaurerMARS05-19\" rel=\"external_link\">[19]<\/a><\/sup> The backend of the used AAS can either be a simple XML file or a web-based application running on a dedicated JBoss Application Server (see Figure 1). For tailored user access, POS applies different user roles, provided by the AAS. Access roles are defined for clinical, NGS, and imaging data as well as for administration. In order to provide custom access profiles, users can be assigned to several roles at once. Furthermore, all web services, used by the POS Image Uploader, authenticate each single connection via the AAS.\n<\/p><p>Finally, POS users can also share their data with other participating users, with the possibility to restrict access to certain data types. It should be noted that as access to patient related data needs to be secured, the implemented sharing concept and the AAS do not support public access to any data type. Rather, users can specify the type of data that can be shared with other users including images, genetic data, or clinical data. If an external user wants access to this integrated data a request must be performed and all participating institutes will be informed about this request. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Installation\">Installation<\/span><\/h3>\n<p>As POS is a web-based application, it can be accessed from any operating system. The only requirements are an up to date web browser and a network connection to the POS server. The server itself is able to run on any operating system with installed Java and R, and has been extensively tested on Linux and Windows machines. The database backend is interchangeable - POS has been tested with the database systems PostgreSQL and <a href=\"https:\/\/www.limswiki.org\/index.php\/MySQL\" title=\"MySQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"35005451bfcd508bce47c58e72260128\">MySQL<\/a>. A detailed installation guide for POS is available at the project home page, accessible at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.icbi.at\/POS\" target=\"_blank\">http:\/\/www.icbi.at\/POS<\/a>. As POS makes use of several Bisque installations, an installation guide for Bisque is provided at the same location.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>POS is a web-based application combining clinical and biomolecular data including NGS data and bioimages. Due to its modular and flexible architecture it can be easily extended and adapted to different requirements. POS provides an intuitive user interface supporting data upload, manipulation and visualization of integrated data types. In summary, POS represents an effective solution for current challenges in clinical cancer research. The suite can be used not only in the context of personalized cancer immunotherapy but also in other studies where NGS data and images of tissue sections are generated.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p><b>Project name:<\/b> Personalized Oncology Suite (POS)\n<\/p><p><b>Project home page:<\/b> <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.icbi.at\/POS\" target=\"_blank\">http:\/\/www.icbi.at\/POS<\/a>\n<\/p><p><b>Operating system:<\/b> Platform independent\n<\/p><p><b>Programming language:<\/b> Java, R\/Bioconductor\n<\/p><p><b>License:<\/b> GNU AGPL\n<\/p><p><b>Any restrictions to use by non-academics:<\/b> No\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p>AAS: Authorization and Authentication System \n<\/p><p>AGPL: Affero General Public License\n<\/p><p>COSMIC: Catalogue of Somatic Mutations in Cancer\n<\/p><p>CRUD: Create, Read, Update and Delete\n<\/p><p>INDEL: Small insertion or deletion\n<\/p><p>JSF: JavaServer Faces\n<\/p><p>POS: Personalized Oncology Suite\n<\/p><p>SNP: Single-nucleotide polymorphism\n<\/p><p>TIL: Tumor-infiltrating lymphocyte\n<\/p><p>TNM: Classification of Malignant Tumors\n<\/p><p>URL: Uniform Resource Locator\n<\/p><p>VCF: Variant Call Format\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h2>\n<p>AD is the principle developer of POS. MB, MS and BH developed parts of the application. SP as well as ZT reviewed and tested POS and provided suggestions for additional features. All authors participated in writing of the manuscript, read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This work was supported by ONCOTYROL (FFG, Austrian Federal Ministries BMVIT\/BMWFJ) and a grant from the Standortagentur Tirol.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-CouzinCancer13-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CouzinCancer13_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Couzin-Frankel, J. (2013). \"Cancer Immunotherapy\". <i>Science<\/i> <b>342<\/b> (6165): 1432-1433. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.342.6165.1432\" target=\"_blank\">10.1126\/science.342.6165.1432<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24357284\" target=\"_blank\">24357284<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cancer+Immunotherapy&rft.jtitle=Science&rft.aulast=Couzin-Frankel%2C+J.&rft.au=Couzin-Frankel%2C+J.&rft.date=2013&rft.volume=342&rft.issue=6165&rft.pages=1432-1433&rft_id=info:doi\/10.1126%2Fscience.342.6165.1432&rft_id=info:pmid\/24357284&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HacklInfo10-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HacklInfo10_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hackl, H.; Stocker, G.; Charoentong, P.; Mlecnik, B.; Bindea, G.; Galon, J.; Trajanoski, Z. (2010). \"Information technology solutions for integration of biomolecular and clinical data in the identification of new cancer biomarkers and targets for therapy\". <i>Pharmacology & Therapeutics<\/i> <b>128<\/b> (3): 488\u2013498. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.pharmthera.2010.08.012\" target=\"_blank\">10.1016\/j.pharmthera.2010.08.012<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20832425\" target=\"_blank\">20832425<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+technology+solutions+for+integration+of+biomolecular+and+clinical+data+in+the+identification+of+new+cancer+biomarkers+and+targets+for+therapy&rft.jtitle=Pharmacology+%26+Therapeutics&rft.aulast=Hackl%2C+H.%3B+Stocker%2C+G.%3B+Charoentong%2C+P.%3B+Mlecnik%2C+B.%3B+Bindea%2C+G.%3B+Galon%2C+J.%3B+Trajanoski%2C+Z.&rft.au=Hackl%2C+H.%3B+Stocker%2C+G.%3B+Charoentong%2C+P.%3B+Mlecnik%2C+B.%3B+Bindea%2C+G.%3B+Galon%2C+J.%3B+Trajanoski%2C+Z.&rft.date=2010&rft.volume=128&rft.issue=3&rft.pages=488%E2%80%93498&rft_id=info:doi\/10.1016%2Fj.pharmthera.2010.08.012&rft_id=info:pmid\/20832425&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LoyekBio11-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LoyekBio11_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Loyek, C.; Rajpoot, N.M.; Khan, M.; Nattkemper, T.W. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3161928\" target=\"_blank\">\"BioIMAX: A Web 2.0 approach for easy exploratory and collaborative access to multivariate bioimage data\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>: 297. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-297\" target=\"_blank\">10.1186\/1471-2105-12-297<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3161928\/\" target=\"_blank\">PMC3161928<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21777450\" target=\"_blank\">21777450<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3161928\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3161928<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioIMAX%3A+A+Web+2.0+approach+for+easy+exploratory+and+collaborative+access+to+multivariate+bioimage+data&rft.jtitle=BMC+Bioinformatics&rft.aulast=Loyek%2C+C.%3B+Rajpoot%2C+N.M.%3B+Khan%2C+M.%3B+Nattkemper%2C+T.W.&rft.au=Loyek%2C+C.%3B+Rajpoot%2C+N.M.%3B+Khan%2C+M.%3B+Nattkemper%2C+T.W.&rft.date=2011&rft.volume=12&rft.pages=297&rft_id=info:doi\/10.1186%2F1471-2105-12-297&rft_id=info:pmc\/PMC3161928&rft_id=info:pmid\/21777450&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3161928&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KvilekvalBis10-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KvilekvalBis10_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-KvilekvalBis10_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kvilekval, K.; Fedorov, D.; Obara, B.; Singh, A.; Manjunath, B.S. (2010). \"Bisque: A platform for bioimage analysis and management\". <i>Bioinformatics<\/i> <b>26<\/b> (4): 544-52. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp699\" target=\"_blank\">10.1093\/bioinformatics\/btp699<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20031971\" target=\"_blank\">20031971<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Bisque%3A+A+platform+for+bioimage+analysis+and+management&rft.jtitle=Bioinformatics&rft.aulast=Kvilekval%2C+K.%3B+Fedorov%2C+D.%3B+Obara%2C+B.%3B+Singh%2C+A.%3B+Manjunath%2C+B.S.&rft.au=Kvilekval%2C+K.%3B+Fedorov%2C+D.%3B+Obara%2C+B.%3B+Singh%2C+A.%3B+Manjunath%2C+B.S.&rft.date=2010&rft.volume=26&rft.issue=4&rft.pages=544-52&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp699&rft_id=info:pmid\/20031971&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScholtalbersGal13-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScholtalbersGal13_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scholtalbers, J.; R\u00f6ssler, J.; Sorn, P.; de Graaf, J.; Boisgu\u00e9rin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". <i>Bioinformatics<\/i> <b>29<\/b> (9): 1233-1234. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt115\" target=\"_blank\">10.1093\/bioinformatics\/btt115<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23479349\" target=\"_blank\">23479349<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy+LIMS+for+next-generation+sequencing&rft.jtitle=Bioinformatics&rft.aulast=Scholtalbers%2C+J.%3B+R%C3%B6ssler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisgu%C3%A9rin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.au=Scholtalbers%2C+J.%3B+R%C3%B6ssler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisgu%C3%A9rin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.date=2013&rft.volume=29&rft.issue=9&rft.pages=1233-1234&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt115&rft_id=info:pmid\/23479349&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarietteNG612-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarietteNG612_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mariette, J.; Escudi\u00e9, F.; Allias, N.; Salin, G.; Noirot, C.; Thomas, S.; Klopp, C. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3444930\" target=\"_blank\">\"NG6: Integrated next generation sequencing storage and processing environment\"<\/a>. <i>BMC Genomics<\/i> <b>13<\/b>: 462. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2164-13-462\" target=\"_blank\">10.1186\/1471-2164-13-462<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3444930\/\" target=\"_blank\">PMC3444930<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22958229\" target=\"_blank\">22958229<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3444930\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3444930<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=NG6%3A+Integrated+next+generation+sequencing+storage+and+processing+environment&rft.jtitle=BMC+Genomics&rft.aulast=Mariette%2C+J.%3B+Escudi%C3%A9%2C+F.%3B+Allias%2C+N.%3B+Salin%2C+G.%3B+Noirot%2C+C.%3B+Thomas%2C+S.%3B+Klopp%2C+C.&rft.au=Mariette%2C+J.%3B+Escudi%C3%A9%2C+F.%3B+Allias%2C+N.%3B+Salin%2C+G.%3B+Noirot%2C+C.%3B+Thomas%2C+S.%3B+Klopp%2C+C.&rft.date=2012&rft.volume=13&rft.pages=462&rft_id=info:doi\/10.1186%2F1471-2164-13-462&rft_id=info:pmc\/PMC3444930&rft_id=info:pmid\/22958229&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3444930&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JenkinsonInt08-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JenkinsonInt08_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jenkinson, A.M.; Albrecht, M.; Birney, E.; Blankenburg, H.; Down, T.; Finn, R.D.; Hermjakob, H.; Hubbard, T.J.P.; Jimenez, R.C.; Jones, P.; K\u00e4h\u00e4ri, A.; Kulesha, E.; Mac\u00edas, J.R.; Reeves, G.A.; Prli\u0107, A. (2008). \"Integrating biological data-the Distributed Annotation System\". <i>BMC Bioinformatics<\/i> <b>9<\/b> (Suppl 8): S3. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-9-S8-S3\" target=\"_blank\">10.1186\/1471-2105-9-S8-S3<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrating+biological+data-the+Distributed+Annotation+System&rft.jtitle=BMC+Bioinformatics&rft.aulast=Jenkinson%2C+A.M.%3B+Albrecht%2C+M.%3B+Birney%2C+E.%3B+Blankenburg%2C+H.%3B+Down%2C+T.%3B+Finn%2C+R.D.%3B+Hermjakob%2C+H.%3B+Hubbard%2C+T.J.P.%3B+Jimenez%2C+R.C.%3B+Jones%2C+P.%3B+K%C3%A4h%C3%A4ri%2C+A.%3B+Kulesha%2C+E.%3B+Mac%C3%ADas%2C+J.R.%3B+Reeves%2C+G.A.%3B+Prli%C4%87%2C+A.&rft.au=Jenkinson%2C+A.M.%3B+Albrecht%2C+M.%3B+Birney%2C+E.%3B+Blankenburg%2C+H.%3B+Down%2C+T.%3B+Finn%2C+R.D.%3B+Hermjakob%2C+H.%3B+Hubbard%2C+T.J.P.%3B+Jimenez%2C+R.C.%3B+Jones%2C+P.%3B+K%C3%A4h%C3%A4ri%2C+A.%3B+Kulesha%2C+E.%3B+Mac%C3%ADas%2C+J.R.%3B+Reeves%2C+G.A.%3B+Prli%C4%87%2C+A.&rft.date=2008&rft.volume=9&rft.issue=Suppl+8&rft.pages=S3&rft_id=info:doi\/10.1186%2F1471-2105-9-S8-S3&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NixNext10-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NixNext10_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nix, D.A.; Di Sera, T.L.; Dalley, B.K.; Milash, B.A.; Cundick, R.M.; Quinn, K.S.; Courdy, S.J. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281\" target=\"_blank\">\"Next generation tools for genomic data generation, distribution, and visualization\"<\/a>. <i>BMC Bioinformatics<\/i> <b>11<\/b>: 455. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-11-455\" target=\"_blank\">10.1186\/1471-2105-11-455<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2944281\/\" target=\"_blank\">PMC2944281<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20828407\" target=\"_blank\">20828407<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Next+generation+tools+for+genomic+data+generation%2C+distribution%2C+and+visualization&rft.jtitle=BMC+Bioinformatics&rft.aulast=Nix%2C+D.A.%3B+Di+Sera%2C+T.L.%3B+Dalley%2C+B.K.%3B+Milash%2C+B.A.%3B+Cundick%2C+R.M.%3B+Quinn%2C+K.S.%3B+Courdy%2C+S.J.&rft.au=Nix%2C+D.A.%3B+Di+Sera%2C+T.L.%3B+Dalley%2C+B.K.%3B+Milash%2C+B.A.%3B+Cundick%2C+R.M.%3B+Quinn%2C+K.S.%3B+Courdy%2C+S.J.&rft.date=2010&rft.volume=11&rft.pages=455&rft_id=info:doi\/10.1186%2F1471-2105-11-455&rft_id=info:pmc\/PMC2944281&rft_id=info:pmid\/20828407&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2944281&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AllenOMERO12-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AllenOMERO12_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Allan, C.; Burel, J-M.; Moore, J.; Blackburn, C.; Linkert, M.; Loynton, S.; MacDonald, D.; Moore, W.J.; Neves, C.; Patterson, A.; Porter, M.; Tarkowska, A.; Loranger, B.; Avondo, J.; Lagerstedt, I.; Lianas, L.; Leo, S.; Hands, K.; Hay, R.T.; Patwardhan, A.; Best, C.; Kleywegt, G.J.; Zanetti, G.; Swedlow, J.R. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3437820\" target=\"_blank\">\"OMERO: flexible, model-driven data management for experimental biology\"<\/a>. <i>Nature Methods<\/i> <b>9<\/b> (3): 245-53. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth.1896\" target=\"_blank\">10.1038\/nmeth.1896<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3437820\/\" target=\"_blank\">PMC3437820<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22373911\" target=\"_blank\">22373911<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3437820\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3437820<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=OMERO%3A+flexible%2C+model-driven+data+management+for+experimental+biology&rft.jtitle=Nature+Methods&rft.aulast=Allan%2C+C.%3B+Burel%2C+J-M.%3B+Moore%2C+J.%3B+Blackburn%2C+C.%3B+Linkert%2C+M.%3B+Loynton%2C+S.%3B+MacDonald%2C+D.%3B+Moore%2C+W.J.%3B+Neves%2C+C.%3B+Patterson%2C+A.%3B+Porter%2C+M.%3B+Tarkowska%2C+A.%3B+Loranger%2C+B.%3B+Avondo%2C+J.%3B+Lagerstedt%2C+I.%3B+Lianas%2C+L.%3B+Leo%2C+S.%3B+Hands%2C+K.%3B+Hay%2C+R.T.%3B+Patwardhan%2C+A.%3B+Best%2C+C.%3B+Kleywegt%2C+G.J.%3B+Zanetti%2C+G.%3B+Swedlow%2C+J.R.&rft.au=Allan%2C+C.%3B+Burel%2C+J-M.%3B+Moore%2C+J.%3B+Blackburn%2C+C.%3B+Linkert%2C+M.%3B+Loynton%2C+S.%3B+MacDonald%2C+D.%3B+Moore%2C+W.J.%3B+Neves%2C+C.%3B+Patterson%2C+A.%3B+Porter%2C+M.%3B+Tarkowska%2C+A.%3B+Loranger%2C+B.%3B+Avondo%2C+J.%3B+Lagerstedt%2C+I.%3B+Lianas%2C+L.%3B+Leo%2C+S.%3B+Hands%2C+K.%3B+Hay%2C+R.T.%3B+Patwardhan%2C+A.%3B+Best%2C+C.%3B+Kleywegt%2C+G.J.%3B+Zanetti%2C+G.%3B+Swedlow%2C+J.R.&rft.date=2012&rft.volume=9&rft.issue=3&rft.pages=245-53&rft_id=info:doi\/10.1038%2Fnmeth.1896&rft_id=info:pmc\/PMC3437820&rft_id=info:pmid\/22373911&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3437820&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SegagniAn12-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SegagniAn12_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Segagni, D.; Tibollo, V.; Dagliati, A.; Zambelli, A.; Priori, S.G.; Bellazzi, R. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3280155\" target=\"_blank\">\"An ICT infrastructure to integrate clinical and molecular data in oncology research\"<\/a>. <i>BMC Bioinformatics<\/i> <b>13<\/b>: 4. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-13-4\" target=\"_blank\">10.1186\/1471-2105-13-4<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3280155\/\" target=\"_blank\">PMC3280155<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22226192\" target=\"_blank\">22226192<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3280155\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3280155<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+ICT+infrastructure+to+integrate+clinical+and+molecular+data+in+oncology+research&rft.jtitle=BMC+Bioinformatics&rft.aulast=Segagni%2C+D.%3B+Tibollo%2C+V.%3B+Dagliati%2C+A.%3B+Zambelli%2C+A.%3B+Priori%2C+S.G.%3B+Bellazzi%2C+R.&rft.au=Segagni%2C+D.%3B+Tibollo%2C+V.%3B+Dagliati%2C+A.%3B+Zambelli%2C+A.%3B+Priori%2C+S.G.%3B+Bellazzi%2C+R.&rft.date=2012&rft.volume=13&rft.pages=4&rft_id=info:doi\/10.1186%2F1471-2105-13-4&rft_id=info:pmc\/PMC3280155&rft_id=info:pmid\/22226192&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3280155&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Bauch_open11-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Bauch_open11_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bauch, A.; Adamczyk, I.; Buczek, P.; Elmer, F-J.; Enimanev, K.; Glyzewski, P.; Kohler, M.; Pylak, T.; Quandt, A.; Ramakrishnan, C.; Beisel, C.; Malmstr\u00f6m, L.; Aebersold, R.; Rinn, B. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639\" target=\"_blank\">\"openBIS: A flexible framework for managing and analyzing complex data in biology research\"<\/a>. <i>BMC Bioinformatics<\/i> <b>12<\/b>: 468. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-12-468\" target=\"_blank\">10.1186\/1471-2105-12-468<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3275639\/\" target=\"_blank\">PMC3275639<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22151573\" target=\"_blank\">22151573<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3275639<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=openBIS%3A+A+flexible+framework+for+managing+and+analyzing+complex+data+in+biology+research&rft.jtitle=BMC+Bioinformatics&rft.aulast=Bauch%2C+A.%3B+Adamczyk%2C+I.%3B+Buczek%2C+P.%3B+Elmer%2C+F-J.%3B+Enimanev%2C+K.%3B+Glyzewski%2C+P.%3B+Kohler%2C+M.%3B+Pylak%2C+T.%3B+Quandt%2C+A.%3B+Ramakrishnan%2C+C.%3B+Beisel%2C+C.%3B+Malmstr%C3%B6m%2C+L.%3B+Aebersold%2C+R.%3B+Rinn%2C+B.&rft.au=Bauch%2C+A.%3B+Adamczyk%2C+I.%3B+Buczek%2C+P.%3B+Elmer%2C+F-J.%3B+Enimanev%2C+K.%3B+Glyzewski%2C+P.%3B+Kohler%2C+M.%3B+Pylak%2C+T.%3B+Quandt%2C+A.%3B+Ramakrishnan%2C+C.%3B+Beisel%2C+C.%3B+Malmstr%C3%B6m%2C+L.%3B+Aebersold%2C+R.%3B+Rinn%2C+B.&rft.date=2011&rft.volume=12&rft.pages=468&rft_id=info:doi\/10.1186%2F1471-2105-12-468&rft_id=info:pmc\/PMC3275639&rft_id=info:pmid\/22151573&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3275639&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WolstencroftTheTav13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WolstencroftTheTav13_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wolstencroft, K.; Haines, R.; Fellows, D.; Williams, A.; Withers, D.; Owen, S.; Soiland-Reyes, S.; Dunlop, I.; Nenadic, A.; Fisher, P.; Bhagat, J.; Belhajjame, K.; Bacall, F.; Hardisty, A.; Nieva de la Hidalga, A.; Balcazar Vargas, M.P.; Sufi, S.; Goble, C. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3692062\" target=\"_blank\">\"The Taverna workflow suite: Designing and executing workflows of Web Services on the desktop, web or in the cloud\"<\/a>. <i>Nucleic Acids Research<\/i> <b>41<\/b> (W1): W557-W561. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkt328\" target=\"_blank\">10.1093\/nar\/gkt328<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3692062\/\" target=\"_blank\">PMC3692062<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23640334\" target=\"_blank\">23640334<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3692062\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3692062<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Taverna+workflow+suite%3A+Designing+and+executing+workflows+of+Web+Services+on+the+desktop%2C+web+or+in+the+cloud&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Wolstencroft%2C+K.%3B+Haines%2C+R.%3B+Fellows%2C+D.%3B+Williams%2C+A.%3B+Withers%2C+D.%3B+Owen%2C+S.%3B+Soiland-Reyes%2C+S.%3B+Dunlop%2C+I.%3B+Nenadic%2C+A.%3B+Fisher%2C+P.%3B+Bhagat%2C+J.%3B+Belhajjame%2C+K.%3B+Bacall%2C+F.%3B+Hardisty%2C+A.%3B+Nieva+de+la+Hidalga%2C+A.%3B+Balcazar+Vargas%2C+M.P.%3B+Sufi%2C+S.%3B+Goble%2C+C.&rft.au=Wolstencroft%2C+K.%3B+Haines%2C+R.%3B+Fellows%2C+D.%3B+Williams%2C+A.%3B+Withers%2C+D.%3B+Owen%2C+S.%3B+Soiland-Reyes%2C+S.%3B+Dunlop%2C+I.%3B+Nenadic%2C+A.%3B+Fisher%2C+P.%3B+Bhagat%2C+J.%3B+Belhajjame%2C+K.%3B+Bacall%2C+F.%3B+Hardisty%2C+A.%3B+Nieva+de+la+Hidalga%2C+A.%3B+Balcazar+Vargas%2C+M.P.%3B+Sufi%2C+S.%3B+Goble%2C+C.&rft.date=2013&rft.volume=41&rft.issue=W1&rft.pages=W557-W561&rft_id=info:doi\/10.1093%2Fnar%2Fgkt328&rft_id=info:pmc\/PMC3692062&rft_id=info:pmid\/23640334&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3692062&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ForbesCOSMIC11-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ForbesCOSMIC11_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-ForbesCOSMIC11_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Forbes, S.A.; Bindal, N.; Bamford, S.; Cole, C.; Kok, C.Y.; Beare, D.; Jia, M.; Shepherd, R.; Leung, K.; Menzies, A.; Teague, J.W.; Campbell, P.J.; Stratton, M.R.; Futreal, P.A. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013785\" target=\"_blank\">\"COSMIC: mining complete cancer genomes in the Catalogue of Somatic Mutations in Cancer\"<\/a>. <i>Nucleic Acids Research<\/i> <b>39<\/b> (Suppl 1): D945-D950. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkq929\" target=\"_blank\">10.1093\/nar\/gkq929<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3013785\/\" target=\"_blank\">PMC3013785<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20952405\" target=\"_blank\">20952405<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013785\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3013785<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=COSMIC%3A+mining+complete+cancer+genomes+in+the+Catalogue+of+Somatic+Mutations+in+Cancer&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Forbes%2C+S.A.%3B+Bindal%2C+N.%3B+Bamford%2C+S.%3B+Cole%2C+C.%3B+Kok%2C+C.Y.%3B+Beare%2C+D.%3B+Jia%2C+M.%3B+Shepherd%2C+R.%3B+Leung%2C+K.%3B+Menzies%2C+A.%3B+Teague%2C+J.W.%3B+Campbell%2C+P.J.%3B+Stratton%2C+M.R.%3B+Futreal%2C+P.A.&rft.au=Forbes%2C+S.A.%3B+Bindal%2C+N.%3B+Bamford%2C+S.%3B+Cole%2C+C.%3B+Kok%2C+C.Y.%3B+Beare%2C+D.%3B+Jia%2C+M.%3B+Shepherd%2C+R.%3B+Leung%2C+K.%3B+Menzies%2C+A.%3B+Teague%2C+J.W.%3B+Campbell%2C+P.J.%3B+Stratton%2C+M.R.%3B+Futreal%2C+P.A.&rft.date=2011&rft.volume=39&rft.issue=Suppl+1&rft.pages=D945-D950&rft_id=info:doi\/10.1093%2Fnar%2Fgkq929&rft_id=info:pmc\/PMC3013785&rft_id=info:pmid\/20952405&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3013785&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchutzeUse12-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchutzeUse12_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sch\u00fctze, B. (2012). \"Use of medical treatment data outside of the patient supply \u2013 best way pseudonymisation\". <i>Deutsche Medizinische Wochenschrift<\/i> <b>137<\/b> (16): 844-50. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1055%2Fs-0031-1299040\" target=\"_blank\">10.1055\/s-0031-1299040<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22495919\" target=\"_blank\">22495919<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Use+of+medical+treatment+data+outside+of+the+patient+supply+%E2%80%93+best+way+pseudonymisation&rft.jtitle=Deutsche+Medizinische+Wochenschrift&rft.aulast=Sch%C3%BCtze%2C+B.&rft.au=Sch%C3%BCtze%2C+B.&rft.date=2012&rft.volume=137&rft.issue=16&rft.pages=844-50&rft_id=info:doi\/10.1055%2Fs-0031-1299040&rft_id=info:pmid\/22495919&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PabingerASurv14-15\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-PabingerASurv14_15-0\" rel=\"external_link\">15.0<\/a><\/sup> <sup><a href=\"#cite_ref-PabingerASurv14_15-1\" rel=\"external_link\">15.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Pabinger, S.; Dander, A.; Fischer, M.; Snajder, R.; Sperk, M.; Efremova, M.; Krabichler, B.; Speicher, M.R.; Zschocke, J.; Trajanoski, Z. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3956068\" target=\"_blank\">\"A survey of tools for variant analysis of next-generation genome sequencing data\"<\/a>. <i>Briefings in Bioinformatics<\/i> <b>15<\/b> (2): 256-78. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbs086\" target=\"_blank\">10.1093\/bib\/bbs086<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3956068\/\" target=\"_blank\">PMC3956068<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23341494\" target=\"_blank\">23341494<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3956068\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3956068<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+survey+of+tools+for+variant+analysis+of+next-generation+genome+sequencing+data&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Pabinger%2C+S.%3B+Dander%2C+A.%3B+Fischer%2C+M.%3B+Snajder%2C+R.%3B+Sperk%2C+M.%3B+Efremova%2C+M.%3B+Krabichler%2C+B.%3B+Speicher%2C+M.R.%3B+Zschocke%2C+J.%3B+Trajanoski%2C+Z.&rft.au=Pabinger%2C+S.%3B+Dander%2C+A.%3B+Fischer%2C+M.%3B+Snajder%2C+R.%3B+Sperk%2C+M.%3B+Efremova%2C+M.%3B+Krabichler%2C+B.%3B+Speicher%2C+M.R.%3B+Zschocke%2C+J.%3B+Trajanoski%2C+Z.&rft.date=2014&rft.volume=15&rft.issue=2&rft.pages=256-78&rft_id=info:doi\/10.1093%2Fbib%2Fbbs086&rft_id=info:pmc\/PMC3956068&rft_id=info:pmid\/23341494&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3956068&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KasprzykBio11-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KasprzykBio11_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kasprzyk, A. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215098\" target=\"_blank\">\"BioMart: Driving a paradigm change in biological data management\"<\/a>. <i>Database<\/i> <b>2011<\/b>: bar049. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fdatabase%2Fbar049\" target=\"_blank\">10.1093\/database\/bar049<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3215098\/\" target=\"_blank\">PMC3215098<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22083790\" target=\"_blank\">22083790<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215098\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215098<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioMart%3A+Driving+a+paradigm+change+in+biological+data+management&rft.jtitle=Database&rft.aulast=Kasprzyk%2C+A.&rft.au=Kasprzyk%2C+A.&rft.date=2011&rft.volume=2011&rft.pages=bar049&rft_id=info:doi\/10.1093%2Fdatabase%2Fbar049&rft_id=info:pmc\/PMC3215098&rft_id=info:pmid\/22083790&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3215098&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoodeOpen13-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoodeOpen13_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goode, A.; Gilbert, B.; Harkes, J.; Jukic, D.; Satyanarayanan, M. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3815078\" target=\"_blank\">\"OpenSlide: A vendor-neutral software foundation for digital pathology\"<\/a>. <i>Journal of Pathology Informatics<\/i> <b>4<\/b>: 27. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.4103%2F2153-3539.119005\" target=\"_blank\">10.4103\/2153-3539.119005<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3815078\/\" target=\"_blank\">PMC3815078<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24244884\" target=\"_blank\">24244884<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3815078\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3815078<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=OpenSlide%3A+A+vendor-neutral+software+foundation+for+digital+pathology&rft.jtitle=Journal+of+Pathology+Informatics&rft.aulast=Goode%2C+A.%3B+Gilbert%2C+B.%3B+Harkes%2C+J.%3B+Jukic%2C+D.%3B+Satyanarayanan%2C+M.&rft.au=Goode%2C+A.%3B+Gilbert%2C+B.%3B+Harkes%2C+J.%3B+Jukic%2C+D.%3B+Satyanarayanan%2C+M.&rft.date=2013&rft.volume=4&rft.pages=27&rft_id=info:doi\/10.4103%2F2153-3539.119005&rft_id=info:pmc\/PMC3815078&rft_id=info:pmid\/24244884&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3815078&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LinkertMeta10-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LinkertMeta10_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Linkert, M.; Rueden, C.T.; Allan, C.; Burel, J-M.; Moore, W.; Patterson, A.; Loranger, B.; Moore, J.; Neves, C.; MacDonald, D.; Tarkowska, A.; Sticco, C.; Hill, E.; Rossner, M.; Eliceiri, K.W.; Swedlow, J.R. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938\" target=\"_blank\">\"Metadata matters: access to image data in the real world\"<\/a>. <i>Journal of Cell Biology<\/i> <b>189<\/b> (5): 777-82. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1083%2Fjcb.201004104\" target=\"_blank\">10.1083\/jcb.201004104<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2878938\/\" target=\"_blank\">PMC2878938<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20513764\" target=\"_blank\">20513764<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2878938<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Metadata+matters%3A+access+to+image+data+in+the+real+world&rft.jtitle=Journal+of+Cell+Biology&rft.aulast=Linkert%2C+M.%3B+Rueden%2C+C.T.%3B+Allan%2C+C.%3B+Burel%2C+J-M.%3B+Moore%2C+W.%3B+Patterson%2C+A.%3B+Loranger%2C+B.%3B+Moore%2C+J.%3B+Neves%2C+C.%3B+MacDonald%2C+D.%3B+Tarkowska%2C+A.%3B+Sticco%2C+C.%3B+Hill%2C+E.%3B+Rossner%2C+M.%3B+Eliceiri%2C+K.W.%3B+Swedlow%2C+J.R.&rft.au=Linkert%2C+M.%3B+Rueden%2C+C.T.%3B+Allan%2C+C.%3B+Burel%2C+J-M.%3B+Moore%2C+W.%3B+Patterson%2C+A.%3B+Loranger%2C+B.%3B+Moore%2C+J.%3B+Neves%2C+C.%3B+MacDonald%2C+D.%3B+Tarkowska%2C+A.%3B+Sticco%2C+C.%3B+Hill%2C+E.%3B+Rossner%2C+M.%3B+Eliceiri%2C+K.W.%3B+Swedlow%2C+J.R.&rft.date=2010&rft.volume=189&rft.issue=5&rft.pages=777-82&rft_id=info:doi\/10.1083%2Fjcb.201004104&rft_id=info:pmc\/PMC2878938&rft_id=info:pmid\/20513764&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2878938&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaurerMARS05-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MaurerMARS05_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maurer, M.; Molidor, R.; Sturn, A.; Hartler, J.; Hackl, H.; Stocker, G.; Prokesch, A.; Scheideler, M.; Trajanoski, Z. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551\" target=\"_blank\">\"MARS: microarray analysis, retrieval, and storage system\"<\/a>. <i>BMC Bioinformatics<\/i> <b>6<\/b>: 101. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-6-101\" target=\"_blank\">10.1186\/1471-2105-6-101<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1090551\/\" target=\"_blank\">PMC1090551<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15836795\" target=\"_blank\">15836795<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MARS%3A+microarray+analysis%2C+retrieval%2C+and+storage+system&rft.jtitle=BMC+Bioinformatics&rft.aulast=Maurer%2C+M.%3B+Molidor%2C+R.%3B+Sturn%2C+A.%3B+Hartler%2C+J.%3B+Hackl%2C+H.%3B+Stocker%2C+G.%3B+Prokesch%2C+A.%3B+Scheideler%2C+M.%3B+Trajanoski%2C+Z.&rft.au=Maurer%2C+M.%3B+Molidor%2C+R.%3B+Sturn%2C+A.%3B+Hartler%2C+J.%3B+Hackl%2C+H.%3B+Stocker%2C+G.%3B+Prokesch%2C+A.%3B+Scheideler%2C+M.%3B+Trajanoski%2C+Z.&rft.date=2005&rft.volume=6&rft.pages=101&rft_id=info:doi\/10.1186%2F1471-2105-6-101&rft_id=info:pmc\/PMC1090551&rft_id=info:pmid\/15836795&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1090551&rfr_id=info:sid\/en.wikipedia.org:Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. Table 1 has also been modified to a Y\/N format from its original check mark\/X format.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210219\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.531 seconds\nReal time usage: 0.565 seconds\nPreprocessor visited node count: 17359\/1000000\nPreprocessor generated node count: 29950\/1000000\nPost\u2010expand include size: 181748\/2097152 bytes\nTemplate argument size: 61841\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 517.823 1 - -total\n 83.21% 430.904 1 - Template:Reflist\n 76.77% 397.523 19 - Template:Cite_journal\n 73.19% 378.988 19 - Template:Citation\/core\n 12.83% 66.435 50 - Template:Citation\/identifier\n 12.11% 62.684 1 - Template:Infobox_journal_article\n 11.61% 60.107 1 - Template:Infobox\n 6.97% 36.089 80 - Template:Infobox\/row\n 4.57% 23.685 113 - Template:Hide_in_print\n 4.26% 22.035 19 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7859-0!*!0!!en!5!* and timestamp 20181213210218 and revision id 23627\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages\">https:\/\/www.limswiki.org\/index.php\/Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","03ae650a31def9630dc93ec76a1927f6_images":["https:\/\/www.limswiki.org\/images\/b\/bf\/Fig1_Dander_BMCBioinformatics2014_15.jpg","https:\/\/www.limswiki.org\/images\/6\/65\/Fig2_Dander_BMCBioinformatics2014_15.jpg","https:\/\/www.limswiki.org\/images\/6\/64\/Fig3_Dander_BMCBioinformatics2014_15.jpg","https:\/\/www.limswiki.org\/images\/9\/9a\/Fig4_Dander_BMCBioinformatics2014_15.jpg"],"03ae650a31def9630dc93ec76a1927f6_timestamp":1544734938,"0ea139cdae16f62db0066623f3ff445c_type":"article","0ea139cdae16f62db0066623f3ff445c_title":"MendeLIMS: A web-based laboratory information management system for clinical genome sequencing (Grimes et al. 2014)","0ea139cdae16f62db0066623f3ff445c_url":"https:\/\/www.limswiki.org\/index.php\/Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing","0ea139cdae16f62db0066623f3ff445c_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:MendeLIMS: A web-based laboratory information management system for clinical genome sequencing\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nMendeLIMS: A web-based laboratory information management system for clinical genome sequencingJournal\n \nBMC BioinformaticsAuthor(s)\n \nGrimes, Susan M.; Ji, Hanlee P.Author affiliation(s)\n \nStanford UniversityPrimary contact\n \nEmail: genomics_ji@stanford.eduYear published\n \n2014Volume and issue\n \n15Page(s)\n \n290DOI\n \n10.1186\/1471-2105-15-290ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 2.0 GenericWebsite\n \nhttp:\/\/www.biomedcentral.com\/1471-2105\/15\/290Download\n \nhttp:\/\/www.biomedcentral.com\/content\/pdf\/1471-2105-15-290.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Implementation \n\n3.1 Sample nomenclature \n3.2 Acquisition of clinical study samples \n3.3 Clinical study sample assessment and processing \n3.4 Tracking molecular and genomic analysis \n3.5 Tracking the next generation sequencing workflow \n3.6 Queries of MendeLIMS \n3.7 Reagent tracking \n3.8 Security \n\n\n4 Discussion \n5 Conclusions \n6 Availability and requirements \n7 Abbreviations \n8 Competing interests \n9 Authors\u2019 contributions \n10 Acknowledgements \n11 References \n12 Notes \n\n\n\nAbstract \nBackground: Large clinical genomics studies using next generation DNA sequencing require the ability to select and track samples from a large population of patients through many experimental steps. With the number of clinical genome sequencing studies increasing, it is critical to maintain adequate laboratory information management systems to manage the thousands of patient samples that are subject to this type of genetic analysis.\nResults: To meet the needs of clinical population studies using genome sequencing, we developed a web-based laboratory information management system (LIMS) with a flexible configuration that is adaptable to continuously evolving experimental protocols of next generation DNA sequencing technologies. Our system is referred to as MendeLIMS, is easily implemented with open source tools and is also highly configurable and extensible. MendeLIMS has been invaluable in the management of our clinical genome sequencing studies.\nConclusions: We maintain a publicly available demonstration version of the application for evaluation purposes at http:\/\/mendelims.stanford.edu. MendeLIMS is programmed in Ruby on Rails (RoR) and accesses data stored in SQL-compliant relational databases. Software is freely available for non-commercial use at http:\/\/dna-discovery.stanford.edu\/software\/mendelims\/.\nKeywords: Next generation sequencing; Clinical studies; Laboratory information management; Pathology; Genomics; Genetics\n\nBackground \nWith next generation DNA sequencing (NGS) now being a commonly adopted technology, the genetic analysis of large clinical populations has become practical and is widely used for identifying disease-related germline and somatic variants such as cancer mutations. The genetic variation from thousands of individuals can now be identified with NGS whole genome, exome, targeted and other resequencing approaches. Due to the dramatic increase in the number of NGS clinical genomics studies, it has become increasingly important to develop adequate laboratory information managements systems (LIMS) to manage the thousands of patient samples that are subject to NGS analysis. Tracking and managing the clinical sample workflow involved in NGS analysis is an extremely difficult task, given the logistical issues of enrolling patients, fragmented procedures for acquisition of clinical study samples, complex molecular preparation steps and the intricacies of the NGS processing pipeline. Commercial systems are available but typically are high cost and require extensive modification to address the specific needs of biomedical research groups conducting genetic analysis on populations.\nAs a general and unique solution to the needs of managing the experimental workflow for clinical genome sequencing projects, we developed MendeLIMS, a web-based, robust and flexible solution for integrating the management of clinical study samples and NGS processes. With respect to genetic studies, MendeLIMS functionality can be grouped into four major categories: (i) enrollment of patients and acquisition of clinical study samples, (ii) sample assessment and processing, (iii) genomic analysis through preparation of next generation DNA sequencing libraries or other molecular assays such as microarrays and finally, (iv) DNA sequencing of samples with associated quality control metrics. Tracking of sequencing steps is currently supported for the following Illumina NGS instruments: GAIIx, MiSeq, HiSeq, HiSeq2500, NextSeq but can easily be configured for any type of NGS instrument which follows a sequencing library to flow cell workflow. We maintain a publicly available demonstration version of the application for evaluation purposes at http:\/\/mendelims.stanford.edu.\n\nImplementation \nMendeLIMS is written in Ruby using the open source web application framework Ruby on Rails (RoR) and implementation is platform-independent. Instructions for installation are provided in Additional file 1. For our own in-house instance of MendeLIMS, our servers run Linux\/Ubuntu and we use the MySQL relational database management system (RDBMS). The application is easily configured to use any other SQL RDBMS supported by RoR. Figure 1 shows a simplified database schema for the major tables. A more comprehensive schema is provided in Additional file 2. \n\n\n\n\n\n\n\n Additional file 1. An installation guide for MendeLIMS.\n\n\nFormat: PDF; Size: 1,016KB Download file\n\n\n\n\n\n\nFigure 1: Database schema for MendeLIMS. Main entities and their relationships are shown in this diagram,\r\nand a complete schema showing other ancillary tables is provided in the supplementary material.\n\n\n\n\n\n\n Additional file 2. MendeLIMS database schema diagram.\n\n\nFormat: PDF; Size: 54KB Download file\n\n\n\nThe web interface is designed to handle a variety of queries in a modular format (Figure 2). To facilitate consistent data entry, MendeLIMS uses drop-down lists for seamless data validation whenever possible. The drop-down lists themselves are user-configurable by users with the appropriate authorization. Examples of user-configurable items include sample types, sequencing library multiplexing schemes, alignment references and DNA sequencers. All of the features are described in the user\u2019s manual (Additional file 3). \n\n\n\nFigure 2: Query web interfaces for MendeLIMS. Database queries are managed by a series of web pages that\r\nhave a modular format. Different search parameters are included with drop down menus used for standardized search\r\nterminology. Based on the needs of any given group, the search interface can be easily modified to accommodate\r\nnew search or entry functions.\n\n\n\n\n\n\n Additional file 3. User\u2019s guide for MendeLIMS.\n\n\nFormat: PDF; Size: 338KB Download file\n\n\n\nThe look and feel of the application is easily changed or customized since all web pages inherit styles from an application-wide cascading style sheet (CSS), and in keeping with RoR convention, overall page layout and navigation is controlled by a single HTML layout file. \n\nSample nomenclature \nTo enable accurate tracking of samples from their initial acquisition, through all key intermediate steps and ultimately to DNA sequencing, we developed a sample labeling nomenclature loosely based on the scheme employed by the Cancer Genome Atlas (TCGA) project (https:\/\/wiki.nci.nih.gov\/display\/TCGA\/Working+with+TCGA+Data). We maintain the original unique sample barcode, and add successive suffixes to indicate processing performed.\n\nAcquisition of clinical study samples \nAfter enrollment into a study, patient samples and their characteristics are entered into MendeLIMS through a web interface (Figure 2). A unique identifier (ID) is assigned for each new clinical sample. The user has the option of entering sample-relevant clinical data including pathology information from clinical reports, digital images originating from pathology slides and general clinical information about the patient (Figure 3). For efficient subsequent retrieval of the physical samples, the storage freezer and container location is entered using a standard nomenclature. If email triggers are configured, an email is automatically sent to an identified central coordinator and\/or to a specified owner for the particular clinical trial giving details of any new sample entered into the system. The web interface enables sample entry to occur at any location thus facilitating sample entry by various researchers and clinical coordinators. \n\n\n\nFigure 3: Workflow of MendeLIMS. Multiple steps of the sample acquisition workflow for clinical studies are\r\nfully integrated with next generation or genomic assay procedures. This allows one to trace the genomic\r\nanalysis of any given sample.\nClinical study sample assessment and processing \nAny manipulation of clinical study samples is tracked (Figure 3). This includes dissection of tissue samples and nucleic acid extraction. Details of these sample workflow operations are stored in MendeLIMS including volumes of any extracted macromolecules such as genomic DNA, concentration metrics and sample storage location. This greatly facilitates the managements of these precious resources for population studies.\n\nTracking molecular and genomic analysis \nMolecular assays and sequencing library steps are also captured in MendeLIMS (Figure 3). Sequencing libraries may be entered as singleplex (e.g. one sample per library), or multiplex (e.g. multiple samples per library with each sample tagged with a unique starting sequence). The multiplex indexing schemes are user-configurable, both for number of samples which can be multiplexed on one lane, and for the actual starting sequences used.\n\nTracking the next generation sequencing workflow \nIn preparation for initiation of an NGS analysis, a sequencing run is entered into the system by selecting existing libraries and placing them into separate lanes or partitions. Normally an entire sequencing run is entered. However, the system is also able to handle partial sequencing runs to accommodate the scenario where sequencing may be performed as a service and the run is shared between multiple groups who are not privy to each other\u2019s results. MendeLIMS generates a unique sequencing run key based on the sequencing date, sequencing machine, and a unique sequential run number. Once the sequencing run has completed, the initial quality control (QC) metrics for the run can be entered into the system. This is currently handled by an offline ruby script, but will in future be incorporated into the web application. MendeLIMS supports any type of sequencing application including whole genome, exome, targeted and RNA-based sequencing studies. The system stores sequencing library and sample lineage, flow cell composition, and sequencing run metadata, along with run status and QC metrics for all runs (Figure 4). The sequencing data files \u2014 for example bam alignment files, or vcf variant calling files \u2014 are not stored in MendeLIMS per se but are on a storage cluster accessible to all researchers in the group. Additionally, since we use the MendeLIMS sequencing run key and sequencing library\/sample nomenclature in the analysis directory and file names, the files are easily cross-referenced between MendeLIMS and the storage cluster. \n\n\n\nFigure 4: Tracking the sequencing of clinical samples. One can follow a clinical sample from enrollment in a clinical study all the way through\r\nto its sequencing. For example, from the sequencing run composition one can back track to the individual libraries and the original source DNA.\r\nScreen shots show the various levels of querying. A sequencing library can be queried for additional information. When required, it is possible to even\r\ndetermine the time of enrollment in a study and pull up relevant images from pathology.\nQueries of MendeLIMS \nAll queries allow specification of multiple filter criteria such as barcode range, date range, owner, protocol which enables users to quickly find the samples of interest, and then drill-down to more detail. For example, when viewing the sample query result set (Figure 2), clicking on the sample barcode will bring up more comprehensive information regarding the sample, including pathology information. Clicking on the \u2018QC\u2019 link from a sequencing library query result set shows QC data for all sequencing runs for that library. Query results may be exported to a tab-delimited file for review or for incorporation with other local data.\n\nReagent tracking \nReagents, equipment and supplies ordering, though not necessarily typical to a LIMS implementation, have been included in MendeLIMS. This feature enables for example the tracking of reagent and supply batches that is useful in troubleshooting failed sequencing runs, or the tracking of all expenditures against a specific funding account.\n\nSecurity \nHypertext Transfer Protocol Secure (HTTPS) is supported and is currently implemented for user login pages, but is easily extended to other pages as needed. User authentication is via a userid and password, and access to functionality is controlled via user roles which are defined and managed from the website by a user with \u2018admin\u2019 role. Other roles available include \u2018clinical\u2019 which allows create\/modify access to clinical study sample information, \u2018clin_admin\u2019 which allows modification to drop-down lists used for system validation for sample data, \u2018researcher\u2019 which allows create\/modify access to sequencing libraries and sequencing runs. The user manual in the supplementary material provides descriptions of all available user roles.\nGiven the extreme complexity of dealing with private health information (PHI), MendeLIMS is not designed to incorporate PHI-related clinical data. MendeLIMS does store a patient identifier that is the link between MendeLIMS and other patient clinical information databases that are securely stored in a very limited access environment. The identifier is stored as a binary encrypted field in the MySQL database and access to this field via the web application is limited to users with a \u2018clinical\u2019 or \u2018clin_admin\u2019role; other users only see a unique system-generated patient identifier which for all intents and purposes is anonymous.\nIn our current implementation used by several groups at Stanford University, MendeLIMS is integrated into an internal network, within a secured firewall. All database transactions are logged and time-stamped to provide an audit trail, and automated database backups are run daily. An administrator can readily generate an audit report to keep track of changes made by users. \n\nDiscussion \nThere are commercial LIMS solutions available for NGS labs, some of which have been implemented at major genomic research centers. For example GeneSifter LAB Edition[1] has been implemented at Vanderbilt University; Progeny LIMS[2] at Pittsburg University and Clarity LIMS[3] at University of Washington. These systems have significant capabilities. However, the cost in time and money to implement them is often out of reach for smaller organizations, particularly those who rely on funding from research grants or who require unique workflows that can not be implemented readily in a system designed for a larger institution. Given these resource constraints, open source options are of greater interest to this category of organizations. \nThere are several simpler LIMS systems covering clinical study samples such as BonsaiLIMS[4], PASSIM[5] and SLIMS.[6] These LIMS offer basic sample management but do not offer comprehensive clinical sample tracking or the ability to define sequencing libraries and flow cell\/sequencing run composition for NGS processing. More recent offerings which are available for open source installation and do support NGS processing include Galaxy LIMS[7] and GNomEx LIMS.[8] These systems address the flow from DNA\/RNA extraction to sequencing library to flow cell\/sequencing run. GNomEx LIMS also provides some analysis workflow capability and integrates data visualization via genome browsers such as UCSC Genome Browser[9] or Integrative Genomics Viewer (IGV).[10] Galaxy LIMS takes advantage of the Galaxy infrastructure to also provide analysis workflow and data visualization. However, none of these systems natively provide tracking of clinical study data such as consent protocols, pathology and histopathology information. Sample lineage and sample tracking via consistent nomenclature, drill down to various levels of source data, and freezer container\/location information is also not addressed. \nAnother open source option is QTREDS.[11] This LIMS has a strong focus on experimental protocols for sample preparation and tracks detailed steps which MendeLIMS and other systems do not specifically track such as sonication, end repair or ligation as part of exome library preparation. QTREDS also manages inventory of reagents for sample preparation and triggers low stock level alerts. However there is no tracking associated with clinical study samples, and NGS support is limited to the sequencing libraries and their associated sequencing status, rather than flow cell composition and sequencing run itself. In contrast, MendeLIMS provides full sample lineage tracking back to patient and the clinic and consent protocol where the sample originated, as well as support for all major processing steps through to the DNA\/RNA sequencing and QC (Table 1). Additional functionality that is useful for tracking is data related to which cluster and sequencing kit versions were used for a particular run, and what publications (if any) reference the results from that run. \n\n\n\n\n\n\n\nTable 1. Comparison among different LIMS systems\n\n\n LIMS software\n\n MendeLIMS\n\n GNomEx\n\n Galaxy LIMS\n\n QTREDS\n\n\nClinical study patient samples\n\n\nPatient data (gender\/race, MRN, pathology, histology)\n\nYes\n\nNo\n\nNo\n\nNo\n\n\nSample processing (dissections, extractions)\n\nYes\n\nNo\n\nNo\n\nYesa\n\n\nSample location tracking\n\nYes\n\nNo\n\nNo\n\nYes\n\n\nArrays, libraries, sequencing runs\n\n\nMolecular assays (genomic arrays, ddPCR, ..)\n\nYes\n\nYes\n\nNo\n\nYes\n\n\nSequencing library prep (singleplex and multiplex)\n\nYes\n\nYes\n\nYes\n\nYesb\n\n\nFlow cell\/sequencing run setup\n\nYes\n\nYes\n\nYes\n\nNo\n\n\nPost-sequencing analysis\n\n\nSequencing QC\n\nYes\n\nnd\n\nYesc\n\nNo\n\n\nAnalysis workflow\n\nNo\n\nYes\n\nYesc\n\nNo\n\n\nSecurity\/Audit trail\n\n\nAuthorization via user roles\n\nYes\n\nYes\n\nYes\n\nYes\n\n\nAudit trail\n\nYes\n\nYes\n\nnd\n\nnd\n\n\nHTTPS\/SSL security\n\nYes\n\nnd\n\nnd\n\nYes\n\n\nGeneral\n\n\nEmail\/notification capability\n\nYes\n\nYes\n\nYesc\n\nYes\n\n\nCustomizable lists for data validation\n\nYes\n\nYes\n\nYes\n\nYes\n\n\nInstrument integration\n\nNo\n\nNo\n\nYesd\n\nNo\n\n\nAttach files to samples\/libraries\/sequencing runs\n\nYes\n\nnd\n\nNo\n\nnd\n\n\nVisualization of results\n\nNo\n\nYese\n\nYesc\n\nNo\n\n\nOther\n\n\nProject based billing\n\nNo\n\nYes\n\nNo\n\nNo\n\n\nReagent inventory management\n\nNo\n\nNo\n\nNo\n\nYes\n\n\nPublicly available User Guide\/Demos\n\nYes\n\nYes\n\nNo\n\nYes\n\n\na) Detailed information tracked regarding sample prep; b) Singleplex libraries only; c) Functionality provided via integration with Galaxy and genome browsers; d) Integration with HiSeq 2000 only; e) Functionality provided via integration with GenoPub.\nnd - Indicates that unable to determine from public documentation whether the functionality is provided. \n\n\n\n\nConclusions \nClinical population studies using NGS sequencing require management of hundreds if not thousands of samples, including various intermediate processing steps, and the resulting sequencing data. A LIMS system is critical to the effective management of this data and the generation of reproducible results. The currently available open source or commercial systems may meet the needs of some research groups, however for those groups where the time and monetary cost of a comprehensive commercial system is prohibitive, there is no end-to-end open source solution covering enrollment of patients in a clinical study through genome sequencing analysis. Our system addresses all of these needs with a specific focus and seamless integration of clinical study enrollment through to NGS.\nIn MendeLIMS, all data is consolidated into one authoritative centrally accessible source repository, eliminating multiple distributed spreadsheets. Samples are traceable from a lane on a sequencing run, back to the patient diagnosis, pathology, and all processing steps in between. In conjunction with a standard barcoding nomenclature and flexible query capability, this significantly reduces errors in sample tracking, provides a comprehensive view of data being sequenced and has resulted in MendeLIMS becoming an invaluable tool for the management of our clinical sequencing studies.\n\nAvailability and requirements \nProject name: MendeLIMS\nProject home page: http:\/\/dna-discovery.stanford.edu\/software\/mendelims\/\nProject demo site: http:\/\/mendelims.stanford.edu\/\nOperating system(s): Platform independent\nProgramming Language(s): Ruby, Ruby on Rails, HTML, Javascript\nServer requirements: Apache2, Mongrel or Passenger, Ruby 1.9.3+, Rails 3.2.x, MySQL 5.0\nWeb browser requirements: Firefox, Chrome, IE, Safari\nLicense: Any restrictions to use by non-academics: None\n\nAbbreviations \nNGS: Next generation sequencing\nLIMS: Laboratory information management system\nRoR: Ruby on rails\nRDBMS: Relational database management system\nCSS: Cascading sheet style\nTCGA: The cancer genome atlas\nID: Identifier\nQC: Quality control\nHTTPS: Hypertext transfer protocol secure\nPHI: Private health information\nIGV: Integrated genome viewer\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors\u2019 contributions \nSMG and HPJ drafted the manuscript. SMG wrote the software, setup the website implementation and wrote the user guide. SMG and HPJ conceived and designed the project. Both authors read and approved the final manuscript.\n\nAcknowledgements \nWe would like to acknowledge Rebecca Kim for help with design for sample acquisition and processing; Jason Buenrostro and Erik Hopmans for help with design for sequencing libraries and flow cells; Jennifer Palm for her assistance with graphics and Curt Palm for his help with server configuration for the MendeLIMS demo site.\nThis work was supported by the following grants from the NIH: RC2HG005570 to S.M.G and H.P.J., DK56339 to H.P.J, 2P01HG000205 to S.M.G. and H.P.J. HPJ was supported by a Research Scholar Grant (RSG-13-297-01-TBG) from the American Cancer Society. In addition, H.P.J. received support from the Doris Duke Clinical Foundation, the Clayville Foundation, the Liu BieJu Cha and Family Fellowship in Cancer and the Howard Hughes Medical Foundation.\n\nReferences \n\n\n\u2191 \"GeneSifter Lab Edition\". PerkinElmer, Inc. http:\/\/www.geospiza.com\/Products\/LabEdition.shtml .   \n\n\u2191 \"Progeny LIMS\". Progeny Genetics, LLC. http:\/\/www.progenygenetics.com\/lims\/ .   \n\n\u2191 \"Clarity LIMS\". GenoLogics Life Sciences Software, Inc. http:\/\/www.genologics.com\/clarity-lims\/ .   \n\n\u2191 Bath, Timothy G.; Bozdag, Selcuk; Afzal, Vackar; Crowther, Daniel (2011). \"LimsPortal and BonsaiLIMS: development of a lab information management system for translational medicine\". Source Code for Biology and Medicine 6: 9. doi:10.1186\/1751-0473-6-9. PMC PMC3113716. PMID 21569484. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3113716 .   \n\n\u2191 Viksna, J.; Celms, E.; Opmanis, M.; Podnieks, K.; Rucevskis, P.; Zarins, A.; Barrett, A.; Neogi, S.G.; Krestyaninova, M.; McCarthy, M.I.; Brazma, A.; Sarkans, U. (2007). \"PASSIM: An open source software system for managing information in biomedical studies\". BMC Bioinformatics 8: 52. doi:10.1186\/1471-2105-8-52. PMC PMC1803798. PMID 17291344. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798 .   \n\n\u2191 Van Rossum, T.; Tripp, B.; Daley, D. (2010). \"SLIMS: A user-friendly sample operations and inventory management system for genotyping labs\". Bioinformatics 26 (14): 1808-1810. doi:10.1093\/bioinformatics\/btq271. PMC PMC2894515. PMID 20513665. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515 .   \n\n\u2191 Scholtalbers, J.; Rossler, J.; Sorn, P.; de Graaf, J.; Boisguerin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". Bioinformatics 29 (9): 1233-1234. doi:10.1093\/bioinformatics\/btt115. PMID 23479349.   \n\n\u2191 Nix, David A.; Di Sera, Tonya L.; Dalley, Brian K.; Milash, Brett A.; Cundick, Robert M.; Quinn, Kevin S.; Courdy, Samir J. (2010). \"Next generation tools for genomic data generation, distribution, and visualization\". BMC Bioinformatics 11: 455. doi:10.1186\/1471-2105-11-455. PMC PMC2944281. PMID 20828407. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281 .   \n\n\u2191 Kent, W.J.; Sugnet, C.W; Furey, T.S.; Roskin, K.M.; Pringle, T.H.; Zahler, A.M.; Haussler, D. (2002). \"The human genome browser at UCSC\". Genome Research 12 (6): 996-1006. doi:10.1101\/gr.229102. PMC PMC186604. PMID 12045153. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604 .   \n\n\u2191 Thorvaldsdottir, H.; Robinson, J.T.; Mesirov, J.P. (2013). \"Integrative Genomics Viewer (IGV): high-performance genomics data visualization and exploration\". Briefings in Bioinformatics 14 (2): 178-192. doi:10.1093\/bib\/bbs017. PMC PMC3603213. PMID 22517427. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3603213 .   \n\n\u2191 Palla, P.; Frau, G.; Vargiu, L.; Rodriguez-Tome, P. (2014). \"QTREDS: A Ruby on Rails-based platform for omics laboratories\". BMC Bioinformatics 15 (Suppl 1): S13. doi:10.1186\/1471-2105-15-S1-S13. PMC PMC4015218. PMID 24564791. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4015218 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\">https:\/\/www.limswiki.org\/index.php\/Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:58.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,754 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","0ea139cdae16f62db0066623f3ff445c_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_MendeLIMS_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:MendeLIMS: A web-based laboratory information management system for clinical genome sequencing<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Large clinical genomics studies using next generation DNA sequencing require the ability to select and track samples from a large population of patients through many experimental steps. With the number of clinical genome sequencing studies increasing, it is critical to maintain adequate <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> to manage the thousands of patient samples that are subject to this type of genetic analysis.\n<\/p><p><b>Results:<\/b> To meet the needs of clinical population studies using genome sequencing, we developed a web-based laboratory information management system (LIMS) with a flexible configuration that is adaptable to continuously evolving experimental protocols of next generation DNA sequencing technologies. Our system is referred to as <a href=\"https:\/\/www.limswiki.org\/index.php\/Stanford_University_School_of_Medicine#MendeLIMS\" title=\"Stanford University School of Medicine\" target=\"_blank\" class=\"wiki-link\" data-key=\"3d5b4c2cca220779c9982302e548f46c\">MendeLIMS<\/a>, is easily implemented with open source tools and is also highly configurable and extensible. MendeLIMS has been invaluable in the management of our clinical genome sequencing studies.\n<\/p><p><b>Conclusions:<\/b> We maintain a publicly available demonstration version of the application for evaluation purposes at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mendelims.stanford.edu\" target=\"_blank\">http:\/\/mendelims.stanford.edu<\/a>. MendeLIMS is programmed in <a href=\"https:\/\/www.limswiki.org\/index.php\/Ruby_on_Rails\" title=\"Ruby on Rails\" target=\"_blank\" class=\"wiki-link\" data-key=\"bec38424b2640687620b42397f4e7c52\">Ruby on Rails<\/a> (RoR) and accesses data stored in SQL-compliant relational databases. Software is freely available for non-commercial use at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dna-discovery.stanford.edu\/software\/mendelims\/\" target=\"_blank\">http:\/\/dna-discovery.stanford.edu\/software\/mendelims\/<\/a>.\n<\/p><p><b>Keywords:<\/b> Next generation sequencing; Clinical studies; Laboratory information management; Pathology; Genomics; Genetics\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>With next generation DNA sequencing (NGS) now being a commonly adopted technology, the genetic analysis of large clinical populations has become practical and is widely used for identifying disease-related germline and somatic variants such as cancer mutations. The genetic variation from thousands of individuals can now be identified with NGS whole genome, exome, targeted and other resequencing approaches. Due to the dramatic increase in the number of NGS clinical genomics studies, it has become increasingly important to develop adequate laboratory information managements systems (LIMS) to manage the thousands of patient samples that are subject to NGS analysis. Tracking and managing the clinical sample workflow involved in NGS analysis is an extremely difficult task, given the logistical issues of enrolling patients, fragmented procedures for acquisition of clinical study samples, complex molecular preparation steps and the intricacies of the NGS processing pipeline. Commercial systems are available but typically are high cost and require extensive modification to address the specific needs of biomedical research groups conducting genetic analysis on populations.\n<\/p><p>As a general and unique solution to the needs of managing the experimental workflow for clinical genome sequencing projects, we developed MendeLIMS, a web-based, robust and flexible solution for integrating the management of clinical study samples and NGS processes. With respect to genetic studies, MendeLIMS functionality can be grouped into four major categories: (i) enrollment of patients and acquisition of clinical study samples, (ii) sample assessment and processing, (iii) genomic analysis through preparation of next generation DNA sequencing libraries or other molecular assays such as microarrays and finally, (iv) DNA sequencing of samples with associated quality control metrics. Tracking of sequencing steps is currently supported for the following Illumina NGS instruments: GAIIx, MiSeq, HiSeq, HiSeq2500, NextSeq but can easily be configured for any type of NGS instrument which follows a sequencing library to flow cell workflow. We maintain a publicly available demonstration version of the application for evaluation purposes at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mendelims.stanford.edu\" target=\"_blank\">http:\/\/mendelims.stanford.edu<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<p>MendeLIMS is written in Ruby using the open source web application framework Ruby on Rails (RoR) and implementation is platform-independent. Instructions for installation are provided in Additional file 1. For our own in-house instance of MendeLIMS, our servers run Linux\/Ubuntu and we use the <a href=\"https:\/\/www.limswiki.org\/index.php\/MySQL\" title=\"MySQL\" target=\"_blank\" class=\"wiki-link\" data-key=\"35005451bfcd508bce47c58e72260128\">MySQL<\/a> relational database management system (RDBMS). The application is easily configured to use any other SQL RDBMS supported by RoR. Figure 1 shows a simplified database schema for the major tables. A more comprehensive schema is provided in Additional file 2. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 1. <i>An installation guide for MendeLIMS.<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Format: PDF; Size: 1,016KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-15-290-s1.pdf\" target=\"_blank\">Download file<\/a>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Grimes_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"38a3e6012b075415a9e316750148fc0a\"><img alt=\"Fig1 Grimes BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/1\/11\/Fig1_Grimes_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1: Database schema for MendeLIMS.<\/b> Main entities and their relationships are shown in this diagram,<br \/>and a complete schema showing other ancillary tables is provided in the supplementary material.<\/blockquote>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 2. <i>MendeLIMS database schema diagram.<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Format: PDF; Size: 54KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-15-290-s2.pdf\" target=\"_blank\">Download file<\/a>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The web interface is designed to handle a variety of queries in a modular format (Figure 2). To facilitate consistent data entry, MendeLIMS uses drop-down lists for seamless data validation whenever possible. The drop-down lists themselves are user-configurable by users with the appropriate authorization. Examples of user-configurable items include sample types, sequencing library multiplexing schemes, alignment references and DNA sequencers. All of the features are described in the user\u2019s manual (Additional file 3). \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Grimes_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"f211e24e31836bb2dd3ada1e6e70de78\"><img alt=\"Fig2 Grimes BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/8\/82\/Fig2_Grimes_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2: Query web interfaces for MendeLIMS.<\/b> Database queries are managed by a series of web pages that<br \/>have a modular format. Different search parameters are included with drop down menus used for standardized search<br \/>terminology. Based on the needs of any given group, the search interface can be easily modified to accommodate<br \/>new search or entry functions.<\/blockquote>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 3. <i>User\u2019s guide for MendeLIMS.<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Format: PDF; Size: 338KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-15-290-s3.pdf\" target=\"_blank\">Download file<\/a>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The look and feel of the application is easily changed or customized since all web pages inherit styles from an application-wide cascading style sheet (CSS), and in keeping with RoR convention, overall page layout and navigation is controlled by a single HTML layout file. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Sample_nomenclature\">Sample nomenclature<\/span><\/h3>\n<p>To enable accurate tracking of samples from their initial acquisition, through all key intermediate steps and ultimately to DNA sequencing, we developed a sample labeling nomenclature loosely based on the scheme employed by the Cancer Genome Atlas (TCGA) project (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/wiki.nci.nih.gov\/display\/TCGA\/Working+with+TCGA+Data\" target=\"_blank\">https:\/\/wiki.nci.nih.gov\/display\/TCGA\/Working+with+TCGA+Data<\/a>). We maintain the original unique sample barcode, and add successive suffixes to indicate processing performed.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Acquisition_of_clinical_study_samples\">Acquisition of clinical study samples<\/span><\/h3>\n<p>After enrollment into a study, patient samples and their characteristics are entered into MendeLIMS through a web interface (Figure 2). A unique identifier (ID) is assigned for each new clinical sample. The user has the option of entering sample-relevant clinical data including pathology information from clinical reports, digital images originating from pathology slides and general clinical information about the patient (Figure 3). For efficient subsequent retrieval of the physical samples, the storage freezer and container location is entered using a standard nomenclature. If email triggers are configured, an email is automatically sent to an identified central coordinator and\/or to a specified owner for the particular clinical trial giving details of any new sample entered into the system. The web interface enables sample entry to occur at any location thus facilitating sample entry by various researchers and clinical coordinators. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Grimes_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"37339dae60858659600f98241487eb78\"><img alt=\"Fig3 Grimes BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/4\/4d\/Fig3_Grimes_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3: Workflow of MendeLIMS.<\/b> Multiple steps of the sample acquisition workflow for clinical studies are<br \/>fully integrated with next generation or genomic assay procedures. This allows one to trace the genomic<br \/>analysis of any given sample.<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Clinical_study_sample_assessment_and_processing\">Clinical study sample assessment and processing<\/span><\/h3>\n<p>Any manipulation of clinical study samples is tracked (Figure 3). This includes dissection of tissue samples and nucleic acid extraction. Details of these sample workflow operations are stored in MendeLIMS including volumes of any extracted macromolecules such as genomic DNA, concentration metrics and sample storage location. This greatly facilitates the managements of these precious resources for population studies.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Tracking_molecular_and_genomic_analysis\">Tracking molecular and genomic analysis<\/span><\/h3>\n<p>Molecular assays and sequencing library steps are also captured in MendeLIMS (Figure 3). Sequencing libraries may be entered as singleplex (e.g. one sample per library), or multiplex (e.g. multiple samples per library with each sample tagged with a unique starting sequence). The multiplex indexing schemes are user-configurable, both for number of samples which can be multiplexed on one lane, and for the actual starting sequences used.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Tracking_the_next_generation_sequencing_workflow\">Tracking the next generation sequencing workflow<\/span><\/h3>\n<p>In preparation for initiation of an NGS analysis, a sequencing run is entered into the system by selecting existing libraries and placing them into separate lanes or partitions. Normally an entire sequencing run is entered. However, the system is also able to handle partial sequencing runs to accommodate the scenario where sequencing may be performed as a service and the run is shared between multiple groups who are not privy to each other\u2019s results. MendeLIMS generates a unique sequencing run key based on the sequencing date, sequencing machine, and a unique sequential run number. Once the sequencing run has completed, the initial quality control (QC) metrics for the run can be entered into the system. This is currently handled by an offline ruby script, but will in future be incorporated into the web application. MendeLIMS supports any type of sequencing application including whole genome, exome, targeted and RNA-based sequencing studies. The system stores sequencing library and sample lineage, flow cell composition, and sequencing run metadata, along with run status and QC metrics for all runs (Figure 4). The sequencing data files \u2014 for example bam alignment files, or vcf variant calling files \u2014 are not stored in MendeLIMS per se but are on a storage cluster accessible to all researchers in the group. Additionally, since we use the MendeLIMS sequencing run key and sequencing library\/sample nomenclature in the analysis directory and file names, the files are easily cross-referenced between MendeLIMS and the storage cluster. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Grimes_BMCBioinformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"9c4e7fad13cfb135d48fb9b6966fe2c6\"><img alt=\"Fig4 Grimes BMCBioinformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/c7\/Fig4_Grimes_BMCBioinformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4: Tracking the sequencing of clinical samples.<\/b> One can follow a clinical sample from enrollment in a clinical study all the way through<br \/>to its sequencing. For example, from the sequencing run composition one can back track to the individual libraries and the original source DNA.<br \/>Screen shots show the various levels of querying. A sequencing library can be queried for additional information. When required, it is possible to even<br \/>determine the time of enrollment in a study and pull up relevant images from pathology.<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Queries_of_MendeLIMS\">Queries of MendeLIMS<\/span><\/h3>\n<p>All queries allow specification of multiple filter criteria such as barcode range, date range, owner, protocol which enables users to quickly find the samples of interest, and then drill-down to more detail. For example, when viewing the sample query result set (Figure 2), clicking on the sample barcode will bring up more comprehensive information regarding the sample, including pathology information. Clicking on the \u2018QC\u2019 link from a sequencing library query result set shows QC data for all sequencing runs for that library. Query results may be exported to a tab-delimited file for review or for incorporation with other local data.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Reagent_tracking\">Reagent tracking<\/span><\/h3>\n<p>Reagents, equipment and supplies ordering, though not necessarily typical to a LIMS implementation, have been included in MendeLIMS. This feature enables for example the tracking of reagent and supply batches that is useful in troubleshooting failed sequencing runs, or the tracking of all expenditures against a specific funding account.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Security\">Security<\/span><\/h3>\n<p>Hypertext Transfer Protocol Secure (HTTPS) is supported and is currently implemented for user login pages, but is easily extended to other pages as needed. User authentication is via a userid and password, and access to functionality is controlled via user roles which are defined and managed from the website by a user with \u2018admin\u2019 role. Other roles available include \u2018clinical\u2019 which allows create\/modify access to clinical study sample information, \u2018clin_admin\u2019 which allows modification to drop-down lists used for system validation for sample data, \u2018researcher\u2019 which allows create\/modify access to sequencing libraries and sequencing runs. The user manual in the supplementary material provides descriptions of all available user roles.\n<\/p><p>Given the extreme complexity of dealing with private health information (PHI), MendeLIMS is not designed to incorporate PHI-related clinical data. MendeLIMS does store a patient identifier that is the link between MendeLIMS and other patient clinical information databases that are securely stored in a very limited access environment. The identifier is stored as a binary encrypted field in the MySQL database and access to this field via the web application is limited to users with a \u2018clinical\u2019 or \u2018clin_admin\u2019role; other users only see a unique system-generated patient identifier which for all intents and purposes is anonymous.\n<\/p><p>In our current implementation used by several groups at Stanford University, MendeLIMS is integrated into an internal network, within a secured firewall. All database transactions are logged and time-stamped to provide an audit trail, and automated database backups are run daily. An administrator can readily generate an audit report to keep track of changes made by users. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Discussion\">Discussion<\/span><\/h2>\n<p>There are commercial LIMS solutions available for NGS labs, some of which have been implemented at major genomic research centers. For example GeneSifter LAB Edition<sup id=\"rdp-ebb-cite_ref-GSLABEd_1-0\" class=\"reference\"><a href=\"#cite_note-GSLABEd-1\" rel=\"external_link\">[1]<\/a><\/sup> has been implemented at Vanderbilt University; <a href=\"https:\/\/www.limswiki.org\/index.php\/Progeny_Software,_LLC\" title=\"Progeny Software, LLC\" target=\"_blank\" class=\"wiki-link\" data-key=\"68a5e1e206d438b6497ca60780c1a3be\">Progeny LIMS<\/a><sup id=\"rdp-ebb-cite_ref-Progeny_2-0\" class=\"reference\"><a href=\"#cite_note-Progeny-2\" rel=\"external_link\">[2]<\/a><\/sup> at Pittsburg University and <a href=\"https:\/\/www.limswiki.org\/index.php\/GenoLogics_Life_Sciences_Software_Inc.\" title=\"GenoLogics Life Sciences Software Inc.\" target=\"_blank\" class=\"wiki-link\" data-key=\"14488d17247e5e4d57f8922bfc1fac13\">Clarity LIMS<\/a><sup id=\"rdp-ebb-cite_ref-Clarity_3-0\" class=\"reference\"><a href=\"#cite_note-Clarity-3\" rel=\"external_link\">[3]<\/a><\/sup> at University of Washington. These systems have significant capabilities. However, the cost in time and money to implement them is often out of reach for smaller organizations, particularly those who rely on funding from research grants or who require unique workflows that can not be implemented readily in a system designed for a larger institution. Given these resource constraints, open source options are of greater interest to this category of organizations. \n<\/p><p>There are several simpler LIMS systems covering clinical study samples such as <a href=\"https:\/\/www.limswiki.org\/index.php\/BonsaiLIMS\" title=\"BonsaiLIMS\" target=\"_blank\" class=\"wiki-link\" data-key=\"e477f6b8a042bd8703a30ef83dbcd0c9\">BonsaiLIMS<\/a><sup id=\"rdp-ebb-cite_ref-BathBons11_4-0\" class=\"reference\"><a href=\"#cite_note-BathBons11-4\" rel=\"external_link\">[4]<\/a><\/sup>, PASSIM<sup id=\"rdp-ebb-cite_ref-ViksnaPASS07_5-0\" class=\"reference\"><a href=\"#cite_note-ViksnaPASS07-5\" rel=\"external_link\">[5]<\/a><\/sup> and SLIMS.<sup id=\"rdp-ebb-cite_ref-VanRossumSLIMS10_6-0\" class=\"reference\"><a href=\"#cite_note-VanRossumSLIMS10-6\" rel=\"external_link\">[6]<\/a><\/sup> These LIMS offer basic sample management but do not offer comprehensive clinical sample tracking or the ability to define sequencing libraries and flow cell\/sequencing run composition for NGS processing. More recent offerings which are available for open source installation and do support NGS processing include <a href=\"https:\/\/www.limswiki.org\/index.php\/Galaxy_(biomedical_software)\" title=\"Galaxy (biomedical software)\" target=\"_blank\" class=\"wiki-link\" data-key=\"ead5d6ebaa8d67744d2f68d454d89ce6\">Galaxy LIMS<\/a><sup id=\"rdp-ebb-cite_ref-ScholtalbersGal13_7-0\" class=\"reference\"><a href=\"#cite_note-ScholtalbersGal13-7\" rel=\"external_link\">[7]<\/a><\/sup> and <a href=\"https:\/\/www.limswiki.org\/index.php\/GNomEx\" title=\"GNomEx\" target=\"_blank\" class=\"wiki-link\" data-key=\"ffe2672873462be87660d574029bf8b2\">GNomEx LIMS<\/a>.<sup id=\"rdp-ebb-cite_ref-NixNext10_8-0\" class=\"reference\"><a href=\"#cite_note-NixNext10-8\" rel=\"external_link\">[8]<\/a><\/sup> These systems address the flow from DNA\/RNA extraction to sequencing library to flow cell\/sequencing run. GNomEx LIMS also provides some analysis workflow capability and integrates data visualization via genome browsers such as UCSC Genome Browser<sup id=\"rdp-ebb-cite_ref-KentTheHuman02_9-0\" class=\"reference\"><a href=\"#cite_note-KentTheHuman02-9\" rel=\"external_link\">[9]<\/a><\/sup> or Integrative Genomics Viewer (IGV).<sup id=\"rdp-ebb-cite_ref-ThorvaldsdottirInt13_10-0\" class=\"reference\"><a href=\"#cite_note-ThorvaldsdottirInt13-10\" rel=\"external_link\">[10]<\/a><\/sup> Galaxy LIMS takes advantage of the Galaxy infrastructure to also provide analysis workflow and data visualization. However, none of these systems natively provide tracking of clinical study data such as consent protocols, pathology and histopathology information. Sample lineage and sample tracking via consistent nomenclature, drill down to various levels of source data, and freezer container\/location information is also not addressed. \n<\/p><p>Another open source option is QTREDS.<sup id=\"rdp-ebb-cite_ref-PallaQT14_11-0\" class=\"reference\"><a href=\"#cite_note-PallaQT14-11\" rel=\"external_link\">[11]<\/a><\/sup> This LIMS has a strong focus on experimental protocols for sample preparation and tracks detailed steps which MendeLIMS and other systems do not specifically track such as sonication, end repair or ligation as part of exome library preparation. QTREDS also manages inventory of reagents for sample preparation and triggers low stock level alerts. However there is no tracking associated with clinical study samples, and NGS support is limited to the sequencing libraries and their associated sequencing status, rather than flow cell composition and sequencing run itself. In contrast, MendeLIMS provides full sample lineage tracking back to patient and the clinic and consent protocol where the sample originated, as well as support for all major processing steps through to the DNA\/RNA sequencing and QC (Table 1). Additional functionality that is useful for tracking is data related to which cluster and sequencing kit versions were used for a particular run, and what publications (if any) reference the results from that run. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"5\"><b>Table 1.<\/b> Comparison among different LIMS systems\n<\/td><\/tr>\n<tr>\n<th> LIMS software\n<\/th>\n<th> MendeLIMS\n<\/th>\n<th> GNomEx\n<\/th>\n<th> Galaxy LIMS\n<\/th>\n<th> QTREDS\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold;\" colspan=\"5\"><i>Clinical study patient samples<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Patient data (gender\/race, MRN, pathology, histology)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sample processing (dissections, extractions)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>a<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sample location tracking\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold;\" colspan=\"5\"><i>Arrays, libraries, sequencing runs<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Molecular assays (genomic arrays, ddPCR, ..)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sequencing library prep (singleplex and multiplex)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>b<\/sup>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Flow cell\/sequencing run setup\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold;\" colspan=\"5\"><i>Post-sequencing analysis<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sequencing QC\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>nd<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>c<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Analysis workflow\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>c<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold;\" colspan=\"5\"><i>Security\/Audit trail<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Authorization via user roles\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Audit trail\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>nd<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>nd<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HTTPS\/SSL security\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>nd<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>nd<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold;\" colspan=\"5\"><i>General<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Email\/notification capability\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>c<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Customizable lists for data validation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Instrument integration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>d<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Attach files to samples\/libraries\/sequencing runs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>nd<\/i>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><i>nd<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Visualization of results\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>e<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes<sup>c<\/sup>\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px; font-weight:bold;\" colspan=\"5\"><i>Other<\/i>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Project based billing\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Reagent inventory management\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Publicly available User Guide\/Demos\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"9\"><sup>a)<\/sup> Detailed information tracked regarding sample prep; <sup>b)<\/sup> Singleplex libraries only; <sup>c)<\/sup> Functionality provided via integration with Galaxy and genome browsers; <sup>d)<\/sup> Integration with HiSeq 2000 only; <sup>e)<\/sup> Functionality provided via integration with GenoPub.\n<p><i>nd<\/i> - Indicates that unable to determine from public documentation whether the functionality is provided. \n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Clinical population studies using NGS sequencing require management of hundreds if not thousands of samples, including various intermediate processing steps, and the resulting sequencing data. A LIMS system is critical to the effective management of this data and the generation of reproducible results. The currently available open source or commercial systems may meet the needs of some research groups, however for those groups where the time and monetary cost of a comprehensive commercial system is prohibitive, there is no end-to-end open source solution covering enrollment of patients in a clinical study through genome sequencing analysis. Our system addresses all of these needs with a specific focus and seamless integration of clinical study enrollment through to NGS.\n<\/p><p>In MendeLIMS, all data is consolidated into one authoritative centrally accessible source repository, eliminating multiple distributed spreadsheets. Samples are traceable from a lane on a sequencing run, back to the patient diagnosis, pathology, and all processing steps in between. In conjunction with a standard barcoding nomenclature and flexible query capability, this significantly reduces errors in sample tracking, provides a comprehensive view of data being sequenced and has resulted in MendeLIMS becoming an invaluable tool for the management of our clinical sequencing studies.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p><b>Project name:<\/b> MendeLIMS\n<\/p><p><b>Project home page:<\/b> <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/dna-discovery.stanford.edu\/software\/mendelims\/\" target=\"_blank\">http:\/\/dna-discovery.stanford.edu\/software\/mendelims\/<\/a>\n<\/p><p><b>Project demo site:<\/b> <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mendelims.stanford.edu\/\" target=\"_blank\">http:\/\/mendelims.stanford.edu\/<\/a>\n<\/p><p><b>Operating system(s):<\/b> Platform independent\n<\/p><p><b>Programming Language(s):<\/b> Ruby, Ruby on Rails, HTML, Javascript\n<\/p><p><b>Server requirements:<\/b> Apache2, Mongrel or Passenger, Ruby 1.9.3+, Rails 3.2.x, MySQL 5.0\n<\/p><p><b>Web browser requirements:<\/b> Firefox, Chrome, IE, Safari\n<\/p><p><b>License:<\/b> Any restrictions to use by non-academics: None\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p>NGS: Next generation sequencing\n<\/p><p>LIMS: Laboratory information management system\n<\/p><p>RoR: Ruby on rails\n<\/p><p>RDBMS: Relational database management system\n<\/p><p>CSS: Cascading sheet style\n<\/p><p>TCGA: The cancer genome atlas\n<\/p><p>ID: Identifier\n<\/p><p>QC: Quality control\n<\/p><p>HTTPS: Hypertext transfer protocol secure\n<\/p><p>PHI: Private health information\n<\/p><p>IGV: Integrated genome viewer\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h2>\n<p>SMG and HPJ drafted the manuscript. SMG wrote the software, setup the website implementation and wrote the user guide. SMG and HPJ conceived and designed the project. Both authors read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>We would like to acknowledge Rebecca Kim for help with design for sample acquisition and processing; Jason Buenrostro and Erik Hopmans for help with design for sequencing libraries and flow cells; Jennifer Palm for her assistance with graphics and Curt Palm for his help with server configuration for the MendeLIMS demo site.\n<\/p><p>This work was supported by the following grants from the NIH: RC2HG005570 to S.M.G and H.P.J., DK56339 to H.P.J, 2P01HG000205 to S.M.G. and H.P.J. HPJ was supported by a Research Scholar Grant (RSG-13-297-01-TBG) from the American Cancer Society. In addition, H.P.J. received support from the Doris Duke Clinical Foundation, the Clayville Foundation, the Liu BieJu Cha and Family Fellowship in Cancer and the Howard Hughes Medical Foundation.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-GSLABEd-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GSLABEd_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.geospiza.com\/Products\/LabEdition.shtml\" target=\"_blank\">\"GeneSifter Lab Edition\"<\/a>. PerkinElmer, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.geospiza.com\/Products\/LabEdition.shtml\" target=\"_blank\">http:\/\/www.geospiza.com\/Products\/LabEdition.shtml<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GeneSifter+Lab+Edition&rft.atitle=&rft.pub=PerkinElmer%2C+Inc&rft_id=http%3A%2F%2Fwww.geospiza.com%2FProducts%2FLabEdition.shtml&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Progeny-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Progeny_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.progenygenetics.com\/lims\/\" target=\"_blank\">\"Progeny LIMS\"<\/a>. Progeny Genetics, LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.progenygenetics.com\/lims\/\" target=\"_blank\">http:\/\/www.progenygenetics.com\/lims\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Progeny+LIMS&rft.atitle=&rft.pub=Progeny+Genetics%2C+LLC&rft_id=http%3A%2F%2Fwww.progenygenetics.com%2Flims%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Clarity-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Clarity_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.genologics.com\/clarity-lims\/\" target=\"_blank\">\"Clarity LIMS\"<\/a>. GenoLogics Life Sciences Software, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.genologics.com\/clarity-lims\/\" target=\"_blank\">http:\/\/www.genologics.com\/clarity-lims\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Clarity+LIMS&rft.atitle=&rft.pub=GenoLogics+Life+Sciences+Software%2C+Inc&rft_id=http%3A%2F%2Fwww.genologics.com%2Fclarity-lims%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BathBons11-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BathBons11_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bath, Timothy G.; Bozdag, Selcuk; Afzal, Vackar; Crowther, Daniel (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3113716\" target=\"_blank\">\"LimsPortal and BonsaiLIMS: development of a lab information management system for translational medicine\"<\/a>. <i>Source Code for Biology and Medicine<\/i> <b>6<\/b>: 9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1751-0473-6-9\" target=\"_blank\">10.1186\/1751-0473-6-9<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3113716\/\" target=\"_blank\">PMC3113716<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21569484\" target=\"_blank\">21569484<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3113716\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3113716<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LimsPortal+and+BonsaiLIMS%3A+development+of+a+lab+information+management+system+for+translational+medicine&rft.jtitle=Source+Code+for+Biology+and+Medicine&rft.aulast=Bath%2C+Timothy+G.%3B+Bozdag%2C+Selcuk%3B+Afzal%2C+Vackar%3B+Crowther%2C+Daniel&rft.au=Bath%2C+Timothy+G.%3B+Bozdag%2C+Selcuk%3B+Afzal%2C+Vackar%3B+Crowther%2C+Daniel&rft.date=2011&rft.volume=6&rft.pages=9&rft_id=info:doi\/10.1186%2F1751-0473-6-9&rft_id=info:pmc\/PMC3113716&rft_id=info:pmid\/21569484&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3113716&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ViksnaPASS07-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ViksnaPASS07_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Viksna, J.; Celms, E.; Opmanis, M.; Podnieks, K.; Rucevskis, P.; Zarins, A.; Barrett, A.; Neogi, S.G.; Krestyaninova, M.; McCarthy, M.I.; Brazma, A.; Sarkans, U. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798\" target=\"_blank\">\"PASSIM: An open source software system for managing information in biomedical studies\"<\/a>. <i>BMC Bioinformatics<\/i> <b>8<\/b>: 52. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-8-52\" target=\"_blank\">10.1186\/1471-2105-8-52<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1803798\/\" target=\"_blank\">PMC1803798<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17291344\" target=\"_blank\">17291344<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1803798<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PASSIM%3A+An+open+source+software+system+for+managing+information+in+biomedical+studies&rft.jtitle=BMC+Bioinformatics&rft.aulast=Viksna%2C+J.%3B+Celms%2C+E.%3B+Opmanis%2C+M.%3B+Podnieks%2C+K.%3B+Rucevskis%2C+P.%3B+Zarins%2C+A.%3B+Barrett%2C+A.%3B+Neogi%2C+S.G.%3B+Krestyaninova%2C+M.%3B+McCarthy%2C+M.I.%3B+Brazma%2C+A.%3B+Sarkans%2C+U.&rft.au=Viksna%2C+J.%3B+Celms%2C+E.%3B+Opmanis%2C+M.%3B+Podnieks%2C+K.%3B+Rucevskis%2C+P.%3B+Zarins%2C+A.%3B+Barrett%2C+A.%3B+Neogi%2C+S.G.%3B+Krestyaninova%2C+M.%3B+McCarthy%2C+M.I.%3B+Brazma%2C+A.%3B+Sarkans%2C+U.&rft.date=2007&rft.volume=8&rft.pages=52&rft_id=info:doi\/10.1186%2F1471-2105-8-52&rft_id=info:pmc\/PMC1803798&rft_id=info:pmid\/17291344&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1803798&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-VanRossumSLIMS10-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-VanRossumSLIMS10_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Van Rossum, T.; Tripp, B.; Daley, D. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515\" target=\"_blank\">\"SLIMS: A user-friendly sample operations and inventory management system for genotyping labs\"<\/a>. <i>Bioinformatics<\/i> <b>26<\/b> (14): 1808-1810. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtq271\" target=\"_blank\">10.1093\/bioinformatics\/btq271<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2894515\/\" target=\"_blank\">PMC2894515<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20513665\" target=\"_blank\">20513665<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2894515<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SLIMS%3A+A+user-friendly+sample+operations+and+inventory+management+system+for+genotyping+labs&rft.jtitle=Bioinformatics&rft.aulast=Van+Rossum%2C+T.%3B+Tripp%2C+B.%3B+Daley%2C+D.&rft.au=Van+Rossum%2C+T.%3B+Tripp%2C+B.%3B+Daley%2C+D.&rft.date=2010&rft.volume=26&rft.issue=14&rft.pages=1808-1810&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtq271&rft_id=info:pmc\/PMC2894515&rft_id=info:pmid\/20513665&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2894515&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScholtalbersGal13-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScholtalbersGal13_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scholtalbers, J.; Rossler, J.; Sorn, P.; de Graaf, J.; Boisguerin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". <i>Bioinformatics<\/i> <b>29<\/b> (9): 1233-1234. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt115\" target=\"_blank\">10.1093\/bioinformatics\/btt115<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23479349\" target=\"_blank\">23479349<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy+LIMS+for+next-generation+sequencing&rft.jtitle=Bioinformatics&rft.aulast=Scholtalbers%2C+J.%3B+Rossler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisguerin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.au=Scholtalbers%2C+J.%3B+Rossler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisguerin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.date=2013&rft.volume=29&rft.issue=9&rft.pages=1233-1234&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt115&rft_id=info:pmid\/23479349&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NixNext10-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NixNext10_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Nix, David A.; Di Sera, Tonya L.; Dalley, Brian K.; Milash, Brett A.; Cundick, Robert M.; Quinn, Kevin S.; Courdy, Samir J. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281\" target=\"_blank\">\"Next generation tools for genomic data generation, distribution, and visualization\"<\/a>. <i>BMC Bioinformatics<\/i> <b>11<\/b>: 455. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-11-455\" target=\"_blank\">10.1186\/1471-2105-11-455<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2944281\/\" target=\"_blank\">PMC2944281<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20828407\" target=\"_blank\">20828407<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2944281<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Next+generation+tools+for+genomic+data+generation%2C+distribution%2C+and+visualization&rft.jtitle=BMC+Bioinformatics&rft.aulast=Nix%2C+David+A.%3B+Di+Sera%2C+Tonya+L.%3B+Dalley%2C+Brian+K.%3B+Milash%2C+Brett+A.%3B+Cundick%2C+Robert+M.%3B+Quinn%2C+Kevin+S.%3B+Courdy%2C+Samir+J.&rft.au=Nix%2C+David+A.%3B+Di+Sera%2C+Tonya+L.%3B+Dalley%2C+Brian+K.%3B+Milash%2C+Brett+A.%3B+Cundick%2C+Robert+M.%3B+Quinn%2C+Kevin+S.%3B+Courdy%2C+Samir+J.&rft.date=2010&rft.volume=11&rft.pages=455&rft_id=info:doi\/10.1186%2F1471-2105-11-455&rft_id=info:pmc\/PMC2944281&rft_id=info:pmid\/20828407&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2944281&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KentTheHuman02-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KentTheHuman02_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kent, W.J.; Sugnet, C.W; Furey, T.S.; Roskin, K.M.; Pringle, T.H.; Zahler, A.M.; Haussler, D. (2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604\" target=\"_blank\">\"The human genome browser at UCSC\"<\/a>. <i>Genome Research<\/i> <b>12<\/b> (6): 996-1006. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.229102\" target=\"_blank\">10.1101\/gr.229102<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC186604\/\" target=\"_blank\">PMC186604<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12045153\" target=\"_blank\">12045153<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC186604<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+human+genome+browser+at+UCSC&rft.jtitle=Genome+Research&rft.aulast=Kent%2C+W.J.%3B+Sugnet%2C+C.W%3B+Furey%2C+T.S.%3B+Roskin%2C+K.M.%3B+Pringle%2C+T.H.%3B+Zahler%2C+A.M.%3B+Haussler%2C+D.&rft.au=Kent%2C+W.J.%3B+Sugnet%2C+C.W%3B+Furey%2C+T.S.%3B+Roskin%2C+K.M.%3B+Pringle%2C+T.H.%3B+Zahler%2C+A.M.%3B+Haussler%2C+D.&rft.date=2002&rft.volume=12&rft.issue=6&rft.pages=996-1006&rft_id=info:doi\/10.1101%2Fgr.229102&rft_id=info:pmc\/PMC186604&rft_id=info:pmid\/12045153&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC186604&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ThorvaldsdottirInt13-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ThorvaldsdottirInt13_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thorvaldsdottir, H.; Robinson, J.T.; Mesirov, J.P. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3603213\" target=\"_blank\">\"Integrative Genomics Viewer (IGV): high-performance genomics data visualization and exploration\"<\/a>. <i>Briefings in Bioinformatics<\/i> <b>14<\/b> (2): 178-192. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbib%2Fbbs017\" target=\"_blank\">10.1093\/bib\/bbs017<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3603213\/\" target=\"_blank\">PMC3603213<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22517427\" target=\"_blank\">22517427<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3603213\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3603213<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrative+Genomics+Viewer+%28IGV%29%3A+high-performance+genomics+data+visualization+and+exploration&rft.jtitle=Briefings+in+Bioinformatics&rft.aulast=Thorvaldsdottir%2C+H.%3B+Robinson%2C+J.T.%3B+Mesirov%2C+J.P.&rft.au=Thorvaldsdottir%2C+H.%3B+Robinson%2C+J.T.%3B+Mesirov%2C+J.P.&rft.date=2013&rft.volume=14&rft.issue=2&rft.pages=178-192&rft_id=info:doi\/10.1093%2Fbib%2Fbbs017&rft_id=info:pmc\/PMC3603213&rft_id=info:pmid\/22517427&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3603213&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PallaQT14-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PallaQT14_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Palla, P.; Frau, G.; Vargiu, L.; Rodriguez-Tome, P. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4015218\" target=\"_blank\">\"QTREDS: A Ruby on Rails-based platform for omics laboratories\"<\/a>. <i>BMC Bioinformatics<\/i> <b>15<\/b> (Suppl 1): S13. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-15-S1-S13\" target=\"_blank\">10.1186\/1471-2105-15-S1-S13<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4015218\/\" target=\"_blank\">PMC4015218<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24564791\" target=\"_blank\">24564791<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4015218\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4015218<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=QTREDS%3A+A+Ruby+on+Rails-based+platform+for+omics+laboratories&rft.jtitle=BMC+Bioinformatics&rft.aulast=Palla%2C+P.%3B+Frau%2C+G.%3B+Vargiu%2C+L.%3B+Rodriguez-Tome%2C+P.&rft.au=Palla%2C+P.%3B+Frau%2C+G.%3B+Vargiu%2C+L.%3B+Rodriguez-Tome%2C+P.&rft.date=2014&rft.volume=15&rft.issue=Suppl+1&rft.pages=S13&rft_id=info:doi\/10.1186%2F1471-2105-15-S1-S13&rft_id=info:pmc\/PMC4015218&rft_id=info:pmid\/24564791&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4015218&rfr_id=info:sid\/en.wikipedia.org:Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210218\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.339 seconds\nReal time usage: 0.369 seconds\nPreprocessor visited node count: 10336\/1000000\nPreprocessor generated node count: 30305\/1000000\nPost\u2010expand include size: 90292\/2097152 bytes\nTemplate argument size: 28817\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 329.425 1 - -total\n 75.68% 249.323 1 - Template:Reflist\n 64.61% 212.857 11 - Template:Citation\/core\n 52.13% 171.736 8 - Template:Cite_journal\n 18.86% 62.123 1 - Template:Infobox_journal_article\n 18.07% 59.518 1 - Template:Infobox\n 17.39% 57.277 3 - Template:Cite_web\n 10.89% 35.879 80 - Template:Infobox\/row\n 9.72% 32.028 23 - Template:Citation\/identifier\n 4.53% 14.915 11 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7873-0!*!0!!en!5!* and timestamp 20181213210218 and revision id 23416\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing\">https:\/\/www.limswiki.org\/index.php\/Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","0ea139cdae16f62db0066623f3ff445c_images":["https:\/\/www.limswiki.org\/images\/1\/11\/Fig1_Grimes_BMCBioinformatics2014_15.jpg","https:\/\/www.limswiki.org\/images\/8\/82\/Fig2_Grimes_BMCBioinformatics2014_15.jpg","https:\/\/www.limswiki.org\/images\/4\/4d\/Fig3_Grimes_BMCBioinformatics2014_15.jpg","https:\/\/www.limswiki.org\/images\/c\/c7\/Fig4_Grimes_BMCBioinformatics2014_15.jpg"],"0ea139cdae16f62db0066623f3ff445c_timestamp":1544734938,"a5e9f45d3e7fa0bbfb4350821efaa31e_type":"article","a5e9f45d3e7fa0bbfb4350821efaa31e_title":"Launching genomics into the cloud: Deployment of Mercury, a next generation sequence analysis pipeline (Reid et al. 2014)","a5e9f45d3e7fa0bbfb4350821efaa31e_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline","a5e9f45d3e7fa0bbfb4350821efaa31e_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Launching genomics into the cloud: Deployment of Mercury, a next generation sequence analysis pipeline\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nLaunching genomics into the cloud: Deployment of Mercury, a next generation sequence analysis pipelineJournal\n \nInternational Journal of Environmental Research and Public HealthAuthor(s)\n \nReid, Jeffrey G.; Carroll, Andrew; Veeraraghavan, Narayanan; Dahdouli, Mahmoud; Sundquist, Andreas;\r\nEnglish, Adam; Bainbridge, Matthew; White, Simon; Salerno, William; Buhay, Christian; Yu, Fuli;\r\nMuzny, Donna; Daly, Richard; Duyk, Geoff; Gibbs, Richard A. Boerwinkle, EricAuthor affiliation(s)\n \nBaylor College of Medicine; DNAnexusPrimary contact\n \nEmail: jgreid@bcm.eduYear published\n \n2014Volume and issue\n \n15Page(s)\n \n30DOI\n \n10.1186\/1471-2105-15-30ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 2.0 GenericWebsite\n \nhttp:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/1471-2105-15-30Download\n \nhttp:\/\/bmcbioinformatics.biomedcentral.com\/track\/pdf\/10.1186\/1471-2105-15-30 (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Results and discussion \n\n3.1 Local workflow management \n3.2 Cloud workflow management \n3.3 Performance \n3.4 Data management, sharing, reliability, and compliance \n3.5 Comparison to existing methods \n3.6 Future directions \n\n\n4 Conclusions \n5 Methods \n\n5.1 Reads and read qualities \n5.2 Mapping and BAM generation \n\n\n6 Variant calling \n\n6.1 Technical validation and reproducibility \n6.2 Annotation \n6.3 Metadata and LIMS \n6.4 Quality assurance, quality control, and error handling \n6.5 Data security and compliance \n\n\n7 Abbreviations \n8 Declarations \n\n8.1 Acknowledgements \n8.2 Competing interests \n8.3 Authors\u2019 contributions \n\n\n9 References \n10 Notes \n\n\n\nAbstract \nBackground: Massively parallel DNA sequencing generates staggering amounts of data. Decreasing cost, increasing throughput, and improved annotation have expanded the diversity of genomics applications in research and clinical practice. This expanding scale creates analytical challenges: accommodating peak compute demand, coordinating secure access for multiple analysts, and sharing validated tools and results.\nResults:\nTo address these challenges, we have developed the Mercury analysis pipeline and deployed it in local hardware and the Amazon Web Services cloud via the DNAnexus platform. Mercury is an automated, flexible, and extensible analysis workflow that provides accurate and reproducible genomic results at scales ranging from individuals to large cohorts.\nConclusions:\nBy taking advantage of cloud computing and with Mercury implemented on the DNAnexus platform, we have demonstrated a powerful combination of a robust and fully validated software pipeline and a scalable computational resource that, to date, we have applied to more than 10,000 whole genome and whole exome samples.\nKeywords: NGS data, Variant calling, Annotation, Clinical sequencing, Cloud computing \n\nBackground \nWhole exome capture sequencing (WES) and whole genome sequencing (WGS) using next generation sequencing (NGS) technologies[1] have emerged as compelling paradigms for routine clinical diagnosis, genetic risk prediction, and patient management.[2] Large numbers of laboratories and hospitals routinely generate terabytes of NGS data, shifting the bottleneck in clinical genetics from DNA sequence production to DNA sequence analysis. Such analysis is most prevalent in three common settings: first, in a clinical diagnostics laboratory (e.g. Baylor\u2019s Whole Genome Laboratory http:\/\/www.bcm.edu\/\u200bgeneticlabs\/\u200b) testing single patients or families with presumed heritable disease; second, in a cancer-analysis setting where the unit of interest is either a normal-tumor tissue pair or normal-primary tumor-recurrence trio[3]; and third, in biomedical research studies sequencing a sample of well-phenotyped individuals. In each case, the input is a DNA sample of appropriate quality having a unique identification number, appropriate informed consent, and relevant clinical and phenotypic information.\nAs these new samples are sequenced, the resulting data is most effectively examined in the context of petabases of existing DNA sequence and the associated meta-data. Such large-scale comparative genomics requires new sequence data to be robustly characterized, consistently reproducible, and easily shared among large collaborations in a secure manner. And while data-management and information technologies have adapted to the processing and storage requirements of emerging sequencing technologies (e.g., the CRAM specification[4]), it is less certain that appropriate informative software interfaces will be made available to the genomics and clinical genetics communities. One element bridging the technology gap between the sequencing instrument and the scientist or clinician is a validated data processing pipeline that takes raw sequencing reads and produces an annotated personal genome ready for further analysis and clinical interpretation.\nTo address this need, we have designed and implemented Mercury, an automated approach that integrates multiple sequence analysis components across many computational steps, from obtaining patient samples to providing a fully annotated list of variant sites for clinical applications. Mercury fully integrates new software with existing routines (e.g., Altas2[5]) and provides the flexibility necessary to respond to changing sequencing technologies and the rapidly increasing volume of relevant data. Mercury has been implemented on both local infrastructure and in a cloud computing platform provided by DNAnexus using Amazon Web Services (AWS). While there are other NGS analysis pipelines, some of which have even been implemented in the cloud[6], the combination of Mercury and DNAnexus together provide for the first time a fully integrated genomic analysis resource that can serve the full spectrum of users.\n\nResults and discussion \nFigure 1 provides an overview of the Mercury data processing pipeline. Source information includes sample and project management data and the characteristics of library preparation and sequencing. This information enters the pipeline either directly from the user or from a laboratory information management system (LIMS). The first step, generating sequencing reads, is based on the vendor\u2019s primary analysis software, which generates sequence reads and base-call confidence values (qualities). The second step maps the reads and qualities to the reference genome using a standard mapping tool, such as BWA[7][8], producing a BAM[9] (binary alignment\/map) file. The third step produces a \u201cfinished\u201d BAM that includes sorting, duplicate marking, indel realignment, base quality recalibration, and indexing (using a combination of tools including SAMtools[9], Picard (http:\/\/picard.sourceforge.net), and GATK[10]). The fourth step in Mercury uses the Atlas2 suite[5][11] (Atlas-SNP and Atlas-indel) to call variants and produce a variant file (VCF). The fifth step adds biological and functional annotation and formats the variant lists for delivery. Each step is described in detail in the Methods section, as is the flow of information between steps.\n\n\n\n\n\n\n\n\n\n Figure 1. Mercury Data Flow. 1) Sequencing Instrument raw data is passed to vendor primary analysis software to generate sequence reads and base call confidence values (qualities). 2) Reads and qualities are passed to a mapping tool (BWA) for comparison to a reference genome to determine the placement of reads on the reference (producing a BAM file). 3) Individual sequence event BAMs are merged to make a single sample-level BAM file that then is processed in preparation for variant calling. 4) Atlas-SNP and Atlas-indel are used to identify variants and produce variant files (VCF). 5) Annotation adds biological and functional information to the variant lists and formats them for delivery.\n\n\n\nMercury has been optimized for the Illumina HiSeq (Illumina, Inc.; San Diego, CA) platform, but the generalized workflow framework is adaptable to other settings. The entire pipeline has been implemented both locally and in a cloud computing environment. All relevant code and documentation are freely available online (https:\/\/www.hgsc.bcm.edu\/\u200bcontent\/\u200bmercury) and the scalable cloud solution is available within the DNAnexus library (http:\/\/www.dnanexus.com\/\u200b). Sensible default parameters have already been determined so that researchers and clinicians can reliably analyze their data with Mercury without needing to configure any of the constituent programs or obtaining access to large computational resources, and they can do so in a manner compliant with multiple regulatory frameworks.\n\nLocal workflow management \nImplementing a robust analysis framework that incorporates a heterogeneous collection of software tools presents many challenges. Running disparate software modules with varying inputs and outputs that depend on each other\u2019s results requires appropriate error checking and centralized output logging. We therefore developed a simple yet extensible workflow management framework, Valence (http:\/\/sourceforge.net\/\u200bprojects\/\u200bvalence\/\u200b), that manages the various steps and dependencies within Mercury and handles internal and external pipeline communication. This formal approach to workflow management helps maximize computational resource utilization and seamlessly guides the data from the sequencing instrument to an annotated variant file ready for clinical interpretation and downstream analysis.\nValence parses and executes an analysis protocol described in XML format with each step treated as either an action or a procedure. An action is defined as a direct call to the system to submit a program or script to the job scheduler for execution; a procedure is defined a collection of actions, which is itself a workflow. This design allows the user to easily add, remove, and modify the steps of any analysis protocol. A protocol description for a specific step must include the required cluster resources, any dependencies on other steps, and a command element that describes how to execute the program or script. To ensure that the XML wrappers are applicable to any run, the command is treated as a string template that allows XML variables to be substituted into the command prior to execution. Thus, a single XML wrapper describing how to run a program can be applied to different inputs. Valence can be deployed on any cluster with a job scheduler (e.g., PBS, LSF, SGE), implementing a database to track both the job (the collection of all the steps in a protocol to be executed) and the status (\u201cNot Started,\u201d \u201cRunning,\u201d \u201cFinished,\u201d \u201cFailed\u201d) of any action associated with the job.\nMercury users can easily incorporate new analysis tools into an existing pipeline. For example, we recently expanded the scope of our pipeline to include Tiresias (https:\/\/sourceforge.net\/\u200bprojects\/\u200btiresias\/\u200b), a structural variant caller focused on mobile elements, and ExCID (https:\/\/github.com\/\u200bcbuhay\/\u200bExCID), an exome coverage analysis tool designed to provide clinical reports on under-covered regions. To incorporate Tiresias and ExCID into the Mercury pipeline, we needed only to specify the compute requirements and add the appropriate command to the existing XML workflow definition; Valence then automatically handles all data passing, logging, and error reporting.\n\nCloud workflow management \nMercury has been instantiated in the cloud via the DNAnexus platform (utilizing AWS\u2019s EC2 and S3). DNAnexus provides a layer of abstraction that simplifies development, execution, monitoring, and collaboration on a cloud infrastructure. The constituent steps of the Mercury pipeline take the form of discrete \u201capplets,\u201d which are then linked to form a workflow within the DNAnexus platform infrastructure. Using the workflow construction GUI, one can add applets (representing each step) to the workflow and create a dependency graph by linking the inputs and outputs of subsequent applets. Inputs are provided to an instance of the workflow, and the entire workflow is run within the cloud infrastructure. The various steps within the workflow are then executed based on the dependency graph. As with Valence, individual applets can be configured to run with a specific set of computational resource requirements such as memory, local disk space, and number of cores and processors. We are currently working to merge the local and cloud infrastructure elements by integrating the upload agent into Valence, allowing Valence to trigger a DNAnexus workflow once all the data is successfully uploaded. Such coordination would serve to transparently support analysis bursts.\nThe Mercury pipeline within DNAnexus comprises code that uses the DNAnexus command-line interface to instantiate the pipeline in the cloud. The Mercury code for DNAnexus is executed on a local terminal. For example, one may provide a list of sample FASTQ files and sample meta-data locations to Mercury, at which point Mercury uploads the data and instantiates the predefined workflow within DNAnexus. On average, on a 100 Mbps connection, we were able to upload at a rate of ~14 MB\/sec. We were able to parallelize this uploading process, yielding an effective upload rate of ~90 MB\/sec. The size of a typical FASTQ file from WES with 150X coverage has a compressed (bzip2) file size of approximately 3 GB. Uploading such a file from a local server took less than five minutes. After sample data are uploaded to the DNAnexus environment, the workflow is instantiated in the cloud.\nProgress can be monitored online using the DNAnexus GUI (Figure 2) or locally through the Mercury monitoring code. To achieve full automation, the monitoring code can be made a part of a local process to poll for analysis status at regular intervals and start analysis of new sequences automatically upon completion of sequencing. When the Mercury monitoring code detects successful completion of analysis an email notification is sent out. The results can either be downloaded to the local server or the user can view various tracks and data with a native genome browser.\n\n\n\n\n\n\n\n\n\n Figure 2. Workflow monitoring in DNAnexus. The GUI for applet monitoring displays the progress as a Gantt chart. The left panel lists the various steps including the parallelization steps with each row corresponding to a compute instance. A particular step can be clicked to determine the exact inputs and output or logs of execution for that step. Here we show a snapshot of the webpage displaying the progress of execution for the NA12878 exome analysis. \n\n\n\nPerformance \nTurn-around time for raw data generation on most NGS platforms is already considered long for many clinical applications, so minimizing analysis time is a primary goal of the Mercury pipeline. By maintaining a high-performance computing cluster consisting of hundreds of 8-core, 48 GB RAM nodes and introducing Mercury into the sequencing pipeline, we can minimize wait times by ensuring that compute resources are always available for all sequence events as the instrument produces the data. To match compute resources to production requirements, we carefully monitor the run times (and RAM and CPU requirements) of each step in the Mercury pipeline. Table 1 describes the pipeline for each compute-intensive step, from the generation of reads and qualities from raw data (bcl to FASTQ) to generation of post-processed annotated variant files (VCFs). Resource requirements for each step are given in terms of fraction of an 8-core node (CPU) and RAM allocated. Note that some steps under-utilize available CPUs because they require most (or all) of the RAM available on a given node. For data generated via WES from human samples, the Mercury pipeline requires less than 36 hours of wall-clock time and 15 node-hours (i.e., the equivalent of one whole node fully dedicated to processing for 15 hours). Run times and resource requirements will vary with data type, reference genome, and computing hardware configuration.\n\n\n\n\n\n\n\nTable 1. Mercury computational resource requirements\n\n\n\n\nData gen\n\nAlignment\n\nBAM finishing\n\nVariants\n\nAnno\n\n\n\n\nBCL to FastQ\n\nBWA align\n\nBWA sample\n\nMates, Dupe, Stats\n\nCap & Cvrg Metrics\n\nGATK indel targets\n\nGATK indel realign\n\nGATK recal\n\nBAM valid\n\nAtlas SNP\n\nAtlas Indel\n\nCassandra\n\n\nNodes\n\n1\n\n1\n\n0.333\n\n0.5\n\n0.125\n\n1\n\n0.333\n\n1\n\n0.125\n\n0.167\n\n0.167\n\n0.167\n\n\nRAM\n\n48\n\n48\n\n15\n\n28\n\n3\n\n48\n\n14\n\n32\n\n4\n\n7\n\n7\n\n8\n\n\nHours\n\n3.62\n\n1.84\n\n1.38\n\n3.39\n\n1.30\n\n0.28\n\n2.25\n\n3.04\n\n0.75\n\n9.00\n\n7.51\n\n1.71\n\n\nNode*Hrs\n\n3.62\n\n1.84\n\n0.46\n\n1.70\n\n0.16\n\n0.28\n\n0.75\n\n3.04\n\n0.09\n\n1.50\n\n1.25\n\n0.29\n\n\nAll estimates are approximate for whole exome and light-skim whole genome (~10-20 Gbp of data) sequenced on Illumina HiSeq and processed with the most recent versions of RTA and Casava. Nodes are 8-core, 48 GB RAM, with ~3 GHz Intel CPUs and ~1 TB of local scratch disk. Steps include all aspects of the pipeline from building reads and qualities (fastQ) from raw data (bcl files), through alignment and BAM generation using the BWA aligner, and BAM finishing with GATK post-processing and duplicate marking, capture and coverage metric calculation, and BAM file validation, finally producing variants from the Atlas2 variant calling suite with annotations from our annotator, Cassandra.\n\n\n\nAfter porting each element of the Mercury pipeline into the DNAnexus environment, the tools (i.e., \u201capps\u201d) can be run on the cloud in environments with different CPU, RAM, disk, and bandwidth resources to optimize wall-clock time and cost-efficiency. Parallelization within a pipeline reduces the time for a single run, which is useful for quick development cycles or time-sensitive applications. In addition, many parallel pipelines can be run simultaneously. The current peak usage for the Mercury pipeline on DNAnexus is approximately 12,000 cores. This throughput is a small fraction of the theoretical maximum that could be achieved in AWS. For the standard implementation of Mercury in the cloud analyzing a validation exome (Hapmap sample NA12878), the total wall time to produce an annotated variant call VCF (starting with paired-end FASTQ files) is approximately one day.\n\nData management, sharing, reliability, and compliance \nLarge projects present a large data management challenge. For example, the Baylor-Hopkins Center for Mendelian Genomics has generated WES data from approximately 2,000 samples and the Human Genome Sequencing Center at Baylor College of Medicine has generated more than 10,000 WES and 3,500 WGS data from research samples from the CHARGE consortium. \nAs a pilot study, we processed 1,000 WES data sets\u2013approximately 80 terabytes of genomic data\u2013from the Center for Mendelian Genomics using Mercury in DNAnexus. Using multi-threaded uploads we were able to deliver data into the cloud at an average rate of ~960 exomes per day. Once uploaded, the data is analyzed with Mercury, and the resulting variants can be accessed for further analysis via a web GUI. Data can also be tagged, and these tags can be filtered or retrieved. Runs of individual pipelines and tools can be queried in a similar way.[12] As datasets become larger, multi-site collaborative consortiums play an increasingly important role in contemporary biomedical research. A major advantage of cloud computing over local computing is that cloud storage can be shared across multiple organizations. Instead of each collaborator maintaining a local copy of the data and working in isolation, cloud users can be given appropriate access permissions so some researchers can view and download the results, others can run analyses on the data and build tools, and those with administrative privileges can determine access to the project. This data paradigm is the only feasible approach to giving patients meaningful access to their own genomic data.\n\nComparison to existing methods \nA number of other tools and services provide similar functionality to Mercury on DNAnexus, with differing approaches to extensibility, ease of use for non-programmers, support for local or cloud infrastructures, and software available by default (Table 2). For example, the academic service Galaxy primarily focuses on extensibility and building a developer community.[13][14][15] Seven Bridges is a commercial service that combines a few fixed pipelines with a visually distinctive workflow editor. Chipster is an academic service that packages a variety of NGS tools in addition to a variety of microarray tools and combines these with visualization of data summaries and QC metrics.[16] Anduril is designed to manage a local cluster and contains packages for a variety of tasks, including alignment and variant calling as well as image analysis and flow cytometry, which are not addressed by the other cloud services surveyed.[17] With respect to the software used in sequence production pipelines, Mercury is most distinguished by its Atlas variant caller and the extensive annotations provided by its Cassandra annotation tool (https:\/\/www.hgsc.bcm.edu\/\u200bsoftware\/\u200bcassandra).\n\n\n\n\n\n\n\nTable 2. A feature summary of Mercury in DNAnexus and similar tools and services\n\n\n\n\n Mercury in DNAnexus\n\n Galaxy\n\n Seven bridges\n\n Chipster\n\n Anduril\n\n\nMapper\n\nBWA\n\nNo canonical (many)\n\nBWA\n\nBWA\n\nBWA\/Bowtie\n\n\nVariant caller\n\nAtlas2\n\nNo canonical (many)\n\nGATK\n\nSamtools mpileup\n\nSamtools\/GATK, Varscan\n\n\nAnnotation\n\nCassandra\n\nsnpEff, AlleleFrequency\n\nsnpEff, ANNOVAR\n\nBioconductor\n\nANNOVAR\n\n\nVisualization\n\nBuilt-in browser\n\nBuilt-in browser\n\nLinks to IGV\n\nBuilt-in browser\n\nNo\n\n\nRuns on local hardware\n\nNo*\n\nYes\n\nNo\n\nYes\n\nYes\n\n\nRuns on cloud infrastructure\n\nYes\n\nYes\n\nYes\n\nJobs queued to public server\n\nNo\n\n\nHIPAA compliant\n\nYes\n\nNo\n\nNo\n\nNo\n\nNot Applicable (local only)\n\n\nRequires setup configuration\n\nNo\n\nYes\n\nNo\n\nYes\n\nYes\n\n\nCan add tools independently\n\nYes\n\nYes\n\nBy request\n\nNo\n\nNo\n\n\nData sharing & collaboration\n\nYes\n\nYes\n\nNo\n\nNo\n\nNo\n\n\n*Mercury can be run on local hardware via Valence, but does not include all of the security and data sharing features of Mercury in DNAnexus.\n\n\n\nFuture directions \nAs genomic studies transition from extensive us of whole exome capture sequencing methods to an emphasis on whole genome sequencing[18], we can take advantage of Mercury\u2019s flexibility to adapt to shifting research priorities. Currently, the sequencing community uses whole exome sequencing because of its comparatively low cost and because it enriches for biological signals that are readily interpretable. However, with changing price structures, the advantages of eliminating the capture step in the laboratory, improvements in lossless compression of sequence data, and improved annotation of the non-protein coding region of the genome (e.g., genome.ucsc.edu\/ENCODE\/), whole genome sequencing may become a more appealing option. Early adopters of whole genome sequencing will certainly include medical sequencing for research and diagnostic purposes. Therefore, a high priority for Mercury is the continual updates to the annotation information using web-based databases. Such web-based annotation updates can incorporate the latest developments related to genome function in near real-time.\nClinically relevant annotation is of particular interest in the growing area of cancer somatic mutation. Currently, disease-related annotation considers inherited disease alleles gleaned from OMIM (http:\/\/www.omim.org), HGMD (http:\/\/www.hgmd.org; assuming the user has the appropriate license to use this data), and genome-wide association studies (http:\/\/www.genome.gov\/\u200bgwastudies). Similarly informative databases, such as the Catalogue of Somatic Mutations in Cancer (COSMIC; http:\/\/www.sanger.ac.uk\/\u200bgenetics\/\u200bCGP\/\u200bcosmic\/\u200b), are emerging and will need to be fully vetted and incorporated into Cassandra. The goal of Mercury is to provide a simple solution for end-to-end sequence analysis so that non-expert users can obtain a list of annotated and prioritized variants as rapidly as possible, and so that expert users can augment and modify the pipeline to meet specialized needs.\n\nConclusions \nThe long-anticipated NGS data deluge[19] has now arrived. The first personal NGS genome was published in 2007[4], and today we estimate that the number of available exomes and genomes approaches one hundred thousand. It is equally impressive that the application of NGS to biomedical research and clinical medicine is rapidly becoming standard. Such applications are driven by the utility of sequence data, as demonstrated by a number of instances where DNA sequencing has been used not only for diagnostic purposes but also to reveal more efficacious therapies.[18]\nBy taking advantage of cloud computing and with Mercury implemented on the DNAnexus platform, we have demonstrated a powerful combination of a robust and fully validated software pipeline and a scalable computational resource. To date, we have analyzed thousands of samples (using the AWS cloud: EC2 and S3), including a population study comprising 3,500 samples and 10,000 samples for which WGS and WES were generated, respectively, more than 1,000 Mendelian disease cases shared with data consumers all over the world, and smaller projects such as 50 exome trios and 30 cancer WGS tumor\/normal pairs. To our knowledge, these projects represent the largest genomic analysis effort to be realized in the cloud to date. They presage a wave of genomic computing that will transform how genomic data is analyzed and delivered to the scientific community, into clinical practice, and eventually directly into the hands of patients and advocates.\n\nMethods \nReads and read qualities \nMaking reads and assigning those reads a quality metric is the only step of Mercury that relies almost exclusively upon vendor-provided software. Processing raw data files down to signal intensities and then taking the signal data through the demultiplexing (i.e., assigning a sequence read to an individual barcoded sample), base calling, and quality scoring processes is integrally tied to the sequencing chemistry and machine instrumentation. Vendors, in collaboration with the user community, are constantly developing and improving this primary data input step. As such, the integrated Mercury pipeline is triggered by the completion of the data transfer from the instrument into the local compute environment, at which point vendor tools are used to generate a FASTQ file containing reads and qualities for each sequence event. The current version of Mercury is optimized for the Illumina HiSeq instrument, so it currently integrates with Bcl2FastQ (http:\/\/support.illumina.com\/\u200bdownloads\/\u200bbcl2fastq_\u200bconversion_\u200bsoftware_\u200b184.\u200bilmn), the Illumina analysis toolkit. Modularity of the Mercury pipeline facilitates incorporation of upgrades to Bcl2FastQ and the replacement of Bcl2FastQ with functionally similar tools as necessary. At the end of this step, data from a flow cell is broken down into individual sequence events associated with each barcode.\n\nMapping and BAM generation \nOnce reads and qualities are generated, each sequence event is mapped to a reference genome. The Burrows-Wheeler Aligner (BWA)[8] is the current preferred mapping tool for Illumina HiSeq data, though Mercury\u2019s modular design allows for alternative mapping tools. The mapping process for short reads consists of a \u201cseeding\u201d step that identifies regions of similarity between the read and reference, and then a local alignment stage that makes comparisons among reads in a region. Through this iterative process, BWA identifies the most likely placement for the reads and the differences between each read and the reference.\nOnce mapping is complete, additional processing steps are taken to improve the resulting BAM files. The data is sorted by mapping position, duplicate reads (artifacts of the sample preparation process) are flagged, and metadata summarizing data production information is added. Further downstream, base quality recalibrations and local realignment around regions with indel calls are performed to further improve the information content of variant calls. For cases in which data from a single sample spans multiple sequence events, Mercury provides the means to merge individual sequence event BAMs into a single sample-level BAM that can then be used for downstream analysis.\n\nVariant calling \nMercury is designed to provide robust variant calling for readily available whole exome sequencing data but is extensible to similar functionality for whole genome data. We have implemented Atlas2[5][11], an SNV- and indel-calling suite. Atlas2 uses a logistic regression approach to detect systematic sequencing errors using context-sensitive input variables, following three hierarchical steps: a logistic regression model to characterize systematic sequencing errors, a probability score that detects and removes errors, and a minimal heuristics filters as post-hoc quality assurances to address mapping and capture biases. Atlas2 has high SNP-discovery sensitivity and specificity when applied to the 1000 Genomes Whole Exome data.[11] The accuracy of homozygous and heterozygous SNPs are as high as 99.8% and 99.6%, respectively, when compared to the SNP array data. We have also demonstrated high quality in short indel discoveries.[5]\n\nTechnical validation and reproducibility \nTo enable clinical applications, we characterized Mercury\u2019s reproducibility of calls in gene-coding regions. We generated VCFs from two independent samples and compared the high quality variant calls from one to the variant calls of any quality from the other sample. The process was then repeated with the sample roles reversed. At the end of this process, we produced three sets of comparisons: SNPs with passing evidence in both samples, SNPs that only have passing evidence in the first sample, and those that only have passing evidence in the second sample. For illustrative purposes, reproducibility through technical replicates for eight samples meeting the basic criteria of greater than 90% of targeted bases covered at 20x or better is shown in Table 3. These observations indicate strong consistency (less than 2% variability between technical replicates), with approximately 24,000 SNPs found consistently in the overlap.\n\n\n\n\n\n\n\nTable 3. Technical replicate data for the Mercury pipeline for eight samples sequenced in duplicate (A and B)\n\n\n Sample #\n\n A only\n\n A only (%)\n\n A and B\n\n A and B (%)\n\n B only\n\n B only (%)\n\n\nHS-1011\n\n133\n\n0.561%\n\n23,320\n\n98.409%\n\n244\n\n1.029%\n\n\nHS-1015\n\n155\n\n0.542%\n\n28,312\n\n98.910%\n\n157\n\n0.548%\n\n\nHS-1016\n\n155\n\n0.644%\n\n23,752\n\n98.732%\n\n150\n\n0.624%\n\n\nHS-1017\n\n105\n\n0.456%\n\n22,767\n\n98.768%\n\n179\n\n0.777%\n\n\nHS-1018\n\n165\n\n0.693%\n\n23,531\n\n98.795%\n\n122\n\n0.512%\n\n\nHS-1019\n\n162\n\n0.682%\n\n23,441\n\n98.686%\n\n150\n\n0.631%\n\n\nHS-1020\n\n493\n\n2.041%\n\n23,518\n\n97.355%\n\n146\n\n0.604%\n\n\nHS-1021\n\n161\n\n0.681%\n\n23,188\n\n98.125%\n\n282\n\n1.193%\n\n\nAverage\n\n191\n\n0.718%\n\n23,979\n\n98.473%\n\n179\n\n0.613%\n\n\n\nWe then assessed Mercury\u2019s accuracy by evaluating the sensitivity and specificity of the Mercury SNP calls. SNP array data and whole-exome sequencing data were generated from a single sample, and seven exome technical replicates were analyzed with Mercury. The array data indicated the presence of 1,927 SNPs and 2,814 homozygous reference sites that overlapped with regions targeted by the exome capture reagent. Table 4 details Mercury\u2019s ability to recover these sites, averaging almost 99% sensitivity across the replicates. Moreover, Mercury made no SNP calls in any of the replicates for any of the SNP array homozygous reference sites, and achieved sufficient coverage to consider those sites reference homozygous.\n\n\n\n\n\n\n\nTable 4. Concordance of SNP array and Mercury data\n\n\n Replicate\n\n PASS\n\n ALL\n\n\n1\n\n1897\n\n1906\n\n\n2\n\n1900\n\n1906\n\n\n3\n\n1898\n\n1904\n\n\n4\n\n1895\n\n1904\n\n\n5\n\n1899\n\n1907\n\n\n6\n\n1904\n\n1907\n\n\n7\n\n1902\n\n1905\n\n\nAverage\n\n1899.28\n\n1905.71\n\n\nAverage %\n\n98.56\n\n98.89\n\n\nFor each exome technical replicate, we report the number of passing SNPs found by Mercury and the total number of SNPs (passing and not passing) in the VCF that overlap the SNP array design. The SNP array contains 1,927 sites that overlap the exome capture region.\n\n\n\nTo facilitate validation by others of local installations of Mercury or modified applications in the cloud, we are making available the FASTQ, BAMs and VCFs of a single individual with a known Mendelian condition.[20] These data are available at http:\/\/www.ncbi.nlm.nih.gov\/\u200bsra?\u200bterm=\u200bSRP023104.\n\nAnnotation \nMercury provides variant annotation via the Cassandra annotation suite, which describes the quality and predicted functional consequences of genomic variants, providing the biological and clinical contexts needed to assess the significance of each variant. Variants are presented to the user with all quality control metrics produced by Atlas2, the \u201cpileup string,\u201d[9] and the theoretical mappability (a measure of sequence degeneracy throughout the region) of the position. The pileup string can be visually or automatically inspected to identify biases in the variant strandedness or determine whether the variant is 5\u2019- or 3\u2019-biased within the supporting reads. Low theoretical mappability can be used to determine whether reads aligned to certain positions could have been mapped confidently or whether they were likely to have been mismapped. AnnoVar[19] is used to determine a variant\u2019s effect on both a conservative (RefSeq) and inclusive (UCSC) gene model set. This assessment also describes whether a variant changes an amino acid residue, whether it creates or removes a stop-codon, and the variant\u2019s location: near an intron-exon boundary, within an intron, within a known non-coding RNA, or in an intergenic region. The annotation also includes five algorithms that predict the deleterious nature of nonsynonymous variants.[21] Variants are additionally annotated according to their frequency and presence in multiple variant collections (e.g., dbSNP, Thousand Genomes). Lastly, variants are annotated based on functional data for the gene in which they occur. These data include the known function of the gene (e.g., Swiss-Prot), any previous association of the gene with a disease (e.g., OMIM), post-translational modifications of the gene, and the expression profile of the gene across human organs. Together, these data are applied to assess the variant quality, its minor allele frequency, and any disease association while also elucidating the potential effects of the variant within a genetic framework.\n\nMetadata and LIMS \nMercury integrates external metadata resources and inputs such as a reference genome, sequence data locations, and a capture design bed file and therefore requires an integrated LIMS within the pipeline. Our LIMS solution is partitioned into three major modules: project management (PM), sample tracking (ST), and reporting. The PM module provides tools to define the purpose of the project and aggregate samples together. Such aggregation allows project-level decisions (e.g., capture design parameters, reference genome for mapping) to be applied to all relevant samples at once. The ST module provides features for tracking samples as they move through the sequencing center pipeline. Samples are tracked via a barcode given to each sample when it is accepted by the sequencing center. Once that barcode is recorded, all lab experiments and informatics analyses performed on the sample track this barcode. The barcode-based LIMS reporting module lets users monitor a sample\u2019s progress in the pipeline and adjust the steps if necessary. The ability to see the history of the samples and sample data grouped or individually, is necessary for troubleshooting problematic samples and for developing and monitoring timelines.\nTo make Mercury portable, we provide communication \u201chooks\u201d for transferring data between Mercury and LIMS. These hooks are scripts that can be modified to query or deliver data to any metadata resource. Examples of information served to Mercury from LIMS are the reference genome and previously generated SNP array data for quality assurance purposes. By decoupling Mercury from the sample tracking data we have built a more portable and compliance-ready pipeline, thus providing increased flexibility.\n\nQuality assurance, quality control, and error handling \nIn local environments, Mercury maintains an off-the-shelf validated open-source pipeline. Such transparency can pose challenges to the strict regulatory requirements that govern clinical sequence analysis. To support analysis best practices, we provide a set of documents, data, and validation tools (detailed manuals available with the code at https:\/\/www.hgsc.bcm.edu\/\u200bcontent\/\u200bmercury) as well as publically available test data.[22] Strict version control is maintained.\nThe Mercury pipeline generates a variety of performance metrics, including the number of pass-filter bases, read mapping fractions, concordance with orthogonal array genotypes, novel SNP rates, and transition-to-transversion ratios, which allow the user to gauge the quality of the final variant call results. Genotype arrays create a fingerprint of each sample upon intake that is then compared to the sequence data to validate sample identity. The governing principle of the process is to generate the quality control (QC) data as soon as possible and deliver that QC data into the LIMS (or other meta-data aggregating resource), but not to automatically interrupt the pipeline based only on QC data. The QC data are used in two ways. First, these data are considered in the clinical sign-out process for an individual run or sample. Second, the QC data can be used to detect systemic problems in the production pipeline, such as contaminated reagents or a malfunctioning sequencing instrument.\n\nData security and compliance \nThe translational nature of modern genomic data and the scope of collaborations demand levels of data security unprecedented in our field. Based on our experience using Mercury on the DNAnexus platform to analyze thousands of samples, we have found a substantially higher security standard for data in the cloud than in most local systems. Managing the regulatory requirements of standards for handling sensitive medical data such as HIPAA, CLIA, dbGaP, and 21 CFR parts 11, 58, and 493 represents a considerable investment in compliance, security, and systems engineering expertise, and most local environments do not have the resources to maintain the required standards. By using cloud computing platforms, we are able to leverage diverse expertise, including modern, high-security data center technologies, best practices in encryption and authentication, and software system design to support clinical applications such as auditability, record retention and destruction, and reproducibility.\nAt the infrastructure level, DNAnexus uses data centers in high-security facilities with SAS-70\/SSAE-16, PCI Level 1, and FISMA Moderate certifications. At the user level, it enforces best practices such as password strength and rotation, session expiration, and client encryption. All data access is carefully controlled, logged for auditing purposes, encrypted end-to-end (both in flight and at rest), integrity-verified, and replicated in at least three physically distinct data centers to ensure against loss. Data analysis is constrained to computing nodes that are sandboxed using virtualization and encryption technologies, and are versioned to ensure reproducibility and the ability to track data provenance. The software has undergone multiple third-party audits, including penetration testing by security experts, and the overall system has been ISO 27001 certified an internationally recognized standard for secure data management processes.\n\nAbbreviations \nAWS: Amazon web services\nBAM: Binary alignment\/map\nBWA: Burrows-wheeler aligner\nCFR: Code of federal regulations\nCLIA: Clinical laboratory improvement amendments\nCOSMIC: Catalogue of somatic mutations in cancer\ndbGaP: The database of Genotypes and Phenotypes\nEC2: Amazon elastic compute cloud\nFISMA: Federal information security management act\nGbp:Giga base-pairs\nHGMD: Human gene mutation database\nHIPPA: Health insurance portability and accountability act\nIndel: Insertion\/Deletion\nLIMS: Laboratory information management system\nNGS: Next generation sequencing\nNHGRI: National human genome research institute\nOMIM: Online mendelian inheritance in man\nPM: Project management\nQC: Quality control\nS3: Amazon simple storage service\nSNP: Single nucleotide polymorphism\nST: Sample tracking\nVCF: Variant call format\nXML: Extensible markup language.\n\nDeclarations \nAcknowledgements \nThis work was supported in part by the US National Human Genome Research Institute (NHGRI) grants U54HG006542 and U54HG003273.\n\nCompeting interests \nJGR is a paid consultant of LaserGen. DNAnexus and AWS derive revenue from cloud computing services. The remaining authors declare that they have no competing interests.\n\nAuthors\u2019 contributions \nJGR conceived and designed the study, developed software, performed data analysis, and participated in the drafting and writing of the manuscript. RAG and EB conceived and designed the study and participated in the drafting and writing of the manuscript. RD and GD conceived and designed the study. AC and NV developed software, performed data analysis, and contributed to the writing of the manuscript. MD and AS developed software and contributed to the writing of the manuscript. AE, MB, SW, and FY developed software. WS participated in the coordination of the study and contributed to the writing of the manuscript. CB and DMM participated in the coordination of the study. All authors read and approved the final manuscript.\n\nReferences \n\n\n\u2191 Metzker, Michael L. (2010). \"Sequencing technologies \u2014 The next generation\". Nature Reviews Genetics 11 (1): 31\u201346. doi:10.1038\/nrg2626. PMID 19997069.   \n\n\u2191 Bainbridge, Matthew N.; Wiszniewski, Wojciech; Murdock, David R. et al. (2011). \"Whole-genome sequencing for optimized patient management\". Science Translational Medicine 3 (87): 87re3. doi:10.1126\/scitranslmed.3002243. PMC PMC3314311. PMID 21677200. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3314311 .   \n\n\u2191 Cancer Genome Atlas Research Network (2011). \"Integrated genomic analyses of ovarian carcinoma\". Nature 474 (7353): 609\u2013615. doi:10.1038\/nature10166. PMC PMC3163504. PMID 21720365. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163504 .   \n\n\u2191 4.0 4.1 Wheeler, David A.; Srinivasan, Maithreyan; Egholm, Michael et al. (2008). \"The complete genome of an individual by massively parallel DNA sequencing\". Nature 452 (7189): 872\u2013876. doi:10.1038\/nature06884. PMID 18421352.   \n\n\u2191 5.0 5.1 5.2 5.3 Challis, D.; Yu, J.; Evani, U.S. et al. (2012). \"An integrative variant analysis suite for whole exome next-generation sequencing data\". BMC Bioinformatics 13: 8. doi:10.1186\/1471-2105-13-8. PMC PMC3292476. PMID 22239737. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3292476 .   \n\n\u2191 O\u2019Driscoll, A.; Daugelaite, J.; Sleator, R.D. (2013). \"'Big data', Hadoop and cloud computing in genomics\". Journal of Biomedical Informatics 46 (5): 774-81. doi:10.1016\/j.jbi.2013.07.001. PMID 23872175.   \n\n\u2191 Li, H.; Durbin, R. (2009). \"Fast and accurate short read alignment with Burrows\u2013Wheeler transform\". Bioinformatics 25 (14): 1754\u201360. doi:10.1093\/bioinformatics\/btp324. PMC PMC2705234. PMID 19451168. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234 .   \n\n\u2191 8.0 8.1 Li, H.; Durbin, R. (2010). \"Fast and accurate long-read alignment with Burrows\u2013Wheeler transform\". Bioinformatics 26 (5): 589-95. doi:10.1093\/bioinformatics\/btp698. PMC PMC2828108. PMID 20080505. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2828108 .   \n\n\u2191 9.0 9.1 9.2 Li, H.; Handsaker, B.; Wysoker, A. et al. (2009). \"The Sequence Alignment\/Map format and SAMtools\". Bioinformatics 25 (16): 2078-9. doi:10.1093\/bioinformatics\/btp352. PMC PMC2723002. PMID 19505943. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002 .   \n\n\u2191 DePristo, M.A.; Banks, E.; Poplin, R. et al. (2011). \"A framework for variation discovery and genotyping using next-generation DNA sequencing data\". Nature Genetics 43 (5): 491-8. doi:10.1038\/ng.806. PMC PMC3083463. PMID 21478889. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3083463 .   \n\n\u2191 11.0 11.1 11.2 Shen, Y.; Wan, Z.; Coarfa, C. et al. (2010). \"A SNP discovery method to assess variant allele probability from next-generation resequencing data\". Genome Research 20 (2): 273-80. doi:10.1101\/gr.096388.109. PMC PMC2813483. PMID 20019143. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813483 .   \n\n\u2191 Morrison, A.C.; Voorman, A.; Johnson, A.D. et al. (2013). \"Whole-genome sequence\u2013based analysis of high-density lipoprotein cholesterol\". Nature Genetics 45 (8): 899-901. doi:10.1038\/ng.2671. PMC PMC4030301. PMID 23770607. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030301 .   \n\n\u2191 Goecks, J.; Nekrutenko, A.; Taylor, J.; The Galaxy Team (2010). \"Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences\". Genome Biology 11 (8): R86. doi:10.1186\/gb-2010-11-8-r86. PMC PMC2945788. PMID 20738864. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788 .   \n\n\u2191 Blankenberg, D.; Von Kuster, G.; Coraor, N. et al. (2010). \"Galaxy: a web-based genome analysis tool for experimentalists\". Current Protocols in Molecular Biology 19 (Unit 19.10.1\u201321). doi:10.1002\/0471142727.mb1910s89. PMC PMC4264107. PMID 20069535. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107 .   \n\n\u2191 Giardine, B.; Riemer, C.; Hardison, R.C. et al. (2005). \"Galaxy: A platform for interactive large-scale genome analysis\". Genome Research 15 (10): 1451\u20131455. doi:10.1101\/gr.4086505. PMC PMC1240089. PMID 16169926. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089 .   \n\n\u2191 Kallio, M.A.; Tuimala, J.T.; Hupponen, T. et al. (2011). \"Chipster: user-friendly analysis software for microarray and other high-throughput data\". BMC Genomics 12: 507. doi:10.1186\/1471-2164-12-507. PMC PMC3215701. PMID 21999641. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215701 .   \n\n\u2191 Ovaska, K.; Laakso, M.; Haapa-Paananen, S. et al. (2010). \"Large-scale data integration framework provides a comprehensive view on glioblastoma multiforme\". Genome Medicine 2 (9): 65. doi:10.1186\/gm186. PMC PMC3092116. PMID 20822536. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092116 .   \n\n\u2191 18.0 18.1 Agrawal, N.; Frederick, M.J.; Pickering, C.R. et al. (2011). \"Exome sequencing of head and neck squamous cell carcinoma reveals inactivating mutations in NOTCH1\". Science 333 (6046): 1154\u201357. doi:10.1126\/science.1206923. PMC PMC3162986. PMID 21798897. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162986 .   \n\n\u2191 19.0 19.1 Wang, K.; Li, M.; Hakonarson, H. (2010). \"ANNOVAR: Functional annotation of genetic variants from high-throughput sequencing data.\". Genetic Acids Research 38 (16): e164. doi:10.1093\/nar\/gkq603. PMC PMC2938201. PMID 20601685. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2938201 .   \n\n\u2191 Lupski, J.R.; Gonzaga-Jauregui. C.; Yang, Y. et al. (2013). \"Exome sequencing resolves apparent incidental findings and reveals further complexity of SH3TC2 variant alleles causing Charcot-Marie-Tooth neuropathy\". Genome Medicine 5 (6): 57. doi:10.1186\/gm461. PMC PMC3706849. PMID 23806086. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3706849 .   \n\n\u2191 Liu, X.; Jian, X.; Boerwinkle, E. (2011). \"dbNSFP: a lightweight database of human nonsynonymous SNPs and their functional predictions\". Human Mutation 32 (8): 894\u2013899. doi:10.1002\/humu.21517. PMC PMC3145015. PMID 21520341. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145015 .   \n\n\u2191 Lupski, J.R; Reid, J.G.; Gonzaga-Jauregui, C. et al. (2010). \"Whole-genome sequencing in a patient with Charcot-Marie-Tooth neuropathy\". The New England Journal of Medicine 362 (13): 1181\u201391. doi:10.1056\/NEJMoa0908094. PMC PMC4036802. PMID 20220177. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4036802 .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\">https:\/\/www.limswiki.org\/index.php\/Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 19 January 2016, at 21:43.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 1,638 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","a5e9f45d3e7fa0bbfb4350821efaa31e_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Launching_genomics_into_the_cloud_Deployment_of_Mercury_a_next_generation_sequence_analysis_pipeline skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Launching genomics into the cloud: Deployment of Mercury, a next generation sequence analysis pipeline<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Massively parallel DNA sequencing generates staggering amounts of data. Decreasing cost, increasing throughput, and improved annotation have expanded the diversity of genomics applications in research and clinical practice. This expanding scale creates analytical challenges: accommodating peak compute demand, coordinating secure access for multiple analysts, and sharing validated tools and results.\n<\/p><p><b>Results:<\/b>\nTo address these challenges, we have developed the Mercury analysis pipeline and deployed it in local hardware and the Amazon Web Services cloud via the DNAnexus platform. Mercury is an automated, flexible, and extensible analysis workflow that provides accurate and reproducible genomic results at scales ranging from individuals to large cohorts.\n<\/p><p><b>Conclusions:<\/b>\nBy taking advantage of cloud computing and with Mercury implemented on the DNAnexus platform, we have demonstrated a powerful combination of a robust and fully validated software pipeline and a scalable computational resource that, to date, we have applied to more than 10,000 whole genome and whole exome samples.\n<\/p><p><b>Keywords:<\/b> NGS data, Variant calling, Annotation, Clinical sequencing, Cloud computing \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Whole exome capture sequencing (WES) and whole genome sequencing (WGS) using next generation sequencing (NGS) technologies<sup id=\"rdp-ebb-cite_ref-MetzkerSeq10_1-0\" class=\"reference\"><a href=\"#cite_note-MetzkerSeq10-1\" rel=\"external_link\">[1]<\/a><\/sup> have emerged as compelling paradigms for routine clinical diagnosis, genetic risk prediction, and patient management.<sup id=\"rdp-ebb-cite_ref-BainbridgeWhole11_2-0\" class=\"reference\"><a href=\"#cite_note-BainbridgeWhole11-2\" rel=\"external_link\">[2]<\/a><\/sup> Large numbers of laboratories and hospitals routinely generate terabytes of NGS data, shifting the bottleneck in clinical genetics from DNA sequence production to DNA sequence analysis. Such analysis is most prevalent in three common settings: first, in a clinical diagnostics laboratory (e.g. Baylor\u2019s Whole Genome Laboratory <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.bcm.edu\/\u200bgeneticlabs\/\u200b\" target=\"_blank\">http:\/\/www.bcm.edu\/\u200bgeneticlabs\/\u200b<\/a>) testing single patients or families with presumed heritable disease; second, in a cancer-analysis setting where the unit of interest is either a normal-tumor tissue pair or normal-primary tumor-recurrence trio<sup id=\"rdp-ebb-cite_ref-CancerInt11_3-0\" class=\"reference\"><a href=\"#cite_note-CancerInt11-3\" rel=\"external_link\">[3]<\/a><\/sup>; and third, in biomedical research studies sequencing a sample of well-phenotyped individuals. In each case, the input is a DNA sample of appropriate quality having a unique identification number, appropriate informed consent, and relevant clinical and phenotypic information.\n<\/p><p>As these new samples are sequenced, the resulting data is most effectively examined in the context of petabases of existing DNA sequence and the associated meta-data. Such large-scale comparative <a href=\"https:\/\/www.limswiki.org\/index.php\/Genomics\" title=\"Genomics\" target=\"_blank\" class=\"wiki-link\" data-key=\"96a82dabf51cf9510dd00c5a03396c44\">genomics<\/a> requires new sequence data to be robustly characterized, consistently reproducible, and easily shared among large collaborations in a secure manner. And while data-management and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> technologies have adapted to the processing and storage requirements of emerging sequencing technologies (e.g., the CRAM specification<sup id=\"rdp-ebb-cite_ref-WheelerTheCom08_4-0\" class=\"reference\"><a href=\"#cite_note-WheelerTheCom08-4\" rel=\"external_link\">[4]<\/a><\/sup>), it is less certain that appropriate informative software interfaces will be made available to the genomics and clinical genetics communities. One element bridging the technology gap between the sequencing instrument and the scientist or clinician is a validated data processing pipeline that takes raw sequencing reads and produces an annotated personal genome ready for further analysis and clinical interpretation.\n<\/p><p>To address this need, we have designed and implemented Mercury, an automated approach that integrates multiple sequence analysis components across many computational steps, from obtaining patient samples to providing a fully annotated list of variant sites for clinical applications. Mercury fully integrates new software with existing routines (e.g., Altas2<sup id=\"rdp-ebb-cite_ref-ChallisAnInt12_5-0\" class=\"reference\"><a href=\"#cite_note-ChallisAnInt12-5\" rel=\"external_link\">[5]<\/a><\/sup>) and provides the flexibility necessary to respond to changing sequencing technologies and the rapidly increasing volume of relevant data. Mercury has been implemented on both local infrastructure and in a cloud computing platform provided by DNAnexus using Amazon Web Services (AWS). While there are other NGS analysis pipelines, some of which have even been implemented in the cloud<sup id=\"rdp-ebb-cite_ref-ODriscollBig13_6-0\" class=\"reference\"><a href=\"#cite_note-ODriscollBig13-6\" rel=\"external_link\">[6]<\/a><\/sup>, the combination of Mercury and DNAnexus together provide for the first time a fully integrated genomic analysis resource that can serve the full spectrum of users.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<p>Figure 1 provides an overview of the Mercury data processing pipeline. Source information includes sample and project management data and the characteristics of library preparation and sequencing. This information enters the pipeline either directly from the user or from a laboratory information management system (LIMS). The first step, generating sequencing reads, is based on the vendor\u2019s primary analysis software, which generates sequence reads and base-call confidence values (qualities). The second step maps the reads and qualities to the reference genome using a standard mapping tool, such as BWA<sup id=\"rdp-ebb-cite_ref-LiFastShort09_7-0\" class=\"reference\"><a href=\"#cite_note-LiFastShort09-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-LiFastLong09_8-0\" class=\"reference\"><a href=\"#cite_note-LiFastLong09-8\" rel=\"external_link\">[8]<\/a><\/sup>, producing a BAM<sup id=\"rdp-ebb-cite_ref-LiTheSeq09_9-0\" class=\"reference\"><a href=\"#cite_note-LiTheSeq09-9\" rel=\"external_link\">[9]<\/a><\/sup> (binary alignment\/map) file. The third step produces a \u201cfinished\u201d BAM that includes sorting, duplicate marking, indel realignment, base quality recalibration, and indexing (using a combination of tools including SAMtools<sup id=\"rdp-ebb-cite_ref-LiTheSeq09_9-1\" class=\"reference\"><a href=\"#cite_note-LiTheSeq09-9\" rel=\"external_link\">[9]<\/a><\/sup>, Picard (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/picard.sourceforge.net\" target=\"_blank\">http:\/\/picard.sourceforge.net<\/a>), and GATK<sup id=\"rdp-ebb-cite_ref-DePristoAFrame11_10-0\" class=\"reference\"><a href=\"#cite_note-DePristoAFrame11-10\" rel=\"external_link\">[10]<\/a><\/sup>). The fourth step in Mercury uses the Atlas2 suite<sup id=\"rdp-ebb-cite_ref-ChallisAnInt12_5-1\" class=\"reference\"><a href=\"#cite_note-ChallisAnInt12-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShenSNP10_11-0\" class=\"reference\"><a href=\"#cite_note-ShenSNP10-11\" rel=\"external_link\">[11]<\/a><\/sup> (Atlas-SNP and Atlas-indel) to call variants and produce a variant file (VCF). The fifth step adds biological and functional annotation and formats the variant lists for delivery. Each step is described in detail in the Methods section, as is the flow of information between steps.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Reid_BMCInformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"34d1f438dc75ac25e9c3142bce86fa80\"><img alt=\"Fig1 Reid BMCInformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/a\/a9\/Fig1_Reid_BMCInformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1. Mercury Data Flow.<\/b> 1) Sequencing Instrument raw data is passed to vendor primary analysis software to generate sequence reads and base call confidence values (qualities). 2) Reads and qualities are passed to a mapping tool (BWA) for comparison to a reference genome to determine the placement of reads on the reference (producing a BAM file). 3) Individual sequence event BAMs are merged to make a single sample-level BAM file that then is processed in preparation for variant calling. 4) Atlas-SNP and Atlas-indel are used to identify variants and produce variant files (VCF). 5) Annotation adds biological and functional information to the variant lists and formats them for delivery.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Mercury has been optimized for the Illumina HiSeq (Illumina, Inc.; San Diego, CA) platform, but the generalized workflow framework is adaptable to other settings. The entire pipeline has been implemented both locally and in a cloud computing environment. All relevant code and documentation are freely available online (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hgsc.bcm.edu\/\u200bcontent\/\u200bmercury\" target=\"_blank\">https:\/\/www.hgsc.bcm.edu\/\u200bcontent\/\u200bmercury<\/a>) and the scalable cloud solution is available within the DNAnexus library (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.dnanexus.com\/\u200b\" target=\"_blank\">http:\/\/www.dnanexus.com\/\u200b<\/a>). Sensible default parameters have already been determined so that researchers and clinicians can reliably analyze their data with Mercury without needing to configure any of the constituent programs or obtaining access to large computational resources, and they can do so in a manner compliant with multiple regulatory frameworks.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Local_workflow_management\">Local workflow management<\/span><\/h3>\n<p>Implementing a robust analysis framework that incorporates a heterogeneous collection of software tools presents many challenges. Running disparate software modules with varying inputs and outputs that depend on each other\u2019s results requires appropriate error checking and centralized output logging. We therefore developed a simple yet extensible workflow management framework, Valence (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/sourceforge.net\/\u200bprojects\/\u200bvalence\/\u200b\" target=\"_blank\">http:\/\/sourceforge.net\/\u200bprojects\/\u200bvalence\/\u200b<\/a>), that manages the various steps and dependencies within Mercury and handles internal and external pipeline communication. This formal approach to workflow management helps maximize computational resource utilization and seamlessly guides the data from the sequencing instrument to an annotated variant file ready for clinical interpretation and downstream analysis.\n<\/p><p>Valence parses and executes an analysis protocol described in XML format with each step treated as either an action or a procedure. An action is defined as a direct call to the system to submit a program or script to the job scheduler for execution; a procedure is defined a collection of actions, which is itself a workflow. This design allows the user to easily add, remove, and modify the steps of any analysis protocol. A protocol description for a specific step must include the required cluster resources, any dependencies on other steps, and a command element that describes how to execute the program or script. To ensure that the XML wrappers are applicable to any run, the command is treated as a string template that allows XML variables to be substituted into the command prior to execution. Thus, a single XML wrapper describing how to run a program can be applied to different inputs. Valence can be deployed on any cluster with a job scheduler (e.g., PBS, LSF, SGE), implementing a database to track both the job (the collection of all the steps in a protocol to be executed) and the status (\u201cNot Started,\u201d \u201cRunning,\u201d \u201cFinished,\u201d \u201cFailed\u201d) of any action associated with the job.\n<\/p><p>Mercury users can easily incorporate new analysis tools into an existing pipeline. For example, we recently expanded the scope of our pipeline to include Tiresias (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/sourceforge.net\/\u200bprojects\/\u200btiresias\/\u200b\" target=\"_blank\">https:\/\/sourceforge.net\/\u200bprojects\/\u200btiresias\/\u200b<\/a>), a structural variant caller focused on mobile elements, and ExCID (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/github.com\/\u200bcbuhay\/\u200bExCID\" target=\"_blank\">https:\/\/github.com\/\u200bcbuhay\/\u200bExCID<\/a>), an exome coverage analysis tool designed to provide clinical reports on under-covered regions. To incorporate Tiresias and ExCID into the Mercury pipeline, we needed only to specify the compute requirements and add the appropriate command to the existing XML workflow definition; Valence then automatically handles all data passing, logging, and error reporting.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Cloud_workflow_management\">Cloud workflow management<\/span><\/h3>\n<p>Mercury has been instantiated in the cloud via the DNAnexus platform (utilizing AWS\u2019s EC2 and S3). DNAnexus provides a layer of abstraction that simplifies development, execution, monitoring, and collaboration on a cloud infrastructure. The constituent steps of the Mercury pipeline take the form of discrete \u201capplets,\u201d which are then linked to form a workflow within the DNAnexus platform infrastructure. Using the workflow construction GUI, one can add applets (representing each step) to the workflow and create a dependency graph by linking the inputs and outputs of subsequent applets. Inputs are provided to an instance of the workflow, and the entire workflow is run within the cloud infrastructure. The various steps within the workflow are then executed based on the dependency graph. As with Valence, individual applets can be configured to run with a specific set of computational resource requirements such as memory, local disk space, and number of cores and processors. We are currently working to merge the local and cloud infrastructure elements by integrating the upload agent into Valence, allowing Valence to trigger a DNAnexus workflow once all the data is successfully uploaded. Such coordination would serve to transparently support analysis bursts.\n<\/p><p>The Mercury pipeline within DNAnexus comprises code that uses the DNAnexus command-line interface to instantiate the pipeline in the cloud. The Mercury code for DNAnexus is executed on a local terminal. For example, one may provide a list of sample FASTQ files and sample meta-data locations to Mercury, at which point Mercury uploads the data and instantiates the predefined workflow within DNAnexus. On average, on a 100 Mbps connection, we were able to upload at a rate of ~14 MB\/sec. We were able to parallelize this uploading process, yielding an effective upload rate of ~90 MB\/sec. The size of a typical FASTQ file from WES with 150X coverage has a compressed (bzip2) file size of approximately 3 GB. Uploading such a file from a local server took less than five minutes. After sample data are uploaded to the DNAnexus environment, the workflow is instantiated in the cloud.\nProgress can be monitored online using the DNAnexus GUI (Figure 2) or locally through the Mercury monitoring code. To achieve full automation, the monitoring code can be made a part of a local process to poll for analysis status at regular intervals and start analysis of new sequences automatically upon completion of sequencing. When the Mercury monitoring code detects successful completion of analysis an email notification is sent out. The results can either be downloaded to the local server or the user can view various tracks and data with a native genome browser.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Reid_BMCInformatics2014_15.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"cae7e5f36458eb104c8876336811ac84\"><img alt=\"Fig2 Reid BMCInformatics2014 15.jpg\" src=\"https:\/\/www.limswiki.org\/images\/d\/d4\/Fig2_Reid_BMCInformatics2014_15.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 2. Workflow monitoring in DNAnexus.<\/b> The GUI for applet monitoring displays the progress as a Gantt chart. The left panel lists the various steps including the parallelization steps with each row corresponding to a compute instance. A particular step can be clicked to determine the exact inputs and output or logs of execution for that step. Here we show a snapshot of the webpage displaying the progress of execution for the NA12878 exome analysis. <\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Performance\">Performance<\/span><\/h3>\n<p>Turn-around time for raw data generation on most NGS platforms is already considered long for many clinical applications, so minimizing analysis time is a primary goal of the Mercury pipeline. By maintaining a high-performance computing cluster consisting of hundreds of 8-core, 48 GB RAM nodes and introducing Mercury into the sequencing pipeline, we can minimize wait times by ensuring that compute resources are always available for all sequence events as the instrument produces the data. To match compute resources to production requirements, we carefully monitor the run times (and RAM and CPU requirements) of each step in the Mercury pipeline. Table 1 describes the pipeline for each compute-intensive step, from the generation of reads and qualities from raw data (bcl to FASTQ) to generation of post-processed annotated variant files (VCFs). Resource requirements for each step are given in terms of fraction of an 8-core node (CPU) and RAM allocated. Note that some steps under-utilize available CPUs because they require most (or all) of the RAM available on a given node. For data generated via WES from human samples, the Mercury pipeline requires less than 36 hours of wall-clock time and 15 node-hours (i.e., the equivalent of one whole node fully dedicated to processing for 15 hours). Run times and resource requirements will vary with data type, reference genome, and computing hardware configuration.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"13\"><b>Table 1.<\/b> Mercury computational resource requirements\n<\/td><\/tr>\n<tr>\n<td style=\"padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"padding-left:10px; padding-right:10px;\"><b>Data gen<\/b>\n<\/td>\n<td style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Alignment<\/b>\n<\/td>\n<td style=\"padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>BAM finishing<\/b>\n<\/td>\n<td style=\"padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Variants<\/b>\n<\/td>\n<td style=\"padding-left:10px; padding-right:10px;\"><b>Anno<\/b>\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BCL to FastQ\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BWA align\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BWA sample\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mates, Dupe, Stats\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cap & Cvrg Metrics\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GATK indel targets\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GATK indel realign\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GATK recal\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BAM valid\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Atlas SNP\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Atlas Indel\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cassandra\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Nodes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.333\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.125\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.333\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.125\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.167\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.167\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.167\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">RAM\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">48\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">48\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">15\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">48\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">14\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">32\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Hours\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.62\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.84\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.38\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.39\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.30\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.75\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9.00\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7.51\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.71\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Node*Hrs\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.62\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.84\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.46\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.70\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.16\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.75\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3.04\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.09\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.50\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.25\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.29\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"13\">All estimates are approximate for whole exome and light-skim whole genome (~10-20 Gbp of data) sequenced on Illumina HiSeq and processed with the most recent versions of RTA and Casava. Nodes are 8-core, 48 GB RAM, with ~3 GHz Intel CPUs and ~1 TB of local scratch disk. Steps include all aspects of the pipeline from building reads and qualities (fastQ) from raw data (bcl files), through alignment and BAM generation using the BWA aligner, and BAM finishing with GATK post-processing and duplicate marking, capture and coverage metric calculation, and BAM file validation, finally producing variants from the Atlas2 variant calling suite with annotations from our annotator, Cassandra.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>After porting each element of the Mercury pipeline into the DNAnexus environment, the tools (i.e., \u201capps\u201d) can be run on the cloud in environments with different CPU, RAM, disk, and bandwidth resources to optimize wall-clock time and cost-efficiency. Parallelization within a pipeline reduces the time for a single run, which is useful for quick development cycles or time-sensitive applications. In addition, many parallel pipelines can be run simultaneously. The current peak usage for the Mercury pipeline on DNAnexus is approximately 12,000 cores. This throughput is a small fraction of the theoretical maximum that could be achieved in AWS. For the standard implementation of Mercury in the cloud analyzing a validation exome (Hapmap sample NA12878), the total wall time to produce an annotated variant call VCF (starting with paired-end FASTQ files) is approximately one day.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_management.2C_sharing.2C_reliability.2C_and_compliance\">Data management, sharing, reliability, and compliance<\/span><\/h3>\n<p>Large projects present a large data management challenge. For example, the Baylor-Hopkins Center for Mendelian Genomics has generated WES data from approximately 2,000 samples and the Human Genome Sequencing Center at Baylor College of Medicine has generated more than 10,000 WES and 3,500 WGS data from research samples from the CHARGE consortium. \n<\/p><p>As a pilot study, we processed 1,000 WES data sets\u2013approximately 80 terabytes of genomic data\u2013from the Center for Mendelian Genomics using Mercury in DNAnexus. Using multi-threaded uploads we were able to deliver data into the cloud at an average rate of ~960 exomes per day. Once uploaded, the data is analyzed with Mercury, and the resulting variants can be accessed for further analysis via a web GUI. Data can also be tagged, and these tags can be filtered or retrieved. Runs of individual pipelines and tools can be queried in a similar way.<sup id=\"rdp-ebb-cite_ref-MorrisonWhole13_12-0\" class=\"reference\"><a href=\"#cite_note-MorrisonWhole13-12\" rel=\"external_link\">[12]<\/a><\/sup> As datasets become larger, multi-site collaborative consortiums play an increasingly important role in contemporary biomedical research. A major advantage of cloud computing over local computing is that cloud storage can be shared across multiple organizations. Instead of each collaborator maintaining a local copy of the data and working in isolation, cloud users can be given appropriate access permissions so some researchers can view and download the results, others can run analyses on the data and build tools, and those with administrative privileges can determine access to the project. This data paradigm is the only feasible approach to giving patients meaningful access to their own genomic data.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Comparison_to_existing_methods\">Comparison to existing methods<\/span><\/h3>\n<p>A number of other tools and services provide similar functionality to Mercury on DNAnexus, with differing approaches to extensibility, ease of use for non-programmers, support for local or cloud infrastructures, and software available by default (Table 2). For example, the academic service Galaxy primarily focuses on extensibility and building a developer community.<sup id=\"rdp-ebb-cite_ref-GoecksGalaxy10_13-0\" class=\"reference\"><a href=\"#cite_note-GoecksGalaxy10-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BlankenbergGalaxy10_14-0\" class=\"reference\"><a href=\"#cite_note-BlankenbergGalaxy10-14\" rel=\"external_link\">[14]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-GiardineGalaxy05_15-0\" class=\"reference\"><a href=\"#cite_note-GiardineGalaxy05-15\" rel=\"external_link\">[15]<\/a><\/sup> Seven Bridges is a commercial service that combines a few fixed pipelines with a visually distinctive workflow editor. Chipster is an academic service that packages a variety of NGS tools in addition to a variety of microarray tools and combines these with visualization of data summaries and QC metrics.<sup id=\"rdp-ebb-cite_ref-KallioChipster11_16-0\" class=\"reference\"><a href=\"#cite_note-KallioChipster11-16\" rel=\"external_link\">[16]<\/a><\/sup> Anduril is designed to manage a local cluster and contains packages for a variety of tasks, including alignment and variant calling as well as image analysis and flow cytometry, which are not addressed by the other cloud services surveyed.<sup id=\"rdp-ebb-cite_ref-OvaskaLarge10_17-0\" class=\"reference\"><a href=\"#cite_note-OvaskaLarge10-17\" rel=\"external_link\">[17]<\/a><\/sup> With respect to the software used in sequence production pipelines, Mercury is most distinguished by its Atlas variant caller and the extensive annotations provided by its Cassandra annotation tool (<a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hgsc.bcm.edu\/\u200bsoftware\/\u200bcassandra\" target=\"_blank\">https:\/\/www.hgsc.bcm.edu\/\u200bsoftware\/\u200bcassandra<\/a>).\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 2.<\/b> A feature summary of Mercury in DNAnexus and similar tools and services\n<\/td><\/tr>\n<tr>\n<th>\n<\/th>\n<th> Mercury in DNAnexus\n<\/th>\n<th> Galaxy\n<\/th>\n<th> Seven bridges\n<\/th>\n<th> Chipster\n<\/th>\n<th> Anduril\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Mapper\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BWA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No canonical (many)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BWA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BWA\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BWA\/Bowtie\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Variant caller\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Atlas2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No canonical (many)\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">GATK\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Samtools mpileup\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Samtools\/GATK, Varscan\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Annotation\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Cassandra\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">snpEff, AlleleFrequency\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">snpEff, ANNOVAR\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Bioconductor\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">ANNOVAR\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Visualization\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Built-in browser\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Built-in browser\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Links to IGV\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Built-in browser\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Runs on local hardware\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No*\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Runs on cloud infrastructure\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Jobs queued to public server\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HIPAA compliant\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Not Applicable (local only)\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Requires setup configuration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Can add tools independently\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">By request\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Data sharing & collaboration\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Yes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">No\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\">*Mercury can be run on local hardware via Valence, but does not include all of the security and data sharing features of Mercury in DNAnexus.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Future_directions\">Future directions<\/span><\/h3>\n<p>As genomic studies transition from extensive us of whole exome capture sequencing methods to an emphasis on whole genome sequencing<sup id=\"rdp-ebb-cite_ref-AgrawalExome11_18-0\" class=\"reference\"><a href=\"#cite_note-AgrawalExome11-18\" rel=\"external_link\">[18]<\/a><\/sup>, we can take advantage of Mercury\u2019s flexibility to adapt to shifting research priorities. Currently, the sequencing community uses whole exome sequencing because of its comparatively low cost and because it enriches for biological signals that are readily interpretable. However, with changing price structures, the advantages of eliminating the capture step in the laboratory, improvements in lossless compression of sequence data, and improved annotation of the non-protein coding region of the genome (e.g., genome.ucsc.edu\/ENCODE\/), whole genome sequencing may become a more appealing option. Early adopters of whole genome sequencing will certainly include medical sequencing for research and diagnostic purposes. Therefore, a high priority for Mercury is the continual updates to the annotation information using web-based databases. Such web-based annotation updates can incorporate the latest developments related to genome function in near real-time.\n<\/p><p>Clinically relevant annotation is of particular interest in the growing area of cancer somatic mutation. Currently, disease-related annotation considers inherited disease alleles gleaned from OMIM (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.omim.org\" target=\"_blank\">http:\/\/www.omim.org<\/a>), HGMD (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.hgmd.org\" target=\"_blank\">http:\/\/www.hgmd.org<\/a>; assuming the user has the appropriate license to use this data), and genome-wide association studies (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.genome.gov\/\u200bgwastudies\" target=\"_blank\">http:\/\/www.genome.gov\/\u200bgwastudies<\/a>). Similarly informative databases, such as the Catalogue of Somatic Mutations in Cancer (COSMIC; <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.sanger.ac.uk\/\u200bgenetics\/\u200bCGP\/\u200bcosmic\/\u200b\" target=\"_blank\">http:\/\/www.sanger.ac.uk\/\u200bgenetics\/\u200bCGP\/\u200bcosmic\/\u200b<\/a>), are emerging and will need to be fully vetted and incorporated into Cassandra. The goal of Mercury is to provide a simple solution for end-to-end sequence analysis so that non-expert users can obtain a list of annotated and prioritized variants as rapidly as possible, and so that expert users can augment and modify the pipeline to meet specialized needs.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>The long-anticipated NGS data deluge<sup id=\"rdp-ebb-cite_ref-WangANNOVAR10_19-0\" class=\"reference\"><a href=\"#cite_note-WangANNOVAR10-19\" rel=\"external_link\">[19]<\/a><\/sup> has now arrived. The first personal NGS genome was published in 2007<sup id=\"rdp-ebb-cite_ref-WheelerTheCom08_4-1\" class=\"reference\"><a href=\"#cite_note-WheelerTheCom08-4\" rel=\"external_link\">[4]<\/a><\/sup>, and today we estimate that the number of available exomes and genomes approaches one hundred thousand. It is equally impressive that the application of NGS to biomedical research and clinical medicine is rapidly becoming standard. Such applications are driven by the utility of sequence data, as demonstrated by a number of instances where DNA sequencing has been used not only for diagnostic purposes but also to reveal more efficacious therapies.<sup id=\"rdp-ebb-cite_ref-AgrawalExome11_18-1\" class=\"reference\"><a href=\"#cite_note-AgrawalExome11-18\" rel=\"external_link\">[18]<\/a><\/sup>\n<\/p><p>By taking advantage of cloud computing and with Mercury implemented on the DNAnexus platform, we have demonstrated a powerful combination of a robust and fully validated software pipeline and a scalable computational resource. To date, we have analyzed thousands of samples (using the AWS cloud: EC2 and S3), including a population study comprising 3,500 samples and 10,000 samples for which WGS and WES were generated, respectively, more than 1,000 Mendelian disease cases shared with data consumers all over the world, and smaller projects such as 50 exome trios and 30 cancer WGS tumor\/normal pairs. To our knowledge, these projects represent the largest genomic analysis effort to be realized in the cloud to date. They presage a wave of genomic computing that will transform how genomic data is analyzed and delivered to the scientific community, into clinical practice, and eventually directly into the hands of patients and advocates.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Reads_and_read_qualities\">Reads and read qualities<\/span><\/h3>\n<p>Making reads and assigning those reads a quality metric is the only step of Mercury that relies almost exclusively upon vendor-provided software. Processing raw data files down to signal intensities and then taking the signal data through the demultiplexing (i.e., assigning a sequence read to an individual barcoded sample), base calling, and quality scoring processes is integrally tied to the sequencing chemistry and machine instrumentation. Vendors, in collaboration with the user community, are constantly developing and improving this primary data input step. As such, the integrated Mercury pipeline is triggered by the completion of the data transfer from the instrument into the local compute environment, at which point vendor tools are used to generate a FASTQ file containing reads and qualities for each sequence event. The current version of Mercury is optimized for the Illumina HiSeq instrument, so it currently integrates with Bcl2FastQ (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/support.illumina.com\/\u200bdownloads\/\u200bbcl2fastq_\u200bconversion_\u200bsoftware_\u200b184.\u200bilmn\" target=\"_blank\">http:\/\/support.illumina.com\/\u200bdownloads\/\u200bbcl2fastq_\u200bconversion_\u200bsoftware_\u200b184.\u200bilmn<\/a>), the Illumina analysis toolkit. Modularity of the Mercury pipeline facilitates incorporation of upgrades to Bcl2FastQ and the replacement of Bcl2FastQ with functionally similar tools as necessary. At the end of this step, data from a flow cell is broken down into individual sequence events associated with each barcode.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Mapping_and_BAM_generation\">Mapping and BAM generation<\/span><\/h3>\n<p>Once reads and qualities are generated, each sequence event is mapped to a reference genome. The Burrows-Wheeler Aligner (BWA)<sup id=\"rdp-ebb-cite_ref-LiFastLong09_8-1\" class=\"reference\"><a href=\"#cite_note-LiFastLong09-8\" rel=\"external_link\">[8]<\/a><\/sup> is the current preferred mapping tool for Illumina HiSeq data, though Mercury\u2019s modular design allows for alternative mapping tools. The mapping process for short reads consists of a \u201cseeding\u201d step that identifies regions of similarity between the read and reference, and then a local alignment stage that makes comparisons among reads in a region. Through this iterative process, BWA identifies the most likely placement for the reads and the differences between each read and the reference.\n<\/p><p>Once mapping is complete, additional processing steps are taken to improve the resulting BAM files. The data is sorted by mapping position, duplicate reads (artifacts of the sample preparation process) are flagged, and metadata summarizing data production information is added. Further downstream, base quality recalibrations and local realignment around regions with indel calls are performed to further improve the information content of variant calls. For cases in which data from a single sample spans multiple sequence events, Mercury provides the means to merge individual sequence event BAMs into a single sample-level BAM that can then be used for downstream analysis.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Variant_calling\">Variant calling<\/span><\/h2>\n<p>Mercury is designed to provide robust variant calling for readily available whole exome sequencing data but is extensible to similar functionality for whole genome data. We have implemented Atlas2<sup id=\"rdp-ebb-cite_ref-ChallisAnInt12_5-2\" class=\"reference\"><a href=\"#cite_note-ChallisAnInt12-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ShenSNP10_11-1\" class=\"reference\"><a href=\"#cite_note-ShenSNP10-11\" rel=\"external_link\">[11]<\/a><\/sup>, an SNV- and indel-calling suite. Atlas2 uses a logistic regression approach to detect systematic sequencing errors using context-sensitive input variables, following three hierarchical steps: a logistic regression model to characterize systematic sequencing errors, a probability score that detects and removes errors, and a minimal heuristics filters as post-hoc quality assurances to address mapping and capture biases. Atlas2 has high SNP-discovery sensitivity and specificity when applied to the 1000 Genomes Whole Exome data.<sup id=\"rdp-ebb-cite_ref-ShenSNP10_11-2\" class=\"reference\"><a href=\"#cite_note-ShenSNP10-11\" rel=\"external_link\">[11]<\/a><\/sup> The accuracy of homozygous and heterozygous SNPs are as high as 99.8% and 99.6%, respectively, when compared to the SNP array data. We have also demonstrated high quality in short indel discoveries.<sup id=\"rdp-ebb-cite_ref-ChallisAnInt12_5-3\" class=\"reference\"><a href=\"#cite_note-ChallisAnInt12-5\" rel=\"external_link\">[5]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Technical_validation_and_reproducibility\">Technical validation and reproducibility<\/span><\/h3>\n<p>To enable clinical applications, we characterized Mercury\u2019s reproducibility of calls in gene-coding regions. We generated VCFs from two independent samples and compared the high quality variant calls from one to the variant calls of any quality from the other sample. The process was then repeated with the sample roles reversed. At the end of this process, we produced three sets of comparisons: SNPs with passing evidence in both samples, SNPs that only have passing evidence in the first sample, and those that only have passing evidence in the second sample. For illustrative purposes, reproducibility through technical replicates for eight samples meeting the basic criteria of greater than 90% of targeted bases covered at 20x or better is shown in Table 3. These observations indicate strong consistency (less than 2% variability between technical replicates), with approximately 24,000 SNPs found consistently in the overlap.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"7\"><b>Table 3.<\/b> Technical replicate data for the Mercury pipeline for eight samples sequenced in duplicate (A and B)\n<\/td><\/tr>\n<tr>\n<th> Sample #\n<\/th>\n<th> A only\n<\/th>\n<th> A only (%)\n<\/th>\n<th> A and B\n<\/th>\n<th> A and B (%)\n<\/th>\n<th> B only\n<\/th>\n<th> B only (%)\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1011\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">133\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.561%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23,320\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.409%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">244\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.029%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1015\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">155\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.542%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">28,312\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.910%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">157\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.548%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1016\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">155\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.644%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23,752\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.732%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">150\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.624%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1017\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">105\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.456%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">22,767\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.768%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">179\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.777%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1018\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">165\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.693%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23,531\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.795%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">122\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.512%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1019\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">162\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.682%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23,441\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.686%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">150\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.631%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1020\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">493\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2.041%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23,518\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">97.355%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">146\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.604%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">HS-1021\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">161\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.681%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23,188\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.125%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">282\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1.193%\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">191\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.718%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">23,979\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.473%\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">179\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">0.613%\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>We then assessed Mercury\u2019s accuracy by evaluating the sensitivity and specificity of the Mercury SNP calls. SNP array data and whole-exome sequencing data were generated from a single sample, and seven exome technical replicates were analyzed with Mercury. The array data indicated the presence of 1,927 SNPs and 2,814 homozygous reference sites that overlapped with regions targeted by the exome capture reagent. Table 4 details Mercury\u2019s ability to recover these sites, averaging almost 99% sensitivity across the replicates. Moreover, Mercury made no SNP calls in any of the replicates for any of the SNP array homozygous reference sites, and achieved sufficient coverage to consider those sites reference homozygous.\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\"><b>Table 4.<\/b> Concordance of SNP array and Mercury data\n<\/td><\/tr>\n<tr>\n<th> Replicate\n<\/th>\n<th> PASS\n<\/th>\n<th> ALL\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1897\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1906\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1900\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1906\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1898\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1904\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1895\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1904\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1899\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1907\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1904\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1907\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1902\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1905\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1899.28\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1905.71\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Average %\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.56\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">98.89\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"3\">For each exome technical replicate, we report the number of passing SNPs found by Mercury and the total number of SNPs (passing and not passing) in the VCF that overlap the SNP array design. The SNP array contains 1,927 sites that overlap the exome capture region.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>To facilitate validation by others of local installations of Mercury or modified applications in the cloud, we are making available the FASTQ, BAMs and VCFs of a single individual with a known Mendelian condition.<sup id=\"rdp-ebb-cite_ref-LupskiExome13_20-0\" class=\"reference\"><a href=\"#cite_note-LupskiExome13-20\" rel=\"external_link\">[20]<\/a><\/sup> These data are available at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/\u200bsra?\u200bterm=\u200bSRP023104\" target=\"_blank\">http:\/\/www.ncbi.nlm.nih.gov\/\u200bsra?\u200bterm=\u200bSRP023104<\/a>.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Annotation\">Annotation<\/span><\/h3>\n<p>Mercury provides variant annotation via the Cassandra annotation suite, which describes the quality and predicted functional consequences of genomic variants, providing the biological and clinical contexts needed to assess the significance of each variant. Variants are presented to the user with all quality control metrics produced by Atlas2, the \u201cpileup string,\u201d<sup id=\"rdp-ebb-cite_ref-LiTheSeq09_9-2\" class=\"reference\"><a href=\"#cite_note-LiTheSeq09-9\" rel=\"external_link\">[9]<\/a><\/sup> and the theoretical mappability (a measure of sequence degeneracy throughout the region) of the position. The pileup string can be visually or automatically inspected to identify biases in the variant strandedness or determine whether the variant is 5\u2019- or 3\u2019-biased within the supporting reads. Low theoretical mappability can be used to determine whether reads aligned to certain positions could have been mapped confidently or whether they were likely to have been mismapped. AnnoVar<sup id=\"rdp-ebb-cite_ref-WangANNOVAR10_19-1\" class=\"reference\"><a href=\"#cite_note-WangANNOVAR10-19\" rel=\"external_link\">[19]<\/a><\/sup> is used to determine a variant\u2019s effect on both a conservative (RefSeq) and inclusive (UCSC) gene model set. This assessment also describes whether a variant changes an amino acid residue, whether it creates or removes a stop-codon, and the variant\u2019s location: near an intron-exon boundary, within an intron, within a known non-coding RNA, or in an intergenic region. The annotation also includes five algorithms that predict the deleterious nature of nonsynonymous variants.<sup id=\"rdp-ebb-cite_ref-Liu_dbNSFP11_21-0\" class=\"reference\"><a href=\"#cite_note-Liu_dbNSFP11-21\" rel=\"external_link\">[21]<\/a><\/sup> Variants are additionally annotated according to their frequency and presence in multiple variant collections (e.g., dbSNP, Thousand Genomes). Lastly, variants are annotated based on functional data for the gene in which they occur. These data include the known function of the gene (e.g., Swiss-Prot), any previous association of the gene with a disease (e.g., OMIM), post-translational modifications of the gene, and the expression profile of the gene across human organs. Together, these data are applied to assess the variant quality, its minor allele frequency, and any disease association while also elucidating the potential effects of the variant within a genetic framework.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Metadata_and_LIMS\">Metadata and LIMS<\/span><\/h3>\n<p>Mercury integrates external metadata resources and inputs such as a reference genome, sequence data locations, and a capture design bed file and therefore requires an integrated LIMS within the pipeline. Our LIMS solution is partitioned into three major modules: project management (PM), sample tracking (ST), and reporting. The PM module provides tools to define the purpose of the project and aggregate samples together. Such aggregation allows project-level decisions (e.g., capture design parameters, reference genome for mapping) to be applied to all relevant samples at once. The ST module provides features for tracking samples as they move through the sequencing center pipeline. Samples are tracked via a barcode given to each sample when it is accepted by the sequencing center. Once that barcode is recorded, all lab experiments and informatics analyses performed on the sample track this barcode. The barcode-based LIMS reporting module lets users monitor a sample\u2019s progress in the pipeline and adjust the steps if necessary. The ability to see the history of the samples and sample data grouped or individually, is necessary for troubleshooting problematic samples and for developing and monitoring timelines.\n<\/p><p>To make Mercury portable, we provide communication \u201chooks\u201d for transferring data between Mercury and LIMS. These hooks are scripts that can be modified to query or deliver data to any metadata resource. Examples of information served to Mercury from LIMS are the reference genome and previously generated SNP array data for quality assurance purposes. By decoupling Mercury from the sample tracking data we have built a more portable and compliance-ready pipeline, thus providing increased flexibility.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Quality_assurance.2C_quality_control.2C_and_error_handling\">Quality assurance, quality control, and error handling<\/span><\/h3>\n<p>In local environments, Mercury maintains an off-the-shelf validated open-source pipeline. Such transparency can pose challenges to the strict regulatory requirements that govern clinical sequence analysis. To support analysis best practices, we provide a set of documents, data, and validation tools (detailed manuals available with the code at <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.hgsc.bcm.edu\/\u200bcontent\/\u200bmercury\" target=\"_blank\">https:\/\/www.hgsc.bcm.edu\/\u200bcontent\/\u200bmercury<\/a>) as well as publically available test data.<sup id=\"rdp-ebb-cite_ref-LupskiWhole10_22-0\" class=\"reference\"><a href=\"#cite_note-LupskiWhole10-22\" rel=\"external_link\">[22]<\/a><\/sup> Strict version control is maintained.\n<\/p><p>The Mercury pipeline generates a variety of performance metrics, including the number of pass-filter bases, read mapping fractions, concordance with orthogonal array genotypes, novel SNP rates, and transition-to-transversion ratios, which allow the user to gauge the quality of the final variant call results. Genotype arrays create a fingerprint of each sample upon intake that is then compared to the sequence data to validate sample identity. The governing principle of the process is to generate the quality control (QC) data as soon as possible and deliver that QC data into the LIMS (or other meta-data aggregating resource), but not to automatically interrupt the pipeline based only on QC data. The QC data are used in two ways. First, these data are considered in the clinical sign-out process for an individual run or sample. Second, the QC data can be used to detect systemic problems in the production pipeline, such as contaminated reagents or a malfunctioning sequencing instrument.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_security_and_compliance\">Data security and compliance<\/span><\/h3>\n<p>The translational nature of modern genomic data and the scope of collaborations demand levels of data security unprecedented in our field. Based on our experience using Mercury on the DNAnexus platform to analyze thousands of samples, we have found a substantially higher security standard for data in the cloud than in most local systems. Managing the regulatory requirements of standards for handling sensitive medical data such as <a href=\"https:\/\/www.limswiki.org\/index.php\/HIPAA\" title=\"HIPAA\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"70050974d1eda9ff8cf9ecf7e4fcd015\">HIPAA<\/a>, <a href=\"https:\/\/www.limswiki.org\/index.php\/CLIA\" title=\"CLIA\" class=\"mw-redirect wiki-link\" target=\"_blank\" data-key=\"9c0b9bbf5c8d2310833a79d137ac621e\">CLIA<\/a>, dbGaP, and <a href=\"https:\/\/www.limswiki.org\/index.php\/21_CFR_Part_11\" title=\"21 CFR Part 11\" target=\"_blank\" class=\"wiki-link\" data-key=\"0ec495c20a8f17894bd0c1388eda8e7a\">21 CFR parts 11<\/a>, 58, and 493 represents a considerable investment in compliance, security, and systems engineering expertise, and most local environments do not have the resources to maintain the required standards. By using cloud computing platforms, we are able to leverage diverse expertise, including modern, high-security data center technologies, best practices in encryption and authentication, and software system design to support clinical applications such as auditability, record retention and destruction, and reproducibility.\n<\/p><p>At the infrastructure level, DNAnexus uses data centers in high-security facilities with SAS-70\/SSAE-16, PCI Level 1, and FISMA Moderate certifications. At the user level, it enforces best practices such as password strength and rotation, session expiration, and client encryption. All data access is carefully controlled, logged for auditing purposes, encrypted end-to-end (both in flight and at rest), integrity-verified, and replicated in at least three physically distinct data centers to ensure against loss. Data analysis is constrained to computing nodes that are sandboxed using virtualization and encryption technologies, and are versioned to ensure reproducibility and the ability to track data provenance. The software has undergone multiple third-party audits, including penetration testing by security experts, and the overall system has been ISO 27001 certified an internationally recognized standard for secure data management processes.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p><b>AWS:<\/b> Amazon web services\n<\/p><p><b>BAM:<\/b> Binary alignment\/map\n<\/p><p><b>BWA:<\/b> Burrows-wheeler aligner\n<\/p><p><b>CFR:<\/b> Code of federal regulations\n<\/p><p><b>CLIA:<\/b> Clinical laboratory improvement amendments\n<\/p><p><b>COSMIC:<\/b> Catalogue of somatic mutations in cancer\n<\/p><p><b>dbGaP:<\/b> The database of Genotypes and Phenotypes\n<\/p><p><b>EC2:<\/b> Amazon elastic compute cloud\n<\/p><p><b>FISMA:<\/b> Federal information security management act\n<\/p><p><b>Gbp:<\/b>Giga base-pairs\n<\/p><p><b>HGMD:<\/b> Human gene mutation database\n<\/p><p><b>HIPPA:<\/b> Health insurance portability and accountability act\n<\/p><p><b>Indel:<\/b> Insertion\/Deletion\n<\/p><p><b>LIMS:<\/b> Laboratory information management system\n<\/p><p><b>NGS:<\/b> Next generation sequencing\n<\/p><p><b>NHGRI:<\/b> National human genome research institute\n<\/p><p><b>OMIM:<\/b> Online mendelian inheritance in man\n<\/p><p><b>PM:<\/b> Project management\n<\/p><p><b>QC:<\/b> Quality control\n<\/p><p><b>S3:<\/b> Amazon simple storage service\n<\/p><p><b>SNP:<\/b> Single nucleotide polymorphism\n<\/p><p><b>ST:<\/b> Sample tracking\n<\/p><p><b>VCF:<\/b> Variant call format\n<\/p><p><b>XML:<\/b> Extensible markup language.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h3>\n<p>This work was supported in part by the US National Human Genome Research Institute (NHGRI) grants U54HG006542 and U54HG003273.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h3>\n<p>JGR is a paid consultant of LaserGen. DNAnexus and AWS derive revenue from cloud computing services. The remaining authors declare that they have no competing interests.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h3>\n<p>JGR conceived and designed the study, developed software, performed data analysis, and participated in the drafting and writing of the manuscript. RAG and EB conceived and designed the study and participated in the drafting and writing of the manuscript. RD and GD conceived and designed the study. AC and NV developed software, performed data analysis, and contributed to the writing of the manuscript. MD and AS developed software and contributed to the writing of the manuscript. AE, MB, SW, and FY developed software. WS participated in the coordination of the study and contributed to the writing of the manuscript. CB and DMM participated in the coordination of the study. All authors read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-MetzkerSeq10-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MetzkerSeq10_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Metzker, Michael L. (2010). \"Sequencing technologies \u2014 The next generation\". <i>Nature Reviews Genetics<\/i> <b>11<\/b> (1): 31\u201346. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnrg2626\" target=\"_blank\">10.1038\/nrg2626<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19997069\" target=\"_blank\">19997069<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sequencing+technologies+%E2%80%94+The+next+generation&rft.jtitle=Nature+Reviews+Genetics&rft.aulast=Metzker%2C+Michael+L.&rft.au=Metzker%2C+Michael+L.&rft.date=2010&rft.volume=11&rft.issue=1&rft.pages=31%E2%80%9346&rft_id=info:doi\/10.1038%2Fnrg2626&rft_id=info:pmid\/19997069&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BainbridgeWhole11-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BainbridgeWhole11_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bainbridge, Matthew N.; Wiszniewski, Wojciech; Murdock, David R. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3314311\" target=\"_blank\">\"Whole-genome sequencing for optimized patient management\"<\/a>. <i>Science Translational Medicine<\/i> <b>3<\/b> (87): 87re3. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscitranslmed.3002243\" target=\"_blank\">10.1126\/scitranslmed.3002243<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3314311\/\" target=\"_blank\">PMC3314311<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21677200\" target=\"_blank\">21677200<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3314311\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3314311<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Whole-genome+sequencing+for+optimized+patient+management&rft.jtitle=Science+Translational+Medicine&rft.aulast=Bainbridge%2C+Matthew+N.%3B+Wiszniewski%2C+Wojciech%3B+Murdock%2C+David+R.+et+al.&rft.au=Bainbridge%2C+Matthew+N.%3B+Wiszniewski%2C+Wojciech%3B+Murdock%2C+David+R.+et+al.&rft.date=2011&rft.volume=3&rft.issue=87&rft.pages=87re3&rft_id=info:doi\/10.1126%2Fscitranslmed.3002243&rft_id=info:pmc\/PMC3314311&rft_id=info:pmid\/21677200&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3314311&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CancerInt11-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CancerInt11_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cancer Genome Atlas Research Network (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163504\" target=\"_blank\">\"Integrated genomic analyses of ovarian carcinoma\"<\/a>. <i>Nature<\/i> <b>474<\/b> (7353): 609\u2013615. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature10166\" target=\"_blank\">10.1038\/nature10166<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3163504\/\" target=\"_blank\">PMC3163504<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21720365\" target=\"_blank\">21720365<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163504\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3163504<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Integrated+genomic+analyses+of+ovarian+carcinoma&rft.jtitle=Nature&rft.aulast=Cancer+Genome+Atlas+Research+Network&rft.au=Cancer+Genome+Atlas+Research+Network&rft.date=2011&rft.volume=474&rft.issue=7353&rft.pages=609%E2%80%93615&rft_id=info:doi\/10.1038%2Fnature10166&rft_id=info:pmc\/PMC3163504&rft_id=info:pmid\/21720365&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3163504&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WheelerTheCom08-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WheelerTheCom08_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-WheelerTheCom08_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wheeler, David A.; Srinivasan, Maithreyan; Egholm, Michael et al. (2008). \"The complete genome of an individual by massively parallel DNA sequencing\". <i>Nature<\/i> <b>452<\/b> (7189): 872\u2013876. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnature06884\" target=\"_blank\">10.1038\/nature06884<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18421352\" target=\"_blank\">18421352<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+complete+genome+of+an+individual+by+massively+parallel+DNA+sequencing&rft.jtitle=Nature&rft.aulast=Wheeler%2C+David+A.%3B+Srinivasan%2C+Maithreyan%3B+Egholm%2C+Michael+et+al.&rft.au=Wheeler%2C+David+A.%3B+Srinivasan%2C+Maithreyan%3B+Egholm%2C+Michael+et+al.&rft.date=2008&rft.volume=452&rft.issue=7189&rft.pages=872%E2%80%93876&rft_id=info:doi\/10.1038%2Fnature06884&rft_id=info:pmid\/18421352&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChallisAnInt12-5\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ChallisAnInt12_5-0\" rel=\"external_link\">5.0<\/a><\/sup> <sup><a href=\"#cite_ref-ChallisAnInt12_5-1\" rel=\"external_link\">5.1<\/a><\/sup> <sup><a href=\"#cite_ref-ChallisAnInt12_5-2\" rel=\"external_link\">5.2<\/a><\/sup> <sup><a href=\"#cite_ref-ChallisAnInt12_5-3\" rel=\"external_link\">5.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Challis, D.; Yu, J.; Evani, U.S. et al. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3292476\" target=\"_blank\">\"An integrative variant analysis suite for whole exome next-generation sequencing data\"<\/a>. <i>BMC Bioinformatics<\/i> <b>13<\/b>: 8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-13-8\" target=\"_blank\">10.1186\/1471-2105-13-8<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3292476\/\" target=\"_blank\">PMC3292476<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22239737\" target=\"_blank\">22239737<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3292476\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3292476<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=An+integrative+variant+analysis+suite+for+whole+exome+next-generation+sequencing+data&rft.jtitle=BMC+Bioinformatics&rft.aulast=Challis%2C+D.%3B+Yu%2C+J.%3B+Evani%2C+U.S.+et+al.&rft.au=Challis%2C+D.%3B+Yu%2C+J.%3B+Evani%2C+U.S.+et+al.&rft.date=2012&rft.volume=13&rft.pages=8&rft_id=info:doi\/10.1186%2F1471-2105-13-8&rft_id=info:pmc\/PMC3292476&rft_id=info:pmid\/22239737&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3292476&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ODriscollBig13-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ODriscollBig13_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">O\u2019Driscoll, A.; Daugelaite, J.; Sleator, R.D. (2013). \"'Big data', Hadoop and cloud computing in genomics\". <i>Journal of Biomedical Informatics<\/i> <b>46<\/b> (5): 774-81. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.jbi.2013.07.001\" target=\"_blank\">10.1016\/j.jbi.2013.07.001<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23872175\" target=\"_blank\">23872175<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%27Big+data%27%2C+Hadoop+and+cloud+computing+in+genomics&rft.jtitle=Journal+of+Biomedical+Informatics&rft.aulast=O%E2%80%99Driscoll%2C+A.%3B+Daugelaite%2C+J.%3B+Sleator%2C+R.D.&rft.au=O%E2%80%99Driscoll%2C+A.%3B+Daugelaite%2C+J.%3B+Sleator%2C+R.D.&rft.date=2013&rft.volume=46&rft.issue=5&rft.pages=774-81&rft_id=info:doi\/10.1016%2Fj.jbi.2013.07.001&rft_id=info:pmid\/23872175&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiFastShort09-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LiFastShort09_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Durbin, R. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234\" target=\"_blank\">\"Fast and accurate short read alignment with Burrows\u2013Wheeler transform\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (14): 1754\u201360. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp324\" target=\"_blank\">10.1093\/bioinformatics\/btp324<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2705234\/\" target=\"_blank\">PMC2705234<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19451168\" target=\"_blank\">19451168<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2705234<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+and+accurate+short+read+alignment+with+Burrows%E2%80%93Wheeler+transform&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.%3B+Durbin%2C+R.&rft.au=Li%2C+H.%3B+Durbin%2C+R.&rft.date=2009&rft.volume=25&rft.issue=14&rft.pages=1754%E2%80%9360&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp324&rft_id=info:pmc\/PMC2705234&rft_id=info:pmid\/19451168&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2705234&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiFastLong09-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiFastLong09_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiFastLong09_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Durbin, R. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2828108\" target=\"_blank\">\"Fast and accurate long-read alignment with Burrows\u2013Wheeler transform\"<\/a>. <i>Bioinformatics<\/i> <b>26<\/b> (5): 589-95. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp698\" target=\"_blank\">10.1093\/bioinformatics\/btp698<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2828108\/\" target=\"_blank\">PMC2828108<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20080505\" target=\"_blank\">20080505<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2828108\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2828108<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Fast+and+accurate+long-read+alignment+with+Burrows%E2%80%93Wheeler+transform&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.%3B+Durbin%2C+R.&rft.au=Li%2C+H.%3B+Durbin%2C+R.&rft.date=2010&rft.volume=26&rft.issue=5&rft.pages=589-95&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp698&rft_id=info:pmc\/PMC2828108&rft_id=info:pmid\/20080505&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2828108&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LiTheSeq09-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-LiTheSeq09_9-0\" rel=\"external_link\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-LiTheSeq09_9-1\" rel=\"external_link\">9.1<\/a><\/sup> <sup><a href=\"#cite_ref-LiTheSeq09_9-2\" rel=\"external_link\">9.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Li, H.; Handsaker, B.; Wysoker, A. et al. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002\" target=\"_blank\">\"The Sequence Alignment\/Map format and SAMtools\"<\/a>. <i>Bioinformatics<\/i> <b>25<\/b> (16): 2078-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtp352\" target=\"_blank\">10.1093\/bioinformatics\/btp352<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2723002\/\" target=\"_blank\">PMC2723002<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/19505943\" target=\"_blank\">19505943<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2723002<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Sequence+Alignment%2FMap+format+and+SAMtools&rft.jtitle=Bioinformatics&rft.aulast=Li%2C+H.%3B+Handsaker%2C+B.%3B+Wysoker%2C+A.+et+al.&rft.au=Li%2C+H.%3B+Handsaker%2C+B.%3B+Wysoker%2C+A.+et+al.&rft.date=2009&rft.volume=25&rft.issue=16&rft.pages=2078-9&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtp352&rft_id=info:pmc\/PMC2723002&rft_id=info:pmid\/19505943&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2723002&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DePristoAFrame11-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DePristoAFrame11_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">DePristo, M.A.; Banks, E.; Poplin, R. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3083463\" target=\"_blank\">\"A framework for variation discovery and genotyping using next-generation DNA sequencing data\"<\/a>. <i>Nature Genetics<\/i> <b>43<\/b> (5): 491-8. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fng.806\" target=\"_blank\">10.1038\/ng.806<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3083463\/\" target=\"_blank\">PMC3083463<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21478889\" target=\"_blank\">21478889<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3083463\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3083463<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+framework+for+variation+discovery+and+genotyping+using+next-generation+DNA+sequencing+data&rft.jtitle=Nature+Genetics&rft.aulast=DePristo%2C+M.A.%3B+Banks%2C+E.%3B+Poplin%2C+R.+et+al.&rft.au=DePristo%2C+M.A.%3B+Banks%2C+E.%3B+Poplin%2C+R.+et+al.&rft.date=2011&rft.volume=43&rft.issue=5&rft.pages=491-8&rft_id=info:doi\/10.1038%2Fng.806&rft_id=info:pmc\/PMC3083463&rft_id=info:pmid\/21478889&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3083463&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShenSNP10-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ShenSNP10_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-ShenSNP10_11-1\" rel=\"external_link\">11.1<\/a><\/sup> <sup><a href=\"#cite_ref-ShenSNP10_11-2\" rel=\"external_link\">11.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shen, Y.; Wan, Z.; Coarfa, C. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813483\" target=\"_blank\">\"A SNP discovery method to assess variant allele probability from next-generation resequencing data\"<\/a>. <i>Genome Research<\/i> <b>20<\/b> (2): 273-80. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.096388.109\" target=\"_blank\">10.1101\/gr.096388.109<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2813483\/\" target=\"_blank\">PMC2813483<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20019143\" target=\"_blank\">20019143<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813483\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2813483<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+SNP+discovery+method+to+assess+variant+allele+probability+from+next-generation+resequencing+data&rft.jtitle=Genome+Research&rft.aulast=Shen%2C+Y.%3B+Wan%2C+Z.%3B+Coarfa%2C+C.+et+al.&rft.au=Shen%2C+Y.%3B+Wan%2C+Z.%3B+Coarfa%2C+C.+et+al.&rft.date=2010&rft.volume=20&rft.issue=2&rft.pages=273-80&rft_id=info:doi\/10.1101%2Fgr.096388.109&rft_id=info:pmc\/PMC2813483&rft_id=info:pmid\/20019143&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2813483&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MorrisonWhole13-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MorrisonWhole13_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Morrison, A.C.; Voorman, A.; Johnson, A.D. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030301\" target=\"_blank\">\"Whole-genome sequence\u2013based analysis of high-density lipoprotein cholesterol\"<\/a>. <i>Nature Genetics<\/i> <b>45<\/b> (8): 899-901. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fng.2671\" target=\"_blank\">10.1038\/ng.2671<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4030301\/\" target=\"_blank\">PMC4030301<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23770607\" target=\"_blank\">23770607<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030301\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4030301<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Whole-genome+sequence%E2%80%93based+analysis+of+high-density+lipoprotein+cholesterol&rft.jtitle=Nature+Genetics&rft.aulast=Morrison%2C+A.C.%3B+Voorman%2C+A.%3B+Johnson%2C+A.D.+et+al.&rft.au=Morrison%2C+A.C.%3B+Voorman%2C+A.%3B+Johnson%2C+A.D.+et+al.&rft.date=2013&rft.volume=45&rft.issue=8&rft.pages=899-901&rft_id=info:doi\/10.1038%2Fng.2671&rft_id=info:pmc\/PMC4030301&rft_id=info:pmid\/23770607&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4030301&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GoecksGalaxy10-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GoecksGalaxy10_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Goecks, J.; Nekrutenko, A.; Taylor, J.; The Galaxy Team (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788\" target=\"_blank\">\"Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences\"<\/a>. <i>Genome Biology<\/i> <b>11<\/b> (8): R86. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2010-11-8-r86\" target=\"_blank\">10.1186\/gb-2010-11-8-r86<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2945788\/\" target=\"_blank\">PMC2945788<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20738864\" target=\"_blank\">20738864<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2945788<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy%3A+a+comprehensive+approach+for+supporting+accessible%2C+reproducible%2C+and+transparent+computational+research+in+the+life+sciences&rft.jtitle=Genome+Biology&rft.aulast=Goecks%2C+J.%3B+Nekrutenko%2C+A.%3B+Taylor%2C+J.%3B+The+Galaxy+Team&rft.au=Goecks%2C+J.%3B+Nekrutenko%2C+A.%3B+Taylor%2C+J.%3B+The+Galaxy+Team&rft.date=2010&rft.volume=11&rft.issue=8&rft.pages=R86&rft_id=info:doi\/10.1186%2Fgb-2010-11-8-r86&rft_id=info:pmc\/PMC2945788&rft_id=info:pmid\/20738864&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2945788&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BlankenbergGalaxy10-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BlankenbergGalaxy10_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Blankenberg, D.; Von Kuster, G.; Coraor, N. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107\" target=\"_blank\">\"Galaxy: a web-based genome analysis tool for experimentalists\"<\/a>. <i>Current Protocols in Molecular Biology<\/i> <b>19<\/b> (Unit 19.10.1\u201321). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2F0471142727.mb1910s89\" target=\"_blank\">10.1002\/0471142727.mb1910s89<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4264107\/\" target=\"_blank\">PMC4264107<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20069535\" target=\"_blank\">20069535<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4264107<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy%3A+a+web-based+genome+analysis+tool+for+experimentalists&rft.jtitle=Current+Protocols+in+Molecular+Biology&rft.aulast=Blankenberg%2C+D.%3B+Von+Kuster%2C+G.%3B+Coraor%2C+N.+et+al.&rft.au=Blankenberg%2C+D.%3B+Von+Kuster%2C+G.%3B+Coraor%2C+N.+et+al.&rft.date=2010&rft.volume=19&rft.issue=Unit+19.10.1%E2%80%9321&rft_id=info:doi\/10.1002%2F0471142727.mb1910s89&rft_id=info:pmc\/PMC4264107&rft_id=info:pmid\/20069535&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4264107&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GiardineGalaxy05-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GiardineGalaxy05_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Giardine, B.; Riemer, C.; Hardison, R.C. et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089\" target=\"_blank\">\"Galaxy: A platform for interactive large-scale genome analysis\"<\/a>. <i>Genome Research<\/i> <b>15<\/b> (10): 1451\u20131455. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.4086505\" target=\"_blank\">10.1101\/gr.4086505<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1240089\/\" target=\"_blank\">PMC1240089<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16169926\" target=\"_blank\">16169926<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1240089<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy%3A+A+platform+for+interactive+large-scale+genome+analysis&rft.jtitle=Genome+Research&rft.aulast=Giardine%2C+B.%3B+Riemer%2C+C.%3B+Hardison%2C+R.C.+et+al.&rft.au=Giardine%2C+B.%3B+Riemer%2C+C.%3B+Hardison%2C+R.C.+et+al.&rft.date=2005&rft.volume=15&rft.issue=10&rft.pages=1451%E2%80%931455&rft_id=info:doi\/10.1101%2Fgr.4086505&rft_id=info:pmc\/PMC1240089&rft_id=info:pmid\/16169926&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1240089&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KallioChipster11-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KallioChipster11_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kallio, M.A.; Tuimala, J.T.; Hupponen, T. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215701\" target=\"_blank\">\"Chipster: user-friendly analysis software for microarray and other high-throughput data\"<\/a>. <i>BMC Genomics<\/i> <b>12<\/b>: 507. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2164-12-507\" target=\"_blank\">10.1186\/1471-2164-12-507<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3215701\/\" target=\"_blank\">PMC3215701<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21999641\" target=\"_blank\">21999641<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215701\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3215701<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Chipster%3A+user-friendly+analysis+software+for+microarray+and+other+high-throughput+data&rft.jtitle=BMC+Genomics&rft.aulast=Kallio%2C+M.A.%3B+Tuimala%2C+J.T.%3B+Hupponen%2C+T.+et+al.&rft.au=Kallio%2C+M.A.%3B+Tuimala%2C+J.T.%3B+Hupponen%2C+T.+et+al.&rft.date=2011&rft.volume=12&rft.pages=507&rft_id=info:doi\/10.1186%2F1471-2164-12-507&rft_id=info:pmc\/PMC3215701&rft_id=info:pmid\/21999641&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3215701&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OvaskaLarge10-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OvaskaLarge10_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ovaska, K.; Laakso, M.; Haapa-Paananen, S. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092116\" target=\"_blank\">\"Large-scale data integration framework provides a comprehensive view on glioblastoma multiforme\"<\/a>. <i>Genome Medicine<\/i> <b>2<\/b> (9): 65. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgm186\" target=\"_blank\">10.1186\/gm186<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3092116\/\" target=\"_blank\">PMC3092116<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20822536\" target=\"_blank\">20822536<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092116\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3092116<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Large-scale+data+integration+framework+provides+a+comprehensive+view+on+glioblastoma+multiforme&rft.jtitle=Genome+Medicine&rft.aulast=Ovaska%2C+K.%3B+Laakso%2C+M.%3B+Haapa-Paananen%2C+S.+et+al.&rft.au=Ovaska%2C+K.%3B+Laakso%2C+M.%3B+Haapa-Paananen%2C+S.+et+al.&rft.date=2010&rft.volume=2&rft.issue=9&rft.pages=65&rft_id=info:doi\/10.1186%2Fgm186&rft_id=info:pmc\/PMC3092116&rft_id=info:pmid\/20822536&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3092116&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AgrawalExome11-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AgrawalExome11_18-0\" rel=\"external_link\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-AgrawalExome11_18-1\" rel=\"external_link\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Agrawal, N.; Frederick, M.J.; Pickering, C.R. et al. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162986\" target=\"_blank\">\"Exome sequencing of head and neck squamous cell carcinoma reveals inactivating mutations in NOTCH1\"<\/a>. <i>Science<\/i> <b>333<\/b> (6046): 1154\u201357. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.1206923\" target=\"_blank\">10.1126\/science.1206923<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3162986\/\" target=\"_blank\">PMC3162986<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21798897\" target=\"_blank\">21798897<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162986\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3162986<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exome+sequencing+of+head+and+neck+squamous+cell+carcinoma+reveals+inactivating+mutations+in+NOTCH1&rft.jtitle=Science&rft.aulast=Agrawal%2C+N.%3B+Frederick%2C+M.J.%3B+Pickering%2C+C.R.+et+al.&rft.au=Agrawal%2C+N.%3B+Frederick%2C+M.J.%3B+Pickering%2C+C.R.+et+al.&rft.date=2011&rft.volume=333&rft.issue=6046&rft.pages=1154%E2%80%9357&rft_id=info:doi\/10.1126%2Fscience.1206923&rft_id=info:pmc\/PMC3162986&rft_id=info:pmid\/21798897&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3162986&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WangANNOVAR10-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-WangANNOVAR10_19-0\" rel=\"external_link\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-WangANNOVAR10_19-1\" rel=\"external_link\">19.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wang, K.; Li, M.; Hakonarson, H. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2938201\" target=\"_blank\">\"ANNOVAR: Functional annotation of genetic variants from high-throughput sequencing data.\"<\/a>. <i>Genetic Acids Research<\/i> <b>38<\/b> (16): e164. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgkq603\" target=\"_blank\">10.1093\/nar\/gkq603<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2938201\/\" target=\"_blank\">PMC2938201<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20601685\" target=\"_blank\">20601685<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2938201\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2938201<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ANNOVAR%3A+Functional+annotation+of+genetic+variants+from+high-throughput+sequencing+data.&rft.jtitle=Genetic+Acids+Research&rft.aulast=Wang%2C+K.%3B+Li%2C+M.%3B+Hakonarson%2C+H.&rft.au=Wang%2C+K.%3B+Li%2C+M.%3B+Hakonarson%2C+H.&rft.date=2010&rft.volume=38&rft.issue=16&rft.pages=e164&rft_id=info:doi\/10.1093%2Fnar%2Fgkq603&rft_id=info:pmc\/PMC2938201&rft_id=info:pmid\/20601685&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2938201&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LupskiExome13-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LupskiExome13_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lupski, J.R.; Gonzaga-Jauregui. C.; Yang, Y. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3706849\" target=\"_blank\">\"Exome sequencing resolves apparent incidental findings and reveals further complexity of SH3TC2 variant alleles causing Charcot-Marie-Tooth neuropathy\"<\/a>. <i>Genome Medicine<\/i> <b>5<\/b> (6): 57. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgm461\" target=\"_blank\">10.1186\/gm461<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3706849\/\" target=\"_blank\">PMC3706849<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23806086\" target=\"_blank\">23806086<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3706849\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3706849<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exome+sequencing+resolves+apparent+incidental+findings+and+reveals+further+complexity+of+SH3TC2+variant+alleles+causing+Charcot-Marie-Tooth+neuropathy&rft.jtitle=Genome+Medicine&rft.aulast=Lupski%2C+J.R.%3B+Gonzaga-Jauregui.+C.%3B+Yang%2C+Y.+et+al.&rft.au=Lupski%2C+J.R.%3B+Gonzaga-Jauregui.+C.%3B+Yang%2C+Y.+et+al.&rft.date=2013&rft.volume=5&rft.issue=6&rft.pages=57&rft_id=info:doi\/10.1186%2Fgm461&rft_id=info:pmc\/PMC3706849&rft_id=info:pmid\/23806086&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3706849&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Liu_dbNSFP11-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Liu_dbNSFP11_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Liu, X.; Jian, X.; Boerwinkle, E. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145015\" target=\"_blank\">\"dbNSFP: a lightweight database of human nonsynonymous SNPs and their functional predictions\"<\/a>. <i>Human Mutation<\/i> <b>32<\/b> (8): 894\u2013899. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1002%2Fhumu.21517\" target=\"_blank\">10.1002\/humu.21517<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3145015\/\" target=\"_blank\">PMC3145015<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21520341\" target=\"_blank\">21520341<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145015\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3145015<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=dbNSFP%3A+a+lightweight+database+of+human+nonsynonymous+SNPs+and+their+functional+predictions&rft.jtitle=Human+Mutation&rft.aulast=Liu%2C+X.%3B+Jian%2C+X.%3B+Boerwinkle%2C+E.&rft.au=Liu%2C+X.%3B+Jian%2C+X.%3B+Boerwinkle%2C+E.&rft.date=2011&rft.volume=32&rft.issue=8&rft.pages=894%E2%80%93899&rft_id=info:doi\/10.1002%2Fhumu.21517&rft_id=info:pmc\/PMC3145015&rft_id=info:pmid\/21520341&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3145015&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LupskiWhole10-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LupskiWhole10_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Lupski, J.R; Reid, J.G.; Gonzaga-Jauregui, C. et al. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4036802\" target=\"_blank\">\"Whole-genome sequencing in a patient with Charcot-Marie-Tooth neuropathy\"<\/a>. <i>The New England Journal of Medicine<\/i> <b>362<\/b> (13): 1181\u201391. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1056%2FNEJMoa0908094\" target=\"_blank\">10.1056\/NEJMoa0908094<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4036802\/\" target=\"_blank\">PMC4036802<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20220177\" target=\"_blank\">20220177<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4036802\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4036802<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Whole-genome+sequencing+in+a+patient+with+Charcot-Marie-Tooth+neuropathy&rft.jtitle=The+New+England+Journal+of+Medicine&rft.aulast=Lupski%2C+J.R%3B+Reid%2C+J.G.%3B+Gonzaga-Jauregui%2C+C.+et+al.&rft.au=Lupski%2C+J.R%3B+Reid%2C+J.G.%3B+Gonzaga-Jauregui%2C+C.+et+al.&rft.date=2010&rft.volume=362&rft.issue=13&rft.pages=1181%E2%80%9391&rft_id=info:doi\/10.1056%2FNEJMoa0908094&rft_id=info:pmc\/PMC4036802&rft_id=info:pmid\/20220177&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4036802&rfr_id=info:sid\/en.wikipedia.org:Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210218\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.663 seconds\nReal time usage: 0.689 seconds\nPreprocessor visited node count: 20251\/1000000\nPreprocessor generated node count: 31110\/1000000\nPost\u2010expand include size: 197499\/2097152 bytes\nTemplate argument size: 61729\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 644.752 1 - -total\n 83.46% 538.106 1 - Template:Reflist\n 77.37% 498.833 22 - Template:Cite_journal\n 74.14% 478.002 22 - Template:Citation\/core\n 13.28% 85.597 63 - Template:Citation\/identifier\n 11.25% 72.542 1 - Template:Infobox_journal_article\n 10.88% 70.152 1 - Template:Infobox\n 6.47% 41.731 80 - Template:Infobox\/row\n 4.80% 30.918 145 - Template:Hide_in_print\n 4.67% 30.080 22 - Template:Citation\/make_link\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7904-0!*!0!!en!5!* and timestamp 20181213210217 and revision id 23703\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline\">https:\/\/www.limswiki.org\/index.php\/Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","a5e9f45d3e7fa0bbfb4350821efaa31e_images":["https:\/\/www.limswiki.org\/images\/a\/a9\/Fig1_Reid_BMCInformatics2014_15.jpg","https:\/\/www.limswiki.org\/images\/d\/d4\/Fig2_Reid_BMCInformatics2014_15.jpg"],"a5e9f45d3e7fa0bbfb4350821efaa31e_timestamp":1544734937,"59816fe6940deb7e43cf53893a6ea71a_type":"article","59816fe6940deb7e43cf53893a6ea71a_title":"iLAP: A workflow-driven software for experimental protocol development, data acquisition and analysis (Stocker et al. 2009)","59816fe6940deb7e43cf53893a6ea71a_url":"https:\/\/www.limswiki.org\/index.php\/Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis","59816fe6940deb7e43cf53893a6ea71a_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:iLAP: A workflow-driven software for experimental protocol development, data acquisition and analysis\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\t \n\nFull article title\n \niLAP: a workflow-driven software for experimental protocol development, data acquisition and analysisJournal\n \nBMC BioinformaticsAuthor(s)\n \nStocker, Gernot; Fischer, Maria; Rieder, Dietmar; Bindea, Gabriela; Kainz, Simon; Oberstolz, Michael; McNally, James G.; Trajanoski, ZlatkoAuthor affiliation(s)\n \nInstitute for Genomics and Bioinformatics, Graz University of Technology; National Cancer Institute, National Institutes of HealthPrimary contact\n \nEmail: zlatko.trajanoski@tugraz.atYear published\n \n2009Volume and issue\n \n10Page(s)\n \n390DOI\n \n10.1186\/1471-2105-10-390ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 2.0 GenericWebsite\n \nhttp:\/\/www.biomedcentral.com\/1471-2105\/10\/390Download\n \nhttp:\/\/www.biomedcentral.com\/content\/pdf\/1471-2105-10-390.pdf (PDF)\n\nContents\n\n1 Abstract \n\n1.1 Background \n1.2 Results \n1.3 Conclusion \n\n\n2 Background \n3 Implementation \n\n3.1 Workflow-driven software design \n3.2 Project definition phase \n3.3 Experimental design and data acquisition \n3.4 Data analysis and processing \n3.5 Data retrieval \n3.6 Software architecture \n3.7 Presentation tier \n3.8 Business tier and runtime environment \n3.9 External programming interfaces \n3.10 Asynchronous data processing \n\n\n4 Results \n\n4.1 Functional overview \n4.2 Hierarchical organization of experimental data \n4.3 Protocol development \n4.4 Data acquisition and analysis \n4.5 Data analysis \n4.6 Integration of external programs \n4.7 Automatic post processing tool chain \n4.8 Data retrieval and information sharing \n4.9 Case Study \n\n\n5 Conclusion \n6 Availability and requirements \n7 Abbreviations \n8 Authors' contributions \n9 Acknowledgements \n10 References \n11 Notes \n\n\n\nAbstract \nBackground \nIn recent years, the genome biology community has expended considerable effort to confront the challenges of managing heterogeneous data in a structured and organized way and developed laboratory information management systems (LIMS) for both raw and processed data. On the other hand, electronic notebooks were developed to record and manage scientific data, and facilitate data-sharing. Software which enables both, management of large datasets and digital recording of laboratory procedures would serve a real need in laboratories using medium and high-throughput techniques.\n\nResults \nWe have developed iLAP (Laboratory data management, Analysis, and Protocol development), a workflow-driven information management system specifically designed to create and manage experimental protocols, and to analyze and share laboratory data. The system combines experimental protocol development, wizard-based data acquisition, and high-throughput data analysis into a single, integrated system. We demonstrate the power and the flexibility of the platform using a microscopy case study based on a combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol and 3D-image reconstruction. iLAP is freely available under the open source license AGPL from http:\/\/genome.tugraz.at\/iLAP\/. (Webcite)\n\nConclusion \niLAP is a flexible and versatile information management system, which has the potential to close the gap between electronic notebooks and LIMS and can therefore be of great value for a broad scientific community.\n\nBackground \nThe development of novel large-scale technologies has considerably changed the way biologists perform experiments. Genome biology experiments do not only generate a wealth of data, but they often rely on sophisticated laboratory protocols comprising hundreds of individual steps. For example, the protocol for chromatin immunoprecipitation on a microarray (Chip-chip) has 90 steps, uses over 30 reagents and 10 different devices.[1] Even adopting an established protocol for large-scale studies represents a daunting challenge for the majority of the labs. The development of novel laboratory protocols and\/or the optimization of existing ones is still more distressing, since this requires systematic changes of many parameters, conditions, and reagents. Such changes are becoming increasingly difficult to trace using paper lab books. A further complication for most protocols is that many laboratory instruments are used, which generate electronic data stored in an unstructured way at disparate locations. Therefore, protocol data files are seldom or never linked to notes in lab books and can be barely shared within or across labs. Finally, once the experimental large-scale data have been generated, they must be analyzed using various software tools, then stored and made available for other users. Thus, it is apparent that software support for current biological research \u2014 be it genomic or performed in a more traditional way \u2014 is urgently needed and inevitable. \nIn recent years, the genome biology community has expended considerable effort to confront the challenges of managing heterogeneous data in a structured and organized way and as a result developed information management systems for both raw and processed data. Laboratory information management systems (LIMS) have been implemented for handling data entry from robotic systems and tracking samples[2][3] as well as data management systems for processed data including microarrays[4][5], proteomics data[6][7][8], and microscopy data.[9] The latter systems support community standards like FUGE[10][11], MIAME[12], MIAPE[13], or MISFISHIE[14] and have proven invaluable in a state-of-the-art laboratory. In general, these sophisticated systems are able to manage and analyze data generated for only a single type or a limited number of instruments, and were designed for only a specific type of molecule. \nOn the other hand, commercial as well as open source electronic notebooks[15][16][17][18][19] were developed to record and manage scientific data, and facilitate data-sharing. The influences encouraging the use of electronic notebooks are twofold.[20] First, much of the data that needs to be recorded in a laboratory notebook is generated electronically. Transcribing data manually into a paper notebook is error-prone, and in many cases, for example, analytical data (spectra, chromatograms, photographs, etc.), transcription of the data is not possible. Second, the incorporation of high-throughput technologies into the research process has resulted in an increased volume of electronic data that need to be transcribed. As opposed to LIMS, which captures highly structured data through rigid user interfaces with standard report formats, electronic notebooks contain unstructured data and have flexible user interfaces.\nSoftware which enables both, management of large datasets and recording of laboratory procedures, would serve a real need in laboratories using medium and high-throughput techniques. To the best of our knowledge, there is no software system available, which supports tedious protocol development in an intuitive way, links the plethora of generated files to the appropriate laboratory steps and integrates further analysis tools. We have therefore developed iLAP, a workflow-driven information management system for protocol development and data management. The system combines experimental protocol development, wizard-based data acquisition, and high-throughput data analysis into a single, integrated system. We demonstrate the power and the flexibility of the platform using a microscopy case study based on combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol and 3D-image reconstruction.\n\nImplementation \nWorkflow-driven software design \nThe design of a software platform that supports the development of protocols and data management in an experimental context has to be based on and directed by the laboratory workflow. The laboratory workflow can be divided into four principal steps: 1) project definition phase, 2) experimental design and data acquisition phase, 3) data analysis and processing phase and 4) data retrieval phase (Figure 1). \n\n\n\nFigure 1: Mapping of the laboratory workflow onto iLAP features. The software design of iLAP is inspired by a typical laboratory workflow in life sciences and offers software assistance during the process.\r\nThe figure illustrates on the left panel the scientific workflow separated into four phases: project definition, data acquisition and analysis, and data retrieval. The right panel shows the main functionalities offered by iLAP.\nProject definition phase \nA scientific project starts with a hypothesis and the choice of methods required to address a specific biological question. Already during this initial phase it is crucial to define the question as specifically as possible and to capture the information in a digital form. Documents collected during the literature research should be collated with the evolving project definition for later review or for sharing with other researchers. All files collected in this period should be attached to the defined projects and experiments in the software.\n\nExperimental design and data acquisition \nFollowing the establishment of a hypothesis and based on preliminary experiments, the detailed design of the biological experiments is then initiated. Usually, the experimental work follows already established standard operating procedures, which have to be modified and optimized for the specific biological experiment. These protocols are defined as a sequence of protocol steps. However, well-established protocols must be kept flexible in a way that particular conditions can be changed. The typically changing parameters of standard protocol steps (e.g. fixation times, temperature changes etc.) are important to record as they are used to improve the experimental reproducibility.\nEquipped with a collection of standard operating procedures, an experiment can be initiated and the data generated. In general, data acquisition comprises not only files but also observations of interest, which might be relevant for the interpretation of the results. Most often these observations disappear in paper notebooks and are not accessible in a digital form. Hence, these experimental notes should be stored and attached to the originating protocol step, experiment or project.\n\nData analysis and processing \nAfter storing the raw result files, additional analysis and post-processing steps must be performed to obtain processed data for subsequent analysis. In order to extract information and to combine it in a statistically meaningful manner, multiple data sets have to be acquired. The software workflow should enable also the inclusion of external analytical steps, so that files resulting from external analysis software can be assigned to their original raw data files. Finally, the data files generated at the analysis stage should be connected to the raw data, allowing connection of the data files with the originating experimental context.\n\nData retrieval \nBy following the experimental workflow, all experimental data e.g. different files, protocols, notes etc. should be organized in a chronological and project-oriented way and continuously registered during their acquisition. An additional advantage should be the ability to search and retrieve the data. Researchers frequently have to search through notebooks to find previously uninterpretable observations. Subsequently, as the project develops, the researchers gain a different perspective and recognize that prior observations could lead to new discoveries. Therefore, the software should offer easy to use interfaces that allow searches through observation notes, projects- and experiment descriptions.\n\nSoftware architecture \niLAP is a multi-tier client-server application and can be subdivided into different functional modules which interact as self-contained units according to their defined responsibilities (see Figure 2). \n\n\n\nFigure 2: Software Architecture. iLAP features a typical three-tier architecture and can hence be divided into a presentation tier, business tier and a persistence tier (from left to right). The presentation tier is formed by a graphical user interface, accessed using a web browser. The following business layer is protected by a security layer, which enforces user authentication and authorization. After access is granted, the security layer passes the user requests to the business layer, which is mainly responsible for guiding the user through the laboratory workflow. This layer also coordinates all background tasks like automatic surveying of analysis jobs on a computing cluster or synchronizing\/exchanging data with further downstream applications. (e.g. OMERO (open microscopy environment) image server). Finally, the persistence layer interacts with the relational database.\nPresentation tier \nThe presentation tier within iLAP is formed by a Web interface, using Tapestry[21] as the model view controller and an Axis Web service[22], which allows programming access to parts of the application logic. Thus, on the client side, a user requires an Internet connection and a recent Web browser with Java Applet support, available for almost every platform. In order to provide a simple, consistent but also attractive Web interface, iLAP follows usability guidelines described in[23][24] and uses Web 2.0 technologies for dynamic content generation. \n\nBusiness tier and runtime environment \nThe business tier is realized as view-independent application logic, which stores and retrieves datasets by communicating with the persistence layer. The internal management of files is also handled from a central service component, which persists the meta-information for acquired files to the database, and stores the file content in a file-system-based data hierarchy. The business layer also holds asynchronous services for application-internal JMS messaging and for integration of external computing resources like high-performance computing clusters. All services of this layer are implemented as Spring[25] beans, for which the Spring-internal interceptor classes provide transactional integrity.\nThe business tier and the persistence tier are bound by the Spring J2EE lightweight container, which manages the component-object life cycle. Furthermore, the Spring context is transparently integrated into the Servlet context of Tapestry using the HiveMind[26] container backend. This is realized by using the automatic dependency injection functionality of HiveMind which avoids integrative glue code for lookups into the Spring container. Since iLAP uses Spring instead of EJB related components, the deployment of the application only requires a standard conformed Servlet container. Therefore, the Servlet container Tomcat[27] is used, which offers not only Servlet functionality but J2EE infrastructure services[28] such as centrally configured data-sources and transaction management realized with the open source library JOTM.[29] This makes the deployment of iLAP on different servers easier, because machine-specific settings for different production environments are kept outside the application configuration. \n\nExternal programming interfaces \nThe SOAP Web service interface for external programmatic access is realized by combining the Web service framework Axis with corresponding iLAP components. The Web service operates as an external access point for Java Applets within the Web application, as well as for external analysis and processing applications such as ImageJ.\nModel driven development\nIn order to reduce coding and to increase the long term maintainability, the model driven development environment AndroMDA[30] is used to generate components of the persistence layer and recurrent parts from the above mentioned business layer. AndroMDA accomplishes this by translating an annotated UML-model into a JEE-platform-specific implementation using Hibernate and Spring as base technology. Due to the flexibility of AndroMDA, application external services, such as the user management system, have a clean integration in the model. Dependencies of internal service components on such externally defined services are cleanly managed by its build system.\nBy changing the build parameters in the AndroMDA configuration, it is also possible to support different relational database management systems. This is because platform specific code with the same functionality is generated for data retrieval. Furthermore, technology lock-in regarding the implementation of the service layers was also addressed by using AndroMDA, as the implementation of the service facade can be switched during the build process from Spring based components to distributed Enterprise Java Beans. At present, iLAP is operating on one local machine and, providing the usage scenarios do not demand it, this architectural configuration will remain. However, chosen technologies are known to work on Web server farms and crucial distribution of the application among server nodes is transparently performed by the chosen technologies. \n\nAsynchronous data processing \nThe asynchronous handling of business processes is realized in iLAP with message-driven Plain Old Java Objects (POJOs). Hence, application tasks, such as the generation of image previews, can be performed asynchronously. If performed immediately, these would unnecessarily block the responsiveness of the Web front-end. iLAP delegates tasks via JMS messages to back-end services, which perform the necessary processing actions in the background.\nThese back-end services are also UML-modelled components and receive messages handled by the JMS provider ActiveMQ. If back-end tasks consume too many calculation resources, the separation of Web front-end and JMS message receiving services can be realized by copying the applications onto two different servers and changing the Spring JMS configuration.\nFor the smooth integration of external computing resources like the high-performance computing cluster or special compute nodes with limited software licenses the JClusterService is used. JClusterService is a separately developed J2EE application which enables a programmer to run generic applications on a remote execution host or high-performance computing cluster. Every application which offers a command line interface can be easily integrated by defining a service definition in XML format and accessing it via a SOAP-based programming interface from any Java-application. The execution of the integrated application is carried out either by using the internal JMS-queuing system for single host installations or by using the open source queuing systems like Sun Grid Engine (Sun Microsystems) or OpenPBS\/Torque. \n\nResults \nFunctional overview \nThe functionality offered by the iLAP web interface can be described by four components: 1) hierarchical organization of the experimental data, 2) protocol development, 3) data acquisition and analysis, and 4) data retrieval and data sharing (Figure 1). iLAP specific terms are summarized in Table 1. \n\n\n\n\n\n\n\nTable 1. iLAP Terminology\n\n\niLAP specific terms\n\n Description\n\n\nProject\n\nLogical unit which can be structured hierarchically and holds experiments, notes and other files (e.g. derived from literature research).\n\n\nExperiment\n\nLogical unit which corresponds to one biological experiment and holds a current working protocol, experiment specific documentation files, parameter values, raw files, notes, and analysis steps.\n\n\nStandard Protocol\n\nFrequently used and well established protocol template also known as standard operating procedures (SOP).\n\n\nCurrent working protocol\n\nSequence of protocol steps for a specific experiment which holds raw files, notes and experiment specific parameter values.\n\n\nProtocol step\n\nOne single step in a protocol which is defined by a name, description, and a list of definable parameters. A sequence of protocol steps defines a protocol.\n\n\nStep group\n\nProtocol step which groups multiple protocol steps to a logical unit. It can be used as a step container for sequentially executed protocol steps or within split steps.\n\n\nSplit step\n\nProtocol step which can contain multiple (step groups) which have to be executed concurrently.\n\n\nProtocol step parameter\n\nChanging parameters which are associated with a step and can hold either textual or numerical values as well as a selection from a predefined value list (enumeration).\n\n\nNote\n\nNotes are textual descriptions which are intended to be used for documenting abnormal observations at almost anywhere within iLAP.\n\n\nRaw file\n\nRaw files are files which are produced by laboratory instruments and are not processed by any analysis step captured within iLAP.\n\n\nAnalysis step\n\nDescription of a processing step which manipulates, analyzes or processes a raw file, and generates processed files which are linked to the original raw file. Analysis steps can be either external e.g. using external software or internal using iLAP-internal analysis modules.\n\n\nAnalysis step parameter\n\nParameters and values used during the analysis step.\n\n\n\nHierarchical organization of experimental data \nThis part of the user interface covers the project definition phase of the experimental workflow. The definition of projects and experiments consists solely in inserting the required descriptive parameters via a Web form. In doing so, a hierarchical structure with projects, sub-projects and experiments is created and displayed in the iLAP overview. The hierarchy (Figure 3) and other screen shots can be found in the iLAP user manual (Additional file 1). This overview is the starting point of iLAP, from which almost every activity can be initiated. By navigating through the tree, an information box appears alongside. This box details information about the current node in the tree and the operations which can be performed on the database managed object represented by the node. Already in this early stage, files derived from literature research can be uploaded to projects and experiments, and ongoing observations can be stored using the general note dialog. If multiple files must be associated with projects and experiments, a Java Applet can be used to upload the files to the generated project\/experiment structure. iLAP can manage every file independent of their file type, and can thus be considered as a generic document management system. File types only need to be considered for subsequent processing and data extraction. \n\n\n\nFigure 3: Hierarchical organization of data in iLAP overview. The continuous use of iLAP inherently leads to structured recording of experiments, conserving the complete experimental context of data records throughout the history of the research project. In doing so, a hierarchical structure with projects, sub-projects and experiments is created and can be displayed in this iLAP overview tree. The P-icons in the tree stand for projects and sub-projects, the E-icon for experiments and the A-icon for analysis steps. Files attached to protocol steps are considered as raw files and are therefore collected under the step container visualized with the R-icon. The consistent association of color schemes to logical units like projects, experiments, etc. can be directly recognized in this overview. By clicking on one of the tree icons on the left hand a detailed overview appears about the selected item. Also actions like creation of new projects etc. can be directly initiated using the quick-links in the \"Actions\" section of \"Details\".\n\n\n\n\n\n\n Additional file 1. iLAP user manual\n\n\nThe iLAP user manual contains a detailed description of the user interface including screen shots.\nFormat: PDF; Size: 1.8MB Download file\n\n\n\n\nProtocol development \nWhen starting experimental work, the iLAP facility manager should define commonly used standard protocols using the protocol development masks. Therefore, a sequence of steps must be defined which describes the typical ongoing experiment in detail. Dynamic protocol parameters, which may be adapted for protocol optimization during the experiment, can be associated with the pre-defined steps. These parameters can be either numerical values, descriptive text or predefined enumeration types, all of which can be preset by default values and marked with appropriate units. In order to force the acquisition of critical parameters in the data acquisition wizard, parameters can be marked as required. According to our experience and the experience of other users, it is helpful to define small and reusable standard protocol units, which can be used as building blocks during the experiment-specific protocol assembly. Automatic internal versioning takes care of changes in standard protocols so that dependent protocols used in previous experiments remain unaffected.\nEquipped with a collection of standard protocols, an experiment can be initiated and should be defined at the beginning of the workflow. The name of each experiment, its general description and specific aims, must be provided in order to be able to distinguish between different experiments. The detailed experiment procedure is defined by its current working protocol which can be composed step by step or by reusing existing current working protocols from already performed experiments. If the experiment is following a standard protocol, the current working protocol should be created by simply copying the predefined standard protocol steps and parameter definitions. In order to consider also the concurrent nature of simultaneously executed steps the experimenter should be able to define different sub-branches (e.g. cells are treated with different drugs in order to study their response) named split steps. These split steps lead to different branches of the experimental workflow called step groups which are separately handled during the data acquisition phase.\nOnce the protocol design phase is completed and all necessary protocol steps with their parameters are defined the researchers should be able to generate a printout of the current working protocol with which the experiment can be performed at the lab bench. \n\nData acquisition and analysis \nAfter having finished all experimental work and having created raw data files with different laboratory instruments the data acquisition within iLAP should be performed. By going through the early defined current working protocol steps, generated raw data files, used protocol parameter values and observation notes must be entered. Wizard-based input masks (wizard), which are derived from the defined current protocol steps, assist the experimenters during this work. On every step the user has to fill in the value fields for required parameters and can attach files and notes to each of the steps. During the creation of the working protocol, it is important to name those steps to which files are attached in a descriptive way. Files that are directly connected to experimental steps are considered as raw files and are protected against deletion. Note, files can be linked to the protocol steps anywhere in iLAP, i.e. also before and after the data acquisition.\nFor this data association, the iLAP workflow offers also the possibility to transfer all generated files to a central repository and associate automatically files with their generating protocol step at once, using a Java Applet. All the internal linkages to protocol steps, experiments or projects are performed automatically without the need of any user interference. As the files are attached to a protocol and an experiment, the overall context is preserved and the likelihood of reproducibility of the same conditions is increased. Within iLAP experimental notes are stored and attached to the originating protocol step, experiment or project and are retrievable using a keyword based search mask\n\nData analysis \nThe analysis steps are recorded in iLAP by either reusing existing analysis templates or describing new analysis steps applied to the previously uploaded raw data files. Additional analysis tools can be developed in Java as described in the iLAP user manual (Additional file 1). According to the file type, internally implemented analysis steps or the description of externally performed analysis steps are associated with the raw data files. Result files from analysis programs together with the used parameters can be easily attached to analysis definitions. As an example, a server analysis tool was implemented for deconvolving three dimensional image stacks, executed on a remote high-performance computing cluster using the JClusterService (see methods). \n\nIntegration of external programs \nA proof of concept about external access of programs using the iLAP application programming interface was shown by the implementation of a plugin for the widely used image processing software ImageJ.[31][32] This Java plugin enables ImageJ to transfer the image files directly from iLAP to the client machine. This functionality appears as a regular dialog in the graphical user interface of ImageJ, and allows upload of result files back into iLAP in a transparent manner. \n\nAutomatic post processing tool chain \nBackground tasks like the generation of previews are performed using the internal post-processing tool chain which is started asynchronously as soon as the files are associated with the originating experiment in iLAP. According to the detected file type, multiple post-processor steps are executed and results are automatically stored back into the database. This flexible system approach is also used to automatically inform and synchronize further downstream applications like OMERO[9] image server from the Open Microscopy Environment project. Therefore, iLAP is able to transfer files \u2014 transparently for the user \u2014 to a server where a comparable project\/dataset structure is created.\n\nData retrieval and information sharing \nThe use of the described data acquisition features inherently leads to structured recording of experiments, conserving the complete experimental context of data records throughout the history of research projects. It is often necessary to go back to already completed experiments and to search through old notes. Therefore, iLAP offers search masks which allow keyword based searching in the recorded projects, experiments and notes. These results are often discussed with collaboration partners to gain different opinions on the same raw data.\nIn order to allow direct collaboration between scientists iLAP is embedded into a central user management system[4] which offers multiple levels of access control to projects and their associated experimental data. The sharing of projects can be done on a per-user basis or on an institutional basis. For small or local single-user installations, the fully featured user management system can be replaced by a file-based user management which still offers the same functionalities from the sharing point of view, but lacks institute-wide functionalities (Additional file 2). This is only possible because iLAP keeps the source of user accounts separated from the internal access control to enable easy integration of additional local or institution wide user management systems. \n\n\n\n\n\n\n\n Additional file 2. iLAP installation and administration manual\n\n\nThe iLAP installation and administration manual contains a detailed description of the installation process for all supported platforms including screen shots.\nFormat: PDF; Size: 1.8MB Download file\n\n\n\n\nSince sophisticated protocols are crucial for successful experiments iLAP-users can export their protocols not only in PDF format (Additional file 3) but also in an exchangeable XML format (Additional file 4 and 5). In that way scientists can directly pass over their optimized protocols to partners who do not share the data using iLAP internally but need to get the protocol information transferred. The same XML files can be also used on a broader basis for protocol exchange using central eScience platforms like MyExperiments.[33] This platform aims for an increased reuse and repurpose of commonly shared workflows achieving at the same time reduced time-to-experiment and avoiding reinvention. Ongoing standardization efforts regarding the XML format like FuGE[10][11] are currently not supported but could be integrated in future versions of iLAP. \n\n\n\n\n\n\n\n Additional file 3. m-FISH protocol in PDF format\n\n\nThis file contains the combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol in PDF format used in the case study section.\nFormat: PDF; Size: 51KB Download file\n\n\n\n\n\n\n\n\n\n\n Additional file 4. m-FISH protocol in XML format\n\n\nThis file contains the combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol in XML format used in the case study section for protocol exchange.\nFormat: XML; Size: 64KB Download file\n\n\n\n\n\n\n\n\n\n\n Additional file 5. Document Type Definition for the XML protocol format\n\n\nThis file contains the Document Type Definition for the XML format used for protocol exchange created in collaboration with Will Moore from the OMERO.editor project.\nFormat: DTD; Size: 4KB Download file\n\n\n\n\nCase Study \nIn order to test the functionality of the system, we used a high-throughput microscopy study. The focus of this study was on the three dimensional nuclear localization of a group of seven genes. This required the development of a combinatorial multicolor fluorescence in situ hybridization (m-FISH) protocol. This protocol enables simultaneous detection and visualization of all seven genes by using a combination of three different fluorescent labels. The elaboration and optimization of m-FISH required many different protocol steps and parameters. Thus it was crucial to keep a record of any parameter and procedure changes during the process of protocol development. These changes were directly connected with data produced in the lab (e.g. concentration of the FISH probes, probe labeling efficiencies etc.) and the resulting imaging data. In the final combinatorial m-FISH protocol, 70 steps and 139 different parameters were present. Using this protocol we conducted 10 experiments and produced 1,441 multicolor 3D-Image stacks of which 984 were subsequently corrected for color shifts and processed by 3D-deconvolution performing 100 iterations of the maximum likelihood estimation algorithm available with the Huygens Deconvolution Software (Scientific Volume Imaging - SVI). These image processing steps were realized as batch analysis in iLAP, which delegated the compute intensive procedure to a high-performance computing cluster and then stored all processed image stacks in the analysis container of the corresponding experiments. Afterwards FISH signals were detected and analyzed using a custom image analysis procedure which was realized as a Matlab (MathWorks Inc.) extension of Imaris (Bitplane Inc.) using the Imaris-XT programming interface. This extension automatically recorded FISH signal coordinates, signal to signal distances, the nuclear volume and several additional parameters of each imaged nucleus. These externally generated data files were transferred back into iLAP and stored in the context of the corresponding experiment as an external analysis step. A summary of the data acquisition and analysis is shown in Figure 4. \n\n\n\nFigure 4: Case study summary. The functionality of iLAP was tested in a high-throughput microscopy study. The figure illustrates a summary of the data acquisition and data analysis performed. In 10 experiments a protocol consisting of 70 steps with 139 different parameters was used to generate three-dimensional multicolor image stacks. Each of the 1,441 raw image stacks consisted of 28 optical sections (slices) where each slice was recorded in 4 different channels. The raw image stacks were stored in the iLAP system and thereby connected with the corresponding experiments and protocols. By utilizing the integrated analysis functionality of iLAP the 984 raw images processed by the Huygens 3D-deconvolution package and analyzed by an external semiautomatic procedure implemented in Matlab and Imaris-XT. The analytical pipeline produced data for 121 different distance measurements of each single image. The resulting images and data were then stored in their experimental context within the iLAP system.\nDuring the course of the study we observed several clear advantages of the iLAP system over a lab-book in paper form, which was maintained in parallel. The first and most valuable feature of iLAP is the direct connection between protocol steps and data files which cannot be realized using a paper lab book. A second notable advantage of the iLAP system was that lab-tasks that were performed in parallel or in overlapping time-frames could also be stored as such, whereas in the traditional lab book all tasks performed in the lab were written sequentially which implied a break-up of connected protocols. A third advantage was that iLAP allowed for rapid searching and finding of experiments, protocols and desired terms, which required only a few mouse clicks as opposite to the cumbersome search using a paper notebook. Moreover, iLAP enabled easy collaboration functionality, data backup or parameter completeness checks. \n\nConclusion \nWe have developed a unique information management system specifically designed to support the creation and management of experimental protocols, and to analyze and share laboratory data. The design of the software was guided by the laboratory workflow and resulted in four unified components accessible through a web interface. The first component allows the hierarchical organization of the experimental data, which is organized in a generic document management system. The second component focuses on protocol development using templates of standard operating procedures. Next, the data acquisition and analysis component offers the possibility to transfer the generated files to a central repository and to associate the files with the corresponding protocol steps. Additionally, external data analysis programs can be integrated and executed on a remote high-performance computing cluster. The last component enables collaboration and data sharing between scientists using iLAP on a user or institutional level as well as protocol transfer with external users.\nAlthough designed in an experimental context for high-throughput protocols like microarray studies of gene expression, DNA-protein binding, proteomics experiments, or high-content image-based screening studies, iLAP has also proven to be valuable in low- and medium-throughput experiments. For example, protocols for qPCR analysis of gene expression using 96 and 384-well formats -a widely used technique- can be easily developed and can contribute significantly to establishment of robust assays. Moreover, since the workflow-oriented concept of iLAP offers the flexibility of a more general scientific data management system it is not limited to a special laboratory protocol, instrument, or type of molecule. For example, its application for next-generation sequencing is straightforward since similar requirements on the computational environment (increasing amount of data, integration of analysis tools, or use of high-performance computing infrastructure) have to be met.\nIn summary, we have developed a flexible and versatile information management system, which has the potential to close the gap between electronic notebooks and LIMS and can therefore be of great value for a broader community. Extensive tests in our and other labs have shown that the benefits of better information access and data sharing immediately result in reduced time spent managing information, increased productivity, better tracking and oversight of research, and enhanced data quality.\n\nAvailability and requirements \nIn order to reach a broader audience of users we have implemented a Java-based installer application, which is guiding an inexperienced computer user through the installation process (see Additional file 2). The basic installer package of iLAP has been tested on most common operating systems for which a Java Virtual Machine Version 1.5 or higher is available, e.g. Unix-based systems (Linux, Solaris, etc.), MacOS and Windows and can be downloaded from http:\/\/genome.tugraz.at\/iLAP\/. In addition to the requirement of a Java VM, a PostgreSQL database must be either locally installed or accessible via network. PostgreSQL comes with an easy-to-use installation wizard, so the complete installation should not be a significant entry level barrier. For further information about installation, please read the installation instructions from the download web site and in case of problems please contact the developers under iLAP@genome.tugraz.at. For initial testing purposes, please see also our test environment http:\/\/ilapdemo.genome.tugraz.at.\nRegarding hardware requirements, the most critical issue is disk space for large data files. These are stored in a directory hierarchy where the base directory must be specified during the installation process. The requirements regarding processor performance and memory depend on the user basis, but PC or server hardware with 2 GB of RAM should be sufficient for most installations.\nThe production environment for our central in-house installation consists of a 4-processor AMD-system X4600 from Sun Microsystems, with 16 GB of RAM which is connected to an 8TB SAN storage. For computational intensive tasks, iLAP delegates the calculations to a 48-node high-performance computing cluster using the JClusterService interface.\n\nAbbreviations \nJavaEE: Java Enterprise Edition platform\nMDA: Model Driven Architecture\nOMERO: Open Microscopy Environment Remote Objects\nSOAP: Simple Object Access Protocol\nFuGE: Data standard for Functional Genomic Experiment\n\nAuthors' contributions \nThe conceptual idea for iLAP goes back to GS, JGM and ZT and was elaborated by GS in the course of the GENAU-mobility\/NIH-visiting-scientist program. GS and MF performed the implementation of the main software modules including persistence-, business- and web tier. SK implemented the data retrieval functionality and worked also on the integration of OMERO. GB together with GS was responsible for the Java-Applet-based file transfer functionality which was additionally extended to work as an ImageJ-Plugin. Archiving functionality and the XML export for general experiment protocol sharing was implemented by MO. DR contributed with extremely useful comments about conceptual ideas, their practical application and their usability. DR's constant input derived from permanent testing under real work conditions lead to major improvements in functionality, usability and responsiveness. The project was coordinated by GS.\n\nAcknowledgements \nThe authors thank the staff of the Institute for Genomics and Bioinformatics for valuable comments and contributions. Special thanks go to Hillary Mueller and Tatiana Karpova from the Laboratory of Receptor Biology and Gene Expression, National Cancer Institute. Special thanks also go to Will Moore for the ongoing successful collaboration regarding the common protocol exchange format. During the large scale analysis on the high-performance computing cluster we were supported by Scientific Volume Imaging with a cluster license of Huygens Deconvolution Software. This work was supported by the Austrian Ministry for Science and Research, GEN-AU Project Bioinformatics Integration Network (BIN).\n\nReferences \n\n\n\u2191 Acevedo, L.G.; Iniguez, A.L.; Holster, H.L.; Zhang, X.; Green, R.; Farnham, P.J. (2007). \"Genome-scale ChIP-chip analysis using 10,000 human cells\". Biotechniques 43 (6): 791-797. PMC PMC2268896. PMID 18251256. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2268896 .   \n\n\u2191 Piggee, C. (2008). \"LIMS and the art of MS proteomics\". Analytical Chemistry 80 (13): 4801-4806. doi:10.1021\/ac0861329. PMID 18609747.   \n\n\u2191 Haquin, S.; Oeuillet, E.; Pajon, A.; Harris, M.; Jones, A.T.; van Tilbeurgh, H.; et al. (2008). \"Data management in structural genomics: an overview\". Methods in Molecular Biology 426: 49-79. doi:10.1007\/978-1-60327-058-8_4. PMID 18542857.   \n\n\u2191 4.0 4.1 Maurer, M.; Molidor, R.; Sturn, A.; Hartler, J.; Hackl, H.; Stocker, G.; et al. (2005). \"MARS: microarray analysis, retrieval, and storage system\". BMC Bioinformatics 6: 101. doi:10.1186\/1471-2105-6-101. PMC PMC1090551. PMID 15836795. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551 .   \n\n\u2191 Saal, L.H.; Troein, C.; Vallon-Christersson, J.; Gruvberger, S.; Borg, A.; Peterson, C. (2002). \"BioArray Software Environment (BASE): a platform for comprehensive management and analysis of microarray data\". Genome Biology 3 (8): SOFTWARE0003. doi:10.1186\/gb-2002-3-8-software0003. PMC PMC139402. PMID 12186655. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC139402 .   \n\n\u2191 Hartler, J.; Thallinger, G.G.; Stocker, G.; Sturn, A.; Burkard, T.R.; Korner, E.; et al. (2007). \"MASPECTRAS: a platform for management and analysis of proteomics LC-MS\/MS data\". BMC Bioinformatics 8: 197. doi:10.1186\/1471-2105-8-197. PMC PMC1906842. PMID 17567892. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1906842 .   \n\n\u2191 Craig, R.; Cortens, J.P.; Beavis, R.C. (2004). \"Open source system for analyzing, validating, and storing protein identification data\". Journal of Proteome Research 3 (6): 1234-1242. doi:10.1021\/pr049882h. PMID 15595733.   \n\n\u2191 Rauch, A.; Bellew, M.; Eng, J.; Fitzgibbon, M.; Holzman, T.; Hussey, P.; et al. (2006). \"Computational Proteomics Analysis System (CPAS): an extensible, open-source analytic system for evaluating and publishing proteomic data and high throughput biological experiments\". Journal of Proteome Research 5 (1): 112-121. doi:10.1021\/pr0503533. PMID 16396501.   \n\n\u2191 9.0 9.1 Moore, J.; Allan, C.; Burel, J.M.; Loranger, B.; MacDonald, D.; Monk, J.; et al. (2008). \"Open tools for storage and management of quantitative image data\". Methods in Cell Biology 85: 555-570. doi:10.1016\/S0091-679X(08)85024-8. PMID 18155479.   \n\n\u2191 10.0 10.1 Jones, A.R.; Pizarro, A.; Spellman, P.; Miller, M. (2006). \"FuGE: Functional Genomics Experiment Object Model\". OMICS 10 (2): 179-184. doi:10.1089\/omi.2006.10.179. PMID 16901224.   \n\n\u2191 11.0 11.1 Jones, A.R.; Miller, M.; Aebersold, R.; Apweiler, R.; Ball, C.A.; Brazma, A.; et al. (2007). \"The Functional Genomics Experiment model (FuGE): an extensible framework for standards in functional genomics\". Nature Biotechnology 25 (10): 1127-1133. doi:10.1038\/nbt1347. PMID 17921998.   \n\n\u2191 Brazma, A.; Hingamp, P.; Quackenbush, J.; Sherlock, G.; Spellman, P.; Stoeckert, C.; et al. (2001). \"Minimum information about a microarray experiment (MIAME)-toward standards for microarray data\". Nature Genetics 29 (4): 365-371. doi:10.1038\/ng1201-365. PMID 11726920.   \n\n\u2191 Taylor, C.F.; Paton, N.W.; Lilley, K.S.; Binz, P.A.; Julian, Jr., R.K.; Jones AR, et al. (2007). \"The minimum information about a proteomics experiment (MIAPE)\". Nature Biotechnology 25 (8): 887-893. doi:10.1038\/nbt1329. PMID 17687369.   \n\n\u2191 Deutsch, E.W.; Ball, C.A.; Berman, J.J.; Bova, G.S.; Brazma, A.; Bumgarner, R.E.; et al. (2008). \"Minimum information specification for in situ hybridization and immunohistochemistry experiments (MISFISHIE)\". Nature Biotechnology 26 (3): 305-312. doi:10.1038\/nbt1391. PMC PMC4367930. PMID 18327244. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4367930 .   \n\n\u2191 Taylor, K.T. (2006). \"The status of electronic laboratory notebooks for chemistry and biology\". Current Opinion in Drug Discovery & Development 9 (3): 348\u201353. PMID 16729731.   \n\n\u2191 Drake, D.J. (2007). \"ELN implementation challenges\". Drug Discovery Today 12 (15-16): 647-649. doi:10.1016\/j.drudis.2007.06.010. PMID 17706546.   \n\n\u2191 Butler, D. (2005). \"Electronic notebooks: a new leaf\". Nature 436 (7047): 20-21. doi:10.1038\/436020a. PMID 16001034.   \n\n\u2191 Kihlen, M. (2005). \"Electronic lab notebooks - do they work in reality?\". Drug Discovery Today 10 (18): 1205-1207. doi:10.1016\/S1359-6446(05)03576-2. PMID 16213407.   \n\n\u2191 Bradley, J.-C.; Samuel, B. (2004). \"SMIRP-A Systems Approach to Laboratory Automation\". Journal of the Association for Laboratory Automation 5 (3): 48-53. doi:10.1016\/S1535-5535(04)\u200b00074-7.   \n\n\u2191 Cite error: Invalid <ref> tag;\nno text was provided for refs named TaylorThe06\n\u2191 \"Tapestry web frame work\". Apache Software Foundation. 2009. Archived from the original on 23 July 2010. http:\/\/www.webcitation.org\/query.php?url=http:\/\/tapestry.apache.org\/&refdoi=10.1186\/1471-2105-10-390 .   \n\n\u2191 \"Java implementation of the SOAP (\"Simple Object Access Protocol\")\". Apache Software Foundation. 2009. Archived from the original on 07 December 2005. http:\/\/www.webcitation.org\/query.php?url=http:\/\/ws.apache.org\/axis\/&refdoi=10.1186\/1471-2105-10-390 .   \n\n\u2191 Krug, S. (2000). Don't make me think! A Common Sense Apporach to Web Usability. Indianapolis, IN: New Riders Publishing.   \n\n\u2191 Johnson, J. (2003). Web Bloopers: 60 Common Web Design Mistakes and How to Avoid Them. San Francisco, CA: Morgan Kaufmann Publishers, Inc.   \n\n\u2191 \"Spring lightweight application container\". SpringSource. 2009. Archived from the original on 27 May 2009. http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.springsource.org\/&refdoi=10.1186\/1471-2105-10-390 .   \n\n\u2191 \"Services and configuration microkernel\". Apache Software Foundation. 2009. http:\/\/hivemind.apache.org\/ .   \n\n\u2191 \"Apache servlet container\". Apache Software Foundation. 2009. Archived from the original on 27 May 2009. http:\/\/www.webcitation.org\/query.php?url=http:\/\/tomcat.apache.org\/&refdoi=10.1186\/1471-2105-10-390 .   \n\n\u2191 Johnson, R.; Hoeller, J. (2004). Expert One-on-One J2EE Development without EJB. Wrox.   \n\n\u2191 \"Java Open Transaction Manager (JOTM)\". OW2 Consortium. 2009. http:\/\/jotm.ow2.org\/xwiki\/bin\/view\/Main\/WebHome? .   \n\n\u2191 Bohlen, M. (2009). \"AndroMDA\". Archived from the original on 27 March 2007. http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.andromda.org\/&refdoi=10.1186\/1471-2105-10-390 .   \n\n\u2191 Rasband, W.S. (2009). \"ImageJ\". Archived from the original on 23 May 2009. http:\/\/www.webcitation.org\/query.php?url=http:\/\/rsb.info.nih.gov\/ij\/&refdoi=10.1186\/1471-2105-10-390 .   \n\n\u2191 Abramoff, M.D.; Magelhaes, P.J.; Ram, S.J.. \"Image Processing with ImageJ\". Biophotonics International 11 (7): 36\u201342.   \n\n\u2191 Roure, D.; Goble, C.; Bhagat, J.; Cruickshank, D.; Goderis, A.; Michaelides, D.; et al.. \"myExperiment: Defining the Social Virtual Research Environment\". IEEE Fourth International Conference on eScience, 2008 2008: 182-189. doi:10.1109\/eScience.2008.86.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformatics\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 1 November 2017, at 18:40.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,554 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","59816fe6940deb7e43cf53893a6ea71a_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_ILAP_A_workflow-driven_software_for_experimental_protocol_development_data_acquisition_and_analysis skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:iLAP: A workflow-driven software for experimental protocol development, data acquisition and analysis<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><p><span><\/span>\n<\/p>\n\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h3>\n<p>In recent years, the genome biology community has expended considerable effort to confront the challenges of managing heterogeneous data in a structured and organized way and developed <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">laboratory information management systems<\/a> (LIMS) for both raw and processed data. On the other hand, <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_laboratory_notebook\" title=\"Electronic laboratory notebook\" target=\"_blank\" class=\"wiki-link\" data-key=\"a9fbbd5e0807980106763fab31f1e72f\">electronic notebooks<\/a> were developed to record and manage scientific data, and facilitate data-sharing. Software which enables both, management of large datasets and digital recording of <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> procedures would serve a real need in laboratories using medium and high-throughput techniques.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Results\">Results<\/span><\/h3>\n<p>We have developed iLAP (Laboratory data management, Analysis, and Protocol development), a workflow-driven information management system specifically designed to create and manage experimental protocols, and to analyze and share laboratory data. The system combines experimental protocol development, wizard-based data acquisition, and high-throughput data analysis into a single, integrated system. We demonstrate the power and the flexibility of the platform using a microscopy case study based on a combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol and 3D-image reconstruction. iLAP is freely available under the open source license AGPL from <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/genome.tugraz.at\/iLAP\/\" target=\"_blank\">http:\/\/genome.tugraz.at\/iLAP\/<\/a>. (<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/genome.tugraz.at\/iLAP\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">Webcite<\/a>)\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Conclusion\">Conclusion<\/span><\/h3>\n<p>iLAP is a flexible and versatile information management system, which has the potential to close the gap between electronic notebooks and LIMS and can therefore be of great value for a broad scientific community.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background_2\">Background<\/span><\/h2>\n<p>The development of novel large-scale technologies has considerably changed the way biologists perform experiments. Genome biology experiments do not only generate a wealth of data, but they often rely on sophisticated laboratory protocols comprising hundreds of individual steps. For example, the protocol for chromatin immunoprecipitation on a microarray (Chip-chip) has 90 steps, uses over 30 reagents and 10 different devices.<sup id=\"rdp-ebb-cite_ref-AcevedoGen07_1-0\" class=\"reference\"><a href=\"#cite_note-AcevedoGen07-1\" rel=\"external_link\">[1]<\/a><\/sup> Even adopting an established protocol for large-scale studies represents a daunting challenge for the majority of the labs. The development of novel laboratory protocols and\/or the optimization of existing ones is still more distressing, since this requires systematic changes of many parameters, conditions, and reagents. Such changes are becoming increasingly difficult to trace using paper lab books. A further complication for most protocols is that many laboratory instruments are used, which generate electronic data stored in an unstructured way at disparate locations. Therefore, protocol data files are seldom or never linked to notes in lab books and can be barely shared within or across labs. Finally, once the experimental large-scale data have been generated, they must be analyzed using various software tools, then stored and made available for other users. Thus, it is apparent that software support for current biological <a href=\"https:\/\/www.limswiki.org\/index.php\/Research\" title=\"Research\" target=\"_blank\" class=\"wiki-link\" data-key=\"409634fd90113f119362927fe222f549\">research<\/a> \u2014 be it genomic or performed in a more traditional way \u2014 is urgently needed and inevitable. \n<\/p><p>In recent years, the genome biology community has expended considerable effort to confront the challenges of managing heterogeneous data in a structured and organized way and as a result developed information management systems for both raw and processed data. Laboratory information management systems (LIMS) have been implemented for handling data entry from robotic systems and tracking samples<sup id=\"rdp-ebb-cite_ref-PiggeeLIMS08_2-0\" class=\"reference\"><a href=\"#cite_note-PiggeeLIMS08-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-HaquinData08_3-0\" class=\"reference\"><a href=\"#cite_note-HaquinData08-3\" rel=\"external_link\">[3]<\/a><\/sup> as well as data management systems for processed data including microarrays<sup id=\"rdp-ebb-cite_ref-MaurerMARS05_4-0\" class=\"reference\"><a href=\"#cite_note-MaurerMARS05-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SaalBio02_5-0\" class=\"reference\"><a href=\"#cite_note-SaalBio02-5\" rel=\"external_link\">[5]<\/a><\/sup>, proteomics data<sup id=\"rdp-ebb-cite_ref-HartlerMAS07_6-0\" class=\"reference\"><a href=\"#cite_note-HartlerMAS07-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-CraigOpen04_7-0\" class=\"reference\"><a href=\"#cite_note-CraigOpen04-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RauchComp06_8-0\" class=\"reference\"><a href=\"#cite_note-RauchComp06-8\" rel=\"external_link\">[8]<\/a><\/sup>, and microscopy data.<sup id=\"rdp-ebb-cite_ref-MooreOpen08_9-0\" class=\"reference\"><a href=\"#cite_note-MooreOpen08-9\" rel=\"external_link\">[9]<\/a><\/sup> The latter systems support community standards like FUGE<sup id=\"rdp-ebb-cite_ref-JonesFuGE06_10-0\" class=\"reference\"><a href=\"#cite_note-JonesFuGE06-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JonesThe07_11-0\" class=\"reference\"><a href=\"#cite_note-JonesThe07-11\" rel=\"external_link\">[11]<\/a><\/sup>, MIAME<sup id=\"rdp-ebb-cite_ref-BrazmaMin01_12-0\" class=\"reference\"><a href=\"#cite_note-BrazmaMin01-12\" rel=\"external_link\">[12]<\/a><\/sup>, MIAPE<sup id=\"rdp-ebb-cite_ref-TaylorThe07_13-0\" class=\"reference\"><a href=\"#cite_note-TaylorThe07-13\" rel=\"external_link\">[13]<\/a><\/sup>, or MISFISHIE<sup id=\"rdp-ebb-cite_ref-DeutschMin08_14-0\" class=\"reference\"><a href=\"#cite_note-DeutschMin08-14\" rel=\"external_link\">[14]<\/a><\/sup> and have proven invaluable in a state-of-the-art laboratory. In general, these sophisticated systems are able to manage and analyze data generated for only a single type or a limited number of instruments, and were designed for only a specific type of molecule. \n<\/p><p>On the other hand, commercial as well as open source electronic notebooks<sup id=\"rdp-ebb-cite_ref-TaylorTheStatus_15-0\" class=\"reference\"><a href=\"#cite_note-TaylorTheStatus-15\" rel=\"external_link\">[15]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DrakeELN07_16-0\" class=\"reference\"><a href=\"#cite_note-DrakeELN07-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ButlerElec05_17-0\" class=\"reference\"><a href=\"#cite_note-ButlerElec05-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-KihlenElec05_18-0\" class=\"reference\"><a href=\"#cite_note-KihlenElec05-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BradleySMIRP04_19-0\" class=\"reference\"><a href=\"#cite_note-BradleySMIRP04-19\" rel=\"external_link\">[19]<\/a><\/sup> were developed to record and manage scientific data, and facilitate data-sharing. The influences encouraging the use of electronic notebooks are twofold.<sup id=\"rdp-ebb-cite_ref-TaylorThe06_20-0\" class=\"reference\"><a href=\"#cite_note-TaylorThe06-20\" rel=\"external_link\">[20]<\/a><\/sup> First, much of the data that needs to be recorded in a laboratory notebook is generated electronically. Transcribing data manually into a paper notebook is error-prone, and in many cases, for example, analytical data (spectra, chromatograms, photographs, etc.), transcription of the data is not possible. Second, the incorporation of high-throughput technologies into the research process has resulted in an increased volume of electronic data that need to be transcribed. As opposed to LIMS, which captures highly structured data through rigid user interfaces with standard report formats, electronic notebooks contain unstructured data and have flexible user interfaces.\n<\/p><p>Software which enables both, management of large datasets and recording of laboratory procedures, would serve a real need in laboratories using medium and high-throughput techniques. To the best of our knowledge, there is no software system available, which supports tedious protocol development in an intuitive way, links the plethora of generated files to the appropriate laboratory steps and integrates further analysis tools. We have therefore developed iLAP, a workflow-driven information management system for protocol development and data management. The system combines experimental protocol development, wizard-based data acquisition, and high-throughput data analysis into a single, integrated system. We demonstrate the power and the flexibility of the platform using a microscopy case study based on combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol and 3D-image reconstruction.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Workflow-driven_software_design\">Workflow-driven software design<\/span><\/h3>\n<p>The design of a software platform that supports the development of protocols and data management in an experimental context has to be based on and directed by the laboratory workflow. The laboratory workflow can be divided into four principal steps: 1) project definition phase, 2) experimental design and data acquisition phase, 3) data analysis and processing phase and 4) data retrieval phase (Figure 1). \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Stocker_BMCBioinformatics2009_10.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a51d9541cbdc59af0fc28acab329bedb\"><img alt=\"Fig1 Stocker BMCBioinformatics2009 10.jpg\" src=\"https:\/\/www.limswiki.org\/images\/a\/ac\/Fig1_Stocker_BMCBioinformatics2009_10.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1: Mapping of the laboratory workflow onto iLAP features.<\/b> The software design of iLAP is inspired by a typical laboratory workflow in life sciences and offers software assistance during the process.<br \/>The figure illustrates on the left panel the scientific workflow separated into four phases: project definition, data acquisition and analysis, and data retrieval. The right panel shows the main functionalities offered by iLAP.<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Project_definition_phase\">Project definition phase<\/span><\/h3>\n<p>A scientific project starts with a hypothesis and the choice of methods required to address a specific biological question. Already during this initial phase it is crucial to define the question as specifically as possible and to capture the information in a digital form. Documents collected during the literature research should be collated with the evolving project definition for later review or for sharing with other researchers. All files collected in this period should be attached to the defined projects and experiments in the software.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Experimental_design_and_data_acquisition\">Experimental design and data acquisition<\/span><\/h3>\n<p>Following the establishment of a hypothesis and based on preliminary experiments, the detailed design of the biological experiments is then initiated. Usually, the experimental work follows already established standard operating procedures, which have to be modified and optimized for the specific biological experiment. These protocols are defined as a sequence of protocol steps. However, well-established protocols must be kept flexible in a way that particular conditions can be changed. The typically changing parameters of standard protocol steps (e.g. fixation times, temperature changes etc.) are important to record as they are used to improve the experimental reproducibility.\n<\/p><p>Equipped with a collection of standard operating procedures, an experiment can be initiated and the data generated. In general, data acquisition comprises not only files but also observations of interest, which might be relevant for the interpretation of the results. Most often these observations disappear in paper notebooks and are not accessible in a digital form. Hence, these experimental notes should be stored and attached to the originating protocol step, experiment or project.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_analysis_and_processing\">Data analysis and processing<\/span><\/h3>\n<p>After storing the raw result files, additional analysis and post-processing steps must be performed to obtain processed data for subsequent analysis. In order to extract information and to combine it in a statistically meaningful manner, multiple data sets have to be acquired. The software workflow should enable also the inclusion of external analytical steps, so that files resulting from external analysis software can be assigned to their original raw data files. Finally, the data files generated at the analysis stage should be connected to the raw data, allowing connection of the data files with the originating experimental context.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_retrieval\">Data retrieval<\/span><\/h3>\n<p>By following the experimental workflow, all experimental data e.g. different files, protocols, notes etc. should be organized in a chronological and project-oriented way and continuously registered during their acquisition. An additional advantage should be the ability to search and retrieve the data. Researchers frequently have to search through notebooks to find previously uninterpretable observations. Subsequently, as the project develops, the researchers gain a different perspective and recognize that prior observations could lead to new discoveries. Therefore, the software should offer easy to use interfaces that allow searches through observation notes, projects- and experiment descriptions.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Software_architecture\">Software architecture<\/span><\/h3>\n<p>iLAP is a multi-tier client-server application and can be subdivided into different functional modules which interact as self-contained units according to their defined responsibilities (see Figure 2). \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Stocker_BMCBioinformatics2009_10.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b9e884bfbee10282fb4b69ec3871708b\"><img alt=\"Fig2 Stocker BMCBioinformatics2009 10.jpg\" src=\"https:\/\/www.limswiki.org\/images\/0\/0c\/Fig2_Stocker_BMCBioinformatics2009_10.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2: Software Architecture.<\/b> iLAP features a typical three-tier architecture and can hence be divided into a presentation tier, business tier and a persistence tier (from left to right). The presentation tier is formed by a graphical user interface, accessed using a web browser. The following business layer is protected by a security layer, which enforces user authentication and authorization. After access is granted, the security layer passes the user requests to the business layer, which is mainly responsible for guiding the user through the laboratory workflow. This layer also coordinates all background tasks like automatic surveying of analysis jobs on a computing cluster or synchronizing\/exchanging data with further downstream applications. (e.g. OMERO (open microscopy environment) image server). Finally, the persistence layer interacts with the relational database.<\/blockquote>\n<h3><span class=\"mw-headline\" id=\"Presentation_tier\">Presentation tier<\/span><\/h3>\n<p>The presentation tier within iLAP is formed by a Web interface, using Tapestry<sup id=\"rdp-ebb-cite_ref-ApacheTap09_21-0\" class=\"reference\"><a href=\"#cite_note-ApacheTap09-21\" rel=\"external_link\">[21]<\/a><\/sup> as the model view controller and an Axis Web service<sup id=\"rdp-ebb-cite_ref-ApacheJava09_22-0\" class=\"reference\"><a href=\"#cite_note-ApacheJava09-22\" rel=\"external_link\">[22]<\/a><\/sup>, which allows programming access to parts of the application logic. Thus, on the client side, a user requires an Internet connection and a recent Web browser with Java Applet support, available for almost every platform. In order to provide a simple, consistent but also attractive Web interface, iLAP follows usability guidelines described in<sup id=\"rdp-ebb-cite_ref-KrugDont00_23-0\" class=\"reference\"><a href=\"#cite_note-KrugDont00-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JohnsonWeb03_24-0\" class=\"reference\"><a href=\"#cite_note-JohnsonWeb03-24\" rel=\"external_link\">[24]<\/a><\/sup> and uses Web 2.0 technologies for dynamic content generation. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Business_tier_and_runtime_environment\">Business tier and runtime environment<\/span><\/h3>\n<p>The business tier is realized as view-independent application logic, which stores and retrieves datasets by communicating with the persistence layer. The internal management of files is also handled from a central service component, which persists the meta-information for acquired files to the database, and stores the file content in a file-system-based data hierarchy. The business layer also holds asynchronous services for application-internal JMS messaging and for integration of external computing resources like high-performance computing clusters. All services of this layer are implemented as Spring<sup id=\"rdp-ebb-cite_ref-SpringSpring09_25-0\" class=\"reference\"><a href=\"#cite_note-SpringSpring09-25\" rel=\"external_link\">[25]<\/a><\/sup> beans, for which the Spring-internal interceptor classes provide transactional integrity.\n<\/p><p>The business tier and the persistence tier are bound by the Spring J2EE lightweight container, which manages the component-object life cycle. Furthermore, the Spring context is transparently integrated into the Servlet context of Tapestry using the HiveMind<sup id=\"rdp-ebb-cite_ref-ApacheServices09_26-0\" class=\"reference\"><a href=\"#cite_note-ApacheServices09-26\" rel=\"external_link\">[26]<\/a><\/sup> container backend. This is realized by using the automatic dependency injection functionality of HiveMind which avoids integrative glue code for lookups into the Spring container. Since iLAP uses Spring instead of EJB related components, the deployment of the application only requires a standard conformed Servlet container. Therefore, the Servlet container Tomcat<sup id=\"rdp-ebb-cite_ref-ApacheApache09_27-0\" class=\"reference\"><a href=\"#cite_note-ApacheApache09-27\" rel=\"external_link\">[27]<\/a><\/sup> is used, which offers not only Servlet functionality but J2EE infrastructure services<sup id=\"rdp-ebb-cite_ref-JohnsonExpert04_28-0\" class=\"reference\"><a href=\"#cite_note-JohnsonExpert04-28\" rel=\"external_link\">[28]<\/a><\/sup> such as centrally configured data-sources and transaction management realized with the open source library JOTM.<sup id=\"rdp-ebb-cite_ref-OW2Java09_29-0\" class=\"reference\"><a href=\"#cite_note-OW2Java09-29\" rel=\"external_link\">[29]<\/a><\/sup> This makes the deployment of iLAP on different servers easier, because machine-specific settings for different production environments are kept outside the application configuration. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"External_programming_interfaces\">External programming interfaces<\/span><\/h3>\n<p>The SOAP Web service interface for external programmatic access is realized by combining the Web service framework Axis with corresponding iLAP components. The Web service operates as an external access point for Java Applets within the Web application, as well as for external analysis and processing applications such as ImageJ.\nModel driven development\n<\/p><p>In order to reduce coding and to increase the long term maintainability, the model driven development environment AndroMDA<sup id=\"rdp-ebb-cite_ref-BohlenAndro09_30-0\" class=\"reference\"><a href=\"#cite_note-BohlenAndro09-30\" rel=\"external_link\">[30]<\/a><\/sup> is used to generate components of the persistence layer and recurrent parts from the above mentioned business layer. AndroMDA accomplishes this by translating an annotated UML-model into a JEE-platform-specific implementation using Hibernate and Spring as base technology. Due to the flexibility of AndroMDA, application external services, such as the user management system, have a clean integration in the model. Dependencies of internal service components on such externally defined services are cleanly managed by its build system.\n<\/p><p>By changing the build parameters in the AndroMDA configuration, it is also possible to support different relational database management systems. This is because platform specific code with the same functionality is generated for data retrieval. Furthermore, technology lock-in regarding the implementation of the service layers was also addressed by using AndroMDA, as the implementation of the service facade can be switched during the build process from Spring based components to distributed Enterprise Java Beans. At present, iLAP is operating on one local machine and, providing the usage scenarios do not demand it, this architectural configuration will remain. However, chosen technologies are known to work on Web server farms and crucial distribution of the application among server nodes is transparently performed by the chosen technologies. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Asynchronous_data_processing\">Asynchronous data processing<\/span><\/h3>\n<p>The asynchronous handling of business processes is realized in iLAP with message-driven Plain Old Java Objects (POJOs). Hence, application tasks, such as the generation of image previews, can be performed asynchronously. If performed immediately, these would unnecessarily block the responsiveness of the Web front-end. iLAP delegates tasks via JMS messages to back-end services, which perform the necessary processing actions in the background.\n<\/p><p>These back-end services are also UML-modelled components and receive messages handled by the JMS provider ActiveMQ. If back-end tasks consume too many calculation resources, the separation of Web front-end and JMS message receiving services can be realized by copying the applications onto two different servers and changing the Spring JMS configuration.\n<\/p><p>For the smooth integration of external computing resources like the high-performance computing cluster or special compute nodes with limited software licenses the JClusterService is used. JClusterService is a separately developed J2EE application which enables a programmer to run generic applications on a remote execution host or high-performance computing cluster. Every application which offers a command line interface can be easily integrated by defining a service definition in XML format and accessing it via a SOAP-based programming interface from any Java-application. The execution of the integrated application is carried out either by using the internal JMS-queuing system for single host installations or by using the open source queuing systems like Sun Grid Engine (Sun Microsystems) or OpenPBS\/Torque. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_2\">Results<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\"Functional_overview\">Functional overview<\/span><\/h3>\n<p>The functionality offered by the iLAP web interface can be described by four components: 1) hierarchical organization of the experimental data, 2) protocol development, 3) data acquisition and analysis, and 4) data retrieval and data sharing (Figure 1). iLAP specific terms are summarized in Table 1. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"2\"><b>Table 1. iLAP Terminology<\/b>\n<\/td><\/tr>\n<tr>\n<th style=\"padding-left:10px; padding-right:10px;\">iLAP specific terms\n<\/th>\n<th> Description\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Project\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Logical unit which can be structured hierarchically and holds experiments, notes and other files (e.g. derived from literature research).\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Experiment\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Logical unit which corresponds to one biological experiment and holds a current working protocol, experiment specific documentation files, parameter values, raw files, notes, and analysis steps.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Standard Protocol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Frequently used and well established protocol template also known as standard operating procedures (SOP).\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Current working protocol\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Sequence of protocol steps for a specific experiment which holds raw files, notes and experiment specific parameter values.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Protocol step\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">One single step in a protocol which is defined by a name, description, and a list of definable parameters. A sequence of protocol steps defines a protocol.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Step group\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Protocol step which groups multiple protocol steps to a logical unit. It can be used as a step container for sequentially executed protocol steps or within split steps.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Split step\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Protocol step which can contain multiple (step groups) which have to be executed concurrently.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Protocol step parameter\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Changing parameters which are associated with a step and can hold either textual or numerical values as well as a selection from a predefined value list (enumeration).\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Note\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Notes are textual descriptions which are intended to be used for documenting abnormal observations at almost anywhere within iLAP.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Raw file\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Raw files are files which are produced by laboratory instruments and are not processed by any analysis step captured within iLAP.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Analysis step\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Description of a processing step which manipulates, analyzes or processes a raw file, and generates processed files which are linked to the original raw file. Analysis steps can be either external e.g. using external software or internal using iLAP-internal analysis modules.\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Analysis step parameter\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Parameters and values used during the analysis step.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Hierarchical_organization_of_experimental_data\">Hierarchical organization of experimental data<\/span><\/h3>\n<p>This part of the user interface covers the project definition phase of the experimental workflow. The definition of projects and experiments consists solely in inserting the required descriptive parameters via a Web form. In doing so, a hierarchical structure with projects, sub-projects and experiments is created and displayed in the iLAP overview. The hierarchy (Figure 3) and other screen shots can be found in the iLAP user manual (Additional file 1). This overview is the starting point of iLAP, from which almost every activity can be initiated. By navigating through the tree, an information box appears alongside. This box details information about the current node in the tree and the operations which can be performed on the database managed object represented by the node. Already in this early stage, files derived from literature research can be uploaded to projects and experiments, and ongoing observations can be stored using the general note dialog. If multiple files must be associated with projects and experiments, a Java Applet can be used to upload the files to the generated project\/experiment structure. iLAP can manage every file independent of their file type, and can thus be considered as a generic document management system. File types only need to be considered for subsequent processing and data extraction. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Stocker_BMCBioinformatics2009_10.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"f09f0d3a710deec0636de835438dde0e\"><img alt=\"Fig3 Stocker BMCBioinformatics2009 10.jpg\" src=\"https:\/\/www.limswiki.org\/images\/4\/44\/Fig3_Stocker_BMCBioinformatics2009_10.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3: Hierarchical organization of data in iLAP overview.<\/b> The continuous use of iLAP inherently leads to structured recording of experiments, conserving the complete experimental context of data records throughout the history of the research project. In doing so, a hierarchical structure with projects, sub-projects and experiments is created and can be displayed in this iLAP overview tree. The P-icons in the tree stand for projects and sub-projects, the E-icon for experiments and the A-icon for analysis steps. Files attached to protocol steps are considered as raw files and are therefore collected under the step container visualized with the R-icon. The consistent association of color schemes to logical units like projects, experiments, etc. can be directly recognized in this overview. By clicking on one of the tree icons on the left hand a detailed overview appears about the selected item. Also actions like creation of new projects etc. can be directly initiated using the quick-links in the \"Actions\" section of \"Details\".<\/blockquote>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 1. iLAP user manual\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The iLAP user manual contains a detailed description of the user interface including screen shots.\n<p>Format: PDF; Size: 1.8MB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-10-390-s1.pdf\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Protocol_development\">Protocol development<\/span><\/h3>\n<p>When starting experimental work, the iLAP facility manager should define commonly used standard protocols using the protocol development masks. Therefore, a sequence of steps must be defined which describes the typical ongoing experiment in detail. Dynamic protocol parameters, which may be adapted for protocol optimization during the experiment, can be associated with the pre-defined steps. These parameters can be either numerical values, descriptive text or predefined enumeration types, all of which can be preset by default values and marked with appropriate units. In order to force the acquisition of critical parameters in the data acquisition wizard, parameters can be marked as required. According to our experience and the experience of other users, it is helpful to define small and reusable standard protocol units, which can be used as building blocks during the experiment-specific protocol assembly. Automatic internal versioning takes care of changes in standard protocols so that dependent protocols used in previous experiments remain unaffected.\n<\/p><p>Equipped with a collection of standard protocols, an experiment can be initiated and should be defined at the beginning of the workflow. The name of each experiment, its general description and specific aims, must be provided in order to be able to distinguish between different experiments. The detailed experiment procedure is defined by its current working protocol which can be composed step by step or by reusing existing current working protocols from already performed experiments. If the experiment is following a standard protocol, the current working protocol should be created by simply copying the predefined standard protocol steps and parameter definitions. In order to consider also the concurrent nature of simultaneously executed steps the experimenter should be able to define different sub-branches (e.g. cells are treated with different drugs in order to study their response) named split steps. These split steps lead to different branches of the experimental workflow called step groups which are separately handled during the data acquisition phase.\n<\/p><p>Once the protocol design phase is completed and all necessary protocol steps with their parameters are defined the researchers should be able to generate a printout of the current working protocol with which the experiment can be performed at the lab bench. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_acquisition_and_analysis\">Data acquisition and analysis<\/span><\/h3>\n<p>After having finished all experimental work and having created raw data files with different laboratory instruments the data acquisition within iLAP should be performed. By going through the early defined current working protocol steps, generated raw data files, used protocol parameter values and observation notes must be entered. Wizard-based input masks (wizard), which are derived from the defined current protocol steps, assist the experimenters during this work. On every step the user has to fill in the value fields for required parameters and can attach files and notes to each of the steps. During the creation of the working protocol, it is important to name those steps to which files are attached in a descriptive way. Files that are directly connected to experimental steps are considered as raw files and are protected against deletion. Note, files can be linked to the protocol steps anywhere in iLAP, i.e. also before and after the data acquisition.\n<\/p><p>For this data association, the iLAP workflow offers also the possibility to transfer all generated files to a central repository and associate automatically files with their generating protocol step at once, using a Java Applet. All the internal linkages to protocol steps, experiments or projects are performed automatically without the need of any user interference. As the files are attached to a protocol and an experiment, the overall context is preserved and the likelihood of reproducibility of the same conditions is increased. Within iLAP experimental notes are stored and attached to the originating protocol step, experiment or project and are retrievable using a keyword based search mask\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_analysis\">Data analysis<\/span><\/h3>\n<p>The analysis steps are recorded in iLAP by either reusing existing analysis templates or describing new analysis steps applied to the previously uploaded raw data files. Additional analysis tools can be developed in Java as described in the iLAP user manual (Additional file 1). According to the file type, internally implemented analysis steps or the description of externally performed analysis steps are associated with the raw data files. Result files from analysis programs together with the used parameters can be easily attached to analysis definitions. As an example, a server analysis tool was implemented for deconvolving three dimensional image stacks, executed on a remote high-performance computing cluster using the JClusterService (see methods). \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Integration_of_external_programs\">Integration of external programs<\/span><\/h3>\n<p>A proof of concept about external access of programs using the iLAP application programming interface was shown by the implementation of a plugin for the widely used image processing software ImageJ.<sup id=\"rdp-ebb-cite_ref-RasbandIma09_31-0\" class=\"reference\"><a href=\"#cite_note-RasbandIma09-31\" rel=\"external_link\">[31]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AbramoffImage04_32-0\" class=\"reference\"><a href=\"#cite_note-AbramoffImage04-32\" rel=\"external_link\">[32]<\/a><\/sup> This Java plugin enables ImageJ to transfer the image files directly from iLAP to the client machine. This functionality appears as a regular dialog in the graphical user interface of ImageJ, and allows upload of result files back into iLAP in a transparent manner. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Automatic_post_processing_tool_chain\">Automatic post processing tool chain<\/span><\/h3>\n<p>Background tasks like the generation of previews are performed using the internal post-processing tool chain which is started asynchronously as soon as the files are associated with the originating experiment in iLAP. According to the detected file type, multiple post-processor steps are executed and results are automatically stored back into the database. This flexible system approach is also used to automatically inform and synchronize further downstream applications like OMERO<sup id=\"rdp-ebb-cite_ref-MooreOpen08_9-1\" class=\"reference\"><a href=\"#cite_note-MooreOpen08-9\" rel=\"external_link\">[9]<\/a><\/sup> image server from the Open Microscopy Environment project. Therefore, iLAP is able to transfer files \u2014 transparently for the user \u2014 to a server where a comparable project\/dataset structure is created.\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Data_retrieval_and_information_sharing\">Data retrieval and information sharing<\/span><\/h3>\n<p>The use of the described data acquisition features inherently leads to structured recording of experiments, conserving the complete experimental context of data records throughout the history of research projects. It is often necessary to go back to already completed experiments and to search through old notes. Therefore, iLAP offers search masks which allow keyword based searching in the recorded projects, experiments and notes. These results are often discussed with collaboration partners to gain different opinions on the same raw data.\n<\/p><p>In order to allow direct collaboration between scientists iLAP is embedded into a central user management system<sup id=\"rdp-ebb-cite_ref-MaurerMARS05_4-1\" class=\"reference\"><a href=\"#cite_note-MaurerMARS05-4\" rel=\"external_link\">[4]<\/a><\/sup> which offers multiple levels of access control to projects and their associated experimental data. The sharing of projects can be done on a per-user basis or on an institutional basis. For small or local single-user installations, the fully featured user management system can be replaced by a file-based user management which still offers the same functionalities from the sharing point of view, but lacks institute-wide functionalities (Additional file 2). This is only possible because iLAP keeps the source of user accounts separated from the internal access control to enable easy integration of additional local or institution wide user management systems. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 2. iLAP installation and administration manual\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The iLAP installation and administration manual contains a detailed description of the installation process for all supported platforms including screen shots.\n<p>Format: PDF; Size: 1.8MB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-10-390-s2.pdf\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Since sophisticated protocols are crucial for successful experiments iLAP-users can export their protocols not only in PDF format (Additional file 3) but also in an exchangeable XML format (Additional file 4 and 5). In that way scientists can directly pass over their optimized protocols to partners who do not share the data using iLAP internally but need to get the protocol information transferred. The same XML files can be also used on a broader basis for protocol exchange using central eScience platforms like MyExperiments.<sup id=\"rdp-ebb-cite_ref-RoureMy08_33-0\" class=\"reference\"><a href=\"#cite_note-RoureMy08-33\" rel=\"external_link\">[33]<\/a><\/sup> This platform aims for an increased reuse and repurpose of commonly shared workflows achieving at the same time reduced time-to-experiment and avoiding reinvention. Ongoing standardization efforts regarding the XML format like FuGE<sup id=\"rdp-ebb-cite_ref-JonesFuGE06_10-1\" class=\"reference\"><a href=\"#cite_note-JonesFuGE06-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-JonesThe07_11-1\" class=\"reference\"><a href=\"#cite_note-JonesThe07-11\" rel=\"external_link\">[11]<\/a><\/sup> are currently not supported but could be integrated in future versions of iLAP. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 3. m-FISH protocol in PDF format\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">This file contains the combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol in PDF format used in the case study section.\n<p>Format: PDF; Size: 51KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-10-390-s3.pdf\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 4. m-FISH protocol in XML format\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">This file contains the combinatorial multiple fluorescence in situ hybridization (m-FISH) protocol in XML format used in the case study section for protocol exchange.\n<p>Format: XML; Size: 64KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-10-390-s4.xml\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 5. Document Type Definition for the XML protocol format\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">This file contains the Document Type Definition for the XML format used for protocol exchange created in collaboration with Will Moore from the OMERO.editor project.\n<p>Format: DTD; Size: 4KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-10-390-s5.dtd\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Case_Study\">Case Study<\/span><\/h3>\n<p>In order to test the functionality of the system, we used a high-throughput microscopy study. The focus of this study was on the three dimensional nuclear localization of a group of seven genes. This required the development of a combinatorial multicolor fluorescence in situ hybridization (m-FISH) protocol. This protocol enables simultaneous detection and visualization of all seven genes by using a combination of three different fluorescent labels. The elaboration and optimization of m-FISH required many different protocol steps and parameters. Thus it was crucial to keep a record of any parameter and procedure changes during the process of protocol development. These changes were directly connected with data produced in the lab (e.g. concentration of the FISH probes, probe labeling efficiencies etc.) and the resulting imaging data. In the final combinatorial m-FISH protocol, 70 steps and 139 different parameters were present. Using this protocol we conducted 10 experiments and produced 1,441 multicolor 3D-Image stacks of which 984 were subsequently corrected for color shifts and processed by 3D-deconvolution performing 100 iterations of the maximum likelihood estimation algorithm available with the Huygens Deconvolution Software (<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.svi.nl\" target=\"_blank\">Scientific Volume Imaging - SVI<\/a>). These image processing steps were realized as batch analysis in iLAP, which delegated the compute intensive procedure to a high-performance computing cluster and then stored all processed image stacks in the analysis container of the corresponding experiments. Afterwards FISH signals were detected and analyzed using a custom image analysis procedure which was realized as a Matlab (MathWorks Inc.) extension of Imaris (Bitplane Inc.) using the Imaris-XT programming interface. This extension automatically recorded FISH signal coordinates, signal to signal distances, the nuclear volume and several additional parameters of each imaged nucleus. These externally generated data files were transferred back into iLAP and stored in the context of the corresponding experiment as an external analysis step. A summary of the data acquisition and analysis is shown in Figure 4. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Stocker_BMCBioinformatics2009_10.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"0a2a235af288f25cd5810750798ae484\"><img alt=\"Fig4 Stocker BMCBioinformatics2009 10.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/ca\/Fig4_Stocker_BMCBioinformatics2009_10.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4: Case study summary.<\/b> The functionality of iLAP was tested in a high-throughput microscopy study. The figure illustrates a summary of the data acquisition and data analysis performed. In 10 experiments a protocol consisting of 70 steps with 139 different parameters was used to generate three-dimensional multicolor image stacks. Each of the 1,441 raw image stacks consisted of 28 optical sections (slices) where each slice was recorded in 4 different channels. The raw image stacks were stored in the iLAP system and thereby connected with the corresponding experiments and protocols. By utilizing the integrated analysis functionality of iLAP the 984 raw images processed by the Huygens 3D-deconvolution package and analyzed by an external semiautomatic procedure implemented in Matlab and Imaris-XT. The analytical pipeline produced data for 121 different distance measurements of each single image. The resulting images and data were then stored in their experimental context within the iLAP system.<\/blockquote>\n<p>During the course of the study we observed several clear advantages of the iLAP system over a lab-book in paper form, which was maintained in parallel. The first and most valuable feature of iLAP is the direct connection between protocol steps and data files which cannot be realized using a paper lab book. A second notable advantage of the iLAP system was that lab-tasks that were performed in parallel or in overlapping time-frames could also be stored as such, whereas in the traditional lab book all tasks performed in the lab were written sequentially which implied a break-up of connected protocols. A third advantage was that iLAP allowed for rapid searching and finding of experiments, protocols and desired terms, which required only a few mouse clicks as opposite to the cumbersome search using a paper notebook. Moreover, iLAP enabled easy collaboration functionality, data backup or parameter completeness checks. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusion_2\">Conclusion<\/span><\/h2>\n<p>We have developed a unique information management system specifically designed to support the creation and management of experimental protocols, and to analyze and share laboratory data. The design of the software was guided by the laboratory workflow and resulted in four unified components accessible through a web interface. The first component allows the hierarchical organization of the experimental data, which is organized in a generic document management system. The second component focuses on protocol development using templates of standard operating procedures. Next, the data acquisition and analysis component offers the possibility to transfer the generated files to a central repository and to associate the files with the corresponding protocol steps. Additionally, external data analysis programs can be integrated and executed on a remote high-performance computing cluster. The last component enables collaboration and data sharing between scientists using iLAP on a user or institutional level as well as protocol transfer with external users.\n<\/p><p>Although designed in an experimental context for high-throughput protocols like microarray studies of gene expression, DNA-protein binding, proteomics experiments, or high-content image-based screening studies, iLAP has also proven to be valuable in low- and medium-throughput experiments. For example, protocols for qPCR analysis of gene expression using 96 and 384-well formats -a widely used technique- can be easily developed and can contribute significantly to establishment of robust assays. Moreover, since the workflow-oriented concept of iLAP offers the flexibility of a more general scientific data management system it is not limited to a special laboratory protocol, instrument, or type of molecule. For example, its application for next-generation sequencing is straightforward since similar requirements on the computational environment (increasing amount of data, integration of analysis tools, or use of high-performance computing infrastructure) have to be met.\n<\/p><p>In summary, we have developed a flexible and versatile information management system, which has the potential to close the gap between electronic notebooks and LIMS and can therefore be of great value for a broader community. Extensive tests in our and other labs have shown that the benefits of better information access and data sharing immediately result in reduced time spent managing information, increased productivity, better tracking and oversight of research, and enhanced data quality.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p>In order to reach a broader audience of users we have implemented a Java-based installer application, which is guiding an inexperienced computer user through the installation process (see Additional file 2). The basic installer package of iLAP has been tested on most common operating systems for which a Java Virtual Machine Version 1.5 or higher is available, e.g. Unix-based systems (Linux, Solaris, etc.), MacOS and Windows and can be downloaded from <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/genome.tugraz.at\/iLAP\/\" target=\"_blank\">http:\/\/genome.tugraz.at\/iLAP\/<\/a>. In addition to the requirement of a Java VM, a PostgreSQL database must be either locally installed or accessible via network. PostgreSQL comes with an easy-to-use installation wizard, so the complete installation should not be a significant entry level barrier. For further information about installation, please read the installation instructions from the download web site and in case of problems please contact the developers under iLAP@genome.tugraz.at. For initial testing purposes, please see also our test environment <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ilapdemo.genome.tugraz.at\" target=\"_blank\">http:\/\/ilapdemo.genome.tugraz.at<\/a>.\n<\/p><p>Regarding hardware requirements, the most critical issue is disk space for large data files. These are stored in a directory hierarchy where the base directory must be specified during the installation process. The requirements regarding processor performance and memory depend on the user basis, but PC or server hardware with 2 GB of RAM should be sufficient for most installations.\n<\/p><p>The production environment for our central in-house installation consists of a 4-processor AMD-system X4600 from Sun Microsystems, with 16 GB of RAM which is connected to an 8TB SAN storage. For computational intensive tasks, iLAP delegates the calculations to a 48-node high-performance computing cluster using the JClusterService interface.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p>JavaEE: Java Enterprise Edition platform\n<\/p><p>MDA: Model Driven Architecture\n<\/p><p>OMERO: Open Microscopy Environment Remote Objects\n<\/p><p>SOAP: Simple Object Access Protocol\n<\/p><p>FuGE: Data standard for Functional Genomic Experiment\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.27_contributions\">Authors' contributions<\/span><\/h2>\n<p>The conceptual idea for iLAP goes back to GS, JGM and ZT and was elaborated by GS in the course of the GENAU-mobility\/NIH-visiting-scientist program. GS and MF performed the implementation of the main software modules including persistence-, business- and web tier. SK implemented the data retrieval functionality and worked also on the integration of OMERO. GB together with GS was responsible for the Java-Applet-based file transfer functionality which was additionally extended to work as an ImageJ-Plugin. Archiving functionality and the XML export for general experiment protocol sharing was implemented by MO. DR contributed with extremely useful comments about conceptual ideas, their practical application and their usability. DR's constant input derived from permanent testing under real work conditions lead to major improvements in functionality, usability and responsiveness. The project was coordinated by GS.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>The authors thank the staff of the Institute for Genomics and Bioinformatics for valuable comments and contributions. Special thanks go to Hillary Mueller and Tatiana Karpova from the Laboratory of Receptor Biology and Gene Expression, National Cancer Institute. Special thanks also go to Will Moore for the ongoing successful collaboration regarding the common protocol exchange format. During the large scale analysis on the high-performance computing cluster we were supported by <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.svi.nl\" target=\"_blank\">Scientific Volume Imaging<\/a> with a cluster license of Huygens Deconvolution Software. This work was supported by the Austrian Ministry for Science and Research, GEN-AU Project Bioinformatics Integration Network (BIN).\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-AcevedoGen07-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AcevedoGen07_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Acevedo, L.G.; Iniguez, A.L.; Holster, H.L.; Zhang, X.; Green, R.; Farnham, P.J. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2268896\" target=\"_blank\">\"Genome-scale ChIP-chip analysis using 10,000 human cells\"<\/a>. <i>Biotechniques<\/i> <b>43<\/b> (6): 791-797. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC2268896\/\" target=\"_blank\">PMC2268896<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18251256\" target=\"_blank\">18251256<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2268896\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC2268896<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genome-scale+ChIP-chip+analysis+using+10%2C000+human+cells&rft.jtitle=Biotechniques&rft.aulast=Acevedo%2C+L.G.%3B+Iniguez%2C+A.L.%3B+Holster%2C+H.L.%3B+Zhang%2C+X.%3B+Green%2C+R.%3B+Farnham%2C+P.J.&rft.au=Acevedo%2C+L.G.%3B+Iniguez%2C+A.L.%3B+Holster%2C+H.L.%3B+Zhang%2C+X.%3B+Green%2C+R.%3B+Farnham%2C+P.J.&rft.date=2007&rft.volume=43&rft.issue=6&rft.pages=791-797&rft_id=info:pmc\/PMC2268896&rft_id=info:pmid\/18251256&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC2268896&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PiggeeLIMS08-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PiggeeLIMS08_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Piggee, C. (2008). \"LIMS and the art of MS proteomics\". <i>Analytical Chemistry<\/i> <b>80<\/b> (13): 4801-4806. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fac0861329\" target=\"_blank\">10.1021\/ac0861329<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18609747\" target=\"_blank\">18609747<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LIMS+and+the+art+of+MS+proteomics&rft.jtitle=Analytical+Chemistry&rft.aulast=Piggee%2C+C.&rft.au=Piggee%2C+C.&rft.date=2008&rft.volume=80&rft.issue=13&rft.pages=4801-4806&rft_id=info:doi\/10.1021%2Fac0861329&rft_id=info:pmid\/18609747&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HaquinData08-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HaquinData08_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Haquin, S.; Oeuillet, E.; Pajon, A.; Harris, M.; Jones, A.T.; van Tilbeurgh, H.; et al. (2008). \"Data management in structural genomics: an overview\". <i>Methods in Molecular Biology<\/i> <b>426<\/b>: 49-79. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1007%2F978-1-60327-058-8_4\" target=\"_blank\">10.1007\/978-1-60327-058-8_4<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18542857\" target=\"_blank\">18542857<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Data+management+in+structural+genomics%3A+an+overview&rft.jtitle=Methods+in+Molecular+Biology&rft.aulast=Haquin%2C+S.%3B+Oeuillet%2C+E.%3B+Pajon%2C+A.%3B+Harris%2C+M.%3B+Jones%2C+A.T.%3B+van+Tilbeurgh%2C+H.%3B+et+al.&rft.au=Haquin%2C+S.%3B+Oeuillet%2C+E.%3B+Pajon%2C+A.%3B+Harris%2C+M.%3B+Jones%2C+A.T.%3B+van+Tilbeurgh%2C+H.%3B+et+al.&rft.date=2008&rft.volume=426&rft.pages=49-79&rft_id=info:doi\/10.1007%2F978-1-60327-058-8_4&rft_id=info:pmid\/18542857&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MaurerMARS05-4\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MaurerMARS05_4-0\" rel=\"external_link\">4.0<\/a><\/sup> <sup><a href=\"#cite_ref-MaurerMARS05_4-1\" rel=\"external_link\">4.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Maurer, M.; Molidor, R.; Sturn, A.; Hartler, J.; Hackl, H.; Stocker, G.; et al. (2005). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551\" target=\"_blank\">\"MARS: microarray analysis, retrieval, and storage system\"<\/a>. <i>BMC Bioinformatics<\/i> <b>6<\/b>: 101. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-6-101\" target=\"_blank\">10.1186\/1471-2105-6-101<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1090551\/\" target=\"_blank\">PMC1090551<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15836795\" target=\"_blank\">15836795<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1090551<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MARS%3A+microarray+analysis%2C+retrieval%2C+and+storage+system&rft.jtitle=BMC+Bioinformatics&rft.aulast=Maurer%2C+M.%3B+Molidor%2C+R.%3B+Sturn%2C+A.%3B+Hartler%2C+J.%3B+Hackl%2C+H.%3B+Stocker%2C+G.%3B+et+al.&rft.au=Maurer%2C+M.%3B+Molidor%2C+R.%3B+Sturn%2C+A.%3B+Hartler%2C+J.%3B+Hackl%2C+H.%3B+Stocker%2C+G.%3B+et+al.&rft.date=2005&rft.volume=6&rft.pages=101&rft_id=info:doi\/10.1186%2F1471-2105-6-101&rft_id=info:pmc\/PMC1090551&rft_id=info:pmid\/15836795&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1090551&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaalBio02-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SaalBio02_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saal, L.H.; Troein, C.; Vallon-Christersson, J.; Gruvberger, S.; Borg, A.; Peterson, C. (2002). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC139402\" target=\"_blank\">\"BioArray Software Environment (BASE): a platform for comprehensive management and analysis of microarray data\"<\/a>. <i>Genome Biology<\/i> <b>3<\/b> (8): SOFTWARE0003. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fgb-2002-3-8-software0003\" target=\"_blank\">10.1186\/gb-2002-3-8-software0003<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC139402\/\" target=\"_blank\">PMC139402<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/12186655\" target=\"_blank\">12186655<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC139402\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC139402<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=BioArray+Software+Environment+%28BASE%29%3A+a+platform+for+comprehensive+management+and+analysis+of+microarray+data&rft.jtitle=Genome+Biology&rft.aulast=Saal%2C+L.H.%3B+Troein%2C+C.%3B+Vallon-Christersson%2C+J.%3B+Gruvberger%2C+S.%3B+Borg%2C+A.%3B+Peterson%2C+C.&rft.au=Saal%2C+L.H.%3B+Troein%2C+C.%3B+Vallon-Christersson%2C+J.%3B+Gruvberger%2C+S.%3B+Borg%2C+A.%3B+Peterson%2C+C.&rft.date=2002&rft.volume=3&rft.issue=8&rft.pages=SOFTWARE0003&rft_id=info:doi\/10.1186%2Fgb-2002-3-8-software0003&rft_id=info:pmc\/PMC139402&rft_id=info:pmid\/12186655&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC139402&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-HartlerMAS07-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-HartlerMAS07_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Hartler, J.; Thallinger, G.G.; Stocker, G.; Sturn, A.; Burkard, T.R.; Korner, E.; et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1906842\" target=\"_blank\">\"MASPECTRAS: a platform for management and analysis of proteomics LC-MS\/MS data\"<\/a>. <i>BMC Bioinformatics<\/i> <b>8<\/b>: 197. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-8-197\" target=\"_blank\">10.1186\/1471-2105-8-197<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1906842\/\" target=\"_blank\">PMC1906842<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17567892\" target=\"_blank\">17567892<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1906842\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1906842<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=MASPECTRAS%3A+a+platform+for+management+and+analysis+of+proteomics+LC-MS%2FMS+data&rft.jtitle=BMC+Bioinformatics&rft.aulast=Hartler%2C+J.%3B+Thallinger%2C+G.G.%3B+Stocker%2C+G.%3B+Sturn%2C+A.%3B+Burkard%2C+T.R.%3B+Korner%2C+E.%3B+et+al.&rft.au=Hartler%2C+J.%3B+Thallinger%2C+G.G.%3B+Stocker%2C+G.%3B+Sturn%2C+A.%3B+Burkard%2C+T.R.%3B+Korner%2C+E.%3B+et+al.&rft.date=2007&rft.volume=8&rft.pages=197&rft_id=info:doi\/10.1186%2F1471-2105-8-197&rft_id=info:pmc\/PMC1906842&rft_id=info:pmid\/17567892&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1906842&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CraigOpen04-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CraigOpen04_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Craig, R.; Cortens, J.P.; Beavis, R.C. (2004). \"Open source system for analyzing, validating, and storing protein identification data\". <i>Journal of Proteome Research<\/i> <b>3<\/b> (6): 1234-1242. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fpr049882h\" target=\"_blank\">10.1021\/pr049882h<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15595733\" target=\"_blank\">15595733<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+source+system+for+analyzing%2C+validating%2C+and+storing+protein+identification+data&rft.jtitle=Journal+of+Proteome+Research&rft.aulast=Craig%2C+R.%3B+Cortens%2C+J.P.%3B+Beavis%2C+R.C.&rft.au=Craig%2C+R.%3B+Cortens%2C+J.P.%3B+Beavis%2C+R.C.&rft.date=2004&rft.volume=3&rft.issue=6&rft.pages=1234-1242&rft_id=info:doi\/10.1021%2Fpr049882h&rft_id=info:pmid\/15595733&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RauchComp06-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RauchComp06_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Rauch, A.; Bellew, M.; Eng, J.; Fitzgibbon, M.; Holzman, T.; Hussey, P.; et al. (2006). \"Computational Proteomics Analysis System (CPAS): an extensible, open-source analytic system for evaluating and publishing proteomic data and high throughput biological experiments\". <i>Journal of Proteome Research<\/i> <b>5<\/b> (1): 112-121. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1021%2Fpr0503533\" target=\"_blank\">10.1021\/pr0503533<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16396501\" target=\"_blank\">16396501<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Computational+Proteomics+Analysis+System+%28CPAS%29%3A+an+extensible%2C+open-source+analytic+system+for+evaluating+and+publishing+proteomic+data+and+high+throughput+biological+experiments&rft.jtitle=Journal+of+Proteome+Research&rft.aulast=Rauch%2C+A.%3B+Bellew%2C+M.%3B+Eng%2C+J.%3B+Fitzgibbon%2C+M.%3B+Holzman%2C+T.%3B+Hussey%2C+P.%3B+et+al.&rft.au=Rauch%2C+A.%3B+Bellew%2C+M.%3B+Eng%2C+J.%3B+Fitzgibbon%2C+M.%3B+Holzman%2C+T.%3B+Hussey%2C+P.%3B+et+al.&rft.date=2006&rft.volume=5&rft.issue=1&rft.pages=112-121&rft_id=info:doi\/10.1021%2Fpr0503533&rft_id=info:pmid\/16396501&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MooreOpen08-9\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MooreOpen08_9-0\" rel=\"external_link\">9.0<\/a><\/sup> <sup><a href=\"#cite_ref-MooreOpen08_9-1\" rel=\"external_link\">9.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Moore, J.; Allan, C.; Burel, J.M.; Loranger, B.; MacDonald, D.; Monk, J.; et al. (2008). \"Open tools for storage and management of quantitative image data\". <i>Methods in Cell Biology<\/i> <b>85<\/b>: 555-570. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS0091-679X%2808%2985024-8\" target=\"_blank\">10.1016\/S0091-679X(08)85024-8<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18155479\" target=\"_blank\">18155479<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+tools+for+storage+and+management+of+quantitative+image+data&rft.jtitle=Methods+in+Cell+Biology&rft.aulast=Moore%2C+J.%3B+Allan%2C+C.%3B+Burel%2C+J.M.%3B+Loranger%2C+B.%3B+MacDonald%2C+D.%3B+Monk%2C+J.%3B+et+al.&rft.au=Moore%2C+J.%3B+Allan%2C+C.%3B+Burel%2C+J.M.%3B+Loranger%2C+B.%3B+MacDonald%2C+D.%3B+Monk%2C+J.%3B+et+al.&rft.date=2008&rft.volume=85&rft.pages=555-570&rft_id=info:doi\/10.1016%2FS0091-679X%2808%2985024-8&rft_id=info:pmid\/18155479&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JonesFuGE06-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JonesFuGE06_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-JonesFuGE06_10-1\" rel=\"external_link\">10.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jones, A.R.; Pizarro, A.; Spellman, P.; Miller, M. (2006). \"FuGE: Functional Genomics Experiment Object Model\". <i>OMICS<\/i> <b>10<\/b> (2): 179-184. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1089%2Fomi.2006.10.179\" target=\"_blank\">10.1089\/omi.2006.10.179<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16901224\" target=\"_blank\">16901224<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=FuGE%3A+Functional+Genomics+Experiment+Object+Model&rft.jtitle=OMICS&rft.aulast=Jones%2C+A.R.%3B+Pizarro%2C+A.%3B+Spellman%2C+P.%3B+Miller%2C+M.&rft.au=Jones%2C+A.R.%3B+Pizarro%2C+A.%3B+Spellman%2C+P.%3B+Miller%2C+M.&rft.date=2006&rft.volume=10&rft.issue=2&rft.pages=179-184&rft_id=info:doi\/10.1089%2Fomi.2006.10.179&rft_id=info:pmid\/16901224&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JonesThe07-11\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-JonesThe07_11-0\" rel=\"external_link\">11.0<\/a><\/sup> <sup><a href=\"#cite_ref-JonesThe07_11-1\" rel=\"external_link\">11.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Jones, A.R.; Miller, M.; Aebersold, R.; Apweiler, R.; Ball, C.A.; Brazma, A.; et al. (2007). \"The Functional Genomics Experiment model (FuGE): an extensible framework for standards in functional genomics\". <i>Nature Biotechnology<\/i> <b>25<\/b> (10): 1127-1133. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt1347\" target=\"_blank\">10.1038\/nbt1347<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17921998\" target=\"_blank\">17921998<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Functional+Genomics+Experiment+model+%28FuGE%29%3A+an+extensible+framework+for+standards+in+functional+genomics&rft.jtitle=Nature+Biotechnology&rft.aulast=Jones%2C+A.R.%3B+Miller%2C+M.%3B+Aebersold%2C+R.%3B+Apweiler%2C+R.%3B+Ball%2C+C.A.%3B+Brazma%2C+A.%3B+et+al.&rft.au=Jones%2C+A.R.%3B+Miller%2C+M.%3B+Aebersold%2C+R.%3B+Apweiler%2C+R.%3B+Ball%2C+C.A.%3B+Brazma%2C+A.%3B+et+al.&rft.date=2007&rft.volume=25&rft.issue=10&rft.pages=1127-1133&rft_id=info:doi\/10.1038%2Fnbt1347&rft_id=info:pmid\/17921998&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BrazmaMin01-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BrazmaMin01_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Brazma, A.; Hingamp, P.; Quackenbush, J.; Sherlock, G.; Spellman, P.; Stoeckert, C.; et al. (2001). \"Minimum information about a microarray experiment (MIAME)-toward standards for microarray data\". <i>Nature Genetics<\/i> <b>29<\/b> (4): 365-371. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fng1201-365\" target=\"_blank\">10.1038\/ng1201-365<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11726920\" target=\"_blank\">11726920<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Minimum+information+about+a+microarray+experiment+%28MIAME%29-toward+standards+for+microarray+data&rft.jtitle=Nature+Genetics&rft.aulast=Brazma%2C+A.%3B+Hingamp%2C+P.%3B+Quackenbush%2C+J.%3B+Sherlock%2C+G.%3B+Spellman%2C+P.%3B+Stoeckert%2C+C.%3B+et+al.&rft.au=Brazma%2C+A.%3B+Hingamp%2C+P.%3B+Quackenbush%2C+J.%3B+Sherlock%2C+G.%3B+Spellman%2C+P.%3B+Stoeckert%2C+C.%3B+et+al.&rft.date=2001&rft.volume=29&rft.issue=4&rft.pages=365-371&rft_id=info:doi\/10.1038%2Fng1201-365&rft_id=info:pmid\/11726920&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorThe07-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorThe07_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, C.F.; Paton, N.W.; Lilley, K.S.; Binz, P.A.; Julian, Jr., R.K.; Jones AR, et al. (2007). \"The minimum information about a proteomics experiment (MIAPE)\". <i>Nature Biotechnology<\/i> <b>25<\/b> (8): 887-893. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt1329\" target=\"_blank\">10.1038\/nbt1329<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17687369\" target=\"_blank\">17687369<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+minimum+information+about+a+proteomics+experiment+%28MIAPE%29&rft.jtitle=Nature+Biotechnology&rft.aulast=Taylor%2C+C.F.%3B+Paton%2C+N.W.%3B+Lilley%2C+K.S.%3B+Binz%2C+P.A.%3B+Julian%2C+Jr.%2C+R.K.%3B+Jones+AR%2C+et+al.&rft.au=Taylor%2C+C.F.%3B+Paton%2C+N.W.%3B+Lilley%2C+K.S.%3B+Binz%2C+P.A.%3B+Julian%2C+Jr.%2C+R.K.%3B+Jones+AR%2C+et+al.&rft.date=2007&rft.volume=25&rft.issue=8&rft.pages=887-893&rft_id=info:doi\/10.1038%2Fnbt1329&rft_id=info:pmid\/17687369&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeutschMin08-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeutschMin08_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Deutsch, E.W.; Ball, C.A.; Berman, J.J.; Bova, G.S.; Brazma, A.; Bumgarner, R.E.; et al. (2008). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4367930\" target=\"_blank\">\"Minimum information specification for in situ hybridization and immunohistochemistry experiments (MISFISHIE)\"<\/a>. <i>Nature Biotechnology<\/i> <b>26<\/b> (3): 305-312. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt1391\" target=\"_blank\">10.1038\/nbt1391<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4367930\/\" target=\"_blank\">PMC4367930<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18327244\" target=\"_blank\">18327244<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4367930\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4367930<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Minimum+information+specification+for+in+situ+hybridization+and+immunohistochemistry+experiments+%28MISFISHIE%29&rft.jtitle=Nature+Biotechnology&rft.aulast=Deutsch%2C+E.W.%3B+Ball%2C+C.A.%3B+Berman%2C+J.J.%3B+Bova%2C+G.S.%3B+Brazma%2C+A.%3B+Bumgarner%2C+R.E.%3B+et+al.&rft.au=Deutsch%2C+E.W.%3B+Ball%2C+C.A.%3B+Berman%2C+J.J.%3B+Bova%2C+G.S.%3B+Brazma%2C+A.%3B+Bumgarner%2C+R.E.%3B+et+al.&rft.date=2008&rft.volume=26&rft.issue=3&rft.pages=305-312&rft_id=info:doi\/10.1038%2Fnbt1391&rft_id=info:pmc\/PMC4367930&rft_id=info:pmid\/18327244&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4367930&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorTheStatus-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorTheStatus_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, K.T. (2006). \"The status of electronic laboratory notebooks for chemistry and biology\". <i>Current Opinion in Drug Discovery & Development<\/i> <b>9<\/b> (3): 348\u201353. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16729731\" target=\"_blank\">16729731<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+status+of+electronic+laboratory+notebooks+for+chemistry+and+biology&rft.jtitle=Current+Opinion+in+Drug+Discovery+%26+Development&rft.aulast=Taylor%2C+K.T.&rft.au=Taylor%2C+K.T.&rft.date=2006&rft.volume=9&rft.issue=3&rft.pages=348%E2%80%9353&rft_id=info:pmid\/16729731&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DrakeELN07-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DrakeELN07_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Drake, D.J. (2007). \"ELN implementation challenges\". <i>Drug Discovery Today<\/i> <b>12<\/b> (15-16): 647-649. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.drudis.2007.06.010\" target=\"_blank\">10.1016\/j.drudis.2007.06.010<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17706546\" target=\"_blank\">17706546<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=ELN+implementation+challenges&rft.jtitle=Drug+Discovery+Today&rft.aulast=Drake%2C+D.J.&rft.au=Drake%2C+D.J.&rft.date=2007&rft.volume=12&rft.issue=15-16&rft.pages=647-649&rft_id=info:doi\/10.1016%2Fj.drudis.2007.06.010&rft_id=info:pmid\/17706546&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ButlerElec05-17\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ButlerElec05_17-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Butler, D. (2005). \"Electronic notebooks: a new leaf\". <i>Nature<\/i> <b>436<\/b> (7047): 20-21. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2F436020a\" target=\"_blank\">10.1038\/436020a<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16001034\" target=\"_blank\">16001034<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+notebooks%3A+a+new+leaf&rft.jtitle=Nature&rft.aulast=Butler%2C+D.&rft.au=Butler%2C+D.&rft.date=2005&rft.volume=436&rft.issue=7047&rft.pages=20-21&rft_id=info:doi\/10.1038%2F436020a&rft_id=info:pmid\/16001034&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KihlenElec05-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KihlenElec05_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Kihlen, M. (2005). \"Electronic lab notebooks - do they work in reality?\". <i>Drug Discovery Today<\/i> <b>10<\/b> (18): 1205-1207. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1359-6446%2805%2903576-2\" target=\"_blank\">10.1016\/S1359-6446(05)03576-2<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16213407\" target=\"_blank\">16213407<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Electronic+lab+notebooks+-+do+they+work+in+reality%3F&rft.jtitle=Drug+Discovery+Today&rft.aulast=Kihlen%2C+M.&rft.au=Kihlen%2C+M.&rft.date=2005&rft.volume=10&rft.issue=18&rft.pages=1205-1207&rft_id=info:doi\/10.1016%2FS1359-6446%2805%2903576-2&rft_id=info:pmid\/16213407&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BradleySMIRP04-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BradleySMIRP04_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bradley, J.-C.; Samuel, B. (2004). \"SMIRP-A Systems Approach to Laboratory Automation\". <i>Journal of the Association for Laboratory Automation<\/i> <b>5<\/b> (3): 48-53. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2FS1535-5535%2804%29%E2%80%8B00074-7\" target=\"_blank\">10.1016\/S1535-5535(04)\u200b00074-7<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=SMIRP-A+Systems+Approach+to+Laboratory+Automation&rft.jtitle=Journal+of+the+Association+for+Laboratory+Automation&rft.aulast=Bradley%2C+J.-C.%3B+Samuel%2C+B.&rft.au=Bradley%2C+J.-C.%3B+Samuel%2C+B.&rft.date=2004&rft.volume=5&rft.issue=3&rft.pages=48-53&rft_id=info:doi\/10.1016%2FS1535-5535%2804%29%E2%80%8B00074-7&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorThe06\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorThe06_0\" rel=\"external_link\">\u2191<\/a><\/span> <strong class=\"error mw-ext-cite-error\">Cite error: Invalid <code><ref><\/code> tag;\nno text was provided for refs named <code>TaylorThe06<\/code><\/strong><\/li>\n<li id=\"cite_note-ApacheTap09-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ApacheTap09_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/tapestry.apache.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">\"Tapestry web frame work\"<\/a>. Apache Software Foundation. 2009. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/tapestry.apache.org\/\" target=\"_blank\">the original<\/a> on 23 July 2010<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/tapestry.apache.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/tapestry.apache.org\/&refdoi=10.1186\/1471-2105-10-390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Tapestry+web+frame+work&rft.atitle=&rft.date=2009&rft.pub=Apache+Software+Foundation&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Ftapestry.apache.org%2F%26refdoi%3D10.1186%2F1471-2105-10-390&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ApacheJava09-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ApacheJava09_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/ws.apache.org\/axis\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">\"Java implementation of the SOAP (\"Simple Object Access Protocol\")\"<\/a>. Apache Software Foundation. 2009. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ws.apache.org\/axis\/\" target=\"_blank\">the original<\/a> on 07 December 2005<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/ws.apache.org\/axis\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/ws.apache.org\/axis\/&refdoi=10.1186\/1471-2105-10-390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Java+implementation+of+the+SOAP+%28%22Simple+Object+Access+Protocol%22%29&rft.atitle=&rft.date=2009&rft.pub=Apache+Software+Foundation&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Fws.apache.org%2Faxis%2F%26refdoi%3D10.1186%2F1471-2105-10-390&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrugDont00-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KrugDont00_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Krug, S. (2000). <i>Don't make me think! A Common Sense Apporach to Web Usability<\/i>. Indianapolis, IN: New Riders Publishing.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Don%27t+make+me+think%21+A+Common+Sense+Apporach+to+Web+Usability&rft.aulast=Krug%2C+S.&rft.au=Krug%2C+S.&rft.date=2000&rft.place=Indianapolis%2C+IN&rft.pub=New+Riders+Publishing&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JohnsonWeb03-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JohnsonWeb03_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Johnson, J. (2003). <i>Web Bloopers: 60 Common Web Design Mistakes and How to Avoid Them<\/i>. San Francisco, CA: Morgan Kaufmann Publishers, Inc.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Web+Bloopers%3A+60+Common+Web+Design+Mistakes+and+How+to+Avoid+Them&rft.aulast=Johnson%2C+J.&rft.au=Johnson%2C+J.&rft.date=2003&rft.place=San+Francisco%2C+CA&rft.pub=Morgan+Kaufmann+Publishers%2C+Inc&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SpringSpring09-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SpringSpring09_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.springsource.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">\"Spring lightweight application container\"<\/a>. SpringSource. 2009. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.springsource.org\/\" target=\"_blank\">the original<\/a> on 27 May 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.springsource.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.springsource.org\/&refdoi=10.1186\/1471-2105-10-390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Spring+lightweight+application+container&rft.atitle=&rft.date=2009&rft.pub=SpringSource&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Fwww.springsource.org%2F%26refdoi%3D10.1186%2F1471-2105-10-390&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ApacheServices09-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ApacheServices09_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/hivemind.apache.org\/\" target=\"_blank\">\"Services and configuration microkernel\"<\/a>. Apache Software Foundation. 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/hivemind.apache.org\/\" target=\"_blank\">http:\/\/hivemind.apache.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Services+and+configuration+microkernel&rft.atitle=&rft.date=2009&rft.pub=Apache+Software+Foundation&rft_id=http%3A%2F%2Fhivemind.apache.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ApacheApache09-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ApacheApache09_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/tomcat.apache.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">\"Apache servlet container\"<\/a>. Apache Software Foundation. 2009. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/tomcat.apache.org\/\" target=\"_blank\">the original<\/a> on 27 May 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/tomcat.apache.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/tomcat.apache.org\/&refdoi=10.1186\/1471-2105-10-390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Apache+servlet+container&rft.atitle=&rft.date=2009&rft.pub=Apache+Software+Foundation&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Ftomcat.apache.org%2F%26refdoi%3D10.1186%2F1471-2105-10-390&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JohnsonExpert04-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JohnsonExpert04_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Johnson, R.; Hoeller, J. (2004). <i>Expert One-on-One J2EE Development without EJB<\/i>. Wrox.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Expert+One-on-One+J2EE+Development+without+EJB&rft.aulast=Johnson%2C+R.%3B+Hoeller%2C+J.&rft.au=Johnson%2C+R.%3B+Hoeller%2C+J.&rft.date=2004&rft.pub=Wrox&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OW2Java09-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OW2Java09_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/jotm.ow2.org\/xwiki\/bin\/view\/Main\/WebHome\" target=\"_blank\">\"Java Open Transaction Manager (JOTM)\"<\/a>. OW2 Consortium. 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/jotm.ow2.org\/xwiki\/bin\/view\/Main\/WebHome\" target=\"_blank\">http:\/\/jotm.ow2.org\/xwiki\/bin\/view\/Main\/WebHome<\/a>?<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Java+Open+Transaction+Manager+%28JOTM%29&rft.atitle=&rft.date=2009&rft.pub=OW2+Consortium&rft_id=http%3A%2F%2Fjotm.ow2.org%2Fxwiki%2Fbin%2Fview%2FMain%2FWebHome%3F&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BohlenAndro09-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BohlenAndro09_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Bohlen, M. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.andromda.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">\"AndroMDA\"<\/a>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.andromda.org\/\" target=\"_blank\">the original<\/a> on 27 March 2007<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.andromda.org\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/www.andromda.org\/&refdoi=10.1186\/1471-2105-10-390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=AndroMDA&rft.atitle=&rft.aulast=Bohlen%2C+M.&rft.au=Bohlen%2C+M.&rft.date=2009&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Fwww.andromda.org%2F%26refdoi%3D10.1186%2F1471-2105-10-390&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RasbandIma09-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RasbandIma09_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Rasband, W.S. (2009). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/rsb.info.nih.gov\/ij\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">\"ImageJ\"<\/a>. Archived from <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/rsb.info.nih.gov\/ij\/\" target=\"_blank\">the original<\/a> on 23 May 2009<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.webcitation.org\/query.php?url=http:\/\/rsb.info.nih.gov\/ij\/&refdoi=10.1186\/1471-2105-10-390\" target=\"_blank\">http:\/\/www.webcitation.org\/query.php?url=http:\/\/rsb.info.nih.gov\/ij\/&refdoi=10.1186\/1471-2105-10-390<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ImageJ&rft.atitle=&rft.aulast=Rasband%2C+W.S.&rft.au=Rasband%2C+W.S.&rft.date=2009&rft_id=http%3A%2F%2Fwww.webcitation.org%2Fquery.php%3Furl%3Dhttp%3A%2F%2Frsb.info.nih.gov%2Fij%2F%26refdoi%3D10.1186%2F1471-2105-10-390&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AbramoffImage04-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AbramoffImage04_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Abramoff, M.D.; Magelhaes, P.J.; Ram, S.J.. \"Image Processing with ImageJ\". <i>Biophotonics International<\/i> <b>11<\/b> (7): 36\u201342.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Image+Processing+with+ImageJ&rft.jtitle=Biophotonics+International&rft.aulast=Abramoff%2C+M.D.%3B+Magelhaes%2C+P.J.%3B+Ram%2C+S.J.&rft.au=Abramoff%2C+M.D.%3B+Magelhaes%2C+P.J.%3B+Ram%2C+S.J.&rft.volume=11&rft.issue=7&rft.pages=36%E2%80%9342&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RoureMy08-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RoureMy08_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Roure, D.; Goble, C.; Bhagat, J.; Cruickshank, D.; Goderis, A.; Michaelides, D.; et al.. \"myExperiment: Defining the Social Virtual Research Environment\". <i>IEEE Fourth International Conference on eScience, 2008<\/i> <b>2008<\/b>: 182-189. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1109%2FeScience.2008.86\" target=\"_blank\">10.1109\/eScience.2008.86<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=myExperiment%3A+Defining+the+Social+Virtual+Research+Environment&rft.jtitle=IEEE+Fourth+International+Conference+on+eScience%2C+2008&rft.aulast=Roure%2C+D.%3B+Goble%2C+C.%3B+Bhagat%2C+J.%3B+Cruickshank%2C+D.%3B+Goderis%2C+A.%3B+Michaelides%2C+D.%3B+et+al.&rft.au=Roure%2C+D.%3B+Goble%2C+C.%3B+Bhagat%2C+J.%3B+Cruickshank%2C+D.%3B+Goderis%2C+A.%3B+Michaelides%2C+D.%3B+et+al.&rft.volume=2008&rft.pages=182-189&rft_id=info:doi\/10.1109%2FeScience.2008.86&rfr_id=info:sid\/en.wikipedia.org:Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In most of the article's references DOIs and PubMed IDs were not given; they've been added to make the references more useful.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210217\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.749 seconds\nReal time usage: 0.787 seconds\nPreprocessor visited node count: 24659\/1000000\nPreprocessor generated node count: 37182\/1000000\nPost\u2010expand include size: 183964\/2097152 bytes\nTemplate argument size: 62900\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 739.386 1 - -total\n 85.56% 632.606 1 - Template:Reflist\n 74.85% 553.411 32 - Template:Citation\/core\n 56.15% 415.148 21 - Template:Cite_journal\n 16.67% 123.228 8 - Template:Cite_web\n 9.01% 66.600 1 - Template:Infobox_journal_article\n 8.65% 63.985 1 - Template:Infobox\n 7.42% 54.836 41 - Template:Citation\/identifier\n 6.68% 49.424 3 - Template:Cite_book\n 5.31% 39.249 80 - Template:Infobox\/row\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7678-0!*!0!!en!5!* and timestamp 20181213210216 and revision id 31954\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis\">https:\/\/www.limswiki.org\/index.php\/Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","59816fe6940deb7e43cf53893a6ea71a_images":["https:\/\/www.limswiki.org\/images\/a\/ac\/Fig1_Stocker_BMCBioinformatics2009_10.jpg","https:\/\/www.limswiki.org\/images\/0\/0c\/Fig2_Stocker_BMCBioinformatics2009_10.jpg","https:\/\/www.limswiki.org\/images\/4\/44\/Fig3_Stocker_BMCBioinformatics2009_10.jpg","https:\/\/www.limswiki.org\/images\/c\/ca\/Fig4_Stocker_BMCBioinformatics2009_10.jpg"],"59816fe6940deb7e43cf53893a6ea71a_timestamp":1544734936,"b55d160b7541498bbeb5c39c2bdb347d_type":"article","b55d160b7541498bbeb5c39c2bdb347d_title":"adLIMS: A customized open source software that allows bridging clinical and basic molecular research studies (Calabria et al. 2015)","b55d160b7541498bbeb5c39c2bdb347d_url":"https:\/\/www.limswiki.org\/index.php\/Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies","b55d160b7541498bbeb5c39c2bdb347d_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:adLIMS: A customized open source software that allows bridging clinical and basic molecular research studies\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\t \n\nFull article title\n \nadLIMS: A customized open source software that allows bridging clinical and basic molecular research studiesJournal\n \nBMC BioinformaticsAuthor(s)\n \nCalabria, Andrea; Spinozzi, Giulio; Benedicenti, Fabrizio; Tenderini, Erika; Montini, EugenioAuthor affiliation(s)\n \nSan Raffaele Telethon Institute for Gene Therapy; University of Milano-BicoccaPrimary contact\n \nEmail: montini.eugenio@hsr.itYear published\n \n2015Volume and issue\n \n16 (Suppl 9)Page(s)\n \nS5DOI\n \n10.1186\/1471-2105-16-S9-S5ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.biomedcentral.com\/1471-2105\/16\/S9\/S5Download\n \nhttp:\/\/www.biomedcentral.com\/content\/pdf\/1471-2105-16-S9-S5.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Methods \n\n3.1 Requirements Analysis \n3.2 Evaluation of available LIMS and alternative solutions \n3.3 Design \n\n\n4 Results and discussion \n5 Conclusions \n6 Competing interests \n7 Authors' contributions \n8 Availability and requirements \n9 Acknowledgements \n10 Declarations \n11 References \n12 Notes \n\n\n\nAbstract \nBackground: Many biological laboratories that deal with genomic samples are facing the problem of sample tracking, both for pure laboratory management and for efficiency. Our laboratory exploits PCR techniques and Next Generation Sequencing (NGS) methods to perform high-throughput integration site monitoring in different clinical trials and scientific projects. Because of the huge amount of samples that we process every year, which result in hundreds of millions of sequencing reads, we need to standardize data management and tracking systems, building up a scalable and flexible structure with web-based interfaces, which are usually called Laboratory Information Management System (LIMS).\nMethods: We started collecting end-users' requirements, composed of desired functionalities of the system and Graphical User Interfaces (GUI), and then we evaluated available tools that could address our requirements, spanning from pure LIMS to Content Management Systems (CMS) up to enterprise information systems. Our analysis identified ADempiere ERP, an open source Enterprise Resource Planning written in Java J2EE, as the best software that also natively implements some highly desirable technological advances, such as the high usability and modularity that grants high use-case flexibility and software scalability for custom solutions.\nResults: We extended and customized ADempiere ERP to fulfil LIMS requirements and we developed adLIMS. It has been validated by our end-users verifying functionalities and GUIs through test cases for PCRs samples and pre-sequencing data and it is currently in use in our laboratories. adLIMS implements authorization and authentication policies, allowing multiple users management and roles definition that enables specific permissions, operations and data views to each user. For example, adLIMS allows creating sample sheets from stored data using available exporting operations. This simplicity and process standardization may avoid manual errors and information backtracking, features that are not granted using track recording on files or spreadsheets.\nConclusions: adLIMS aims to combine sample tracking and data reporting features with higher accessibility and usability of GUIs, thus allowing time to be saved on doing repetitive laboratory tasks, and reducing errors with respect to manual data collection methods. Moreover, adLIMS implements automated data entry, exploiting sample data multiplexing and parallel\/transactional processing. adLIMS is natively extensible to cope with laboratory automation through platform-dependent API interfaces, and could be extended to genomic facilities due to the ERP functionalities.\nKeywords: LIMS; Open Source Software; Information Systems; ADempiere ERP; Sample Tracking\n\nBackground \nIn many biological laboratories, sample tracking is an outstanding issue and often represents a bottleneck for the correct handling and interpretation of experimental data. This issue is becoming particularly critical when automation and high-throughput technologies are introduced in the laboratory practice. Our laboratory performs high-throughput characterization of vector-genomic integration sites in the context of gene therapy applications based on the delivery of therapeutic genes by viral vectors that stably integrate into the genome of targeted cells, as well as gene therapy preclinical models and insertional mutagenesis research projects.[1][2][3][4][5][6][7][8][9] Vector integration sites are retrieved and mapped in the genome through a combination of Polymerase Chain Reaction (PCR)-based techniques[10], next generation sequencing (NGS) and bioinformatics analyses.[11] We process and analyze around 2000 samples\/year resulting in hundreds of millions of sequencing reads. Despite the fact that adopting robotic automation for sample manipulation in our laboratory has provided many advantages in terms of manual error-reduction and data production scalability, drawbacks related to sample information volume and tracking are still present. These reasons prompted us to develop a Laboratory Information Management System (LIMS)[12] for sample tracking on a scalable and flexible infrastructure with an easily accessible and web-based interface. LIMS is a type of information system implemented as a software utility specifically designed to improve the data acquisition and sample monitoring along laboratory workflows, and supporting sample reporting. An information system is a combination of information technologies developed to grant business processes efficiency and monitoring. Extended IS are the Enterprise Resource Planning (ERP) solutions[13] that integrate the standard information system features with accounting and administrative operations for performance monitoring through dashboards and data mining tools.\nIn this work we describe our LIMS, developed on an existing open source ERP framework that natively implements all technological functionalities, as software customization and parameterization. After a brief introduction of the ERP framework with the motivation of the specific choice, we will describe in details our implementation with custom use cases and scenarios derived from our laboratory requirements and experience. \n\nMethods \nWe followed a typical software engineering approach, \"waterfall\"[14], to design our solution. We first collected end-users' requirements, composed of desired functionalities of the systems and graphical user interfaces (GUI). Then we evaluated available tools that could address our requirements, spanning from pure LIMS to Content Management Systems (CMS) up to enterprise information systems. In the next step we selected our candidate solution and we designed its configuration to best fulfil our requirements. \n\nRequirements Analysis \nThe analysis of requirements, also called requirements engineering, is the process of acquiring software expectations from users\/clients in terms of functionalities and interaction that are translated in software requirements.[15]\nThe first step is to define which types of users will access the LIMS (roles and groups) and which authorization policies we have to apply to specific users (permissions on data and operations). Using a Unified Modeling Language (UML)[16] use case diagram (Figure 1), we collected desired roles (called \"actors\" in UML) and operations (called \"actions\" in UML). The two principal roles of the system are distinguished for the area in which they operate. In this instance we identified two main areas: a sample management area where samples are collected from patients or subjects and a wet experimental area in which collected samples are processed to retrieve vector-cellular genomic junctions. For this reason we identified two corresponding roles, a SampleManager and a WetManager, and their permissions and operations scope correspond to the data that they can access: the SampleManager role is related to users that input and manage sample metadata (project, DNA and so on), while the WetManager role is related to users that input and manage experimental procedures and workflows on the DNA samples (PCR steps, NGS pooling, and so on). Then we collected the desired authorizations associated to each role as reported in the CRUD (Create, Read, Update, Delete) policies (Table 1). \n\n\n\n\n\n\n\n\n\n Figure 1: adLIMS' use case: The UML Use Case Diagram of our case study. In our case we have two actors: SampleManager and WetManager. A user with SampleManager role can insert, edit, manage, search and filter only clinical data, and deals with data about DNA, project and patient. A user with WetManager role can view clinical data, and it can insert, edit, manage, search and filter data about LAM-PCR and NGS pool.\n\n\n\n\n\n\n\n\n\nTable 1. User's Rules in \"Create, Read, Update and Delete\" format (CRUD). The table is a summary of the role permissions and policies in adLIMS in terms of data access. We use the CRUD syntax: create (C), read (R), update (U), delete (D).\n\n\n\n\n Clinic data\n\n Sequencing data\n\n Projects\n\n LIMS managing\n\n Users managing\n\n\nWetManager\n\nR\n\nCRUD\n\nCRUD\n\n--\n\n--\n\n\nSampleManager\n\nCRUD\n\n--\n\nCRUD\n\n--\n\n--\n\n\nAdministrator\n\n--\n\n--\n\n--\n\nCRUD\n\nCRUD\n\n\n\nThe SampleManager role can access and edit metadata related to all clinical and preclinical aspects such as anonymized patients' IDs, DNA (concentrations, volume, and so on) and cell types (markers, lineage, and so on). The visibility spectrum of a user with the SampleManager role is restricted to sample metadata only and will not be able to read other information of post sample processing.\nThe WetManager role is able to control all aspects related to experimental procedures and workflows on the samples. A user assigned to the WetManager role will not be able to modify input metadata from the SampleManager role.\nWe added an administrator role (or SuperUser) that is able to control and edit every component of the system, customizing all the functional and graphical levels related to the model-view-controller paradigm.[17] However this role is not authorized to modify input data. At the functional level, the administrator has the privileges to: (1) create, delete, and suspend accounts and (2) define rules to access to the database (such as tables, views, fields, etc.). At the graphical level, the administrator can create and edit windows and layouts, control how to graphically visualize and access data and define how to export and import data in the information system.\nFrom this analysis of requirements we derived the interaction workflow (Figure 2). In the first step of our workflow, a user with a SampleManager role inputs data related to the specific project and the samples that need to be processed for integration site analysis, in particular information relative to DNA. These samples will be then processed (upon a request of analysis from the SampleManager) in the custom workflow by a user with a WetManager role for linear amplification mediated (LAM)-PCR.[10] The procedure of the LAM-PCR protocol has to be implemented in a LIMS process that allows tracking each sample manipulation. LAM-PCR products are then selected and pooled for deep sequencing. All the information of the pooled products are exported and formatted as required by Illumina MiSeq sample sheet. Therefore the information system allows monitoring and tracking sample metadata, LAM-PCR processing and the pool preparation for NGS supporting DNA barcodes association without directly handling NGS files. Additional file 1 summarizes the laboratory workflow and system interaction based on the above-mentioned requirements. \n\n\n\n\n\n\n\n\n\nFigure 2: Sample case study of adLIMS A user with SampleManager role inserts data of the sample harvested from a patient (in this scenario) into adLIMS and then activate a request for wet processing (arrow number 3). A user with WetManager role processes the sample first adding other sample related information like data quality or concentration, then sets-up a LAM-PCR experiment (arrow number 5), prepares a pool for sequencing (arrow number 6), and finally exports a sample sheet. The arrow colors represent different conceptual steps of the workflow.\n\n\n\n\n\n\n\n\n\n Additional file 1. General data workflow of vector integration sites identification at TIGET Vector Integration Core\n\n\n The standard flow of a vector integration site project in the Vector Integration Core at the Telethon Institute for Gene Therapy in Milan\nFormat: DOCX; Size: 153KB Download file\n\n\n\n\nAs additional requirements, the system has to be web-based and supported by reliable technologies with a backup system thus enabling data maintenance and recovery in case of failures (e.g. electrical supply problems). The centralized nature of the system requires standard hardware performances such as the simultaneous interaction with dozens of users keeping response times faster than 3 seconds. \n\nEvaluation of available LIMS and alternative solutions \nWe evaluated available LIMS solutions, from commercial to open source ones. Commercial or stand-alone LIMSs[18][19] are often very expensive and\/or lack the flexibility and scalability needed to manage different types of sample data, procedures and analyses specifically designed for each research project. We also explored open source software designed for biological laboratories, such as Bika LIMS[20], LabKey[21] or Galaxy[22], and content management systems (CMS) like Plone[23] and Drupal[24] under the perspective of customizing them and exploiting built-in functionalities such as user management, workflow management and configuration. Unfortunately, none of them fully satisfied our requirements because most of the features, that are the peculiarity of a LIMS (like export, report, etc.), have not been implemented. In this context, we analyzed Enterprise Resource Planning (ERP) solutions and ADempiere ERP[25] was the best available software. ADempiere ERP has been developed under GPL license in Java J2EE with Model-View-Controller design pattern[17] and database-driven logic.[26] ADempiere ERP implements all required features and presents some highly desirable technological advances (see Additional file 2 for the list of desirable features with a comparative analysis among Bika LIMS, LabKey and ADempiere), such as high usability (web and mobile interfaces) and modularity that grants high use-case flexibility (plug and play approach) and software scalability for custom solutions (adaptable to all use cases). The application server is JBoss and supported databases are Oracle and PostgreSQL. Web interfaces exploit the latest technologies with ZK.[27] Moreover, it natively supports multiple languages, accounting procedures and dashboards for process monitoring thanks to the ERP functionalities that can be easily adapted to high-standard industrial and commercial contexts. \n\n\n\n\n\n\n\n Additional file 2. Features' comparison between Bika, LabKey and ADempiere\n\n\nComparison among the features available in Bika, LabKey and ADempiere\nFormat: DOCX; Size: 26KB Download file\n\n\n\n\nDesign \nThe last process in the \"waterfall\" approach is the system design. In order to translate the requirements into both functionalities and GUI within ADempiere ERP, we developed an extension of the core database by applying the required operations and user's policies resulting in a scalable system that natively supports Java fat client and web interfaces. For each end-user interaction and functionality, we designed a custom view of the workflow (Figure 2) with dedicated GUIs based on use cases (Figure 1). Since ADempiere ERP is database-driven, the design of the database related to the LIMS extension is a key aspect that drives and manages both the workflow instance and the GUIs. Our database extension is compliant with the core-system table design that required the addition of ten pre-defined fields (see Additional file 3). This operation is required because ADempiere ERP leverages on inner tables to directly create forms (windows) and GUIs. The \"application dictionary\" is one of the most powerful aspects of ADempiere ERP that acts as the engine of the database-driven model. All metadata needed to build data forms, windows and GUI are contained in the application dictionary that operates at the application layer and generates windows, tabs, menu, forms, nested elements connection, and so on. The application dictionary allows dynamic and flexible changes in GUI and data forms by changing its table content without requiring programming development that is thus drastically reduced. As a direct consequence, GUI changes or customizations can be configured directly in the application dictionary without requiring software compiling or re-building. For example, to create a new window with proper title, menu bar, tool bar, and status bar, ADempiere ERP automatically adds elements in the application dictionary and generates all required fields starting from the database table. \n\n\n\n\n\n\n\n Additional file 3. Customization set-up\n\n\nThe report file of all the customization steps required for creating an instance of adLIMS, with an example of table creation and trigger definition\nFormat: DOCX; Size: 28KB Download file\n\n\n\n\nBased on the general workflow modeled in the analysis of requirements (Figure 2), we designed and implemented the LIMS database as an extension of the core ADempiere ERP database (Additional file 4), here reported as Entity-Relationship model in which each basic entity is associated to a custom table (\"project\", \"subject\", \"DNA\", \"vector\", \"sample\", and so on). We implemented the model in PostgreSQL and we used BLOB (Binary Large Object) to manage external files as attachments (such as images, pdf files, and so on) that users can upload into any entry. We then created custom GUIs related to the previously described workflow for the management of all data tables (Figure 2). Each window and the relative data are accessed by users according to their role (SampleManager, WetManager and administrator role) and authorization policies. Is always possible to modify existing roles and names and to add new roles according to new requirements or specifications. In our laboratory practice we routinely collect LAM-PCR data, gel images and sequencing quality reports and we store these data in adLIMS. The LAM-PCR workflow has been automated by implementing dedicated tables in the system (\"experiment\", \"lam_pcr_linear\", \"lam_pcr_1st_exp\", \"lam_pcr_2nd_exp\") with corresponding GUI input forms. Custom database triggers support the multiplexing of samples during the experimental operations required in different steps of the process (see Additional file 3 for trigger details). Similar automated procedures have been developed to support the generation of sequencing pools for high-throughput NGS platforms by defining custom tables (\"fusion\", \"pool\", \"pool_details\") and associated GUI. To avoid hard drive bottlenecks, raw NGS files are included in the system as links (absolute path with host server) and not as attachments. \n\n\n\n\n\n\n\n Additional file 4. adLIMS entity-relationship database model adLIMS\n\n\nDatabase schema represented as Entity-Relationship model\nFormat: DOCX; Size: 716KB Download file\n\n\n\n\nResults and discussion \nWe developed adLIMS as an open source customization of ADempiere ERP, which implements the general workflow and requirements previously described (Figure 3 and Additional file 5 for details of roles and GUIs). adLIMS exploits all the core functionalities of ADempiere resulting in a completely database-driven solution that, from the extended database, is able to build forms and GUIs. We maximized the use of itemized form elements (dropdown menus, checkboxes, etc.) as GUI's design principle to reduce users' errors or typos and we limited the free text boxes as much as possible applying some basic form validators (e.g. e-mail check). As a direct consequence of the authorization policies, we restricted the user interaction and scope to the specific sections they are supposed to use with dedicated privileges. Each user can access adLIMS both using web GUI and the Java fat client. The web client is the most used interface that does not require any installation on the target user machine nor requires any configuration and all computational requests are processed server-side lightening client-side workload. Moreover adLIMS allows mobile interfaces for smartphones and tablets, features inherited from ADempiere ERP. We implemented data backup by creating custom server side jobs, scheduled as incremental backup with daily dump of the whole database. \n\n\n\n\n\n\n\n\n\nFigure 3: adLIMS workflow. The circular schema represents the adLIMS workflow from sample input to NGS pool generation. In the first step, a user with the SampleManager role fills the \"Cell\" and \"Sample\" forms that contain data about tissue source, lineage, cell type, cell marker, project, subject, DNA metadata. The two subplots related to this step highlight examples of real input data, in which on the back is visible a sample DNA list in the context of the adLIMS window layout whereas on the foreground is opened a sample image of a DNA quantification from an input data, stored as attachment in adLIMS. In the second step a user with the WetManager role processes samples along the LAM-PCR protocol setting-up the required Linear, 1\u00b0 and 2\u00b0 Exponential PCR amplification. Here the two subplots represent a sample window layout on the back and a picture of an agarose gel electrophoresis of LAM-PCR products in foreground. In the third step a subset of PCR samples are pooled together through the NGS Pool Setup and the sample window layout allows visualizing all required fields of the forms. The final step is the NGS sequencing and sample sheet acquisition, in which is possible to upload raw data quality reports such as the per-base sequence quality visible on the windowed subplot.\n\n\n\n\n\n\n\n\n\n Additional file 5. Data elements managed by adLIMS\n\n\nIn this file we reported all data types inserted in adLIMS, following the workflow in Figure 3.\nFormat: DOCX; Size: 23KB Download file\n\n\n\n\nThe LAM-PCR and pre-sequencing activities are directly supported by the adLIMS automated procedures that allow the creation of parallel processes by database triggers, required in case of laboratories with high multiplexing needs (such as PCRs liquid handler with 96 wells\/plate format). For the LAM-PCR workflow (Figure 4A), adLIMS supports the user with dedicated GUI (Figure 4B). \n\r\n\n\n\n\n\n\n\n\n\n\nFigure 4: LAM-PCR workflow and the corresponding adLIMS automated process. The LAM-PCR procedure[10] is reported as diagram (A) in which from a selected sample, different linear amplifications may be processed followed then by a first and a second exponential PCR amplification. The procedure is developed in adLIMS starting from the\r\nexperiment set-up (B) in which a user fills the form with the data relative to the experiment such as \"Name\", number of LAM-PCRs (N-LAM), \"LAM ID\" and \"date\". Then adLIMS creates a number of entries as reported in the field \"N-LAM\", for the subsequent step \"LAM-PCR linear\" (C), that will be then processed and expanded in the \"LAM-PCR 1st exp\" step (D) with details such as \"Name\", \"Plate barcode\" and \"Enzyme\". The last step is the \"LAM-PCR 2nd exp\" (E), corresponding to the second exponential amplification, in which the user completes additional data of the experiment (\"concentration\" and \"quality\").\n\n\n\nOur end-users validated adLIMS by verifying functionalities and GUIs through test cases for PCRs samples and pre-sequencing data. Here we reported two settings related to two distinct sections of the general workflow (Figure 3) with different scopes and actors, thus requiring different user policies and sample activities. The first workflow concerns the interaction of a user with SampleManager role that insert patient sample metadata into adLIMS (Figure 5). The system supports and guides the user to select and input the data with ad hoc forms, for example cell types (Cell Marker CD14, Myeloid in Figure 5A), tissues (peripheral blood, PB, in Figure 5A), DNA purity and concentration, project, sample ID and so on (Figure 5B-C). The SampleManager user profile is authorized to modify only the data relative to a specific project and subject from the menu list through two accessible windows (\"Sample\" and \"Cell Forms\"). In the second scenario, a user with the WetManager role is required to process DNA samples from specific cell populations (Figure 6), performing (1) LAM-PCR amplification (Figure 6A-B), (2) storing the images of the gel-electrophoresis PCR products, (3) preparing a sequencing pool (Figure 6C) and (4) exporting a report in PDF file format (Figure 6D). In all the steps of the process, adLIMS supports the user and enables the parallel sample processing and multiplexing. \n\n\n\n\n\n\n\n\n\nFigure 5: Workflow scenario of a user with SampleManager role An example of an adLIMS' workflow for the SampleManager role in the context of the web interface in ZK. (A) The Cell Form window is filled with some details of the Cell Marker tab, reached from the previous related tabs (Source, Lineage, Cell Type) after user log-in. (B) The Sample window filled with some details of a project in which an example of specie dropdown menu is expanded (data from the related table). (C) The Sample window filled with some details of the tab Sample.\n\n\n\n\n\n\n\n\n\n\n\n\nFigure 6: Workflow scenario of a user with WetManager role An example of an adLIMS' workflow for the WetManager role in the context of the web interface in ZK where a user with the WetManager role can insert\/edit\/delete data about the LAM-PCR process and obtain the report of the selected LAM_PCR experiments (pool) to sequence. (A) The LAM-PCR window filled with some details of a new experiment tab after user log-in. (B) The LAM-PCR window filled with some details of a first exponential data reached from the previous related tabs (Experiment and Linear). (C) The Pool window filled with some details of the tab Pool Details reached from the previous related tabs (Setup, Barcode LTR, Barcode LC). (D) Example of exported report in PDF file format from samples of a pool.\n\n\n\nMoreover adLIMS allows report generation from all input data, both using all data and a selection of filtered data with user defined criteria (Figure 7). adLIMS exploits the data-driven logic to export data reports from tables to a wide range of file formats (PDF, XLS, CSV, etc.) through the JasperReports library.[28] JasperReports is a flexible report library that can be customized for each instance of the system. The default visualization allows reporting data in the same table view. In our daily practice, we use data reports to create sample-sheets of Illumina sequencing pools. \n\n\n\n\n\n\n\n\n\nFigure 7: Data filtering and reporting in adLIMS The function of integrated reporting (via JasperReport) allows the users to configure customized reports from a selection of data fields. In this example, a user starts from the DNA table (A), then clicks on the button \"report\" on the main menu to initiate the report generation, obtaining a new tab (B) with the list of the whole entries of the table. For example, to set a filter of the DNA samples with concentrations between 100 and 250 ng \/ uL, the user clicks on the \"search\" button (icon button with a magnifying glass) that opens a new window. Then the user switches in the advanced tab (C) of the search window and specifies the two needed filters. Once completed the search form, adLIMS filters the dataset and releases the PDF file that the user can save locally.\n\n\n\nadLIMS in our laboratory practice has brought an improvement through simplification and traceability of the data entry in sample workflows and sample reporting with respect to traditional data storage methods. As an example, adLIMS allows to create a sample sheets by selecting samples information in a few clicks whereas without adLIMS this activity may require hours, potentially introducing typing errors.\nadLIMS can be adapted and extended to any laboratory working with biological samples with limited efforts, which are mainly related to the database design and customization, thus defining the business and view logics (workflows and GUI) and creating the data structures (PostgreSQL or Oracle database schema and tables) that are automatically converted into interfaces and data views by the ADempiere ERP engine.\nadLIMS currently does not provide direct interfaces with laboratory instruments but integrates local server connections (storage mount access, network printers, and so on). Future extensions of the system will include API integration of laboratory platforms for sequencing, PCR product quantification and liquid handlers. On the other hand, client side interfaces (such as barcode readers and scanners) can be directly integrated into adLIMS, enhancing sample tracking and data acquisition.\nThe potentialities of adLIMS in the context of genomic companies and facilities are conveyed in the ready-to-use availability of the ERP features that would only require customization. \n\nConclusions \nCurrently, in many laboratories the procedures for data tracking and storage of sample information are based on spreadsheets without a real information management or standardization. This data management resulted in inefficiencies and redundancies, potentially generating many errors (typically typos) hard to backtrack or resolve. The use of a LIMS allows bypassing spreadsheets or local file management, supporting automation of all standard procedures for sample tracking. Exploiting adLIMS, our laboratory sustained the critical issues of sample tracking, data standardization and automation derived by the NGS and robotics revolution. This successful application of process engineering and monitoring allowed our laboratory to increase efficiency and reduce manual errors, thus posing concrete bases to sustain business scalability and potentially to approach pharmaco-vigilance monitoring of gene therapy patients in a highly standardized fashion compliant to regulatory requirements. Moreover adLIMS can natively be extended to incorporate ERP solutions, such as CRM, supply chain management, billing and accounting, integrated features that are critical for many genomics facilities.\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors' contributions \nA.C. conceived of the study, participated in its design and coordination and drafted the manuscript. G.S. created the application (adLIMS) and drafted the manuscript. F.B., ET participated in the requirements analysis. E.T. tested the system. E.M. revised the manuscript and supervised the project. All authors read and approved the final manuscript.\n\nAvailability and requirements \nThe source code, user guide and appliance of adLIMS are freely available at the project homepage http:\/\/sourceforge.net\/projects\/adlims. We also provide a live demo for users who want to evaluate adLIMS without installation. Release notes and other information will be also updated on the project homepage.\nProject name: adLIMS\nProject homepage: http:\/\/sourceforge.net\/projects\/adlims\nAuthor: Giulio Spinozzi\nOperating system: Platform independent\nSupported web browsers: Chrome, Firefox 3.5 +, Safari 4+\nProgramming language: Java EE\nDatabase: PostgreSQL\/Oracle\nLicense: GNU GPLv2\nAny restrictions to use by non-academics: None\n\nAcknowledgements \nThis work was supported by Telethon Grant D1 to EM. We would like to thank Giorgio Cafasso for his support with ADempiere ERP and for his gentle availability. We thank Luca Biasco for suggestions in LAM-PCR workflow set-up.\n\nDeclarations \nPublication of this article has been funded by Telethon.\nThis article has been published as part of BMC Bioinformatics Volume 16 Supplement 9, 2015: Proceedings of the Italian Society of Bioinformatics (BITS): Annual Meeting 2014: Bioinformatics. The full contents of the supplement are available online at http:\/\/www.biomedcentral.com\/bmcbioinformatics\/supplements\/16\/S9.\n\nReferences \n\n\n\u2191 Ranzani, M.; Annunziato, S.; Calabria, A.; Brasca, S.; Benedicenti, F.; Gallina, P.; Naldini, L.; Montini, E. (2014). \"Lentiviral vector-based insertional mutagenesis identifies genes involved in the resistance to targeted anticancer therapies\". Molecular Therapy 22 (12): 2056-2068. doi:10.1038\/mt.2014.174. PMC PMC4429698. PMID 25195596. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4429698 .   \n\n\u2191 Aiuti, A.; Biasco, L.; Scaramuzza, S.; Ferrua, F.; Cicalese, M.P.; Baricordi, C. et al. (2013). \"Lentiviral hematopoietic stem cell gene therapy in patients with Wiskott-Aldrich syndrome\". Science 341 (6148): 1233151. doi:10.1126\/science.1233151. PMC PMC4375961. PMID 23845947. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4375961 .   \n\n\u2191 Biffi, A.; Montini, E.; Lorioli, L.; Cesani, M.; Fumagalli, F.; Plati, T. et al. (2013). \"Lentiviral hematopoietic stem cell gene therapy benefits metachromatic leukodystrophy\". Science 341 (6148): 1233158. doi:10.1126\/science.1233158. PMID 23845948.   \n\n\u2191 Aiuti, A.; Cassani, B.; Andolfi, G.; Mirolo, M.; Biasco, L.; Recchia, A. et al. (2007). \"Multilineage hematopoietic reconstitution without clonal selection in ADA-SCID patients treated with stem cell gene therapy\". Journal of Clinical Investigation 117 (8): 2233-40. doi:10.1172\/JCI31666. PMC PMC1934603. PMID 17671653. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934603 .   \n\n\u2191 Braun, C.J.; Boztug, K.; Paruzynski, A.; Witzel, M.; Schwarzer, A.; Rothe, M. et al. (2014). \"Gene therapy for Wiskott-Aldrich syndrome: Long-term efficacy and genotoxicity\". Science Translational Medicine 6 (227): 227\u201333. doi:10.1126\/scitranslmed.3007280. PMID 24622513.   \n\n\u2191 Deichmann, A.; Hacein-Bey-Abina, S.; Schmidt, M.; Garrigue, A.; Brugman, M.H.; Hu, J.; Glimm, H.; Gyapay, G.; Prum, B.; Fraser, C.C. et al. (2007). \"Vector integration is nonrandom and clustered and influences the fate of lymphopoiesis in SCID-X1 gene therapy\". Journal of Clinical Investigation 117 (8): 2225\u20132232. doi:10.1172\/JCI31659. PMC PMC1934585. PMID 17671652. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934585 .   \n\n\u2191 Ott, M.G; Schmidt, M.; Schwarzwaelder, K.; Stein, S.; Siler, U.; Koehl, U. et al. (2006). \"Correction of X-linked chronic granulomatous disease by gene therapy, augmented by insertional activation of MDS1-EVI1, PRDM16 or SETBP1\". Nature Medicine 12 (4): 401-409. doi:10.1038\/nm1393. PMID 16582916.   \n\n\u2191 Schwarzwaelder, K.; Howe, S.J.; Schmidt, M.; Brugman, M.H.; Deichmann, A.; Glimm, H. et al. (2007). \"Gammaretrovirus-mediated correction of SCID-X1 is associated with skewed vector integration site distribution in vivo\". Journal of Clinical Investigation 117 (8): 2241-9. doi:10.1172\/JCI31661. PMC PMC1934556. PMID 17671654. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934556 .   \n\n\u2191 Stein, S.; Ott, M.G.; Schultze-Strasser, S.; Jauch, A.; Burwinkel, B.; Kinner, A. et al. (2010). \"Genomic instability and myelodysplasia with monosomy 7 consequent to EVI1 activation after gene therapy for chronic granulomatous disease\". Nature Medicine 16 (2): 198-204. doi:10.1038\/nm.2088. PMID 20098431.   \n\n\u2191 10.0 10.1 10.2 Schmidt, M.; Schwarzwaelder, K.; Bartholomae, C.; Zaoui, K.; Ball, C.; Pilz, I. (2007). \"High-resolution insertion-site analysis by linear amplification-mediated PCR (LAM-PCR)\". Nature Methods 4 (12): 1051-1057. doi:10.1038\/nmeth1103. PMID 18049469.   \n\n\u2191 Calabria, A.; Leo, S.; Benedicenti, F.; Cesana, D.; Spinozzi, G.; Orsini, M. et al. (2014). \"VISPA: A computational pipeline for the identification and analysis of genomic vector integration sites\". Genome Medicine 6 (9): 67. doi:10.1186\/s13073-014-0067-5. PMC PMC4169225. PMID 25342980. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169225 .   \n\n\u2191 Prasad, P.J.; Bodhe, G.L. (2012). \"Trends in laboratory information management system\". Chemometrics and Intelligent Laboratory Systems 118: 187\u2013192. doi:10.1016\/j.chemolab.2012.07.001.   \n\n\u2191 Soh, C.; Kien, S.S.; Tay-Yap, J. (April 2000). \"Enterprise resource planning: Cultural fits and misfits: Is ERP a universal solution?\". Communications of the ACM 43 (4): 47-51. doi:10.1145\/332051.332070.   \n\n\u2191 Aoyama, Mikio (1997). \"Managing the concurrent development of large-scale software systems\". International Journal of Technology Management 14 (6\u20138): 739\u2013765. doi:10.1504\/IJTM.1997.002580.   \n\n\u2191 Department of Defense Systems Management College (January 2001) (PDF). Systems Engineering Fundamentals. Fort Belvoir, VA: Defense Acquisition University Press. pp. 216. http:\/\/ocw.mit.edu\/courses\/aeronautics-and-astronautics\/16-885j-aircraft-systems-engineering-fall-2005\/readings\/sefguide_01_01.pdf .   \n\n\u2191 Booch, Grady; Rumbaugh, James; Jacobson, Ivar (1996). The Unified Modeling Language for Object-Oriented Development. Santa Clara, CA: Rational Software Corporation.   \n\n\u2191 17.0 17.1 Krasner, Glenn E.; Pope, Stephen T. (1988) (PDF). A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System. Mountain View, CA: ParcPlace Systems, Inc. pp. 34. http:\/\/www.create.ucsb.edu\/~stp\/PostScript\/mvc.pdf .   \n\n\u2191 \"Clarity LIMS\". GenoLogics Life Sciences Software Inc. http:\/\/www.genologics.com\/clarity-lims\/ .   \n\n\u2191 \"MiniLIMS\". BioTeam Inc. http:\/\/bioteam.net\/minilims\/ .   \n\n\u2191 \"Bika: Open Source laboratory information management systems\". Bika Lab Systems (Pty) Ltd. http:\/\/www.bikalabs.com\/ .   \n\n\u2191 \"LabKey Software\". LabKey Corporation. https:\/\/labkey.com\/ .   \n\n\u2191 Scholtalbers, J.; R\u00f6ssler, J.; Sorn, P.; de Graaf, J.; Boisgu\u00e9rin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". Bioinformatics 29 (9): 1233-1234. doi:10.1093\/bioinformatics\/btt115. PMID 23479349.   \n\n\u2191 \"Plone CMS\". Plone Foundation. https:\/\/plone.org\/ .   \n\n\u2191 \"Drupal\". Drupal Association. https:\/\/drupal.org\/ .   \n\n\u2191 \"ADempiere ERP\". ADempiere Community. http:\/\/www.adempiere.com\/ADempiere_ERP .   \n\n\u2191 Wirfs-Brock, Rebecca; Wilkerson, Brian (1989). \"Object-oriented design: A responsibility-driven approach\". OOPSLA '89 Conference Proceedings on Object-oriented Programming Systems, Languages and Applications 24 (10): 71\u201375. doi:10.1145\/74878.74885.   \n\n\u2191 \"ZK\". Potix Corporation. http:\/\/www.zkoss.org\/ .   \n\n\u2191 \"Jaspersoft\". TIBCO Software, Inc. http:\/\/www.jaspersoft.com\/ .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\">https:\/\/www.limswiki.org\/index.php\/Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:50.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,894 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","b55d160b7541498bbeb5c39c2bdb347d_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_AdLIMS_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:adLIMS: A customized open source software that allows bridging clinical and basic molecular research studies<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\"><p><span><\/span>\n<\/p>\n\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Many biological laboratories that deal with genomic samples are facing the problem of sample tracking, both for pure <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory\" title=\"Laboratory\" target=\"_blank\" class=\"wiki-link\" data-key=\"c57fc5aac9e4abf31dccae81df664c33\">laboratory<\/a> management and for efficiency. Our laboratory exploits PCR techniques and Next Generation Sequencing (NGS) methods to perform high-throughput integration site monitoring in different clinical trials and scientific projects. Because of the huge amount of samples that we process every year, which result in hundreds of millions of sequencing reads, we need to standardize data management and tracking systems, building up a scalable and flexible structure with web-based interfaces, which are usually called <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_information_management_system\" title=\"Laboratory information management system\" target=\"_blank\" class=\"wiki-link\" data-key=\"8ff56a51d34c9b1806fcebdcde634d00\">Laboratory Information Management System<\/a> (LIMS).\n<\/p><p><b>Methods:<\/b> We started collecting end-users' requirements, composed of desired functionalities of the system and Graphical User Interfaces (GUI), and then we evaluated available tools that could address our requirements, spanning from pure LIMS to Content Management Systems (CMS) up to enterprise information systems. Our analysis identified <a href=\"https:\/\/www.limswiki.org\/index.php\/ADempiere\" title=\"ADempiere\" target=\"_blank\" class=\"wiki-link\" data-key=\"034c7fd48264a5270d6e2f75b1e9519b\">ADempiere<\/a> ERP, an open source Enterprise Resource Planning written in Java J2EE, as the best software that also natively implements some highly desirable technological advances, such as the high usability and modularity that grants high use-case flexibility and software scalability for custom solutions.\n<\/p><p><b>Results:<\/b> We extended and customized ADempiere ERP to fulfil LIMS requirements and we developed adLIMS. It has been validated by our end-users verifying functionalities and GUIs through test cases for PCRs samples and pre-sequencing data and it is currently in use in our laboratories. adLIMS implements authorization and authentication policies, allowing multiple users management and roles definition that enables specific permissions, operations and data views to each user. For example, adLIMS allows creating sample sheets from stored data using available exporting operations. This simplicity and process standardization may avoid manual errors and <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> backtracking, features that are not granted using track recording on files or spreadsheets.\n<\/p><p><b>Conclusions:<\/b> adLIMS aims to combine sample tracking and data reporting features with higher accessibility and usability of GUIs, thus allowing time to be saved on doing repetitive laboratory tasks, and reducing errors with respect to manual data collection methods. Moreover, adLIMS implements automated data entry, exploiting sample data multiplexing and parallel\/transactional processing. adLIMS is natively extensible to cope with <a href=\"https:\/\/www.limswiki.org\/index.php\/Laboratory_automation\" title=\"Laboratory automation\" target=\"_blank\" class=\"wiki-link\" data-key=\"0061880849aeaca05f8aa27ae171f331\">laboratory automation<\/a> through platform-dependent API interfaces, and could be extended to genomic facilities due to the ERP functionalities.\n<\/p><p><b>Keywords:<\/b> LIMS; Open Source Software; Information Systems; ADempiere ERP; Sample Tracking\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>In many biological laboratories, sample tracking is an outstanding issue and often represents a bottleneck for the correct handling and interpretation of experimental data. This issue is becoming particularly critical when automation and high-throughput technologies are introduced in the laboratory practice. Our laboratory performs high-throughput characterization of vector-genomic integration sites in the context of gene therapy applications based on the delivery of therapeutic genes by viral vectors that stably integrate into the genome of targeted cells, as well as gene therapy preclinical models and insertional mutagenesis research projects.<sup id=\"rdp-ebb-cite_ref-RanzaniLenti14_1-0\" class=\"reference\"><a href=\"#cite_note-RanzaniLenti14-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AiutiLenti13_2-0\" class=\"reference\"><a href=\"#cite_note-AiutiLenti13-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BiffiLenti13_3-0\" class=\"reference\"><a href=\"#cite_note-BiffiLenti13-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AiutiMulti07_4-0\" class=\"reference\"><a href=\"#cite_note-AiutiMulti07-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-BraunGene14_5-0\" class=\"reference\"><a href=\"#cite_note-BraunGene14-5\" rel=\"external_link\">[5]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-DeichmannVector07_6-0\" class=\"reference\"><a href=\"#cite_note-DeichmannVector07-6\" rel=\"external_link\">[6]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-OttCorrect06_7-0\" class=\"reference\"><a href=\"#cite_note-OttCorrect06-7\" rel=\"external_link\">[7]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SchwarzwaelderGamma07_8-0\" class=\"reference\"><a href=\"#cite_note-SchwarzwaelderGamma07-8\" rel=\"external_link\">[8]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SteinGeno10_9-0\" class=\"reference\"><a href=\"#cite_note-SteinGeno10-9\" rel=\"external_link\">[9]<\/a><\/sup> Vector integration sites are retrieved and mapped in the genome through a combination of Polymerase Chain Reaction (PCR)-based techniques<sup id=\"rdp-ebb-cite_ref-SchmidtHigh07_10-0\" class=\"reference\"><a href=\"#cite_note-SchmidtHigh07-10\" rel=\"external_link\">[10]<\/a><\/sup>, next generation sequencing (NGS) and <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> analyses.<sup id=\"rdp-ebb-cite_ref-CalabriaVISPA14_11-0\" class=\"reference\"><a href=\"#cite_note-CalabriaVISPA14-11\" rel=\"external_link\">[11]<\/a><\/sup> We process and analyze around 2000 samples\/year resulting in hundreds of millions of sequencing reads. Despite the fact that adopting robotic automation for sample manipulation in our laboratory has provided many advantages in terms of manual error-reduction and data production scalability, drawbacks related to sample information volume and tracking are still present. These reasons prompted us to develop a Laboratory Information Management System (LIMS)<sup id=\"rdp-ebb-cite_ref-PrasadTrends12_12-0\" class=\"reference\"><a href=\"#cite_note-PrasadTrends12-12\" rel=\"external_link\">[12]<\/a><\/sup> for sample tracking on a scalable and flexible infrastructure with an easily accessible and web-based interface. LIMS is a type of information system implemented as a software utility specifically designed to improve the data acquisition and sample monitoring along laboratory workflows, and supporting sample reporting. An information system is a combination of information technologies developed to grant business processes efficiency and monitoring. Extended IS are the Enterprise Resource Planning (ERP) solutions<sup id=\"rdp-ebb-cite_ref-SohEnt00_13-0\" class=\"reference\"><a href=\"#cite_note-SohEnt00-13\" rel=\"external_link\">[13]<\/a><\/sup> that integrate the standard information system features with accounting and administrative operations for performance monitoring through dashboards and data mining tools.\n<\/p><p>In this work we describe our LIMS, developed on an existing open source ERP framework that natively implements all technological functionalities, as software customization and parameterization. After a brief introduction of the ERP framework with the motivation of the specific choice, we will describe in details our implementation with custom use cases and scenarios derived from our laboratory requirements and experience. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Methods\">Methods<\/span><\/h2>\n<p>We followed a typical software engineering approach, \"waterfall\"<sup id=\"rdp-ebb-cite_ref-AoyamaManaging97_14-0\" class=\"reference\"><a href=\"#cite_note-AoyamaManaging97-14\" rel=\"external_link\">[14]<\/a><\/sup>, to design our solution. We first collected end-users' requirements, composed of desired functionalities of the systems and graphical user interfaces (GUI). Then we evaluated available tools that could address our requirements, spanning from pure LIMS to Content Management Systems (CMS) up to enterprise information systems. In the next step we selected our candidate solution and we designed its configuration to best fulfil our requirements. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Requirements_Analysis\">Requirements Analysis<\/span><\/h3>\n<p>The analysis of requirements, also called requirements engineering, is the process of acquiring software expectations from users\/clients in terms of functionalities and interaction that are translated in software requirements.<sup id=\"rdp-ebb-cite_ref-DoDSys01_15-0\" class=\"reference\"><a href=\"#cite_note-DoDSys01-15\" rel=\"external_link\">[15]<\/a><\/sup>\n<\/p><p>The first step is to define which types of users will access the LIMS (roles and groups) and which authorization policies we have to apply to specific users (permissions on data and operations). Using a Unified Modeling Language (UML)<sup id=\"rdp-ebb-cite_ref-BoochTheUni96_16-0\" class=\"reference\"><a href=\"#cite_note-BoochTheUni96-16\" rel=\"external_link\">[16]<\/a><\/sup> use case diagram (Figure 1), we collected desired roles (called \"actors\" in UML) and operations (called \"actions\" in UML). The two principal roles of the system are distinguished for the area in which they operate. In this instance we identified two main areas: a sample management area where samples are collected from patients or subjects and a wet experimental area in which collected samples are processed to retrieve vector-cellular genomic junctions. For this reason we identified two corresponding roles, a SampleManager and a WetManager, and their permissions and operations scope correspond to the data that they can access: the SampleManager role is related to users that input and manage sample metadata (project, DNA and so on), while the WetManager role is related to users that input and manage experimental procedures and workflows on the DNA samples (PCR steps, NGS pooling, and so on). Then we collected the desired authorizations associated to each role as reported in the CRUD (Create, Read, Update, Delete) policies (Table 1). \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"f2c96810e6f78a993e12e558b7b73662\"><img alt=\"Fig1 Calabria BMCBioinformatics2015 16-Suppl9.jpg\" src=\"https:\/\/www.limswiki.org\/images\/c\/c4\/Fig1_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> <blockquote><b>Figure 1: adLIMS' use case: The UML Use Case Diagram of our case study.<\/b> In our case we have two actors: SampleManager and WetManager. A user with SampleManager role can insert, edit, manage, search and filter only clinical data, and deals with data about DNA, project and patient. A user with WetManager role can view clinical data, and it can insert, edit, manage, search and filter data about LAM-PCR and NGS pool.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"6\"><b>Table 1.<\/b> User's Rules in \"Create, Read, Update and Delete\" format (CRUD). The table is a summary of the role permissions and policies in adLIMS in terms of data access. We use the CRUD syntax: create (C), read (R), update (U), delete (D).\n<\/td><\/tr>\n<tr>\n<th>\n<\/th>\n<th> Clinic data\n<\/th>\n<th> Sequencing data\n<\/th>\n<th> Projects\n<\/th>\n<th> LIMS managing\n<\/th>\n<th> Users managing\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">WetManager\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">R\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CRUD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CRUD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">SampleManager\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CRUD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CRUD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Administrator\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">--\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CRUD\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">CRUD\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The SampleManager role can access and edit metadata related to all clinical and preclinical aspects such as anonymized patients' IDs, DNA (concentrations, volume, and so on) and cell types (markers, lineage, and so on). The visibility spectrum of a user with the SampleManager role is restricted to sample metadata only and will not be able to read other information of post sample processing.\n<\/p><p>The WetManager role is able to control all aspects related to experimental procedures and workflows on the samples. A user assigned to the WetManager role will not be able to modify input metadata from the SampleManager role.\n<\/p><p>We added an administrator role (or SuperUser) that is able to control and edit every component of the system, customizing all the functional and graphical levels related to the model-view-controller paradigm.<sup id=\"rdp-ebb-cite_ref-KrasnerADesc88_17-0\" class=\"reference\"><a href=\"#cite_note-KrasnerADesc88-17\" rel=\"external_link\">[17]<\/a><\/sup> However this role is not authorized to modify input data. At the functional level, the administrator has the privileges to: (1) create, delete, and suspend accounts and (2) define rules to access to the database (such as tables, views, fields, etc.). At the graphical level, the administrator can create and edit windows and layouts, control how to graphically visualize and access data and define how to export and import data in the information system.\n<\/p><p>From this analysis of requirements we derived the interaction workflow (Figure 2). In the first step of our workflow, a user with a SampleManager role inputs data related to the specific project and the samples that need to be processed for integration site analysis, in particular information relative to DNA. These samples will be then processed (upon a request of analysis from the SampleManager) in the custom workflow by a user with a WetManager role for linear amplification mediated (LAM)-PCR.<sup id=\"rdp-ebb-cite_ref-SchmidtHigh07_10-1\" class=\"reference\"><a href=\"#cite_note-SchmidtHigh07-10\" rel=\"external_link\">[10]<\/a><\/sup> The procedure of the LAM-PCR protocol has to be implemented in a LIMS process that allows tracking each sample manipulation. LAM-PCR products are then selected and pooled for deep sequencing. All the information of the pooled products are exported and formatted as required by Illumina MiSeq sample sheet. Therefore the information system allows monitoring and tracking sample metadata, LAM-PCR processing and the pool preparation for NGS supporting DNA barcodes association without directly handling NGS files. Additional file 1 summarizes the laboratory workflow and system interaction based on the above-mentioned requirements. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"b8cb6e3be41014947ed83a2eb46ef4b9\"><img alt=\"Fig2 Calabria BMCBioinformatics2015 16-Suppl9.jpg\" src=\"https:\/\/www.limswiki.org\/images\/d\/d4\/Fig2_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 2: Sample case study of adLIMS A user with SampleManager role inserts data of the sample harvested from a patient (in this scenario) into adLIMS and then activate a request for wet processing (arrow number 3).<\/b> A user with WetManager role processes the sample first adding other sample related information like data quality or concentration, then sets-up a LAM-PCR experiment (arrow number 5), prepares a pool for sequencing (arrow number 6), and finally exports a sample sheet. The arrow colors represent different conceptual steps of the workflow.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 1. <i>General data workflow of vector integration sites identification at TIGET Vector Integration Core<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"> The standard flow of a vector integration site project in the Vector Integration Core at the Telethon Institute for Gene Therapy in Milan\n<p>Format: DOCX; Size: 153KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-16-s9-s5-s1.docx\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As additional requirements, the system has to be web-based and supported by reliable technologies with a backup system thus enabling data maintenance and recovery in case of failures (e.g. electrical supply problems). The centralized nature of the system requires standard hardware performances such as the simultaneous interaction with dozens of users keeping response times faster than 3 seconds. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Evaluation_of_available_LIMS_and_alternative_solutions\">Evaluation of available LIMS and alternative solutions<\/span><\/h3>\n<p>We evaluated available LIMS solutions, from commercial to open source ones. Commercial or stand-alone LIMSs<sup id=\"rdp-ebb-cite_ref-Clarity_18-0\" class=\"reference\"><a href=\"#cite_note-Clarity-18\" rel=\"external_link\">[18]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MiniLIMS_19-0\" class=\"reference\"><a href=\"#cite_note-MiniLIMS-19\" rel=\"external_link\">[19]<\/a><\/sup> are often very expensive and\/or lack the flexibility and scalability needed to manage different types of sample data, procedures and analyses specifically designed for each research project. We also explored open source software designed for biological laboratories, such as Bika LIMS<sup id=\"rdp-ebb-cite_ref-Bika_20-0\" class=\"reference\"><a href=\"#cite_note-Bika-20\" rel=\"external_link\">[20]<\/a><\/sup>, LabKey<sup id=\"rdp-ebb-cite_ref-LabKey_21-0\" class=\"reference\"><a href=\"#cite_note-LabKey-21\" rel=\"external_link\">[21]<\/a><\/sup> or Galaxy<sup id=\"rdp-ebb-cite_ref-ScholtalbersGal13_22-0\" class=\"reference\"><a href=\"#cite_note-ScholtalbersGal13-22\" rel=\"external_link\">[22]<\/a><\/sup>, and content management systems (CMS) like Plone<sup id=\"rdp-ebb-cite_ref-Plone_23-0\" class=\"reference\"><a href=\"#cite_note-Plone-23\" rel=\"external_link\">[23]<\/a><\/sup> and Drupal<sup id=\"rdp-ebb-cite_ref-Drupal_24-0\" class=\"reference\"><a href=\"#cite_note-Drupal-24\" rel=\"external_link\">[24]<\/a><\/sup> under the perspective of customizing them and exploiting built-in functionalities such as user management, workflow management and configuration. Unfortunately, none of them fully satisfied our requirements because most of the features, that are the peculiarity of a LIMS (like export, report, etc.), have not been implemented. In this context, we analyzed Enterprise Resource Planning (ERP) solutions and ADempiere ERP<sup id=\"rdp-ebb-cite_ref-ADempiere_25-0\" class=\"reference\"><a href=\"#cite_note-ADempiere-25\" rel=\"external_link\">[25]<\/a><\/sup> was the best available software. ADempiere ERP has been developed under GPL license in Java J2EE with Model-View-Controller design pattern<sup id=\"rdp-ebb-cite_ref-KrasnerADesc88_17-1\" class=\"reference\"><a href=\"#cite_note-KrasnerADesc88-17\" rel=\"external_link\">[17]<\/a><\/sup> and database-driven logic.<sup id=\"rdp-ebb-cite_ref-Wirfs-BrockObject89_26-0\" class=\"reference\"><a href=\"#cite_note-Wirfs-BrockObject89-26\" rel=\"external_link\">[26]<\/a><\/sup> ADempiere ERP implements all required features and presents some highly desirable technological advances (see Additional file 2 for the list of desirable features with a comparative analysis among Bika LIMS, LabKey and ADempiere), such as high usability (web and mobile interfaces) and modularity that grants high use-case flexibility (plug and play approach) and software scalability for custom solutions (adaptable to all use cases). The application server is JBoss and supported databases are Oracle and PostgreSQL. Web interfaces exploit the latest technologies with ZK.<sup id=\"rdp-ebb-cite_ref-ZK_27-0\" class=\"reference\"><a href=\"#cite_note-ZK-27\" rel=\"external_link\">[27]<\/a><\/sup> Moreover, it natively supports multiple languages, accounting procedures and dashboards for process monitoring thanks to the ERP functionalities that can be easily adapted to high-standard industrial and commercial contexts. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 2. <i>Features' comparison between Bika, LabKey and ADempiere<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Comparison among the features available in Bika, LabKey and ADempiere\n<p>Format: DOCX; Size: 26KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-16-s9-s5-s2.docx\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h3><span class=\"mw-headline\" id=\"Design\">Design<\/span><\/h3>\n<p>The last process in the \"waterfall\" approach is the system design. In order to translate the requirements into both functionalities and GUI within ADempiere ERP, we developed an extension of the core database by applying the required operations and user's policies resulting in a scalable system that natively supports Java fat client and web interfaces. For each end-user interaction and functionality, we designed a custom view of the workflow (Figure 2) with dedicated GUIs based on use cases (Figure 1). Since ADempiere ERP is database-driven, the design of the database related to the LIMS extension is a key aspect that drives and manages both the workflow instance and the GUIs. Our database extension is compliant with the core-system table design that required the addition of ten pre-defined fields (see Additional file 3). This operation is required because ADempiere ERP leverages on inner tables to directly create forms (windows) and GUIs. The \"application dictionary\" is one of the most powerful aspects of ADempiere ERP that acts as the engine of the database-driven model. All metadata needed to build data forms, windows and GUI are contained in the application dictionary that operates at the application layer and generates windows, tabs, menu, forms, nested elements connection, and so on. The application dictionary allows dynamic and flexible changes in GUI and data forms by changing its table content without requiring programming development that is thus drastically reduced. As a direct consequence, GUI changes or customizations can be configured directly in the application dictionary without requiring software compiling or re-building. For example, to create a new window with proper title, menu bar, tool bar, and status bar, ADempiere ERP automatically adds elements in the application dictionary and generates all required fields starting from the database table. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 3. <i>Customization set-up<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The report file of all the customization steps required for creating an instance of adLIMS, with an example of table creation and trigger definition\n<p>Format: DOCX; Size: 28KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-16-s9-s5-s3.docx\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Based on the general workflow modeled in the analysis of requirements (Figure 2), we designed and implemented the LIMS database as an extension of the core ADempiere ERP database (Additional file 4), here reported as Entity-Relationship model in which each basic entity is associated to a custom table (\"project\", \"subject\", \"DNA\", \"vector\", \"sample\", and so on). We implemented the model in PostgreSQL and we used BLOB (Binary Large Object) to manage external files as attachments (such as images, pdf files, and so on) that users can upload into any entry. We then created custom GUIs related to the previously described workflow for the management of all data tables (Figure 2). Each window and the relative data are accessed by users according to their role (SampleManager, WetManager and administrator role) and authorization policies. Is always possible to modify existing roles and names and to add new roles according to new requirements or specifications. In our laboratory practice we routinely collect LAM-PCR data, gel images and sequencing quality reports and we store these data in adLIMS. The LAM-PCR workflow has been automated by implementing dedicated tables in the system (\"experiment\", \"lam_pcr_linear\", \"lam_pcr_1st_exp\", \"lam_pcr_2nd_exp\") with corresponding GUI input forms. Custom database triggers support the multiplexing of samples during the experimental operations required in different steps of the process (see Additional file 3 for trigger details). Similar automated procedures have been developed to support the generation of sequencing pools for high-throughput NGS platforms by defining custom tables (\"fusion\", \"pool\", \"pool_details\") and associated GUI. To avoid hard drive bottlenecks, raw NGS files are included in the system as links (absolute path with host server) and not as attachments. \n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 4. <i>adLIMS entity-relationship database model adLIMS<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Database schema represented as Entity-Relationship model\n<p>Format: DOCX; Size: 716KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-16-s9-s5-s4.docx\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<p>We developed adLIMS as an open source customization of ADempiere ERP, which implements the general workflow and requirements previously described (Figure 3 and Additional file 5 for details of roles and GUIs). adLIMS exploits all the core functionalities of ADempiere resulting in a completely database-driven solution that, from the extended database, is able to build forms and GUIs. We maximized the use of itemized form elements (dropdown menus, checkboxes, etc.) as GUI's design principle to reduce users' errors or typos and we limited the free text boxes as much as possible applying some basic form validators (e.g. e-mail check). As a direct consequence of the authorization policies, we restricted the user interaction and scope to the specific sections they are supposed to use with dedicated privileges. Each user can access adLIMS both using web GUI and the Java fat client. The web client is the most used interface that does not require any installation on the target user machine nor requires any configuration and all computational requests are processed server-side lightening client-side workload. Moreover adLIMS allows mobile interfaces for smartphones and tablets, features inherited from ADempiere ERP. We implemented data backup by creating custom server side jobs, scheduled as incremental backup with daily dump of the whole database. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"129acbd3a95855de7dc7dadb0a5e79a3\"><img alt=\"Fig3 Calabria BMCBioinformatics2015 16-Suppl9.jpg\" src=\"https:\/\/www.limswiki.org\/images\/8\/88\/Fig3_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 3: adLIMS workflow<\/b>. The circular schema represents the adLIMS workflow from sample input to NGS pool generation. In the first step, a user with the SampleManager role fills the \"Cell\" and \"Sample\" forms that contain data about tissue source, lineage, cell type, cell marker, project, subject, DNA metadata. The two subplots related to this step highlight examples of real input data, in which on the back is visible a sample DNA list in the context of the adLIMS window layout whereas on the foreground is opened a sample image of a DNA quantification from an input data, stored as attachment in adLIMS. In the second step a user with the WetManager role processes samples along the LAM-PCR protocol setting-up the required Linear, 1\u00b0 and 2\u00b0 Exponential PCR amplification. Here the two subplots represent a sample window layout on the back and a picture of an agarose gel electrophoresis of LAM-PCR products in foreground. In the third step a subset of PCR samples are pooled together through the NGS Pool Setup and the sample window layout allows visualizing all required fields of the forms. The final step is the NGS sequencing and sample sheet acquisition, in which is possible to upload raw data quality reports such as the per-base sequence quality visible on the windowed subplot.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 5. <i>Data elements managed by adLIMS<\/i>\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">In this file we reported all data types inserted in adLIMS, following the workflow in Figure 3.\n<p>Format: DOCX; Size: 23KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-16-s9-s5-s5.docx\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>The LAM-PCR and pre-sequencing activities are directly supported by the adLIMS automated procedures that allow the creation of parallel processes by database triggers, required in case of laboratories with high multiplexing needs (such as PCRs liquid handler with 96 wells\/plate format). For the LAM-PCR workflow (Figure 4A), adLIMS supports the user with dedicated GUI (Figure 4B). \n<\/p><p><br \/>\n<a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"7c56096e817dfcfb9e4c40b66eb0eb06\"><img alt=\"Fig4 Calabria BMCBioinformatics2015 16-Suppl9.jpg\" src=\"https:\/\/www.limswiki.org\/images\/3\/37\/Fig4_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 4: LAM-PCR workflow and the corresponding adLIMS automated process<\/b>. The LAM-PCR procedure<sup id=\"rdp-ebb-cite_ref-SchmidtHigh07_10-2\" class=\"reference\"><a href=\"#cite_note-SchmidtHigh07-10\" rel=\"external_link\">[10]<\/a><\/sup> is reported as diagram (A) in which from a selected sample, different linear amplifications may be processed followed then by a first and a second exponential PCR amplification. The procedure is developed in adLIMS starting from the<br \/>experiment set-up (B) in which a user fills the form with the data relative to the experiment such as \"Name\", number of LAM-PCRs (N-LAM), \"LAM ID\" and \"date\". Then adLIMS creates a number of entries as reported in the field \"N-LAM\", for the subsequent step \"LAM-PCR linear\" (C), that will be then processed and expanded in the \"LAM-PCR 1st exp\" step (D) with details such as \"Name\", \"Plate barcode\" and \"Enzyme\". The last step is the \"LAM-PCR 2nd exp\" (E), corresponding to the second exponential amplification, in which the user completes additional data of the experiment (\"concentration\" and \"quality\").<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Our end-users validated adLIMS by verifying functionalities and GUIs through test cases for PCRs samples and pre-sequencing data. Here we reported two settings related to two distinct sections of the general workflow (Figure 3) with different scopes and actors, thus requiring different user policies and sample activities. The first workflow concerns the interaction of a user with SampleManager role that insert patient sample metadata into adLIMS (Figure 5). The system supports and guides the user to select and input the data with ad hoc forms, for example cell types (Cell Marker CD14, Myeloid in Figure 5A), tissues (peripheral blood, PB, in Figure 5A), DNA purity and concentration, project, sample ID and so on (Figure 5B-C). The SampleManager user profile is authorized to modify only the data relative to a specific project and subject from the menu list through two accessible windows (\"Sample\" and \"Cell Forms\"). In the second scenario, a user with the WetManager role is required to process DNA samples from specific cell populations (Figure 6), performing (1) LAM-PCR amplification (Figure 6A-B), (2) storing the images of the gel-electrophoresis PCR products, (3) preparing a sequencing pool (Figure 6C) and (4) exporting a report in PDF file format (Figure 6D). In all the steps of the process, adLIMS supports the user and enables the parallel sample processing and multiplexing. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"348bb3ae1b186993f7cb52ee6c069558\"><img alt=\"Fig5 Calabria BMCBioinformatics2015 16-Suppl9.jpg\" src=\"https:\/\/www.limswiki.org\/images\/5\/54\/Fig5_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 5: Workflow scenario of a user with SampleManager role An example of an adLIMS' workflow for the SampleManager role in the context of the web interface in ZK<\/b>. (A) The Cell Form window is filled with some details of the Cell Marker tab, reached from the previous related tabs (Source, Lineage, Cell Type) after user log-in. (B) The Sample window filled with some details of a project in which an example of specie dropdown menu is expanded (data from the related table). (C) The Sample window filled with some details of the tab Sample.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"8e73db3852c0c2ea17208d323c161e93\"><img alt=\"Fig6 Calabria BMCBioinformatics2015 16-Suppl9.jpg\" src=\"https:\/\/www.limswiki.org\/images\/0\/00\/Fig6_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 6: Workflow scenario of a user with WetManager role An example of an adLIMS' workflow for the WetManager role in the context of the web interface in ZK where a user with the WetManager role can insert\/edit\/delete data about the LAM-PCR process and obtain the report of the selected LAM_PCR experiments (pool) to sequence<\/b>. (A) The LAM-PCR window filled with some details of a new experiment tab after user log-in. (B) The LAM-PCR window filled with some details of a first exponential data reached from the previous related tabs (Experiment and Linear). (C) The Pool window filled with some details of the tab Pool Details reached from the previous related tabs (Setup, Barcode LTR, Barcode LC). (D) Example of exported report in PDF file format from samples of a pool.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Moreover adLIMS allows report generation from all input data, both using all data and a selection of filtered data with user defined criteria (Figure 7). adLIMS exploits the data-driven logic to export data reports from tables to a wide range of file formats (PDF, XLS, CSV, etc.) through the JasperReports library.<sup id=\"rdp-ebb-cite_ref-JasperReports_28-0\" class=\"reference\"><a href=\"#cite_note-JasperReports-28\" rel=\"external_link\">[28]<\/a><\/sup> JasperReports is a flexible report library that can be customized for each instance of the system. The default visualization allows reporting data in the same table view. In our daily practice, we use data reports to create sample-sheets of Illumina sequencing pools. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"705647e12e1e5850e3fd07bdf05ea796\"><img alt=\"Fig7 Calabria BMCBioinformatics2015 16-Suppl9.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/9f\/Fig7_Calabria_BMCBioinformatics2015_16-Suppl9.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table border=\"0\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\"><blockquote><b>Figure 7: Data filtering and reporting in adLIMS The function of integrated reporting (via JasperReport) allows the users to configure customized reports from a selection of data fields<\/b>. In this example, a user starts from the DNA table (A), then clicks on the button \"report\" on the main menu to initiate the report generation, obtaining a new tab (B) with the list of the whole entries of the table. For example, to set a filter of the DNA samples with concentrations between 100 and 250 ng \/ uL, the user clicks on the \"search\" button (icon button with a magnifying glass) that opens a new window. Then the user switches in the advanced tab (C) of the search window and specifies the two needed filters. Once completed the search form, adLIMS filters the dataset and releases the PDF file that the user can save locally.<\/blockquote>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>adLIMS in our laboratory practice has brought an improvement through simplification and traceability of the data entry in sample workflows and sample reporting with respect to traditional data storage methods. As an example, adLIMS allows to create a sample sheets by selecting samples information in a few clicks whereas without adLIMS this activity may require hours, potentially introducing typing errors.\n<\/p><p>adLIMS can be adapted and extended to any laboratory working with biological samples with limited efforts, which are mainly related to the database design and customization, thus defining the business and view logics (workflows and GUI) and creating the data structures (PostgreSQL or Oracle database schema and tables) that are automatically converted into interfaces and data views by the ADempiere ERP engine.\n<\/p><p>adLIMS currently does not provide direct interfaces with laboratory instruments but integrates local server connections (storage mount access, network printers, and so on). Future extensions of the system will include API integration of laboratory platforms for sequencing, PCR product quantification and liquid handlers. On the other hand, client side interfaces (such as barcode readers and scanners) can be directly integrated into adLIMS, enhancing sample tracking and data acquisition.\n<\/p><p>The potentialities of adLIMS in the context of genomic companies and facilities are conveyed in the ready-to-use availability of the ERP features that would only require customization. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Currently, in many laboratories the procedures for data tracking and storage of sample information are based on spreadsheets without a real information management or standardization. This data management resulted in inefficiencies and redundancies, potentially generating many errors (typically typos) hard to backtrack or resolve. The use of a LIMS allows bypassing spreadsheets or local file management, supporting automation of all standard procedures for sample tracking. Exploiting adLIMS, our laboratory sustained the critical issues of sample tracking, data standardization and automation derived by the NGS and robotics revolution. This successful application of process engineering and monitoring allowed our laboratory to increase efficiency and reduce manual errors, thus posing concrete bases to sustain business scalability and potentially to approach pharmaco-vigilance monitoring of gene therapy patients in a highly standardized fashion compliant to regulatory requirements. Moreover adLIMS can natively be extended to incorporate ERP solutions, such as CRM, supply chain management, billing and accounting, integrated features that are critical for many genomics facilities.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.27_contributions\">Authors' contributions<\/span><\/h2>\n<p>A.C. conceived of the study, participated in its design and coordination and drafted the manuscript. G.S. created the application (adLIMS) and drafted the manuscript. F.B., ET participated in the requirements analysis. E.T. tested the system. E.M. revised the manuscript and supervised the project. All authors read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p>The source code, user guide and appliance of adLIMS are freely available at the project homepage <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/sourceforge.net\/projects\/adlims\" target=\"_blank\">http:\/\/sourceforge.net\/projects\/adlims<\/a>. We also provide a live demo for users who want to evaluate adLIMS without installation. Release notes and other information will be also updated on the project homepage.\n<\/p><p><b>Project name:<\/b> adLIMS\n<\/p><p><b>Project homepage:<\/b> <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/sourceforge.net\/projects\/adlims\" target=\"_blank\">http:\/\/sourceforge.net\/projects\/adlims<\/a>\n<\/p><p><b>Author:<\/b> Giulio Spinozzi\n<\/p><p><b>Operating system:<\/b> Platform independent\n<\/p><p><b>Supported web browsers:<\/b> Chrome, Firefox 3.5 +, Safari 4+\n<\/p><p><b>Programming language:<\/b> Java EE\n<\/p><p><b>Database:<\/b> PostgreSQL\/Oracle\n<\/p><p><b>License:<\/b> GNU GPLv2\n<\/p><p><b>Any restrictions to use by non-academics:<\/b> None\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>This work was supported by Telethon Grant D1 to EM. We would like to thank Giorgio Cafasso for his support with ADempiere ERP and for his gentle availability. We thank Luca Biasco for suggestions in LAM-PCR workflow set-up.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Declarations\">Declarations<\/span><\/h2>\n<p>Publication of this article has been funded by Telethon.\n<\/p><p>This article has been published as part of BMC Bioinformatics Volume 16 Supplement 9, 2015: Proceedings of the Italian Society of Bioinformatics (BITS): Annual Meeting 2014: Bioinformatics. The full contents of the supplement are available online at <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.biomedcentral.com\/bmcbioinformatics\/supplements\/16\/S9\" target=\"_blank\">http:\/\/www.biomedcentral.com\/bmcbioinformatics\/supplements\/16\/S9<\/a>.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-RanzaniLenti14-1\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RanzaniLenti14_1-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ranzani, M.; Annunziato, S.; Calabria, A.; Brasca, S.; Benedicenti, F.; Gallina, P.; Naldini, L.; Montini, E. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4429698\" target=\"_blank\">\"Lentiviral vector-based insertional mutagenesis identifies genes involved in the resistance to targeted anticancer therapies\"<\/a>. <i>Molecular Therapy<\/i> <b>22<\/b> (12): 2056-2068. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fmt.2014.174\" target=\"_blank\">10.1038\/mt.2014.174<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4429698\/\" target=\"_blank\">PMC4429698<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25195596\" target=\"_blank\">25195596<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4429698\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4429698<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lentiviral+vector-based+insertional+mutagenesis+identifies+genes+involved+in+the+resistance+to+targeted+anticancer+therapies&rft.jtitle=Molecular+Therapy&rft.aulast=Ranzani%2C+M.%3B+Annunziato%2C+S.%3B+Calabria%2C+A.%3B+Brasca%2C+S.%3B+Benedicenti%2C+F.%3B+Gallina%2C+P.%3B+Naldini%2C+L.%3B+Montini%2C+E.&rft.au=Ranzani%2C+M.%3B+Annunziato%2C+S.%3B+Calabria%2C+A.%3B+Brasca%2C+S.%3B+Benedicenti%2C+F.%3B+Gallina%2C+P.%3B+Naldini%2C+L.%3B+Montini%2C+E.&rft.date=2014&rft.volume=22&rft.issue=12&rft.pages=2056-2068&rft_id=info:doi\/10.1038%2Fmt.2014.174&rft_id=info:pmc\/PMC4429698&rft_id=info:pmid\/25195596&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4429698&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AiutiLenti13-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AiutiLenti13_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aiuti, A.; Biasco, L.; Scaramuzza, S.; Ferrua, F.; Cicalese, M.P.; Baricordi, C. et al. (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4375961\" target=\"_blank\">\"Lentiviral hematopoietic stem cell gene therapy in patients with Wiskott-Aldrich syndrome\"<\/a>. <i>Science<\/i> <b>341<\/b> (6148): 1233151. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.1233151\" target=\"_blank\">10.1126\/science.1233151<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4375961\/\" target=\"_blank\">PMC4375961<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23845947\" target=\"_blank\">23845947<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4375961\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4375961<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lentiviral+hematopoietic+stem+cell+gene+therapy+in+patients+with+Wiskott-Aldrich+syndrome&rft.jtitle=Science&rft.aulast=Aiuti%2C+A.%3B+Biasco%2C+L.%3B+Scaramuzza%2C+S.%3B+Ferrua%2C+F.%3B+Cicalese%2C+M.P.%3B+Baricordi%2C+C.+et+al.&rft.au=Aiuti%2C+A.%3B+Biasco%2C+L.%3B+Scaramuzza%2C+S.%3B+Ferrua%2C+F.%3B+Cicalese%2C+M.P.%3B+Baricordi%2C+C.+et+al.&rft.date=2013&rft.volume=341&rft.issue=6148&rft.pages=1233151&rft_id=info:doi\/10.1126%2Fscience.1233151&rft_id=info:pmc\/PMC4375961&rft_id=info:pmid\/23845947&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4375961&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BiffiLenti13-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BiffiLenti13_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Biffi, A.; Montini, E.; Lorioli, L.; Cesani, M.; Fumagalli, F.; Plati, T. et al. (2013). \"Lentiviral hematopoietic stem cell gene therapy benefits metachromatic leukodystrophy\". <i>Science<\/i> <b>341<\/b> (6148): 1233158. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscience.1233158\" target=\"_blank\">10.1126\/science.1233158<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23845948\" target=\"_blank\">23845948<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Lentiviral+hematopoietic+stem+cell+gene+therapy+benefits+metachromatic+leukodystrophy&rft.jtitle=Science&rft.aulast=Biffi%2C+A.%3B+Montini%2C+E.%3B+Lorioli%2C+L.%3B+Cesani%2C+M.%3B+Fumagalli%2C+F.%3B+Plati%2C+T.+et+al.&rft.au=Biffi%2C+A.%3B+Montini%2C+E.%3B+Lorioli%2C+L.%3B+Cesani%2C+M.%3B+Fumagalli%2C+F.%3B+Plati%2C+T.+et+al.&rft.date=2013&rft.volume=341&rft.issue=6148&rft.pages=1233158&rft_id=info:doi\/10.1126%2Fscience.1233158&rft_id=info:pmid\/23845948&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AiutiMulti07-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AiutiMulti07_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aiuti, A.; Cassani, B.; Andolfi, G.; Mirolo, M.; Biasco, L.; Recchia, A. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934603\" target=\"_blank\">\"Multilineage hematopoietic reconstitution without clonal selection in ADA-SCID patients treated with stem cell gene therapy\"<\/a>. <i>Journal of Clinical Investigation<\/i> <b>117<\/b> (8): 2233-40. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1172%2FJCI31666\" target=\"_blank\">10.1172\/JCI31666<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1934603\/\" target=\"_blank\">PMC1934603<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17671653\" target=\"_blank\">17671653<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934603\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934603<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Multilineage+hematopoietic+reconstitution+without+clonal+selection+in+ADA-SCID+patients+treated+with+stem+cell+gene+therapy&rft.jtitle=Journal+of+Clinical+Investigation&rft.aulast=Aiuti%2C+A.%3B+Cassani%2C+B.%3B+Andolfi%2C+G.%3B+Mirolo%2C+M.%3B+Biasco%2C+L.%3B+Recchia%2C+A.+et+al.&rft.au=Aiuti%2C+A.%3B+Cassani%2C+B.%3B+Andolfi%2C+G.%3B+Mirolo%2C+M.%3B+Biasco%2C+L.%3B+Recchia%2C+A.+et+al.&rft.date=2007&rft.volume=117&rft.issue=8&rft.pages=2233-40&rft_id=info:doi\/10.1172%2FJCI31666&rft_id=info:pmc\/PMC1934603&rft_id=info:pmid\/17671653&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1934603&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BraunGene14-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BraunGene14_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Braun, C.J.; Boztug, K.; Paruzynski, A.; Witzel, M.; Schwarzer, A.; Rothe, M. et al. (2014). \"Gene therapy for Wiskott-Aldrich syndrome: Long-term efficacy and genotoxicity\". <i>Science Translational Medicine<\/i> <b>6<\/b> (227): 227\u201333. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1126%2Fscitranslmed.3007280\" target=\"_blank\">10.1126\/scitranslmed.3007280<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/24622513\" target=\"_blank\">24622513<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+therapy+for+Wiskott-Aldrich+syndrome%3A+Long-term+efficacy+and+genotoxicity&rft.jtitle=Science+Translational+Medicine&rft.aulast=Braun%2C+C.J.%3B+Boztug%2C+K.%3B+Paruzynski%2C+A.%3B+Witzel%2C+M.%3B+Schwarzer%2C+A.%3B+Rothe%2C+M.+et+al.&rft.au=Braun%2C+C.J.%3B+Boztug%2C+K.%3B+Paruzynski%2C+A.%3B+Witzel%2C+M.%3B+Schwarzer%2C+A.%3B+Rothe%2C+M.+et+al.&rft.date=2014&rft.volume=6&rft.issue=227&rft.pages=227%E2%80%9333&rft_id=info:doi\/10.1126%2Fscitranslmed.3007280&rft_id=info:pmid\/24622513&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DeichmannVector07-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DeichmannVector07_6-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Deichmann, A.; Hacein-Bey-Abina, S.; Schmidt, M.; Garrigue, A.; Brugman, M.H.; Hu, J.; Glimm, H.; Gyapay, G.; Prum, B.; Fraser, C.C. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934585\" target=\"_blank\">\"Vector integration is nonrandom and clustered and influences the fate of lymphopoiesis in SCID-X1 gene therapy\"<\/a>. <i>Journal of Clinical Investigation<\/i> <b>117<\/b> (8): 2225\u20132232. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1172%2FJCI31659\" target=\"_blank\">10.1172\/JCI31659<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1934585\/\" target=\"_blank\">PMC1934585<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17671652\" target=\"_blank\">17671652<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934585\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934585<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Vector+integration+is+nonrandom+and+clustered+and+influences+the+fate+of+lymphopoiesis+in+SCID-X1+gene+therapy&rft.jtitle=Journal+of+Clinical+Investigation&rft.aulast=Deichmann%2C+A.%3B+Hacein-Bey-Abina%2C+S.%3B+Schmidt%2C+M.%3B+Garrigue%2C+A.%3B+Brugman%2C+M.H.%3B+Hu%2C+J.%3B+Glimm%2C+H.%3B+Gyapay%2C+G.%3B+Prum%2C+B.%3B+Fraser%2C+C.C.+et+al.&rft.au=Deichmann%2C+A.%3B+Hacein-Bey-Abina%2C+S.%3B+Schmidt%2C+M.%3B+Garrigue%2C+A.%3B+Brugman%2C+M.H.%3B+Hu%2C+J.%3B+Glimm%2C+H.%3B+Gyapay%2C+G.%3B+Prum%2C+B.%3B+Fraser%2C+C.C.+et+al.&rft.date=2007&rft.volume=117&rft.issue=8&rft.pages=2225%E2%80%932232&rft_id=info:doi\/10.1172%2FJCI31659&rft_id=info:pmc\/PMC1934585&rft_id=info:pmid\/17671652&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1934585&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-OttCorrect06-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-OttCorrect06_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ott, M.G; Schmidt, M.; Schwarzwaelder, K.; Stein, S.; Siler, U.; Koehl, U. et al. (2006). \"Correction of X-linked chronic granulomatous disease by gene therapy, augmented by insertional activation of MDS1-EVI1, PRDM16 or SETBP1\". <i>Nature Medicine<\/i> <b>12<\/b> (4): 401-409. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnm1393\" target=\"_blank\">10.1038\/nm1393<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16582916\" target=\"_blank\">16582916<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Correction+of+X-linked+chronic+granulomatous+disease+by+gene+therapy%2C+augmented+by+insertional+activation+of+MDS1-EVI1%2C+PRDM16+or+SETBP1&rft.jtitle=Nature+Medicine&rft.aulast=Ott%2C+M.G%3B+Schmidt%2C+M.%3B+Schwarzwaelder%2C+K.%3B+Stein%2C+S.%3B+Siler%2C+U.%3B+Koehl%2C+U.+et+al.&rft.au=Ott%2C+M.G%3B+Schmidt%2C+M.%3B+Schwarzwaelder%2C+K.%3B+Stein%2C+S.%3B+Siler%2C+U.%3B+Koehl%2C+U.+et+al.&rft.date=2006&rft.volume=12&rft.issue=4&rft.pages=401-409&rft_id=info:doi\/10.1038%2Fnm1393&rft_id=info:pmid\/16582916&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchwarzwaelderGamma07-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SchwarzwaelderGamma07_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schwarzwaelder, K.; Howe, S.J.; Schmidt, M.; Brugman, M.H.; Deichmann, A.; Glimm, H. et al. (2007). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934556\" target=\"_blank\">\"Gammaretrovirus-mediated correction of SCID-X1 is associated with skewed vector integration site distribution in vivo\"<\/a>. <i>Journal of Clinical Investigation<\/i> <b>117<\/b> (8): 2241-9. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1172%2FJCI31661\" target=\"_blank\">10.1172\/JCI31661<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1934556\/\" target=\"_blank\">PMC1934556<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17671654\" target=\"_blank\">17671654<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934556\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1934556<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gammaretrovirus-mediated+correction+of+SCID-X1+is+associated+with+skewed+vector+integration+site+distribution+in+vivo&rft.jtitle=Journal+of+Clinical+Investigation&rft.aulast=Schwarzwaelder%2C+K.%3B+Howe%2C+S.J.%3B+Schmidt%2C+M.%3B+Brugman%2C+M.H.%3B+Deichmann%2C+A.%3B+Glimm%2C+H.+et+al.&rft.au=Schwarzwaelder%2C+K.%3B+Howe%2C+S.J.%3B+Schmidt%2C+M.%3B+Brugman%2C+M.H.%3B+Deichmann%2C+A.%3B+Glimm%2C+H.+et+al.&rft.date=2007&rft.volume=117&rft.issue=8&rft.pages=2241-9&rft_id=info:doi\/10.1172%2FJCI31661&rft_id=info:pmc\/PMC1934556&rft_id=info:pmid\/17671654&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1934556&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SteinGeno10-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SteinGeno10_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Stein, S.; Ott, M.G.; Schultze-Strasser, S.; Jauch, A.; Burwinkel, B.; Kinner, A. et al. (2010). \"Genomic instability and myelodysplasia with monosomy 7 consequent to EVI1 activation after gene therapy for chronic granulomatous disease\". <i>Nature Medicine<\/i> <b>16<\/b> (2): 198-204. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnm.2088\" target=\"_blank\">10.1038\/nm.2088<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20098431\" target=\"_blank\">20098431<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Genomic+instability+and+myelodysplasia+with+monosomy+7+consequent+to+EVI1+activation+after+gene+therapy+for+chronic+granulomatous+disease&rft.jtitle=Nature+Medicine&rft.aulast=Stein%2C+S.%3B+Ott%2C+M.G.%3B+Schultze-Strasser%2C+S.%3B+Jauch%2C+A.%3B+Burwinkel%2C+B.%3B+Kinner%2C+A.+et+al.&rft.au=Stein%2C+S.%3B+Ott%2C+M.G.%3B+Schultze-Strasser%2C+S.%3B+Jauch%2C+A.%3B+Burwinkel%2C+B.%3B+Kinner%2C+A.+et+al.&rft.date=2010&rft.volume=16&rft.issue=2&rft.pages=198-204&rft_id=info:doi\/10.1038%2Fnm.2088&rft_id=info:pmid\/20098431&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SchmidtHigh07-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SchmidtHigh07_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-SchmidtHigh07_10-1\" rel=\"external_link\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-SchmidtHigh07_10-2\" rel=\"external_link\">10.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Schmidt, M.; Schwarzwaelder, K.; Bartholomae, C.; Zaoui, K.; Ball, C.; Pilz, I. (2007). \"High-resolution insertion-site analysis by linear amplification-mediated PCR (LAM-PCR)\". <i>Nature Methods<\/i> <b>4<\/b> (12): 1051-1057. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnmeth1103\" target=\"_blank\">10.1038\/nmeth1103<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18049469\" target=\"_blank\">18049469<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=High-resolution+insertion-site+analysis+by+linear+amplification-mediated+PCR+%28LAM-PCR%29&rft.jtitle=Nature+Methods&rft.aulast=Schmidt%2C+M.%3B+Schwarzwaelder%2C+K.%3B+Bartholomae%2C+C.%3B+Zaoui%2C+K.%3B+Ball%2C+C.%3B+Pilz%2C+I.&rft.au=Schmidt%2C+M.%3B+Schwarzwaelder%2C+K.%3B+Bartholomae%2C+C.%3B+Zaoui%2C+K.%3B+Ball%2C+C.%3B+Pilz%2C+I.&rft.date=2007&rft.volume=4&rft.issue=12&rft.pages=1051-1057&rft_id=info:doi\/10.1038%2Fnmeth1103&rft_id=info:pmid\/18049469&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-CalabriaVISPA14-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-CalabriaVISPA14_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Calabria, A.; Leo, S.; Benedicenti, F.; Cesana, D.; Spinozzi, G.; Orsini, M. et al. (2014). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169225\" target=\"_blank\">\"VISPA: A computational pipeline for the identification and analysis of genomic vector integration sites\"<\/a>. <i>Genome Medicine<\/i> <b>6<\/b> (9): 67. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2Fs13073-014-0067-5\" target=\"_blank\">10.1186\/s13073-014-0067-5<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4169225\/\" target=\"_blank\">PMC4169225<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/25342980\" target=\"_blank\">25342980<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169225\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC4169225<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=VISPA%3A+A+computational+pipeline+for+the+identification+and+analysis+of+genomic+vector+integration+sites&rft.jtitle=Genome+Medicine&rft.aulast=Calabria%2C+A.%3B+Leo%2C+S.%3B+Benedicenti%2C+F.%3B+Cesana%2C+D.%3B+Spinozzi%2C+G.%3B+Orsini%2C+M.+et+al.&rft.au=Calabria%2C+A.%3B+Leo%2C+S.%3B+Benedicenti%2C+F.%3B+Cesana%2C+D.%3B+Spinozzi%2C+G.%3B+Orsini%2C+M.+et+al.&rft.date=2014&rft.volume=6&rft.issue=9&rft.pages=67&rft_id=info:doi\/10.1186%2Fs13073-014-0067-5&rft_id=info:pmc\/PMC4169225&rft_id=info:pmid\/25342980&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC4169225&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PrasadTrends12-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PrasadTrends12_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Prasad, P.J.; Bodhe, G.L. (2012). \"Trends in laboratory information management system\". <i>Chemometrics and Intelligent Laboratory Systems<\/i> <b>118<\/b>: 187\u2013192. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1016%2Fj.chemolab.2012.07.001\" target=\"_blank\">10.1016\/j.chemolab.2012.07.001<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Trends+in+laboratory+information+management+system&rft.jtitle=Chemometrics+and+Intelligent+Laboratory+Systems&rft.aulast=Prasad%2C+P.J.%3B+Bodhe%2C+G.L.&rft.au=Prasad%2C+P.J.%3B+Bodhe%2C+G.L.&rft.date=2012&rft.volume=118&rft.pages=187%E2%80%93192&rft_id=info:doi\/10.1016%2Fj.chemolab.2012.07.001&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SohEnt00-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SohEnt00_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Soh, C.; Kien, S.S.; Tay-Yap, J. (April 2000). \"Enterprise resource planning: Cultural fits and misfits: Is ERP a universal solution?\". <i>Communications of the ACM<\/i> <b>43<\/b> (4): 47-51. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F332051.332070\" target=\"_blank\">10.1145\/332051.332070<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Enterprise+resource+planning%3A+Cultural+fits+and+misfits%3A+Is+ERP+a+universal+solution%3F&rft.jtitle=Communications+of+the+ACM&rft.aulast=Soh%2C+C.%3B+Kien%2C+S.S.%3B+Tay-Yap%2C+J.&rft.au=Soh%2C+C.%3B+Kien%2C+S.S.%3B+Tay-Yap%2C+J.&rft.date=April+2000&rft.volume=43&rft.issue=4&rft.pages=47-51&rft_id=info:doi\/10.1145%2F332051.332070&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AoyamaManaging97-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AoyamaManaging97_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Aoyama, Mikio (1997). \"Managing the concurrent development of large-scale software systems\". <i>International Journal of Technology Management<\/i> <b>14<\/b> (6\u20138): 739\u2013765. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1504%2FIJTM.1997.002580\" target=\"_blank\">10.1504\/IJTM.1997.002580<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Managing+the+concurrent+development+of+large-scale+software+systems&rft.jtitle=International+Journal+of+Technology+Management&rft.aulast=Aoyama%2C+Mikio&rft.au=Aoyama%2C+Mikio&rft.date=1997&rft.volume=14&rft.issue=6%E2%80%938&rft.pages=739%E2%80%93765&rft_id=info:doi\/10.1504%2FIJTM.1997.002580&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-DoDSys01-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-DoDSys01_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Department of Defense Systems Management College (January 2001) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ocw.mit.edu\/courses\/aeronautics-and-astronautics\/16-885j-aircraft-systems-engineering-fall-2005\/readings\/sefguide_01_01.pdf\" target=\"_blank\"><i>Systems Engineering Fundamentals<\/i><\/a>. Fort Belvoir, VA: Defense Acquisition University Press. pp. 216<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ocw.mit.edu\/courses\/aeronautics-and-astronautics\/16-885j-aircraft-systems-engineering-fall-2005\/readings\/sefguide_01_01.pdf\" target=\"_blank\">http:\/\/ocw.mit.edu\/courses\/aeronautics-and-astronautics\/16-885j-aircraft-systems-engineering-fall-2005\/readings\/sefguide_01_01.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Systems+Engineering+Fundamentals&rft.aulast=Department+of+Defense+Systems+Management+College&rft.au=Department+of+Defense+Systems+Management+College&rft.date=January+2001&rft.pages=pp.%26nbsp%3B216&rft.place=Fort+Belvoir%2C+VA&rft.pub=Defense+Acquisition+University+Press&rft_id=http%3A%2F%2Focw.mit.edu%2Fcourses%2Faeronautics-and-astronautics%2F16-885j-aircraft-systems-engineering-fall-2005%2Freadings%2Fsefguide_01_01.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BoochTheUni96-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BoochTheUni96_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Booch, Grady; Rumbaugh, James; Jacobson, Ivar (1996). <i>The Unified Modeling Language for Object-Oriented Development<\/i>. Santa Clara, CA: Rational Software Corporation.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Unified+Modeling+Language+for+Object-Oriented+Development&rft.aulast=Booch%2C+Grady%3B+Rumbaugh%2C+James%3B+Jacobson%2C+Ivar&rft.au=Booch%2C+Grady%3B+Rumbaugh%2C+James%3B+Jacobson%2C+Ivar&rft.date=1996&rft.place=Santa+Clara%2C+CA&rft.pub=Rational+Software+Corporation&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KrasnerADesc88-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KrasnerADesc88_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-KrasnerADesc88_17-1\" rel=\"external_link\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Krasner, Glenn E.; Pope, Stephen T. (1988) (PDF). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.create.ucsb.edu\/~stp\/PostScript\/mvc.pdf\" target=\"_blank\"><i>A Description of the Model-View-Controller User Interface Paradigm in the Smalltalk-80 System<\/i><\/a>. Mountain View, CA: ParcPlace Systems, Inc. pp. 34<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.create.ucsb.edu\/~stp\/PostScript\/mvc.pdf\" target=\"_blank\">http:\/\/www.create.ucsb.edu\/~stp\/PostScript\/mvc.pdf<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=A+Description+of+the+Model-View-Controller+User+Interface+Paradigm+in+the+Smalltalk-80+System&rft.aulast=Krasner%2C+Glenn+E.%3B+Pope%2C+Stephen+T.&rft.au=Krasner%2C+Glenn+E.%3B+Pope%2C+Stephen+T.&rft.date=1988&rft.pages=pp.%26nbsp%3B34&rft.place=Mountain+View%2C+CA&rft.pub=ParcPlace+Systems%2C+Inc&rft_id=http%3A%2F%2Fwww.create.ucsb.edu%2F%7Estp%2FPostScript%2Fmvc.pdf&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Clarity-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Clarity_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.genologics.com\/clarity-lims\/\" target=\"_blank\">\"Clarity LIMS\"<\/a>. GenoLogics Life Sciences Software Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.genologics.com\/clarity-lims\/\" target=\"_blank\">http:\/\/www.genologics.com\/clarity-lims\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Clarity+LIMS&rft.atitle=&rft.pub=GenoLogics+Life+Sciences+Software+Inc&rft_id=http%3A%2F%2Fwww.genologics.com%2Fclarity-lims%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MiniLIMS-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MiniLIMS_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/bioteam.net\/minilims\/\" target=\"_blank\">\"MiniLIMS\"<\/a>. BioTeam Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/bioteam.net\/minilims\/\" target=\"_blank\">http:\/\/bioteam.net\/minilims\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=MiniLIMS&rft.atitle=&rft.pub=BioTeam+Inc&rft_id=http%3A%2F%2Fbioteam.net%2Fminilims%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Bika-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Bika_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.bikalabs.com\/\" target=\"_blank\">\"Bika: Open Source laboratory information management systems\"<\/a>. Bika Lab Systems (Pty) Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.bikalabs.com\/\" target=\"_blank\">http:\/\/www.bikalabs.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bika%3A+Open+Source+laboratory+information+management+systems&rft.atitle=&rft.pub=Bika+Lab+Systems+%28Pty%29+Ltd&rft_id=http%3A%2F%2Fwww.bikalabs.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LabKey-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LabKey_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/labkey.com\/\" target=\"_blank\">\"LabKey Software\"<\/a>. LabKey Corporation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/labkey.com\/\" target=\"_blank\">https:\/\/labkey.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=LabKey+Software&rft.atitle=&rft.pub=LabKey+Corporation&rft_id=https%3A%2F%2Flabkey.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ScholtalbersGal13-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ScholtalbersGal13_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Scholtalbers, J.; R\u00f6ssler, J.; Sorn, P.; de Graaf, J.; Boisgu\u00e9rin, V.; Castle, J.; Sahin, U. (2013). \"Galaxy LIMS for next-generation sequencing\". <i>Bioinformatics<\/i> <b>29<\/b> (9): 1233-1234. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fbioinformatics%2Fbtt115\" target=\"_blank\">10.1093\/bioinformatics\/btt115<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23479349\" target=\"_blank\">23479349<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Galaxy+LIMS+for+next-generation+sequencing&rft.jtitle=Bioinformatics&rft.aulast=Scholtalbers%2C+J.%3B+R%C3%B6ssler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisgu%C3%A9rin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.au=Scholtalbers%2C+J.%3B+R%C3%B6ssler%2C+J.%3B+Sorn%2C+P.%3B+de+Graaf%2C+J.%3B+Boisgu%C3%A9rin%2C+V.%3B+Castle%2C+J.%3B+Sahin%2C+U.&rft.date=2013&rft.volume=29&rft.issue=9&rft.pages=1233-1234&rft_id=info:doi\/10.1093%2Fbioinformatics%2Fbtt115&rft_id=info:pmid\/23479349&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Plone-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Plone_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/plone.org\/\" target=\"_blank\">\"Plone CMS\"<\/a>. Plone Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/plone.org\/\" target=\"_blank\">https:\/\/plone.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Plone+CMS&rft.atitle=&rft.pub=Plone+Foundation&rft_id=https%3A%2F%2Fplone.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Drupal-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Drupal_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/drupal.org\/\" target=\"_blank\">\"Drupal\"<\/a>. Drupal Association<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/drupal.org\/\" target=\"_blank\">https:\/\/drupal.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Drupal&rft.atitle=&rft.pub=Drupal+Association&rft_id=https%3A%2F%2Fdrupal.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ADempiere-25\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ADempiere_25-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.adempiere.com\/ADempiere_ERP\" target=\"_blank\">\"ADempiere ERP\"<\/a>. ADempiere Community<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.adempiere.com\/ADempiere_ERP\" target=\"_blank\">http:\/\/www.adempiere.com\/ADempiere_ERP<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ADempiere+ERP&rft.atitle=&rft.pub=ADempiere+Community&rft_id=http%3A%2F%2Fwww.adempiere.com%2FADempiere_ERP&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Wirfs-BrockObject89-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Wirfs-BrockObject89_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Wirfs-Brock, Rebecca; Wilkerson, Brian (1989). \"Object-oriented design: A responsibility-driven approach\". <i>OOPSLA '89 Conference Proceedings on Object-oriented Programming Systems, Languages and Applications<\/i> <b>24<\/b> (10): 71\u201375. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1145%2F74878.74885\" target=\"_blank\">10.1145\/74878.74885<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Object-oriented+design%3A+A+responsibility-driven+approach&rft.jtitle=OOPSLA+%2789+Conference+Proceedings+on+Object-oriented+Programming+Systems%2C+Languages+and+Applications&rft.aulast=Wirfs-Brock%2C+Rebecca%3B+Wilkerson%2C+Brian&rft.au=Wirfs-Brock%2C+Rebecca%3B+Wilkerson%2C+Brian&rft.date=1989&rft.volume=24&rft.issue=10&rft.pages=71%E2%80%9375&rft_id=info:doi\/10.1145%2F74878.74885&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ZK-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ZK_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.zkoss.org\/\" target=\"_blank\">\"ZK\"<\/a>. Potix Corporation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.zkoss.org\/\" target=\"_blank\">http:\/\/www.zkoss.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ZK&rft.atitle=&rft.pub=Potix+Corporation&rft_id=http%3A%2F%2Fwww.zkoss.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-JasperReports-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-JasperReports_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.jaspersoft.com\/\" target=\"_blank\">\"Jaspersoft\"<\/a>. TIBCO Software, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.jaspersoft.com\/\" target=\"_blank\">http:\/\/www.jaspersoft.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Jaspersoft&rft.atitle=&rft.pub=TIBCO+Software%2C+Inc&rft_id=http%3A%2F%2Fwww.jaspersoft.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210216\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.655 seconds\nReal time usage: 0.705 seconds\nPreprocessor visited node count: 21320\/1000000\nPreprocessor generated node count: 36130\/1000000\nPost\u2010expand include size: 163407\/2097152 bytes\nTemplate argument size: 53249\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 643.595 1 - -total\n 81.55% 524.825 1 - Template:Reflist\n 70.54% 453.983 28 - Template:Citation\/core\n 48.92% 314.878 16 - Template:Cite_journal\n 18.73% 120.562 9 - Template:Cite_web\n 12.58% 80.935 1 - Template:Infobox_journal_article\n 12.20% 78.508 1 - Template:Infobox\n 7.77% 50.027 3 - Template:Cite_book\n 7.39% 47.588 80 - Template:Infobox\/row\n 7.01% 45.141 34 - Template:Citation\/identifier\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7888-0!*!0!!en!5!* and timestamp 20181213210215 and revision id 23402\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies\">https:\/\/www.limswiki.org\/index.php\/Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","b55d160b7541498bbeb5c39c2bdb347d_images":["https:\/\/www.limswiki.org\/images\/c\/c4\/Fig1_Calabria_BMCBioinformatics2015_16-Suppl9.jpg","https:\/\/www.limswiki.org\/images\/d\/d4\/Fig2_Calabria_BMCBioinformatics2015_16-Suppl9.jpg","https:\/\/www.limswiki.org\/images\/8\/88\/Fig3_Calabria_BMCBioinformatics2015_16-Suppl9.jpg","https:\/\/www.limswiki.org\/images\/3\/37\/Fig4_Calabria_BMCBioinformatics2015_16-Suppl9.jpg","https:\/\/www.limswiki.org\/images\/5\/54\/Fig5_Calabria_BMCBioinformatics2015_16-Suppl9.jpg","https:\/\/www.limswiki.org\/images\/0\/00\/Fig6_Calabria_BMCBioinformatics2015_16-Suppl9.jpg","https:\/\/www.limswiki.org\/images\/9\/9f\/Fig7_Calabria_BMCBioinformatics2015_16-Suppl9.jpg"],"b55d160b7541498bbeb5c39c2bdb347d_timestamp":1544734935,"1916b796a2be08f4f1d2fd01df17d5fa_type":"article","1916b796a2be08f4f1d2fd01df17d5fa_title":"4273\u03c0: Bioinformatics education on low cost ARM hardware (Barker et al. 2013)","1916b796a2be08f4f1d2fd01df17d5fa_url":"https:\/\/www.limswiki.org\/index.php\/Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware","1916b796a2be08f4f1d2fd01df17d5fa_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:4273\u03c0: Bioinformatics education on low cost ARM hardware\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \n4273\u03c0: Bioinformatics education on low cost ARM hardwareJournal\n \nBMC BioinformaticsAuthor(s)\n \nBarker, Daniel; Ferrier, David E.K.; Holland, Peter W.H.; Mitchell, John B.O.; Plaisier, Heleen; Ritchie, Michael G.; Smart, Steven D.Author affiliation(s)\n \nUniversity of St. Andrews, University of OxfordPrimary contact\n \nEmail: db60@st-andrews.ac.ukYear published\n \n2013Volume and issue\n \n14Page(s)\n \n243DOI\n \n10.1186\/1471-2105-14-243ISSN\n \n1471-2105Distribution license\n \nCreative Commons Attribution 2.0 GenericWebsite\n \nhttp:\/\/www.biomedcentral.com\/1471-2105\/14\/243Download\n \nhttp:\/\/www.biomedcentral.com\/content\/pdf\/1471-2105-14-243.pdf (PDF)\n\nContents\n\n1 Abstract \n2 Background \n3 Implementation \n4 Results and discussion \n5 Conclusions \n\n5.1 (Administrator) power to the people \n5.2 The virtue of patience \n5.3 Low-cost teaching and learning \n5.4 Open learning \n\n\n6 Availability and requirements \n7 Endnotes \n8 Abbreviations \n9 Competing interests \n10 Authors\u2019 contributions \n11 Acknowledgements \n12 References \n13 Notes \n\n\n\nAbstract \nBackground: Teaching bioinformatics at universities is complicated by typical computer classroom settings. As well as running software locally and online, students should gain experience of systems administration. For a future career in biology or bioinformatics, the installation of software is a useful skill. We propose that this may be taught by running the course on GNU\/Linux running on inexpensive Raspberry Pi computer hardware, for which students may be granted full administrator access.\nResults: We release 4273\u03c0, an operating system image for Raspberry Pi based on Raspbian Linux. This includes minor customisations for classroom use and includes our Open Access bioinformatics course, 4273\u03c0 Bioinformatics for Biologists. This is based on the final-year undergraduate module BL4273, run on Raspberry Pi computers at the University of St Andrews, Semester 1, academic year 2012\u20132013.\nConclusions: 4273\u03c0 is a means to teach bioinformatics, including systems administration tasks, to undergraduates at low cost.\nKeywords: Bioinformatics education; Teaching material; Raspberry Pi; Linux\n\nBackground \nBioinformatics is increasingly included in the undergraduate curriculum for biology students. Teaching bioinformatics is made difficult, however, by the constraints of a typical university computer classroom. Some areas of basic bioinformatics may be taught using such classrooms, where all that is required is an Internet connection and Web browser (e.g. BLAST[1] searches at the NCBI[2]). More in-depth teaching requires the re-creation of a bioinformatics research environment, consisting of a Linux or UNIX operating system, standard GNU utilities[3], specialist bioinformatics software, and sequence databases.\nUndergraduate modules ought, ideally, to prepare students for research in an academic research group. Students who do pursue a research career will often find that institutional computer support is targeted to generic computer use (e.g. Microsoft software) rather than installing and maintaining systems suitable for bioinformatics. Particularly outside of bioinformatics research (but also occasionally within it), the principal investigator of the research group may never have used Linux, may have a limited idea of the procedures, and may expect group members to \u2018pick things up\u2019 and deal with problems themselves. This requires researchers to have a high level of proficiency with Linux, including the ability to install both standard Linux packages and software for which no standard package may be available. A single taught module cannot prepare a student for all eventualities, but ought to leave the student with the basic skills and confidence to be able to discover solutions, and implement them, as required. Hence, a certain amount of system administration should appear in an undergraduate bioinformatics module for biologists. \nTraditionally, the environment required for an undergraduate bioinformatics module has been created in one of four ways. Firstly, one may set up a central GNU\/Linux server on the campus and allow students to connect by Secure Shell, ssh (\u2018the server approach\u2019). The server will typically run either a standard Linux distribution or a specialist bioinformatics distribution such as NEBC Bio-Linux.[4] The server approach allows the instructor to have full control over the server, and allows students to connect from existing computing classrooms with little or no adjustment to the classroom software. For students to connect to the server via the intranet, classroom computers only require an ssh client, the X Window System (X11), and a means of file transfer such as secure copy (scp). Students may also connect to the server from home (typically requiring them to install virtual private network software in addition to ssh, X11 and an scp client) or elsewhere on campus. Secondly, one may provide students with a virtual machine, consisting of an environment similar to that which they might experience on the Linux server but running on a classroom computer, either with a standard Linux distribution or a specialist bioinformatics distribution such as DNA Linux Virtual Desktop Edition[5] (\u2018the VM approach\u2019). This has the advantage that students may be given administrator access to their virtual machine. Thirdly, one may provide students with a Linux system on removable media (\u2018the USB stick approach\u2019, for example[6]; where files and settings do not have to be saved, a DVD may be used instead[7]). So long as students have the media to hand, this allows them to boot into \u2018their own\u2019 Linux. As with the VM approach, students may be given administrator access. The additional advantage is that the media may be portable between computer classrooms and home computers, without requiring students to move virtual machine image files. Fourthly, students may be loaned or required to buy laptops of a specific kind, with a suitable operating system, data and software installed (\u2018the laptop approach\u2019). This avoids hardware incompatibilities that the USB stick approach may, in practice, experience.[6]\nBecause administrator access cannot be allowed, the server approach fails to give students experience of the standard mechanism of software installation. It also involves competition for resources such as CPU time, especially if the class is large or the server is also shared with research colleagues. The VM approach solves both these problems but is less portable. Although, in theory, students may transfer a VM from one computer to another (assuming the destination has the necessary virtualisation software installed), the task is non-trivial, and more time consuming than a simple transfer of data or documents. The USB stick approach reduces the portability problem, since it is trivial to move a USB stick from one computer to another. However, smooth operation on all hardware is not guaranteed and requires ongoing efforts from the developers of the Linux distribution as new hardware is released. The laptop approach avoids all these problems by providing a portable computer holding everything required for the course. However, it is expensive.\nAs a fifth approach, we propose loaning a Raspberry Pi computer[8] and associated peripherals to students for the duration of the course (\u2018the Raspberry Pi approach\u2019). This includes a customised version of Linux, appropriate software and data. This allows students full administrator access to a suitable operating system, without the difficulties of the VM or USB stick approaches. Should the student accidentally damage critical files, the system can be re-written from a master image.\nThe Raspberry Pi Model B \u2014 with 256 MB (now 512 MB) RAM, an ARM11 CPU running at 700 MHz before over clocking and a Video core IV GPU \u2014 was released for public sale in 2012[9] and costs \u00a328.07a or \u00a331.20.[10][11] Though additional items are required to turn it into a functioning, general-purpose computer (case, charger, SD card, mouse, keyboard, monitor and cable; and an entirely separate computer for initialising the SD card), it is still relatively low-cost (Additional file 1: Table S1). The existence of the Raspberry Pi is partly a celebration of the early days of popular computing in the 1980s, and an attempt to recreate that excitement among young people today.[12] It is also a symptom of the rapidly decreasing costs and increasing performance of computer hardware. The Raspberry Pi uses an ARM CPU.[13] Because of their high performance-per-watt, ARM CPUs are frequently found in small electronic appliances such as mobile phones and tablets. With CPU innovation increasingly driven by such applications, as opposed to more traditional areas such as desktop, laptop and server computers, the prevalence and utility of ARM-based computer hardware is likely to increase. Indeed, ARM-based servers are starting to appear in data centres, due to their modest requirements for power.[14]\n\n\n\n\n\n\n\n Additional file 1. Table S1\n\n\nExample prices of Raspberry Pi peripherals we found to work well in practice. These are presented without any endorsement. A case for the Raspberry Pi (various models and suppliers; ~\u00a35-\u00a310), the Raspberry Pi itself (see main text) and a monitor are not shown. Standard consumer prices, including UK tax but excluding any delivery charge, were obtained from the Insight UK Web site or via the Amazon UK Web site on 7 April 2013.\nFormat: PDF; Size: 110KB Download file\n\n\n\n\nThough far slower than current desktop and laptop computers, the Raspberry Pi is notably faster than the Cray 1 supercomputer[15], a marvel of computer speed in its day. The valid question arises as to how much computer power is actually required to teach undergraduates bioinformatics? We propose that the answer is, by current standards, \u2018not much\u2019. The Raspberry Pi is more than adequate for the task. The Raspberry Pi approach includes all the benefits of the laptop approach, above, but at lower cost. In addition, the Raspberry Pi is a new and exciting computer system, which in itself can add interest to the course. \nA variety of operating systems is available for the Raspberry Pi.[16] These include Raspbian[17], which is based on Debian GNU\/Linux.[18] Over 35,000 Debian software packages are available pre-compiled for Raspbian, including Web browsers, text editors, word processors, and a wide range of bioinformatics packages.[19] Other software will usually compile and run without problems. Some features of recent CPUs (e.g. 64-bit addressing or vector operations) are absent, but we have not found these to be at all necessary for our proposed use of the Raspberry Pi. The most serious limitation, affecting structure visualisation software in particular, is limited graphics performance. Even so, some structural visualisation software does work on the Raspberry Pi.[20] New system software, improving graphics performance by making better use of the Raspberry Pi\u2019s GPU, is under development.[21]\nWe provide 4273\u03c0, a customised version of Linux for Raspberry Pi computer hardware. 4273\u03c0 includes an Open Access bioinformatics course, 4273\u03c0 Bioinformatics for Biologists. \n\nImplementation \n4273\u03c0 Bioinformatics for Biologists is based on the module BL4273 Bioinformatics for Biologists at the University of St Andrews[22], an optional module of 15 SCOTCAT credits, equivalent to 7.5 ECTS credits or ~4 US credits. BL4273 is intended for final-year undergraduate students on BSc(Hons) Biology, BSc(Hons) Biochemistry and other degree courses taught by the School of Biology. BL4273 was taught on Raspberry Pi hardware in Semester 1 of academic year 2012\u20132013. During this time, students were loaned a Raspberry Pi, SD card and USB stick for backup. Teaching was carried out in a small computer room in which students connected to monitors more generally used with Windows desktop PCs. Students were allowed to take all equipment home on loan (including keyboard, mouse and cables), with the exception of the monitors; several copies of the main textbook[23] were also available. Course material was released week-by-week using an rsync server (running on a Raspberry Pi) on the university intranet. Following the conclusion of the module, material was edited to remove closed-access material (e.g. images in lectures), typographical errors were corrected, and the SD card image was re-created using a recent version of Raspbian.[17]\nPreparation of a release of 4273\u03c0 begins with the latest Raspbian SD card image.[24] This is then customised to produce a \u2018master\u2019 SD card for the release, partly by a series of scripts which alter the configuration and use Raspbian\u2019s port of the APT mechanism of Debian[18] to install specialist bioinformatics packages[19] and more general packages, and partly by a series of commands entered manually (e.g. to install BLAST databases and 4273\u03c0 Bioinformatics for Biologists in the ~\/4273pi\/ directory). The master SD card image is stored on a separate computer and uploaded to the 4273\u03c0 Web site. A \u2018work instruction\u2019 detailing the steps performed to convert Raspbian into 4273\u03c0, and all scripts used, are distributed with 4273\u03c0.\nFor the permanent record, the teaching material included in the current release \u2013 excluding Linux, software and BLAST databases \u2013 is available as Additional file 2. The latest version may be downloaded from the 4273\u03c0 Web site.[25]\n\n\n\n\n\n\n\n Additional file 2. 4273\u03c0 Bioinformatics for Biologists teaching material, Version 1.01\n\n\nThe module handbook, lectures and practicals are included. The latest version, including Linux, software and BLAST databases, is available at the 4273\u03c0 Web site[25]\nFormat: ZIP; Size: 7MB Download file\n\n\n\n\nAs the Raspbian operating system, Raspberry Pi firmware and hardware and 4273\u03c0 Bioinformatics for Biologists teaching material develop, further releases of 4273\u03c0 will be made available. It is anticipated that there will be a minimum of two releases per year during the next four years. \n\nResults and discussion \n4273\u03c0 provides an attractive, general-purpose computing environment, within which the course 4273\u03c0 Bioinformatics for Biologists is embedded (Figure 1; Table 1). Student feedback questionnaires for the module BL4273, upon which it is based, are too few to draw strong conclusions (five returned from a class of six students; Additional file 1: Table S2). However, they were sufficiently positive to continue the module without modification. \n\n\n\nFigure 1: 4273\u03c0 screenshot. Practical, Week 7. Predicted gene structures of the Or98b gene in the Drosophila sechellia genome are being compared. Different predictions are made de novo by SNAP[26], run in a terminal on the Raspberry Pi; GeneWise[27], run online at the European Bioinformatics Institute[28], with the D. melanogaster protein being aligned against D. sechellia genomic DNA; and using TBLASTN[1], run online at FlyBase.[29] FlyBase also displays the annotation of this gene from its database. This case study arose during research for a comparative study of the chemoreceptor superfamily in Drosophila.[30] The 4273\u03c0 background desktop image includes hand-written versions of material from 4273\u03c0 Bioinformatics for Biologists and from the module BL3320 Practical Statistics for Biologists, which most students at the University of St Andrews would have taken earlier. It also includes a copy of one of the first published phylogenetic trees, from Vestiges of the Natural History of Creation, written in St Andrews.[31]\n\n\n\n\n\n\nTable 1. Timetable for 4273\u03c0 Bioinformatics for Biologists\n\n\n Week\n\n Lecture\n\n Practical class\n\n\n1\n\nGenomes, sequences and bioinformatics data\n\nLinux and Perl\n\n\n2\n\n-\n\nLinux, Perl and protein BLAST\n\n\n3\n\n-\n\nLinux, Perl and delimiting gene\/protein families\n\n\n4\n\nMultiple alignment and phylogeny\n\nMultiple alignment and phylogeny\n\n\n5\n\nGene family evolution\n\nGene family evolution\n\n\n6\n\nBLAST; DNA sequence analysis\n\nDNA sequence analysis\n\n\n7\n\nLooking at species differences\n\nDetecting positive selection\n\n\n8\n\nFunction and evolution of enzymes\n\nFunction and evolution of enzymes\n\n\n9\n\nSeminar: student presentations\n\n-\n\n\nLectures last slightly less than an hour. Practical classes all use the Raspberry Pi, and last two hours. In addition, in their own time, students engage in self-study and complete a coursework \u2018practical project\u2019.\n\n\n\nConclusions \n(Administrator) power to the people \nRather than present systems administration as a complex task to be delegated to a technical support unit \u2014 which in students\u2019 future careers may not be available \u2014 4273\u03c0 Bioinformatics for Biologists introduces software installation on Linux through standard package management (APT), through compilation (GNU \u2018make\u2019 for SNAP[26] and PAML[32]) and through command-line launch of a JAR (Modelgenerator[33] and Mesquite[34]). Other administrative tasks covered include upgrading Linux and installing a MySQL server. Although these matters of system administration are incidental to an intellectual understanding of bioinformatics, they do not take much time to teach, and we believe they leave students well-prepared for a bioinformatics research career \u2013 in many cases, better prepared than any other members of the research group within which they are working. \n\nThe virtue of patience \nNo serious compromises in content were required to teach bioinformatics on the Raspberry Pi. A BLAST search of the GenPept database (\u2018nr\u2019) is, in practice, too slow. However a BLAST search of the SwissProt database takes only a few minutes. Delimitation of protein families across two prokaryotic genomes using BLAST and OrthoMCL is entirely feasible, and is central to the coursework component of 4273\u03c0 Bioinformatics for Biologists. Java (Modelgenerator, Mesquite) programs run slowly, but not unbearably so. Speed will likely improve once Oracle Java becomes available for the platform. Floating-point-intensive tasks (PhyML[35] and PAML) are also slow, but feasible. Waiting an hour for an analysis is, in fact, realistic training for bioinformatics research. Research will tend to use a far faster computer or cluster, but with far larger input. Teaching bioinformatics on rather slow hardware was not universally popular among students (Additional file 1: Table S2). However, it is a valuable lesson in the transferrable virtue of patience, and the value of checking the configuration of analyses before launching them. \n\nLow-cost teaching and learning \nWhen the undergraduate module BL4273 at the University of St Andrews ran previously, in Semester 1 of academic year 2008\u20132009, it used the server approach. Students used desktop computers running Windows in a computer classroom to connect to a 64-bit IBM System x3755 8877 server with 4 AMD CPU cores and 16 GB RAM, running Debian Linux. When purchased in 2008, this server cost \u00a319,537 after educational discount, including 20 TB external storage (14 TB usable as RAID5), uninterruptable power supply (UPS), tape library for backup, and 3 years\u2019 hardware support. It was housed in a secure air-conditioned server room, where it required the University\u2019s IT Services department for installation and maintenance (and for setting up the tape library in a different building) and the University\u2019s Estates department to set up a new 15A electrical connection for the UPS. Although this did not happen, there was always the fear of down-time at some crucial stage in teaching, which is more problematic for a single server being used by all students than for (say) one desktop computer in a classroom. As well as teaching, the server was used for research purposes, leading to worries about conflicting resource use between students and researchers.\nTo use 4273\u03c0, the investment in hardware per set of equipment, including the Raspberry Pi but excluding the monitor, before any quantity or educational discount is ~ \u00a3147-\u00a3159, depending on what exactly is bought and from where (Additional file 1: Table S1; Background). This is far cheaper than the server used in 2008\u20132009, but is not extremely cheap. However, these are mostly one-off expenses, since the equipment may be re-used; some components are likely already present in an educational establishment; the maximum cost of any one part is no more than \u00a331.20, allowing cheap repairs compared to \u2018the laptop approach\u2019; and the cost of some parts of the equipment (e.g. SD card) continues to fall noticeably. Among its other advantages, the Raspberry Pi approach is a low-cost method for bioinformatics teaching and learning. \n\nOpen learning \nBy including an explicit Open Access licence, and removing or replacing material incompatible with this from 4273\u03c0 Bioinformatics for Biologists, we have been able to share it with anyone interested, the world over, in such a way that they can \u2014 with minimal care \u2014 re-use and adapt it without accusation of plagiarism or copyright violation. This approach is broadly in common with the pioneering EcoEd Digital Library[36] and related portals[37], but is in contrast to most of the teaching material that can be found by an online search, for which the licence is unclear. We expect our approach will lead to mutual benefits, for example the contribution of corrections or teaching material by others. As Open Access publication is becoming more standard for research, we predict that Open Access will become more standard for teaching material. \n\nAvailability and requirements \nProject name: 4273\u03c0\nProject home page: http:\/\/eggg.st-andrews.ac.uk\/4273pi \nOperating Systems: Linux\nOther requirements: Raspberry Pi computer hardware\nLicence: 4273\u03c0 Bioinformatics for Biologists has a Creative Commons Attribution licence (http:\/\/creativecommons.org\/licenses\/by\/2.0)\nAny restrictions to use by non-academics: no\n\nEndnotes \naPrices in British Pounds (GBP), including UK tax but excluding delivery charges, obtained on 26 June 2013. \u00a31 converts to approximately $1.54 US Dollars, \u20ac1.18 Euro, or R93.04 Indian Rupees.[38]\n\nAbbreviations \nAMD: Advanced Micro Devices\nAPT: A package tool\nBSc(Hons): Bachelor of Science with Honours\nCPU: Central processing unit\nECTS: European credit transfer and accumulation system\nGB: Gigabyte\nGNU: GNU\u2019s not UNIX!\nGPU: Graphics processing unit\nIBM: International Business Machines\nIT: Information technology\nJAR: Java archive\nMHz: Megahertz\nNCBI: National Center for Biotechnology Information\nNEBC: NERC Environmental Bioinformatics Centre\nNERC: Natural Environment Research Council\nPC: Personal computer\nRAID5: Redundant array of independent disks, level 5\nRAM: Random-access memory\nSCOTCAT: Scottish Credit Accumulation and Transfer\nscp: Secure copy\nSD: Secure digital\nssh: secure shell\nTB: Terabyte\nUSB: Universal serial bus\nUPS: Uninterruptable power supply\nUS: United States\nVM: Virtual machine\nX11: The X Window System\n\nCompeting interests \nThe authors declare that they have no competing interests.\n\nAuthors\u2019 contributions \nDB conceived the study, and wrote the manuscript with contributions from other authors. DB, DEKF, PWHH, JBOM and MGR designed and wrote teaching material, which was edited by DB and HP. DB configured Raspbian and prepared the 4273\u03c0 release, with contributions from SDS and HP. All authors read and approved the final manuscript.\n\nAcknowledgements \nLosia Lagisz contributed to the lecture material on Looking at Species Differences. Ian Grieve and Lianne Baker assisted with the task of obtaining sufficient Raspberry Pi computers, during summer and autumn 2012 when they were scarce. All students of BL4273 Bioinformatics for Biologists at the University of St Andrews are thanked for testing teaching material, particularly those in academic year 2012\u20132013, who used the Raspberry Pi. Their contribution has been invaluable. We thank the developers of Raspbian Linux, upon which 4273\u03c0 is based, for their rapid and professional work. Clare Peddie\u2019s enthusiasm (as Director of Teaching for the School of Biology, University of St Andrews) allowed the necessary equipment purchases to go ahead. JBOM is grateful to the Scottish Universities Life Sciences Alliance (SULSA) for financial support and thanks the BBSRC for funding his research through grant BB\/I00596X\/1. The University of St Andrews provided funding for the Open Access charge.\n\nReferences \n\n\n\u2191 1.0 1.1 Altschul, S.F.; Madden, T.L.; Sch\u00e4ffer, A.A.; Zhang, J.; Zhang, Z.; Miller, W.; Lipman, D.J. (1997). \"Gapped BLAST and PSI-BLAST: a new generation of protein database search programs\". Nucleic Acids Research 25 (17): 3389-3402. doi:10.1093\/nar\/25.17.3389. PMC PMC146917. PMID 9254694. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917 .   \n\n\u2191 \"BLAST: Basic Local Alignment Search Tool\". National Center for Biotechnology Information. http:\/\/blast.ncbi.nlm.nih.gov\/Blast.cgi .   \n\n\u2191 \"The GNU Operating Systems and the Free Software Movement\". Free Software Movement, Inc. http:\/\/www.gnu.org\/ .   \n\n\u2191 Field, D.; Tiwari, B. Booth, T.; Houten, S.; Swan, D.; Bertrand, N.; Thurston, M. (2006). \"Open software for biologists: from famine to feast\". Nature Biotechnology 24 (7): 801\u2013803. doi:10.1038\/nbt0706-801. PMID 16841067.   \n\n\u2191 Bassi, S.; Gonzalez, V.C. (2007). \"DNALinux Virtual Desktop Edition\". Nature Precedings. doi:10.1038\/npre.2007.670.1.   \n\n\u2191 6.0 6.1 \"Bio-Linux 7\". Environmental Omics Network. http:\/\/nebc.nerc.ac.uk\/tools\/bio-linux\/live-usbkey .   \n\n\u2191 Yu, G.; Wang, L.G.; Meng, X.H.; He, Q.Y. (2012). \"LXtoo: an integrated live Linux distribution for the bioinformatics community\". BMC Research Notes 5: 360. doi:10.1186\/1756-0500-5-360. PMC PMC3461469. PMID 22813356. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3461469 .   \n\n\u2191 \"Raspberry Pi\". Raspberry Pi Foundation. https:\/\/www.raspberrypi.org\/ .   \n\n\u2191 \"The Raspberry Pi computer goes on general sale\". BBC. 29 February 2012. http:\/\/www.bbc.com\/news\/technology-17190918 .   \n\n\u2191 \"Raspberry Pi\". Element14 Community. Premier Farnell plc. http:\/\/www.element14.com\/community\/community\/raspberry-pi .   \n\n\u2191 \"Raspberry Pi\". RS Components Ltd. http:\/\/uk.rs-online.com\/web\/generalDisplay.html?id=raspberrypi .   \n\n\u2191 Roberts, Jonathan (01 May 2012). \"Is the Raspberry Pi the future of computing?\". TechRadar Pro. Future Publishing Limited. http:\/\/www.techradar.com\/news\/computing\/pc\/is-the-raspberry-pi-the-future-of-computing-1078276 .   \n\n\u2191 \"ARM: The Architecture for the Digital World\". ARM Ltd. http:\/\/www.arm.com\/ .   \n\n\u2191 Latif, Lawrence (05 April 2013). \"ARM sees its 32-bit chips being deployed in future servers: Not everything needs 64-bit addressing\". The Inquirer. Incisive Business Media Limited. http:\/\/www.theinquirer.net\/inquirer\/news\/2259386\/arm-sees-its-32bit-chips-being-deployed-in-future-servers .   \n\n\u2191 Nic (19 November 2012). \"A Cray for $35\". 2000 Nickels. http:\/\/2000nickels.com\/blog\/2012\/11\/19\/a-cray-for-35-dollars\/ .   \n\n\u2191 \"Raspberry Pi - Operating system distributions\". Raspberry Pi Foundation. https:\/\/www.raspberrypi.org\/forums\/viewforum.php?f=18 .   \n\n\u2191 17.0 17.1 \"Raspbian\". Mike Thompson and the Raspbian community. http:\/\/www.raspbian.org\/ .   \n\n\u2191 18.0 18.1 \"Debian\". Software in the Public Interest, Inc. http:\/\/www.debian.org\/ .   \n\n\u2191 19.0 19.1 M\u00f6ller, S.; Krabbenh\u00f6ft, H.N.; Tille, A.; Paleino, D.; Williams, A.; Wolstencroft, K.; Goble, C.; Holland, R.; Belhachemi, D.; Plessy, C. (2010). \"Community-driven computational biology with Debian Linux\". BMC Bioinformatics 11 (Suppl 12): S5. doi:10.1186\/1471-2105-11-S12-S5. PMC PMC3040531. PMID 21210984. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040531 .   \n\n\u2191 O'Boyle, Noel (19 January 2013). \"Chemistrify your Raspberry Pi Part III\". Noel O'Blog. http:\/\/baoilleach.blogspot.co.uk\/2013\/01\/chemistrify-your-raspberry-pi-part-iii_19.html .   \n\n\u2191 Upton, Eben (24 May 2013). \"Wayland project\". Raspberry Pi Blog. http:\/\/www.raspberrypi.org\/archives\/4053 .   \n\n\u2191 \"Undergraduate Course Catalogue 2012-2013\". University of St. Andrews. 11 December 2012. http:\/\/www.st-andrews.ac.uk\/coursecatalogue\/ug\/2012-2013\/ .   \n\n\u2191 Bradnam, Keith; Korf, Ian (2012). UNIX and Perl to the Rescue! - A Field Guide for the Life Sciences (and Other Data-rich Pursuits). Cambridge, England: Cambridge University Press. pp. 428. ISBN 9780521169820. http:\/\/www.cambridge.org\/us\/academic\/subjects\/life-sciences\/genomics-bioinformatics-and-systems-biology\/unix-and-perl-rescue-field-guide-life-sciences-and-other-data-rich-pursuits .   \n\n\u2191 \"Downloads\". Raspberry Pi Foundation. https:\/\/www.raspberrypi.org\/downloads\/ .   \n\n\u2191 25.0 25.1 \"4273\u03c0\". University of St. Andrews. http:\/\/4273pi.org\/ .   \n\n\u2191 26.0 26.1 Korf, I. (2004). \"Gene finding in novel genomes\". BMC Bioinformatics 14 (5): 59. doi:10.1186\/1471-2105-5-59. PMC PMC421630. PMID 15144565. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630 .   \n\n\u2191 Birney, E.; Clamp, M.; Durbin, R. (2004). \"GeneWise and Genomewise\". Genome Research 14 (5): 988-995. doi:10.1101\/gr.1865504. PMC PMC479130. PMID 15123596. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130 .   \n\n\u2191 \"GeneWise\". European Bioinformatics Institute. http:\/\/www.ebi.ac.uk\/Tools\/psa\/genewise\/ .   \n\n\u2191 Marygold, S.J.; Leyland, P.C.; Seal, R.L.; Goodman, J.L.; Thurmond, J.; Strelets, V.B.; Wilson, R.J.; FlyBase Consortium (2013). \"FlyBase: improvements to the bibliography\". Nucleic Acids Research 41 (Database issue): D751-7. doi:10.1093\/nar\/gks1024. PMC PMC3531214. PMID 23125371. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531214 .   \n\n\u2191 Gardiner, A.; Barker, D.; Butlin, R.K.; Jordan, W.C.; Ritchie, M.G. (2008). \"Drosophila chemoreceptor gene evolution: selection, specialization and genome size\". Molecular Ecology 17 (7): 1648-57. doi:10.1111\/j.1365-294X.2008.03713.x. PMID 18371013.   \n\n\u2191 Chambers, R. (1844). Vestiges of the natural history of creation. London, England: John Churchill.   \n\n\u2191 Yang, Z. (2007). \"PAML 4: Phylogenetic analysis by maximum likelihood\". Molecular Biology and Evolution 24 (8): 1586-91. doi:10.1093\/molbev\/msm088. PMID 17483113.   \n\n\u2191 Keane, T.M.; Creevey, C.J.; Pentony, M.P.; Naughton, T.J.; Mclnerney, J.O. (2006). \"Assessment of methods for amino acid matrix selection and their use on empirical data shows that ad hoc assumptions for choice of matrix are not justified\". BMC Evolutionary Biology 6: 29. doi:10.1186\/1471-2148-6-29. PMC PMC1435933. PMID 16563161. http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1435933 .   \n\n\u2191 Maddison, W.P.; Maddison, D.R. (2011). \"Mesquite\". Tangient, LLC. http:\/\/mesquiteproject.org\/ .   \n\n\u2191 Guindon, S.; Dufayard, J.F.; Lefort, V.; Anisimova, M.; Hordijk, W.; Gascuel, O. (2010). \"New algorithms and methods to estimate maximum-likelihood phylogenies: assessing the performance of PhyML 3.0\". Systematic Biology 59 (3): 307-21. doi:10.1093\/sysbio\/syq010. PMID 20525638.   \n\n\u2191 \"EcoEd Digital Library\". Ecological Society of America. http:\/\/ecoed.esa.org\/ .   \n\n\u2191 \"EvoEd Digital Library\". Society for the Study of Evolution. http:\/\/evoed.evolutionsociety.org\/index.php?P=DRD_People .   \n\n\u2191 \"XE Currency Converter\". XE.com, Inc. http:\/\/www.xe.com\/currencyconverter\/ .   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\">https:\/\/www.limswiki.org\/index.php\/Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on bioinformaticsLIMSwiki journal articles on education\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:50.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 3,535 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","1916b796a2be08f4f1d2fd01df17d5fa_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_4273\u03c0_Bioinformatics_education_on_low_cost_ARM_hardware skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:4273\u03c0: Bioinformatics education on low cost ARM hardware<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p><b>Background:<\/b> Teaching <a href=\"https:\/\/www.limswiki.org\/index.php\/Bioinformatics\" title=\"Bioinformatics\" target=\"_blank\" class=\"wiki-link\" data-key=\"8f506695fdbb26e3f314da308f8c053b\">bioinformatics<\/a> at universities is complicated by typical computer classroom settings. As well as running software locally and online, students should gain experience of systems administration. For a future career in biology or bioinformatics, the installation of software is a useful skill. We propose that this may be taught by running the course on GNU\/Linux running on inexpensive Raspberry Pi computer hardware, for which students may be granted full administrator access.\n<\/p><p><b>Results:<\/b> We release 4273<i>\u03c0<\/i>, an operating system image for Raspberry Pi based on Raspbian Linux. This includes minor customisations for classroom use and includes our Open Access bioinformatics course, <i>4273\u03c0 Bioinformatics for Biologists<\/i>. This is based on the final-year undergraduate module BL4273, run on Raspberry Pi computers at the University of St Andrews, Semester 1, academic year 2012\u20132013.\n<\/p><p><b>Conclusions:<\/b> 4273<i>\u03c0<\/i> is a means to teach bioinformatics, including systems administration tasks, to undergraduates at low cost.\n<\/p><p><b>Keywords:<\/b> Bioinformatics education; Teaching material; Raspberry Pi; Linux\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Background\">Background<\/span><\/h2>\n<p>Bioinformatics is increasingly included in the undergraduate curriculum for biology students. Teaching bioinformatics is made difficult, however, by the constraints of a typical university computer classroom. Some areas of basic bioinformatics may be taught using such classrooms, where all that is required is an Internet connection and Web browser (e.g. BLAST<sup id=\"rdp-ebb-cite_ref-AltschulBLAST97_1-0\" class=\"reference\"><a href=\"#cite_note-AltschulBLAST97-1\" rel=\"external_link\">[1]<\/a><\/sup> searches at the NCBI<sup id=\"rdp-ebb-cite_ref-NIHBLAST_2-0\" class=\"reference\"><a href=\"#cite_note-NIHBLAST-2\" rel=\"external_link\">[2]<\/a><\/sup>). More in-depth teaching requires the re-creation of a bioinformatics research environment, consisting of a Linux or UNIX operating system, standard GNU utilities<sup id=\"rdp-ebb-cite_ref-FSFGNU_3-0\" class=\"reference\"><a href=\"#cite_note-FSFGNU-3\" rel=\"external_link\">[3]<\/a><\/sup>, specialist bioinformatics software, and sequence databases.\n<\/p><p>Undergraduate modules ought, ideally, to prepare students for research in an academic research group. Students who do pursue a research career will often find that institutional computer support is targeted to generic computer use (e.g. Microsoft software) rather than installing and maintaining systems suitable for bioinformatics. Particularly outside of bioinformatics research (but also occasionally within it), the principal investigator of the research group may never have used Linux, may have a limited idea of the procedures, and may expect group members to \u2018pick things up\u2019 and deal with problems themselves. This requires researchers to have a high level of proficiency with Linux, including the ability to install both standard Linux packages and software for which no standard package may be available. A single taught module cannot prepare a student for all eventualities, but ought to leave the student with the basic skills and confidence to be able to discover solutions, and implement them, as required. Hence, a certain amount of system administration should appear in an undergraduate bioinformatics module for biologists. \n<\/p><p>Traditionally, the environment required for an undergraduate bioinformatics module has been created in one of four ways. Firstly, one may set up a central GNU\/Linux server on the campus and allow students to connect by Secure Shell, ssh (\u2018the server approach\u2019). The server will typically run either a standard Linux distribution or a specialist bioinformatics distribution such as NEBC Bio-Linux.<sup id=\"rdp-ebb-cite_ref-FieldOpen06_4-0\" class=\"reference\"><a href=\"#cite_note-FieldOpen06-4\" rel=\"external_link\">[4]<\/a><\/sup> The server approach allows the instructor to have full control over the server, and allows students to connect from existing computing classrooms with little or no adjustment to the classroom software. For students to connect to the server via the intranet, classroom computers only require an ssh client, the X Window System (X11), and a means of file transfer such as secure copy (scp). Students may also connect to the server from home (typically requiring them to install virtual private network software in addition to ssh, X11 and an scp client) or elsewhere on campus. Secondly, one may provide students with a virtual machine, consisting of an environment similar to that which they might experience on the Linux server but running on a classroom computer, either with a standard Linux distribution or a specialist bioinformatics distribution such as DNA Linux Virtual Desktop Edition<sup id=\"rdp-ebb-cite_ref-BassiDNA07_5-0\" class=\"reference\"><a href=\"#cite_note-BassiDNA07-5\" rel=\"external_link\">[5]<\/a><\/sup> (\u2018the VM approach\u2019). This has the advantage that students may be given administrator access to their virtual machine. Thirdly, one may provide students with a Linux system on removable media (\u2018the USB stick approach\u2019, for example<sup id=\"rdp-ebb-cite_ref-USBBio_6-0\" class=\"reference\"><a href=\"#cite_note-USBBio-6\" rel=\"external_link\">[6]<\/a><\/sup>; where files and settings do not have to be saved, a DVD may be used instead<sup id=\"rdp-ebb-cite_ref-YuLX12_7-0\" class=\"reference\"><a href=\"#cite_note-YuLX12-7\" rel=\"external_link\">[7]<\/a><\/sup>). So long as students have the media to hand, this allows them to boot into \u2018their own\u2019 Linux. As with the VM approach, students may be given administrator access. The additional advantage is that the media may be portable between computer classrooms and home computers, without requiring students to move virtual machine image files. Fourthly, students may be loaned or required to buy laptops of a specific kind, with a suitable operating system, data and software installed (\u2018the laptop approach\u2019). This avoids hardware incompatibilities that the USB stick approach may, in practice, experience.<sup id=\"rdp-ebb-cite_ref-USBBio_6-1\" class=\"reference\"><a href=\"#cite_note-USBBio-6\" rel=\"external_link\">[6]<\/a><\/sup>\n<\/p><p>Because administrator access cannot be allowed, the server approach fails to give students experience of the standard mechanism of software installation. It also involves competition for resources such as CPU time, especially if the class is large or the server is also shared with research colleagues. The VM approach solves both these problems but is less portable. Although, in theory, students may transfer a VM from one computer to another (assuming the destination has the necessary virtualisation software installed), the task is non-trivial, and more time consuming than a simple transfer of data or documents. The USB stick approach reduces the portability problem, since it is trivial to move a USB stick from one computer to another. However, smooth operation on all hardware is not guaranteed and requires ongoing efforts from the developers of the Linux distribution as new hardware is released. The laptop approach avoids all these problems by providing a portable computer holding everything required for the course. However, it is expensive.\n<\/p><p>As a fifth approach, we propose loaning a Raspberry Pi computer<sup id=\"rdp-ebb-cite_ref-RasPi_8-0\" class=\"reference\"><a href=\"#cite_note-RasPi-8\" rel=\"external_link\">[8]<\/a><\/sup> and associated peripherals to students for the duration of the course (\u2018the Raspberry Pi approach\u2019). This includes a customised version of Linux, appropriate software and data. This allows students full administrator access to a suitable operating system, without the difficulties of the VM or USB stick approaches. Should the student accidentally damage critical files, the system can be re-written from a master image.\n<\/p><p>The Raspberry Pi Model B \u2014 with 256 MB (now 512 MB) RAM, an ARM11 CPU running at 700 MHz before over clocking and a Video core IV GPU \u2014 was released for public sale in 2012<sup id=\"rdp-ebb-cite_ref-BBCPi_9-0\" class=\"reference\"><a href=\"#cite_note-BBCPi-9\" rel=\"external_link\">[9]<\/a><\/sup> and costs \u00a328.07<sup>a<\/sup> or \u00a331.20.<sup id=\"rdp-ebb-cite_ref-E14Pi_10-0\" class=\"reference\"><a href=\"#cite_note-E14Pi-10\" rel=\"external_link\">[10]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RSComp_11-0\" class=\"reference\"><a href=\"#cite_note-RSComp-11\" rel=\"external_link\">[11]<\/a><\/sup> Though additional items are required to turn it into a functioning, general-purpose computer (case, charger, SD card, mouse, keyboard, monitor and cable; and an entirely separate computer for initialising the SD card), it is still relatively low-cost (Additional file 1: Table S1). The existence of the Raspberry Pi is partly a celebration of the early days of popular computing in the 1980s, and an attempt to recreate that excitement among young people today.<sup id=\"rdp-ebb-cite_ref-RobertsIs12_12-0\" class=\"reference\"><a href=\"#cite_note-RobertsIs12-12\" rel=\"external_link\">[12]<\/a><\/sup> It is also a symptom of the rapidly decreasing costs and increasing performance of computer hardware. The Raspberry Pi uses an ARM CPU.<sup id=\"rdp-ebb-cite_ref-ARM_13-0\" class=\"reference\"><a href=\"#cite_note-ARM-13\" rel=\"external_link\">[13]<\/a><\/sup> Because of their high performance-per-watt, ARM CPUs are frequently found in small electronic appliances such as mobile phones and tablets. With CPU innovation increasingly driven by such applications, as opposed to more traditional areas such as desktop, laptop and server computers, the prevalence and utility of ARM-based computer hardware is likely to increase. Indeed, ARM-based servers are starting to appear in data centres, due to their modest requirements for power.<sup id=\"rdp-ebb-cite_ref-LatifARM13_14-0\" class=\"reference\"><a href=\"#cite_note-LatifARM13-14\" rel=\"external_link\">[14]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 1. Table S1\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Example prices of Raspberry Pi peripherals we found to work well in practice. These are presented without any endorsement. A case for the Raspberry Pi (various models and suppliers; ~\u00a35-\u00a310), the Raspberry Pi itself (see main text) and a monitor are not shown. Standard consumer prices, including UK tax but excluding any delivery charge, were obtained from the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/uk.insight.com\" target=\"_blank\">Insight UK<\/a> Web site or via the <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.amazon.co.uk\" target=\"_blank\">Amazon UK<\/a> Web site on 7 April 2013.\n<p>Format: PDF; Size: 110KB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-14-243-s1.pdf\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>Though far slower than current desktop and laptop computers, the Raspberry Pi is notably faster than the Cray 1 supercomputer<sup id=\"rdp-ebb-cite_ref-NicACray12_15-0\" class=\"reference\"><a href=\"#cite_note-NicACray12-15\" rel=\"external_link\">[15]<\/a><\/sup>, a marvel of computer speed in its day. The valid question arises as to how much computer power is actually required to teach undergraduates bioinformatics? We propose that the answer is, by current standards, \u2018not much\u2019. The Raspberry Pi is more than adequate for the task. The Raspberry Pi approach includes all the benefits of the laptop approach, above, but at lower cost. In addition, the Raspberry Pi is a new and exciting computer system, which in itself can add interest to the course. \n<\/p><p>A variety of operating systems is available for the Raspberry Pi.<sup id=\"rdp-ebb-cite_ref-PiOS_16-0\" class=\"reference\"><a href=\"#cite_note-PiOS-16\" rel=\"external_link\">[16]<\/a><\/sup> These include Raspbian<sup id=\"rdp-ebb-cite_ref-Raspbian_17-0\" class=\"reference\"><a href=\"#cite_note-Raspbian-17\" rel=\"external_link\">[17]<\/a><\/sup>, which is based on Debian GNU\/Linux.<sup id=\"rdp-ebb-cite_ref-Debian_18-0\" class=\"reference\"><a href=\"#cite_note-Debian-18\" rel=\"external_link\">[18]<\/a><\/sup> Over 35,000 Debian software packages are available pre-compiled for Raspbian, including Web browsers, text editors, word processors, and a wide range of bioinformatics packages.<sup id=\"rdp-ebb-cite_ref-MollerComm10_19-0\" class=\"reference\"><a href=\"#cite_note-MollerComm10-19\" rel=\"external_link\">[19]<\/a><\/sup> Other software will usually compile and run without problems. Some features of recent CPUs (e.g. 64-bit addressing or vector operations) are absent, but we have not found these to be at all necessary for our proposed use of the Raspberry Pi. The most serious limitation, affecting structure visualisation software in particular, is limited graphics performance. Even so, some structural visualisation software does work on the Raspberry Pi.<sup id=\"rdp-ebb-cite_ref-O.27BoyleChem13_20-0\" class=\"reference\"><a href=\"#cite_note-O.27BoyleChem13-20\" rel=\"external_link\">[20]<\/a><\/sup> New system software, improving graphics performance by making better use of the Raspberry Pi\u2019s GPU, is under development.<sup id=\"rdp-ebb-cite_ref-UptonWay13_21-0\" class=\"reference\"><a href=\"#cite_note-UptonWay13-21\" rel=\"external_link\">[21]<\/a><\/sup>\n<\/p><p>We provide 4273<i>\u03c0<\/i>, a customised version of Linux for Raspberry Pi computer hardware. 4273<i>\u03c0<\/i> includes an Open Access bioinformatics course, <i>4273\u03c0 Bioinformatics for Biologists<\/i>. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Implementation\">Implementation<\/span><\/h2>\n<p><i>4273\u03c0 Bioinformatics for Biologists<\/i> is based on the module BL4273 Bioinformatics for Biologists at the University of St Andrews<sup id=\"rdp-ebb-cite_ref-Andrews1213_22-0\" class=\"reference\"><a href=\"#cite_note-Andrews1213-22\" rel=\"external_link\">[22]<\/a><\/sup>, an optional module of 15 SCOTCAT credits, equivalent to 7.5 ECTS credits or ~4 US credits. BL4273 is intended for final-year undergraduate students on BSc(Hons) Biology, BSc(Hons) Biochemistry and other degree courses taught by the School of Biology. BL4273 was taught on Raspberry Pi hardware in Semester 1 of academic year 2012\u20132013. During this time, students were loaned a Raspberry Pi, SD card and USB stick for backup. Teaching was carried out in a small computer room in which students connected to monitors more generally used with Windows desktop PCs. Students were allowed to take all equipment home on loan (including keyboard, mouse and cables), with the exception of the monitors; several copies of the main textbook<sup id=\"rdp-ebb-cite_ref-BradnamUNIX12_23-0\" class=\"reference\"><a href=\"#cite_note-BradnamUNIX12-23\" rel=\"external_link\">[23]<\/a><\/sup> were also available. Course material was released week-by-week using an rsync server (running on a Raspberry Pi) on the university intranet. Following the conclusion of the module, material was edited to remove closed-access material (e.g. images in lectures), typographical errors were corrected, and the SD card image was re-created using a recent version of Raspbian.<sup id=\"rdp-ebb-cite_ref-Raspbian_17-1\" class=\"reference\"><a href=\"#cite_note-Raspbian-17\" rel=\"external_link\">[17]<\/a><\/sup>\n<\/p><p>Preparation of a release of 4273<i>\u03c0<\/i> begins with the latest Raspbian SD card image.<sup id=\"rdp-ebb-cite_ref-RPDown_24-0\" class=\"reference\"><a href=\"#cite_note-RPDown-24\" rel=\"external_link\">[24]<\/a><\/sup> This is then customised to produce a \u2018master\u2019 SD card for the release, partly by a series of scripts which alter the configuration and use Raspbian\u2019s port of the APT mechanism of Debian<sup id=\"rdp-ebb-cite_ref-Debian_18-1\" class=\"reference\"><a href=\"#cite_note-Debian-18\" rel=\"external_link\">[18]<\/a><\/sup> to install specialist bioinformatics packages<sup id=\"rdp-ebb-cite_ref-MollerComm10_19-1\" class=\"reference\"><a href=\"#cite_note-MollerComm10-19\" rel=\"external_link\">[19]<\/a><\/sup> and more general packages, and partly by a series of commands entered manually (e.g. to install BLAST databases and <i>4273\u03c0 Bioinformatics for Biologists<\/i> in the ~\/4273pi\/ directory). The master SD card image is stored on a separate computer and uploaded to the 4273<i>\u03c0<\/i> Web site. A \u2018work instruction\u2019 detailing the steps performed to convert Raspbian into 4273<i>\u03c0<\/i>, and all scripts used, are distributed with 4273<i>\u03c0<\/i>.\n<\/p><p>For the permanent record, the teaching material included in the current release \u2013 excluding Linux, software and BLAST databases \u2013 is available as Additional file 2. The latest version may be downloaded from the 4273<i>\u03c0<\/i> Web site.<sup id=\"rdp-ebb-cite_ref-4273pi_25-0\" class=\"reference\"><a href=\"#cite_note-4273pi-25\" rel=\"external_link\">[25]<\/a><\/sup>\n<\/p>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<th> Additional file 2. <i>4273\u03c0 Bioinformatics for Biologists<\/i> teaching material, Version 1.01\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">The module handbook, lectures and practicals are included. The latest version, including Linux, software and BLAST databases, is available at the 4273<i>\u03c0<\/i> Web site<sup id=\"rdp-ebb-cite_ref-4273pi_25-1\" class=\"reference\"><a href=\"#cite_note-4273pi-25\" rel=\"external_link\">[25]<\/a><\/sup>\n<p>Format: ZIP; Size: 7MB <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.biomedcentral.com\/content\/supplementary\/1471-2105-14-243-s2.zip\" target=\"_blank\">Download file<\/a>\n<\/p>\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<p>As the Raspbian operating system, Raspberry Pi firmware and hardware and <i>4273\u03c0 Bioinformatics for Biologists<\/i> teaching material develop, further releases of 4273<i>\u03c0<\/i> will be made available. It is anticipated that there will be a minimum of two releases per year during the next four years. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Results_and_discussion\">Results and discussion<\/span><\/h2>\n<p>4273<i>\u03c0<\/i> provides an attractive, general-purpose computing environment, within which the course <i>4273\u03c0 Bioinformatics for Biologists<\/i> is embedded (Figure 1; Table 1). Student feedback questionnaires for the module BL4273, upon which it is based, are too few to draw strong conclusions (five returned from a class of six students; Additional file 1: Table S2). However, they were sufficiently positive to continue the module without modification. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Barker_BMCBio2013_14.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5b9e402b64945b9881130505804bc9ae\"><img alt=\"Fig1 Barker BMCBio2013 14.jpg\" src=\"https:\/\/www.limswiki.org\/images\/0\/03\/Fig1_Barker_BMCBio2013_14.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1: 4273<i>\u03c0<\/i> screenshot.<\/b> Practical, Week 7. Predicted gene structures of the Or98b gene in the Drosophila sechellia genome are being compared. Different predictions are made de novo by SNAP<sup id=\"rdp-ebb-cite_ref-KorfGene04_26-0\" class=\"reference\"><a href=\"#cite_note-KorfGene04-26\" rel=\"external_link\">[26]<\/a><\/sup>, run in a terminal on the Raspberry Pi; GeneWise<sup id=\"rdp-ebb-cite_ref-BirneyGene04_27-0\" class=\"reference\"><a href=\"#cite_note-BirneyGene04-27\" rel=\"external_link\">[27]<\/a><\/sup>, run online at the European Bioinformatics Institute<sup id=\"rdp-ebb-cite_ref-GWEBI_28-0\" class=\"reference\"><a href=\"#cite_note-GWEBI-28\" rel=\"external_link\">[28]<\/a><\/sup>, with the D. melanogaster protein being aligned against D. sechellia genomic DNA; and using TBLASTN<sup id=\"rdp-ebb-cite_ref-AltschulBLAST97_1-1\" class=\"reference\"><a href=\"#cite_note-AltschulBLAST97-1\" rel=\"external_link\">[1]<\/a><\/sup>, run online at FlyBase.<sup id=\"rdp-ebb-cite_ref-MarygoldFly13_29-0\" class=\"reference\"><a href=\"#cite_note-MarygoldFly13-29\" rel=\"external_link\">[29]<\/a><\/sup> FlyBase also displays the annotation of this gene from its database. This case study arose during research for a comparative study of the chemoreceptor superfamily in Drosophila.<sup id=\"rdp-ebb-cite_ref-GardinerDros08_30-0\" class=\"reference\"><a href=\"#cite_note-GardinerDros08-30\" rel=\"external_link\">[30]<\/a><\/sup> The 4273<i>\u03c0<\/i> background desktop image includes hand-written versions of material from <i>4273\u03c0 Bioinformatics for Biologists<\/i> and from the module BL3320 Practical Statistics for Biologists, which most students at the University of St Andrews would have taken earlier. It also includes a copy of one of the first published phylogenetic trees, from <i>Vestiges of the Natural History of Creation<\/i>, written in St Andrews.<sup id=\"rdp-ebb-cite_ref-ChambersVest1844_31-0\" class=\"reference\"><a href=\"#cite_note-ChambersVest1844-31\" rel=\"external_link\">[31]<\/a><\/sup><\/blockquote>\n<table style=\"\">\n<tr>\n<td style=\"vertical-align:top;\">\n<table class=\"wikitable\" border=\"1\" cellpadding=\"5\" cellspacing=\"0\" style=\"\">\n\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\"><b>Table 1.<\/b> Timetable for <i>4273\u03c0 Bioinformatics for Biologists<\/i>\n<\/td><\/tr>\n<tr>\n<th> Week\n<\/th>\n<th> Lecture\n<\/th>\n<th> Practical class\n<\/th><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">1\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Genomes, sequences and bioinformatics data\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Linux and Perl\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">2\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Linux, Perl and protein BLAST\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">3\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Linux, Perl and delimiting gene\/protein families\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">4\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple alignment and phylogeny\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Multiple alignment and phylogeny\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">5\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Gene family evolution\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Gene family evolution\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">6\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">BLAST; DNA sequence analysis\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">DNA sequence analysis\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">7\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Looking at species differences\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Detecting positive selection\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">8\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Function and evolution of enzymes\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Function and evolution of enzymes\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">9\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">Seminar: student presentations\n<\/td>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\">-\n<\/td><\/tr>\n<tr>\n<td style=\"background-color:white; padding-left:10px; padding-right:10px;\" colspan=\"8\">Lectures last slightly less than an hour. Practical classes all use the Raspberry Pi, and last two hours. In addition, in their own time, students engage in self-study and complete a coursework \u2018practical project\u2019.\n<\/td><\/tr>\n<\/table>\n<\/td><\/tr><\/table>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<h3><span class=\"mw-headline\" id=\".28Administrator.29_power_to_the_people\">(Administrator) power to the people<\/span><\/h3>\n<p>Rather than present systems administration as a complex task to be delegated to a technical support unit \u2014 which in students\u2019 future careers may not be available \u2014 <i>4273\u03c0 Bioinformatics for Biologists<\/i> introduces software installation on Linux through standard package management (APT), through compilation (GNU \u2018make\u2019 for SNAP<sup id=\"rdp-ebb-cite_ref-KorfGene04_26-1\" class=\"reference\"><a href=\"#cite_note-KorfGene04-26\" rel=\"external_link\">[26]<\/a><\/sup> and PAML<sup id=\"rdp-ebb-cite_ref-YangPAML07_32-0\" class=\"reference\"><a href=\"#cite_note-YangPAML07-32\" rel=\"external_link\">[32]<\/a><\/sup>) and through command-line launch of a JAR (Modelgenerator<sup id=\"rdp-ebb-cite_ref-KeaneAss06_33-0\" class=\"reference\"><a href=\"#cite_note-KeaneAss06-33\" rel=\"external_link\">[33]<\/a><\/sup> and Mesquite<sup id=\"rdp-ebb-cite_ref-Mesquite_34-0\" class=\"reference\"><a href=\"#cite_note-Mesquite-34\" rel=\"external_link\">[34]<\/a><\/sup>). Other administrative tasks covered include upgrading Linux and installing a MySQL server. Although these matters of system administration are incidental to an intellectual understanding of bioinformatics, they do not take much time to teach, and we believe they leave students well-prepared for a bioinformatics research career \u2013 in many cases, better prepared than any other members of the research group within which they are working. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"The_virtue_of_patience\">The virtue of patience<\/span><\/h3>\n<p>No serious compromises in content were required to teach bioinformatics on the Raspberry Pi. A BLAST search of the GenPept database (\u2018nr\u2019) is, in practice, too slow. However a BLAST search of the SwissProt database takes only a few minutes. Delimitation of protein families across two prokaryotic genomes using BLAST and OrthoMCL is entirely feasible, and is central to the coursework component of <i>4273\u03c0 Bioinformatics for Biologists<\/i>. Java (Modelgenerator, Mesquite) programs run slowly, but not unbearably so. Speed will likely improve once Oracle Java becomes available for the platform. Floating-point-intensive tasks (PhyML<sup id=\"rdp-ebb-cite_ref-GuindonNew10_35-0\" class=\"reference\"><a href=\"#cite_note-GuindonNew10-35\" rel=\"external_link\">[35]<\/a><\/sup> and PAML) are also slow, but feasible. Waiting an hour for an analysis is, in fact, realistic training for bioinformatics research. Research will tend to use a far faster computer or cluster, but with far larger input. Teaching bioinformatics on rather slow hardware was not universally popular among students (Additional file 1: Table S2). However, it is a valuable lesson in the transferrable virtue of patience, and the value of checking the configuration of analyses before launching them. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Low-cost_teaching_and_learning\">Low-cost teaching and learning<\/span><\/h3>\n<p>When the undergraduate module BL4273 at the University of St Andrews ran previously, in Semester 1 of academic year 2008\u20132009, it used the server approach. Students used desktop computers running Windows in a computer classroom to connect to a 64-bit IBM System x3755 8877 server with 4 AMD CPU cores and 16 GB RAM, running Debian Linux. When purchased in 2008, this server cost \u00a319,537 after educational discount, including 20 TB external storage (14 TB usable as RAID5), uninterruptable power supply (UPS), tape library for backup, and 3 years\u2019 hardware support. It was housed in a secure air-conditioned server room, where it required the University\u2019s IT Services department for installation and maintenance (and for setting up the tape library in a different building) and the University\u2019s Estates department to set up a new 15A electrical connection for the UPS. Although this did not happen, there was always the fear of down-time at some crucial stage in teaching, which is more problematic for a single server being used by all students than for (say) one desktop computer in a classroom. As well as teaching, the server was used for research purposes, leading to worries about conflicting resource use between students and researchers.\n<\/p><p>To use 4273<i>\u03c0<\/i>, the investment in hardware per set of equipment, including the Raspberry Pi but excluding the monitor, before any quantity or educational discount is ~ \u00a3147-\u00a3159, depending on what exactly is bought and from where (Additional file 1: Table S1; Background). This is far cheaper than the server used in 2008\u20132009, but is not extremely cheap. However, these are mostly one-off expenses, since the equipment may be re-used; some components are likely already present in an educational establishment; the maximum cost of any one part is no more than \u00a331.20, allowing cheap repairs compared to \u2018the laptop approach\u2019; and the cost of some parts of the equipment (e.g. SD card) continues to fall noticeably. Among its other advantages, the Raspberry Pi approach is a low-cost method for bioinformatics teaching and learning. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Open_learning\">Open learning<\/span><\/h3>\n<p>By including an explicit Open Access licence, and removing or replacing material incompatible with this from <i>4273\u03c0 Bioinformatics for Biologists<\/i>, we have been able to share it with anyone interested, the world over, in such a way that they can \u2014 with minimal care \u2014 re-use and adapt it without accusation of plagiarism or copyright violation. This approach is broadly in common with the pioneering EcoEd Digital Library<sup id=\"rdp-ebb-cite_ref-EcoEdDL_36-0\" class=\"reference\"><a href=\"#cite_note-EcoEdDL-36\" rel=\"external_link\">[36]<\/a><\/sup> and related portals<sup id=\"rdp-ebb-cite_ref-EvoEdDL_37-0\" class=\"reference\"><a href=\"#cite_note-EvoEdDL-37\" rel=\"external_link\">[37]<\/a><\/sup>, but is in contrast to most of the teaching material that can be found by an online search, for which the licence is unclear. We expect our approach will lead to mutual benefits, for example the contribution of corrections or teaching material by others. As Open Access publication is becoming more standard for research, we predict that Open Access will become more standard for teaching material. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Availability_and_requirements\">Availability and requirements<\/span><\/h2>\n<p><b>Project name:<\/b> 4273<i>\u03c0<\/i>\n<\/p><p><b>Project home page:<\/b> <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/eggg.st-andrews.ac.uk\/4273pi\" target=\"_blank\">http:\/\/eggg.st-andrews.ac.uk\/4273pi<\/a> \n<\/p><p><b>Operating Systems:<\/b> Linux\n<\/p><p><b>Other requirements:<\/b> Raspberry Pi computer hardware\n<\/p><p><b>Licence:<\/b> <i>4273\u03c0 Bioinformatics for Biologists<\/i> has a Creative Commons Attribution licence (<a rel=\"external_link\" class=\"external free\" href=\"http:\/\/creativecommons.org\/licenses\/by\/2.0\" target=\"_blank\">http:\/\/creativecommons.org\/licenses\/by\/2.0<\/a>)\n<\/p><p><b>Any restrictions to use by non-academics:<\/b> no\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Endnotes\">Endnotes<\/span><\/h2>\n<p><sup>a<\/sup>Prices in British Pounds (GBP), including UK tax but excluding delivery charges, obtained on 26 June 2013. \u00a31 converts to approximately $1.54 US Dollars, \u20ac1.18 Euro, or R93.04 Indian Rupees.<sup id=\"rdp-ebb-cite_ref-XEConv_38-0\" class=\"reference\"><a href=\"#cite_note-XEConv-38\" rel=\"external_link\">[38]<\/a><\/sup>\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Abbreviations\">Abbreviations<\/span><\/h2>\n<p>AMD: Advanced Micro Devices\n<\/p><p>APT: A package tool\n<\/p><p>BSc(Hons): Bachelor of Science with Honours\n<\/p><p>CPU: Central processing unit\n<\/p><p>ECTS: European credit transfer and accumulation system\n<\/p><p>GB: Gigabyte\n<\/p><p>GNU: GNU\u2019s not UNIX!\n<\/p><p>GPU: Graphics processing unit\n<\/p><p>IBM: International Business Machines\n<\/p><p>IT: Information technology\n<\/p><p>JAR: Java archive\n<\/p><p>MHz: Megahertz\n<\/p><p>NCBI: National Center for Biotechnology Information\n<\/p><p>NEBC: NERC Environmental Bioinformatics Centre\n<\/p><p>NERC: Natural Environment Research Council\n<\/p><p>PC: Personal computer\n<\/p><p>RAID5: Redundant array of independent disks, level 5\n<\/p><p>RAM: Random-access memory\n<\/p><p>SCOTCAT: Scottish Credit Accumulation and Transfer\n<\/p><p>scp: Secure copy\n<\/p><p>SD: Secure digital\n<\/p><p>ssh: secure shell\n<\/p><p>TB: Terabyte\n<\/p><p>USB: Universal serial bus\n<\/p><p>UPS: Uninterruptable power supply\n<\/p><p>US: United States\n<\/p><p>VM: Virtual machine\n<\/p><p>X11: The X Window System\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Competing_interests\">Competing interests<\/span><\/h2>\n<p>The authors declare that they have no competing interests.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Authors.E2.80.99_contributions\">Authors\u2019 contributions<\/span><\/h2>\n<p>DB conceived the study, and wrote the manuscript with contributions from other authors. DB, DEKF, PWHH, JBOM and MGR designed and wrote teaching material, which was edited by DB and HP. DB configured Raspbian and prepared the 4273\u03c0 release, with contributions from SDS and HP. All authors read and approved the final manuscript.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Acknowledgements\">Acknowledgements<\/span><\/h2>\n<p>Losia Lagisz contributed to the lecture material on Looking at Species Differences. Ian Grieve and Lianne Baker assisted with the task of obtaining sufficient Raspberry Pi computers, during summer and autumn 2012 when they were scarce. All students of BL4273 Bioinformatics for Biologists at the University of St Andrews are thanked for testing teaching material, particularly those in academic year 2012\u20132013, who used the Raspberry Pi. Their contribution has been invaluable. We thank the developers of Raspbian Linux, upon which 4273<i>\u03c0<\/i> is based, for their rapid and professional work. Clare Peddie\u2019s enthusiasm (as Director of Teaching for the School of Biology, University of St Andrews) allowed the necessary equipment purchases to go ahead. JBOM is grateful to the Scottish Universities Life Sciences Alliance (SULSA) for financial support and thanks the BBSRC for funding his research through grant BB\/I00596X\/1. The University of St Andrews provided funding for the Open Access charge.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<div class=\"reflist references-column-width\" style=\"-moz-column-width: 30em; -webkit-column-width: 30em; column-width: 30em; list-style-type: decimal;\">\n<ol class=\"references\">\n<li id=\"cite_note-AltschulBLAST97-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AltschulBLAST97_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-AltschulBLAST97_1-1\" rel=\"external_link\">1.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Altschul, S.F.; Madden, T.L.; Sch\u00e4ffer, A.A.; Zhang, J.; Zhang, Z.; Miller, W.; Lipman, D.J. (1997). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917\" target=\"_blank\">\"Gapped BLAST and PSI-BLAST: a new generation of protein database search programs\"<\/a>. <i>Nucleic Acids Research<\/i> <b>25<\/b> (17): 3389-3402. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2F25.17.3389\" target=\"_blank\">10.1093\/nar\/25.17.3389<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC146917\/\" target=\"_blank\">PMC146917<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/9254694\" target=\"_blank\">9254694<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC146917<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gapped+BLAST+and+PSI-BLAST%3A+a+new+generation+of+protein+database+search+programs&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Altschul%2C+S.F.%3B+Madden%2C+T.L.%3B+Sch%C3%A4ffer%2C+A.A.%3B+Zhang%2C+J.%3B+Zhang%2C+Z.%3B+Miller%2C+W.%3B+Lipman%2C+D.J.&rft.au=Altschul%2C+S.F.%3B+Madden%2C+T.L.%3B+Sch%C3%A4ffer%2C+A.A.%3B+Zhang%2C+J.%3B+Zhang%2C+Z.%3B+Miller%2C+W.%3B+Lipman%2C+D.J.&rft.date=1997&rft.volume=25&rft.issue=17&rft.pages=3389-3402&rft_id=info:doi\/10.1093%2Fnar%2F25.17.3389&rft_id=info:pmc\/PMC146917&rft_id=info:pmid\/9254694&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC146917&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NIHBLAST-2\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NIHBLAST_2-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/blast.ncbi.nlm.nih.gov\/Blast.cgi\" target=\"_blank\">\"BLAST: Basic Local Alignment Search Tool\"<\/a>. National Center for Biotechnology Information<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/blast.ncbi.nlm.nih.gov\/Blast.cgi\" target=\"_blank\">http:\/\/blast.ncbi.nlm.nih.gov\/Blast.cgi<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=BLAST%3A+Basic+Local+Alignment+Search+Tool&rft.atitle=&rft.pub=National+Center+for+Biotechnology+Information&rft_id=http%3A%2F%2Fblast.ncbi.nlm.nih.gov%2FBlast.cgi&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FSFGNU-3\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FSFGNU_3-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.gnu.org\/\" target=\"_blank\">\"The GNU Operating Systems and the Free Software Movement\"<\/a>. Free Software Movement, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.gnu.org\/\" target=\"_blank\">http:\/\/www.gnu.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+GNU+Operating+Systems+and+the+Free+Software+Movement&rft.atitle=&rft.pub=Free+Software+Movement%2C+Inc&rft_id=http%3A%2F%2Fwww.gnu.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FieldOpen06-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FieldOpen06_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Field, D.; Tiwari, B. Booth, T.; Houten, S.; Swan, D.; Bertrand, N.; Thurston, M. (2006). \"Open software for biologists: from famine to feast\". <i>Nature Biotechnology<\/i> <b>24<\/b> (7): 801\u2013803. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnbt0706-801\" target=\"_blank\">10.1038\/nbt0706-801<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16841067\" target=\"_blank\">16841067<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Open+software+for+biologists%3A+from+famine+to+feast&rft.jtitle=Nature+Biotechnology&rft.aulast=Field%2C+D.%3B+Tiwari%2C+B.+Booth%2C+T.%3B+Houten%2C+S.%3B+Swan%2C+D.%3B+Bertrand%2C+N.%3B+Thurston%2C+M.&rft.au=Field%2C+D.%3B+Tiwari%2C+B.+Booth%2C+T.%3B+Houten%2C+S.%3B+Swan%2C+D.%3B+Bertrand%2C+N.%3B+Thurston%2C+M.&rft.date=2006&rft.volume=24&rft.issue=7&rft.pages=801%E2%80%93803&rft_id=info:doi\/10.1038%2Fnbt0706-801&rft_id=info:pmid\/16841067&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BassiDNA07-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BassiDNA07_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Bassi, S.; Gonzalez, V.C. (2007). \"DNALinux Virtual Desktop Edition\". <i>Nature Precedings<\/i>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1038%2Fnpre.2007.670.1\" target=\"_blank\">10.1038\/npre.2007.670.1<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=DNALinux+Virtual+Desktop+Edition&rft.jtitle=Nature+Precedings&rft.aulast=Bassi%2C+S.%3B+Gonzalez%2C+V.C.&rft.au=Bassi%2C+S.%3B+Gonzalez%2C+V.C.&rft.date=2007&rft_id=info:doi\/10.1038%2Fnpre.2007.670.1&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-USBBio-6\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-USBBio_6-0\" rel=\"external_link\">6.0<\/a><\/sup> <sup><a href=\"#cite_ref-USBBio_6-1\" rel=\"external_link\">6.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/nebc.nerc.ac.uk\/tools\/bio-linux\/live-usbkey\" target=\"_blank\">\"Bio-Linux 7\"<\/a>. Environmental Omics Network<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/nebc.nerc.ac.uk\/tools\/bio-linux\/live-usbkey\" target=\"_blank\">http:\/\/nebc.nerc.ac.uk\/tools\/bio-linux\/live-usbkey<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Bio-Linux+7&rft.atitle=&rft.pub=Environmental+Omics+Network&rft_id=http%3A%2F%2Fnebc.nerc.ac.uk%2Ftools%2Fbio-linux%2Flive-usbkey&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YuLX12-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YuLX12_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yu, G.; Wang, L.G.; Meng, X.H.; He, Q.Y. (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3461469\" target=\"_blank\">\"LXtoo: an integrated live Linux distribution for the bioinformatics community\"<\/a>. <i>BMC Research Notes<\/i> <b>5<\/b>: 360. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1756-0500-5-360\" target=\"_blank\">10.1186\/1756-0500-5-360<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3461469\/\" target=\"_blank\">PMC3461469<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/22813356\" target=\"_blank\">22813356<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3461469\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3461469<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=LXtoo%3A+an+integrated+live+Linux+distribution+for+the+bioinformatics+community&rft.jtitle=BMC+Research+Notes&rft.aulast=Yu%2C+G.%3B+Wang%2C+L.G.%3B+Meng%2C+X.H.%3B+He%2C+Q.Y.&rft.au=Yu%2C+G.%3B+Wang%2C+L.G.%3B+Meng%2C+X.H.%3B+He%2C+Q.Y.&rft.date=2012&rft.volume=5&rft.pages=360&rft_id=info:doi\/10.1186%2F1756-0500-5-360&rft_id=info:pmc\/PMC3461469&rft_id=info:pmid\/22813356&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3461469&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RasPi-8\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RasPi_8-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.raspberrypi.org\/\" target=\"_blank\">\"Raspberry Pi\"<\/a>. Raspberry Pi Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.raspberrypi.org\/\" target=\"_blank\">https:\/\/www.raspberrypi.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Raspberry+Pi&rft.atitle=&rft.pub=Raspberry+Pi+Foundation&rft_id=https%3A%2F%2Fwww.raspberrypi.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BBCPi-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BBCPi_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.bbc.com\/news\/technology-17190918\" target=\"_blank\">\"The Raspberry Pi computer goes on general sale\"<\/a>. BBC. 29 February 2012<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.bbc.com\/news\/technology-17190918\" target=\"_blank\">http:\/\/www.bbc.com\/news\/technology-17190918<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=The+Raspberry+Pi+computer+goes+on+general+sale&rft.atitle=&rft.date=29+February+2012&rft.pub=BBC&rft_id=http%3A%2F%2Fwww.bbc.com%2Fnews%2Ftechnology-17190918&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-E14Pi-10\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-E14Pi_10-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.element14.com\/community\/community\/raspberry-pi\" target=\"_blank\">\"Raspberry Pi\"<\/a>. <i>Element14 Community<\/i>. Premier Farnell plc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.element14.com\/community\/community\/raspberry-pi\" target=\"_blank\">http:\/\/www.element14.com\/community\/community\/raspberry-pi<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Raspberry+Pi&rft.atitle=Element14+Community&rft.pub=Premier+Farnell+plc&rft_id=http%3A%2F%2Fwww.element14.com%2Fcommunity%2Fcommunity%2Fraspberry-pi&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RSComp-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RSComp_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/uk.rs-online.com\/web\/generalDisplay.html?id=raspberrypi\" target=\"_blank\">\"Raspberry Pi\"<\/a>. RS Components Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/uk.rs-online.com\/web\/generalDisplay.html?id=raspberrypi\" target=\"_blank\">http:\/\/uk.rs-online.com\/web\/generalDisplay.html?id=raspberrypi<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Raspberry+Pi&rft.atitle=&rft.pub=RS+Components+Ltd&rft_id=http%3A%2F%2Fuk.rs-online.com%2Fweb%2FgeneralDisplay.html%3Fid%3Draspberrypi&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RobertsIs12-12\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RobertsIs12_12-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Roberts, Jonathan (01 May 2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.techradar.com\/news\/computing\/pc\/is-the-raspberry-pi-the-future-of-computing-1078276\" target=\"_blank\">\"Is the Raspberry Pi the future of computing?\"<\/a>. <i>TechRadar Pro<\/i>. Future Publishing Limited<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.techradar.com\/news\/computing\/pc\/is-the-raspberry-pi-the-future-of-computing-1078276\" target=\"_blank\">http:\/\/www.techradar.com\/news\/computing\/pc\/is-the-raspberry-pi-the-future-of-computing-1078276<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Is+the+Raspberry+Pi+the+future+of+computing%3F&rft.atitle=TechRadar+Pro&rft.aulast=Roberts%2C+Jonathan&rft.au=Roberts%2C+Jonathan&rft.date=01+May+2012&rft.pub=Future+Publishing+Limited&rft_id=http%3A%2F%2Fwww.techradar.com%2Fnews%2Fcomputing%2Fpc%2Fis-the-raspberry-pi-the-future-of-computing-1078276&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ARM-13\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ARM_13-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.arm.com\/\" target=\"_blank\">\"ARM: The Architecture for the Digital World\"<\/a>. ARM Ltd<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.arm.com\/\" target=\"_blank\">http:\/\/www.arm.com\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ARM%3A+The+Architecture+for+the+Digital+World&rft.atitle=&rft.pub=ARM+Ltd&rft_id=http%3A%2F%2Fwww.arm.com%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-LatifARM13-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-LatifARM13_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Latif, Lawrence (05 April 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.theinquirer.net\/inquirer\/news\/2259386\/arm-sees-its-32bit-chips-being-deployed-in-future-servers\" target=\"_blank\">\"ARM sees its 32-bit chips being deployed in future servers: Not everything needs 64-bit addressing\"<\/a>. <i>The Inquirer<\/i>. Incisive Business Media Limited<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.theinquirer.net\/inquirer\/news\/2259386\/arm-sees-its-32bit-chips-being-deployed-in-future-servers\" target=\"_blank\">http:\/\/www.theinquirer.net\/inquirer\/news\/2259386\/arm-sees-its-32bit-chips-being-deployed-in-future-servers<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=ARM+sees+its+32-bit+chips+being+deployed+in+future+servers%3A+Not+everything+needs+64-bit+addressing&rft.atitle=The+Inquirer&rft.aulast=Latif%2C+Lawrence&rft.au=Latif%2C+Lawrence&rft.date=05+April+2013&rft.pub=Incisive+Business+Media+Limited&rft_id=http%3A%2F%2Fwww.theinquirer.net%2Finquirer%2Fnews%2F2259386%2Farm-sees-its-32bit-chips-being-deployed-in-future-servers&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-NicACray12-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-NicACray12_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Nic (19 November 2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/2000nickels.com\/blog\/2012\/11\/19\/a-cray-for-35-dollars\/\" target=\"_blank\">\"A Cray for $35\"<\/a>. <i>2000 Nickels<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/2000nickels.com\/blog\/2012\/11\/19\/a-cray-for-35-dollars\/\" target=\"_blank\">http:\/\/2000nickels.com\/blog\/2012\/11\/19\/a-cray-for-35-dollars\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=A+Cray+for+%2435&rft.atitle=2000+Nickels&rft.aulast=Nic&rft.au=Nic&rft.date=19+November+2012&rft_id=http%3A%2F%2F2000nickels.com%2Fblog%2F2012%2F11%2F19%2Fa-cray-for-35-dollars%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PiOS-16\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PiOS_16-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.raspberrypi.org\/forums\/viewforum.php?f=18\" target=\"_blank\">\"Raspberry Pi - Operating system distributions\"<\/a>. Raspberry Pi Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.raspberrypi.org\/forums\/viewforum.php?f=18\" target=\"_blank\">https:\/\/www.raspberrypi.org\/forums\/viewforum.php?f=18<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Raspberry+Pi+-+Operating+system+distributions&rft.atitle=&rft.pub=Raspberry+Pi+Foundation&rft_id=https%3A%2F%2Fwww.raspberrypi.org%2Fforums%2Fviewforum.php%3Ff%3D18&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Raspbian-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Raspbian_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-Raspbian_17-1\" rel=\"external_link\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.raspbian.org\/\" target=\"_blank\">\"Raspbian\"<\/a>. Mike Thompson and the Raspbian community<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.raspbian.org\/\" target=\"_blank\">http:\/\/www.raspbian.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Raspbian&rft.atitle=&rft.pub=Mike+Thompson+and+the+Raspbian+community&rft_id=http%3A%2F%2Fwww.raspbian.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Debian-18\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-Debian_18-0\" rel=\"external_link\">18.0<\/a><\/sup> <sup><a href=\"#cite_ref-Debian_18-1\" rel=\"external_link\">18.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.debian.org\/\" target=\"_blank\">\"Debian\"<\/a>. Software in the Public Interest, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.debian.org\/\" target=\"_blank\">http:\/\/www.debian.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Debian&rft.atitle=&rft.pub=Software+in+the+Public+Interest%2C+Inc&rft_id=http%3A%2F%2Fwww.debian.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MollerComm10-19\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MollerComm10_19-0\" rel=\"external_link\">19.0<\/a><\/sup> <sup><a href=\"#cite_ref-MollerComm10_19-1\" rel=\"external_link\">19.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">M\u00f6ller, S.; Krabbenh\u00f6ft, H.N.; Tille, A.; Paleino, D.; Williams, A.; Wolstencroft, K.; Goble, C.; Holland, R.; Belhachemi, D.; Plessy, C. (2010). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040531\" target=\"_blank\">\"Community-driven computational biology with Debian Linux\"<\/a>. <i>BMC Bioinformatics<\/i> <b>11<\/b> (Suppl 12): S5. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-11-S12-S5\" target=\"_blank\">10.1186\/1471-2105-11-S12-S5<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3040531\/\" target=\"_blank\">PMC3040531<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/21210984\" target=\"_blank\">21210984<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040531\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3040531<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Community-driven+computational+biology+with+Debian+Linux&rft.jtitle=BMC+Bioinformatics&rft.aulast=M%C3%B6ller%2C+S.%3B+Krabbenh%C3%B6ft%2C+H.N.%3B+Tille%2C+A.%3B+Paleino%2C+D.%3B+Williams%2C+A.%3B+Wolstencroft%2C+K.%3B+Goble%2C+C.%3B+Holland%2C+R.%3B+Belhachemi%2C+D.%3B+Plessy%2C+C.&rft.au=M%C3%B6ller%2C+S.%3B+Krabbenh%C3%B6ft%2C+H.N.%3B+Tille%2C+A.%3B+Paleino%2C+D.%3B+Williams%2C+A.%3B+Wolstencroft%2C+K.%3B+Goble%2C+C.%3B+Holland%2C+R.%3B+Belhachemi%2C+D.%3B+Plessy%2C+C.&rft.date=2010&rft.volume=11&rft.issue=Suppl+12&rft.pages=S5&rft_id=info:doi\/10.1186%2F1471-2105-11-S12-S5&rft_id=info:pmc\/PMC3040531&rft_id=info:pmid\/21210984&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3040531&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-O.27BoyleChem13-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-O.27BoyleChem13_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">O'Boyle, Noel (19 January 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/baoilleach.blogspot.co.uk\/2013\/01\/chemistrify-your-raspberry-pi-part-iii_19.html\" target=\"_blank\">\"Chemistrify your Raspberry Pi Part III\"<\/a>. <i>Noel O'Blog<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/baoilleach.blogspot.co.uk\/2013\/01\/chemistrify-your-raspberry-pi-part-iii_19.html\" target=\"_blank\">http:\/\/baoilleach.blogspot.co.uk\/2013\/01\/chemistrify-your-raspberry-pi-part-iii_19.html<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Chemistrify+your+Raspberry+Pi+Part+III&rft.atitle=Noel+O%27Blog&rft.aulast=O%27Boyle%2C+Noel&rft.au=O%27Boyle%2C+Noel&rft.date=19+January+2013&rft_id=http%3A%2F%2Fbaoilleach.blogspot.co.uk%2F2013%2F01%2Fchemistrify-your-raspberry-pi-part-iii_19.html&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-UptonWay13-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-UptonWay13_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Upton, Eben (24 May 2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.raspberrypi.org\/archives\/4053\" target=\"_blank\">\"Wayland project\"<\/a>. <i>Raspberry Pi Blog<\/i><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.raspberrypi.org\/archives\/4053\" target=\"_blank\">http:\/\/www.raspberrypi.org\/archives\/4053<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Wayland+project&rft.atitle=Raspberry+Pi+Blog&rft.aulast=Upton%2C+Eben&rft.au=Upton%2C+Eben&rft.date=24+May+2013&rft_id=http%3A%2F%2Fwww.raspberrypi.org%2Farchives%2F4053&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Andrews1213-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Andrews1213_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.st-andrews.ac.uk\/coursecatalogue\/ug\/2012-2013\/\" target=\"_blank\">\"Undergraduate Course Catalogue 2012-2013\"<\/a>. University of St. Andrews. 11 December 2012<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.st-andrews.ac.uk\/coursecatalogue\/ug\/2012-2013\/\" target=\"_blank\">http:\/\/www.st-andrews.ac.uk\/coursecatalogue\/ug\/2012-2013\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Undergraduate+Course+Catalogue+2012-2013&rft.atitle=&rft.date=11+December+2012&rft.pub=University+of+St.+Andrews&rft_id=http%3A%2F%2Fwww.st-andrews.ac.uk%2Fcoursecatalogue%2Fug%2F2012-2013%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BradnamUNIX12-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BradnamUNIX12_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Bradnam, Keith; Korf, Ian (2012). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cambridge.org\/us\/academic\/subjects\/life-sciences\/genomics-bioinformatics-and-systems-biology\/unix-and-perl-rescue-field-guide-life-sciences-and-other-data-rich-pursuits\" target=\"_blank\"><i>UNIX and Perl to the Rescue! - A Field Guide for the Life Sciences (and Other Data-rich Pursuits)<\/i><\/a>. Cambridge, England: Cambridge University Press. pp. 428. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/International_Standard_Book_Number\" target=\"_blank\">ISBN<\/a> 9780521169820<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cambridge.org\/us\/academic\/subjects\/life-sciences\/genomics-bioinformatics-and-systems-biology\/unix-and-perl-rescue-field-guide-life-sciences-and-other-data-rich-pursuits\" target=\"_blank\">http:\/\/www.cambridge.org\/us\/academic\/subjects\/life-sciences\/genomics-bioinformatics-and-systems-biology\/unix-and-perl-rescue-field-guide-life-sciences-and-other-data-rich-pursuits<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=UNIX+and+Perl+to+the+Rescue%21+-+A+Field+Guide+for+the+Life+Sciences+%28and+Other+Data-rich+Pursuits%29&rft.aulast=Bradnam%2C+Keith%3B+Korf%2C+Ian&rft.au=Bradnam%2C+Keith%3B+Korf%2C+Ian&rft.date=2012&rft.pages=pp.%26nbsp%3B428&rft.place=Cambridge%2C+England&rft.pub=Cambridge+University+Press&rft.isbn=9780521169820&rft_id=http%3A%2F%2Fwww.cambridge.org%2Fus%2Facademic%2Fsubjects%2Flife-sciences%2Fgenomics-bioinformatics-and-systems-biology%2Funix-and-perl-rescue-field-guide-life-sciences-and-other-data-rich-pursuits&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RPDown-24\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RPDown_24-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"https:\/\/www.raspberrypi.org\/downloads\/\" target=\"_blank\">\"Downloads\"<\/a>. Raspberry Pi Foundation<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"https:\/\/www.raspberrypi.org\/downloads\/\" target=\"_blank\">https:\/\/www.raspberrypi.org\/downloads\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Downloads&rft.atitle=&rft.pub=Raspberry+Pi+Foundation&rft_id=https%3A%2F%2Fwww.raspberrypi.org%2Fdownloads%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-4273pi-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-4273pi_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-4273pi_25-1\" rel=\"external_link\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/4273pi.org\/\" target=\"_blank\">\"4273<i>\u03c0<\/i>\"<\/a>. University of St. Andrews<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/4273pi.org\/\" target=\"_blank\">http:\/\/4273pi.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=4273%27%27%CF%80%27%27&rft.atitle=&rft.pub=University+of+St.+Andrews&rft_id=http%3A%2F%2F4273pi.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KorfGene04-26\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KorfGene04_26-0\" rel=\"external_link\">26.0<\/a><\/sup> <sup><a href=\"#cite_ref-KorfGene04_26-1\" rel=\"external_link\">26.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Korf, I. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630\" target=\"_blank\">\"Gene finding in novel genomes\"<\/a>. <i>BMC Bioinformatics<\/i> <b>14<\/b> (5): 59. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2105-5-59\" target=\"_blank\">10.1186\/1471-2105-5-59<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC421630\/\" target=\"_blank\">PMC421630<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15144565\" target=\"_blank\">15144565<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC421630<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Gene+finding+in+novel+genomes&rft.jtitle=BMC+Bioinformatics&rft.aulast=Korf%2C+I.&rft.au=Korf%2C+I.&rft.date=2004&rft.volume=14&rft.issue=5&rft.pages=59&rft_id=info:doi\/10.1186%2F1471-2105-5-59&rft_id=info:pmc\/PMC421630&rft_id=info:pmid\/15144565&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC421630&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-BirneyGene04-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-BirneyGene04_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Birney, E.; Clamp, M.; Durbin, R. (2004). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130\" target=\"_blank\">\"GeneWise and Genomewise\"<\/a>. <i>Genome Research<\/i> <b>14<\/b> (5): 988-995. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1101%2Fgr.1865504\" target=\"_blank\">10.1101\/gr.1865504<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC479130\/\" target=\"_blank\">PMC479130<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/15123596\" target=\"_blank\">15123596<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC479130<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=GeneWise+and+Genomewise&rft.jtitle=Genome+Research&rft.aulast=Birney%2C+E.%3B+Clamp%2C+M.%3B+Durbin%2C+R.&rft.au=Birney%2C+E.%3B+Clamp%2C+M.%3B+Durbin%2C+R.&rft.date=2004&rft.volume=14&rft.issue=5&rft.pages=988-995&rft_id=info:doi\/10.1101%2Fgr.1865504&rft_id=info:pmc\/PMC479130&rft_id=info:pmid\/15123596&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC479130&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GWEBI-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GWEBI_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ebi.ac.uk\/Tools\/psa\/genewise\/\" target=\"_blank\">\"GeneWise\"<\/a>. European Bioinformatics Institute<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.ebi.ac.uk\/Tools\/psa\/genewise\/\" target=\"_blank\">http:\/\/www.ebi.ac.uk\/Tools\/psa\/genewise\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=GeneWise&rft.atitle=&rft.pub=European+Bioinformatics+Institute&rft_id=http%3A%2F%2Fwww.ebi.ac.uk%2FTools%2Fpsa%2Fgenewise%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MarygoldFly13-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MarygoldFly13_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Marygold, S.J.; Leyland, P.C.; Seal, R.L.; Goodman, J.L.; Thurmond, J.; Strelets, V.B.; Wilson, R.J.; FlyBase Consortium (2013). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531214\" target=\"_blank\">\"FlyBase: improvements to the bibliography\"<\/a>. <i>Nucleic Acids Research<\/i> <b>41<\/b> (Database issue): D751-7. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fnar%2Fgks1024\" target=\"_blank\">10.1093\/nar\/gks1024<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3531214\/\" target=\"_blank\">PMC3531214<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/23125371\" target=\"_blank\">23125371<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531214\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC3531214<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=FlyBase%3A+improvements+to+the+bibliography&rft.jtitle=Nucleic+Acids+Research&rft.aulast=Marygold%2C+S.J.%3B+Leyland%2C+P.C.%3B+Seal%2C+R.L.%3B+Goodman%2C+J.L.%3B+Thurmond%2C+J.%3B+Strelets%2C+V.B.%3B+Wilson%2C+R.J.%3B+FlyBase+Consortium&rft.au=Marygold%2C+S.J.%3B+Leyland%2C+P.C.%3B+Seal%2C+R.L.%3B+Goodman%2C+J.L.%3B+Thurmond%2C+J.%3B+Strelets%2C+V.B.%3B+Wilson%2C+R.J.%3B+FlyBase+Consortium&rft.date=2013&rft.volume=41&rft.issue=Database+issue&rft.pages=D751-7&rft_id=info:doi\/10.1093%2Fnar%2Fgks1024&rft_id=info:pmc\/PMC3531214&rft_id=info:pmid\/23125371&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC3531214&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GardinerDros08-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GardinerDros08_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gardiner, A.; Barker, D.; Butlin, R.K.; Jordan, W.C.; Ritchie, M.G. (2008). \"<i>Drosophila<\/i> chemoreceptor gene evolution: selection, specialization and genome size\". <i>Molecular Ecology<\/i> <b>17<\/b> (7): 1648-57. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1111%2Fj.1365-294X.2008.03713.x\" target=\"_blank\">10.1111\/j.1365-294X.2008.03713.x<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/18371013\" target=\"_blank\">18371013<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%27%27Drosophila%27%27+chemoreceptor+gene+evolution%3A+selection%2C+specialization+and+genome+size&rft.jtitle=Molecular+Ecology&rft.aulast=Gardiner%2C+A.%3B+Barker%2C+D.%3B+Butlin%2C+R.K.%3B+Jordan%2C+W.C.%3B+Ritchie%2C+M.G.&rft.au=Gardiner%2C+A.%3B+Barker%2C+D.%3B+Butlin%2C+R.K.%3B+Jordan%2C+W.C.%3B+Ritchie%2C+M.G.&rft.date=2008&rft.volume=17&rft.issue=7&rft.pages=1648-57&rft_id=info:doi\/10.1111%2Fj.1365-294X.2008.03713.x&rft_id=info:pmid\/18371013&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ChambersVest1844-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ChambersVest1844_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Chambers, R. (1844). <i>Vestiges of the natural history of creation<\/i>. London, England: John Churchill.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Vestiges+of+the+natural+history+of+creation&rft.aulast=Chambers%2C+R.&rft.au=Chambers%2C+R.&rft.date=1844&rft.place=London%2C+England&rft.pub=John+Churchill&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-YangPAML07-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-YangPAML07_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Yang, Z. (2007). \"PAML 4: Phylogenetic analysis by maximum likelihood\". <i>Molecular Biology and Evolution<\/i> <b>24<\/b> (8): 1586-91. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fmolbev%2Fmsm088\" target=\"_blank\">10.1093\/molbev\/msm088<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/17483113\" target=\"_blank\">17483113<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=PAML+4%3A+Phylogenetic+analysis+by+maximum+likelihood&rft.jtitle=Molecular+Biology+and+Evolution&rft.aulast=Yang%2C+Z.&rft.au=Yang%2C+Z.&rft.date=2007&rft.volume=24&rft.issue=8&rft.pages=1586-91&rft_id=info:doi\/10.1093%2Fmolbev%2Fmsm088&rft_id=info:pmid\/17483113&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KeaneAss06-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KeaneAss06_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Keane, T.M.; Creevey, C.J.; Pentony, M.P.; Naughton, T.J.; Mclnerney, J.O. (2006). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1435933\" target=\"_blank\">\"Assessment of methods for amino acid matrix selection and their use on empirical data shows that ad hoc assumptions for choice of matrix are not justified\"<\/a>. <i>BMC Evolutionary Biology<\/i> <b>6<\/b>: 29. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1186%2F1471-2148-6-29\" target=\"_blank\">10.1186\/1471-2148-6-29<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Central\" target=\"_blank\">PMC<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC1435933\/\" target=\"_blank\">PMC1435933<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/16563161\" target=\"_blank\">16563161<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1435933\" target=\"_blank\">http:\/\/www.pubmedcentral.nih.gov\/articlerender.fcgi?tool=pmcentrez&artid=PMC1435933<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Assessment+of+methods+for+amino+acid+matrix+selection+and+their+use+on+empirical+data+shows+that+ad+hoc+assumptions+for+choice+of+matrix+are+not+justified&rft.jtitle=BMC+Evolutionary+Biology&rft.aulast=Keane%2C+T.M.%3B+Creevey%2C+C.J.%3B+Pentony%2C+M.P.%3B+Naughton%2C+T.J.%3B+Mclnerney%2C+J.O.&rft.au=Keane%2C+T.M.%3B+Creevey%2C+C.J.%3B+Pentony%2C+M.P.%3B+Naughton%2C+T.J.%3B+Mclnerney%2C+J.O.&rft.date=2006&rft.volume=6&rft.pages=29&rft_id=info:doi\/10.1186%2F1471-2148-6-29&rft_id=info:pmc\/PMC1435933&rft_id=info:pmid\/16563161&rft_id=http%3A%2F%2Fwww.pubmedcentral.nih.gov%2Farticlerender.fcgi%3Ftool%3Dpmcentrez%26artid%3DPMC1435933&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-Mesquite-34\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-Mesquite_34-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Maddison, W.P.; Maddison, D.R. (2011). <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/mesquiteproject.org\/\" target=\"_blank\">\"Mesquite\"<\/a>. Tangient, LLC<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/mesquiteproject.org\/\" target=\"_blank\">http:\/\/mesquiteproject.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Mesquite&rft.atitle=&rft.aulast=Maddison%2C+W.P.%3B+Maddison%2C+D.R.&rft.au=Maddison%2C+W.P.%3B+Maddison%2C+D.R.&rft.date=2011&rft.pub=Tangient%2C+LLC&rft_id=http%3A%2F%2Fmesquiteproject.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GuindonNew10-35\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GuindonNew10_35-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Guindon, S.; Dufayard, J.F.; Lefort, V.; Anisimova, M.; Hordijk, W.; Gascuel, O. (2010). \"New algorithms and methods to estimate maximum-likelihood phylogenies: assessing the performance of PhyML 3.0\". <i>Systematic Biology<\/i> <b>59<\/b> (3): 307-21. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/Digital_object_identifier\" target=\"_blank\">doi<\/a>:<a rel=\"external_link\" class=\"external text\" href=\"http:\/\/dx.doi.org\/10.1093%2Fsysbio%2Fsyq010\" target=\"_blank\">10.1093\/sysbio\/syq010<\/a>. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/en.wikipedia.org\/wiki\/PubMed_Identifier\" target=\"_blank\">PMID<\/a> <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.ncbi.nlm.nih.gov\/pubmed\/20525638\" target=\"_blank\">20525638<\/a>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=New+algorithms+and+methods+to+estimate+maximum-likelihood+phylogenies%3A+assessing+the+performance+of+PhyML+3.0&rft.jtitle=Systematic+Biology&rft.aulast=Guindon%2C+S.%3B+Dufayard%2C+J.F.%3B+Lefort%2C+V.%3B+Anisimova%2C+M.%3B+Hordijk%2C+W.%3B+Gascuel%2C+O.&rft.au=Guindon%2C+S.%3B+Dufayard%2C+J.F.%3B+Lefort%2C+V.%3B+Anisimova%2C+M.%3B+Hordijk%2C+W.%3B+Gascuel%2C+O.&rft.date=2010&rft.volume=59&rft.issue=3&rft.pages=307-21&rft_id=info:doi\/10.1093%2Fsysbio%2Fsyq010&rft_id=info:pmid\/20525638&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EcoEdDL-36\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EcoEdDL_36-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/ecoed.esa.org\/\" target=\"_blank\">\"EcoEd Digital Library\"<\/a>. Ecological Society of America<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/ecoed.esa.org\/\" target=\"_blank\">http:\/\/ecoed.esa.org\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=EcoEd+Digital+Library&rft.atitle=&rft.pub=Ecological+Society+of+America&rft_id=http%3A%2F%2Fecoed.esa.org%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EvoEdDL-37\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EvoEdDL_37-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/evoed.evolutionsociety.org\/index.php?P=DRD_People\" target=\"_blank\">\"EvoEd Digital Library\"<\/a>. Society for the Study of Evolution<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/evoed.evolutionsociety.org\/index.php?P=DRD_People\" target=\"_blank\">http:\/\/evoed.evolutionsociety.org\/index.php?P=DRD_People<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=EvoEd+Digital+Library&rft.atitle=&rft.pub=Society+for+the+Study+of+Evolution&rft_id=http%3A%2F%2Fevoed.evolutionsociety.org%2Findex.php%3FP%3DDRD_People&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-XEConv-38\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-XEConv_38-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\"><a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.xe.com\/currencyconverter\/\" target=\"_blank\">\"XE Currency Converter\"<\/a>. XE.com, Inc<span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.xe.com\/currencyconverter\/\" target=\"_blank\">http:\/\/www.xe.com\/currencyconverter\/<\/a><\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=XE+Currency+Converter&rft.atitle=&rft.pub=XE.com%2C+Inc&rft_id=http%3A%2F%2Fwww.xe.com%2Fcurrencyconverter%2F&rfr_id=info:sid\/en.wikipedia.org:Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol><\/div>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210215\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.799 seconds\nReal time usage: 0.836 seconds\nPreprocessor visited node count: 26514\/1000000\nPreprocessor generated node count: 37368\/1000000\nPost\u2010expand include size: 175743\/2097152 bytes\nTemplate argument size: 57926\/2097152 bytes\nHighest expansion depth: 18\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 806.959 1 - -total\n 86.34% 696.693 1 - Template:Reflist\n 74.61% 602.071 38 - Template:Citation\/core\n 41.04% 331.146 24 - Template:Cite_web\n 33.88% 273.394 12 - Template:Cite_journal\n 8.00% 64.569 1 - Template:Infobox_journal_article\n 7.58% 61.182 1 - Template:Infobox\n 6.07% 48.985 31 - Template:Citation\/identifier\n 4.69% 37.827 44 - Template:Citation\/make_link\n 4.64% 37.415 2 - Template:Cite_book\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7839-0!*!0!!en!5!* and timestamp 20181213210214 and revision id 23401\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware\">https:\/\/www.limswiki.org\/index.php\/Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","1916b796a2be08f4f1d2fd01df17d5fa_images":["https:\/\/www.limswiki.org\/images\/0\/03\/Fig1_Barker_BMCBio2013_14.jpg"],"1916b796a2be08f4f1d2fd01df17d5fa_timestamp":1544734934,"f514955508ff84c1a3bca10dd93a3266":{"type":"chapter","title":"1. Bioinformatics","key":"f514955508ff84c1a3bca10dd93a3266"}},"link":"https:\/\/www.limswiki.org\/index.php\/Book:LIMSjournal:_2015_Edition","price_currency":"","price_amount":"","book_size":"","download_url":"https:\/\/www.limsforum.com?ebb_action=book_download&book_id=78004","language":"","cta_button_content":"","toc":[{"type":"chapter","name":"1. Bioinformatics","id":"f514955508ff84c1a3bca10dd93a3266","children":[{"type":"article","name":"4273\u03c0: Bioinformatics education on low cost ARM hardware (Barker et al. 2013)","id":"1916b796a2be08f4f1d2fd01df17d5fa","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:4273%CF%80:_Bioinformatics_education_on_low_cost_ARM_hardware"},{"type":"article","name":"adLIMS: A customized open source software that allows bridging clinical and basic molecular research studies (Calabria et al. 2015)","id":"b55d160b7541498bbeb5c39c2bdb347d","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:AdLIMS:_A_customized_open_source_software_that_allows_bridging_clinical_and_basic_molecular_research_studies"},{"type":"article","name":"iLAP: A workflow-driven software for experimental protocol development, data acquisition and analysis (Stocker et al. 2009)","id":"59816fe6940deb7e43cf53893a6ea71a","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:ILAP:_A_workflow-driven_software_for_experimental_protocol_development,_data_acquisition_and_analysis"},{"type":"article","name":"Launching genomics into the cloud: Deployment of Mercury, a next generation sequence analysis pipeline (Reid et al. 2014)","id":"a5e9f45d3e7fa0bbfb4350821efaa31e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Launching_genomics_into_the_cloud:_Deployment_of_Mercury,_a_next_generation_sequence_analysis_pipeline"},{"type":"article","name":"MendeLIMS: A web-based laboratory information management system for clinical genome sequencing (Grimes et al. 2014)","id":"0ea139cdae16f62db0066623f3ff445c","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:MendeLIMS:_A_web-based_laboratory_information_management_system_for_clinical_genome_sequencing"},{"type":"article","name":"Personalized Oncology Suite: Integrating next-generation sequencing data and whole-slide bioimages (Dander et al. 2014)","id":"03ae650a31def9630dc93ec76a1927f6","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Personalized_Oncology_Suite:_Integrating_next-generation_sequencing_data_and_whole-slide_bioimages"},{"type":"article","name":"Requirements for data integration platforms in biomedical research networks: A reference model (Ganzinger et al. 2015)","id":"0a86ce45c610776ca0038eb4e754f3c5","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Requirements_for_data_integration_platforms_in_biomedical_research_networks:_A_reference_model"},{"type":"article","name":"SaDA: From sampling to data analysis\u2014An extensible open source infrastructure for rapid, robust and automated management and analysis of modern ecological high-throughput microarray data (Singh et al. 2015)","id":"9847a6beea0bba9383335c5a41e65647","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:SaDA:_From_sampling_to_data_analysis%E2%80%94An_extensible_open_source_infrastructure_for_rapid,_robust_and_automated_management_and_analysis_of_modern_ecological_high-throughput_microarray_data"},{"type":"article","name":"SeqWare Query Engine: Storing and searching sequence data in the cloud (O\u2019Connor et al. 2010)","id":"8d2093f9b49a2cde0a0f2b14e9a154a1","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:SeqWare_Query_Engine:_Storing_and_searching_sequence_data_in_the_cloud"},{"type":"article","name":"University-level practical activities in bioinformatics benefit voluntary groups of pupils in the last 2 years of school (Barker et al. 2015)","id":"e85e2a721257f9447a4d6f2238effb07","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:University-level_practical_activities_in_bioinformatics_benefit_voluntary_groups_of_pupils_in_the_last_2_years_of_school"}]},{"type":"chapter","name":"2. Health informatics","id":"e8741c9012e81ca525e09827f947d4d0","children":[{"type":"article","name":"Analyzing huge pathology images with open source software (Deroulers et al. 2013)","id":"89de2c69e566bc890426667a52566bb6","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Analyzing_huge_pathology_images_with_open_source_software"},{"type":"article","name":"Basics of case report form designing in clinical research (Bellary et al. 2014)","id":"548bc0dcb34959573b02fdcd77beb67e","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Basics_of_case_report_form_designing_in_clinical_research"},{"type":"article","name":"Beyond information retrieval and electronic health record use: Competencies in clinical informatics for medical education (Hersh et al. 2014)","id":"43a50f3587a6a8e1ea355835ec07b4a1","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Beyond_information_retrieval_and_electronic_health_record_use:_Competencies_in_clinical_informatics_for_medical_education"},{"type":"article","name":"Factors associated with adoption of health information technology: A conceptual model based on a systematic review (Kruse et al. 2014)","id":"3eb9d25818bfd5a80515d585fb6a6712","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Factors_associated_with_adoption_of_health_information_technology:_A_conceptual_model_based_on_a_systematic_review"},{"type":"article","name":"Incorporating domain knowledge in chemical and biomedical named entity recognition with word representations (Munkhdalai et al. 2015)","id":"fee95fa41e5e6f66dc7579d2ee634b76","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Incorporating_domain_knowledge_in_chemical_and_biomedical_named_entity_recognition_with_word_representations"},{"type":"article","name":"Making big data useful for health care: A summary of the inaugural MIT Critical Data Conference (Badawi et al. 2014)","id":"bcdf1ad9d7bcb8661bc008ff0ca9fb02","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Making_big_data_useful_for_health_care:_A_summary_of_the_inaugural_MIT_Critical_Data_Conference"},{"type":"article","name":"Medical informatics specialty in the developed English-speaking countries: The terminology comparative analysis (Kobryn 2015)","id":"80b7c208090cc5cb594724fd09a469c6","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Medical_informatics_specialty_in_the_developed_English-speaking_countries:_The_terminology_comparative_analysis"},{"type":"article","name":"Return on investment in electronic health records in primary care practices: A mixed-methods study (Jang et al. 2014)","id":"5d49b7c0866e1210c1696240ca7f59fd","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Return_on_investment_in_electronic_health_records_in_primary_care_practices:_A_mixed-methods_study"},{"type":"article","name":"Support patient search on pathology reports with interactive online learning based data extraction (Zheng et al. 2015)","id":"65b45b4f6a653c627bbc452940cf174f","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Support_patient_search_on_pathology_reports_with_interactive_online_learning_based_data_extraction"},{"type":"article","name":"The evolution, use, and effects of integrated personal health records: A narrative review (Zieth et al. 2014)","id":"5c6e2bf767feb103119262aff1c6473b","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:The_evolution,_use,_and_effects_of_integrated_personal_health_records:_A_narrative_review"},{"type":"article","name":"Undertaking sociotechnical evaluations of health information technologies (Cresswell et al. 2014)","id":"302bc46eb57fbea71e5d7745898c7261","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Undertaking_sociotechnical_evaluations_of_health_information_technologies"},{"type":"article","name":"Unravelling the tangled taxonomies of health informatics (Barrett et al. 2014)","id":"4d5313302570d2f6c92b0f91883ad392","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Unravelling_the_tangled_taxonomies_of_health_informatics"},{"type":"article","name":"Use of handheld computers in clinical practice: A systematic review (Mickan et al. 2014)","id":"26a517162c5d1f67f3c6bbb8c4cd1a80","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Use_of_handheld_computers_in_clinical_practice:_A_systematic_review"},{"type":"article","name":"Why health services research needs geoinformatics: Rationale and case example (Onega et al. 2014)","id":"6865b9c249ccc54417e80c99d92bb99c","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Why_health_services_research_needs_geoinformatics:_Rationale_and_case_example"}]},{"type":"chapter","name":"3. Miscellaneous","id":"d34281196982f80b853bdcb2e4b10681","children":[{"type":"article","name":"Benefits of the community for partners of open source vendors (Groganz 2011)","id":"2d0bddc8ac63fbe42ecf34c128173d02","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Benefits_of_the_community_for_partners_of_open_source_vendors"},{"type":"article","name":"Generalized procedure for screening free software and open-source software applications (Joyce 2015)","id":"13b3b6c47f60a3b75f007d8cd017d140","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version"},{"type":"article","name":"Human\u2013information interaction with complex information for decision-making (Albers 2015)","id":"be30bf8a40599ea6d6b707ba00d0baa1","pageUrl":"https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making"}]}],"settings":{"show_cover":"1","show_title":"1","show_subtitle":"0","show_full_title":"1","show_editor":"1","show_editor_pic":"1","show_publisher":"1","show_language":"1","show_size":"1","show_toc":"1","show_content_beneath_cover":"1","cta_button":"1","content_location":"1","toc_links":"disabled","log_in_msg":"<span><\/span> Please log in to read online.","cover_size":"medium"},"title_image":"https:\/\/s3.limsforum.com\/www.limsforum.com\/wp-content\/uploads\/Fig1_Dander_BMCBioinformatics2014_15.jpg"}}
LIMSjournal - 2015 Edition
Editor: Shawn Douglas
Publisher: LabLynx Press
Copyright LabLynx Inc. All rights reserved.