{"ID":78004,"post_author":"9208550","post_date":"2018-12-13 16:02:11","post_date_gmt":"0000-00-00 00:00:00","post_content":"","post_title":"LIMSjournal - 2015 Edition","post_excerpt":"","post_status":"draft","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"","to_ping":"","pinged":"","post_modified":"2018-12-13 16:02:11","post_modified_gmt":"2018-12-13 21:02:11","post_content_filtered":"","post_parent":0,"guid":"https:\/\/www.limsforum.com\/?post_type=ebook&p=78004","menu_order":0,"post_type":"ebook","post_mime_type":"","comment_count":"0","filter":"","_ebook_metadata":{"enabled":"on","private":"0","guid":"A7260F82-C570-4D27-997E-6EE74C766B54","title":"LIMSjournal - 2015 Edition","subtitle":"","cover_theme":"nico_3","cover_image":"https:\/\/www.limsforum.com\/wp-content\/plugins\/rdp-ebook-builder\/pl\/cover.php?cover_style=nico_3&subtitle=&editor=Shawn+Douglas&title=LIMSjournal+-+2015+Edition&title_image=https%3A%2F%2Fs3.limsforum.com%2Fwww.limsforum.com%2Fwp-content%2Fuploads%2FFig1_Dander_BMCBioinformatics2014_15.jpg&publisher=LabLynx+Press","editor":"Shawn Douglas","publisher":"LabLynx Press","author_id":"26","image_url":"","items":{"be30bf8a40599ea6d6b707ba00d0baa1_type":"article","be30bf8a40599ea6d6b707ba00d0baa1_title":"Human\u2013information interaction with complex information for decision-making (Albers 2015)","be30bf8a40599ea6d6b707ba00d0baa1_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making","be30bf8a40599ea6d6b707ba00d0baa1_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Human\u2013information interaction with complex information for decision-making\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nHuman\u2013information interaction with complex information for decision-makingJournal\n \nInformaticsAuthor(s)\n \nAlbers, Michael J.Author affiliation(s)\n \nDepartment of English, East Carolina UniversityPrimary contact\n \nE-Mail: albersm@ecu.edu; Tel.: +1-252-328-6374Editors\n \nSedig, Kamran; Parsons, PaulYear published\n \n2015Volume and issue\n \n2 (2)Page(s)\n \n4\u201319DOI\n \n10.3390\/informatics2020004ISSN\n \n2227-9709Distribution license\n \nCreative Commons Attribution 4.0 InternationalWebsite\n \nhttp:\/\/www.mdpi.com\/2227-9709\/2\/2\/4Download\n \nhttp:\/\/www.mdpi.com\/2227-9709\/2\/2\/4\/pdf (PDF)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Explanation of terms \n\n3.1 Simple information \n3.2 Complex information \n3.3 Information relationships \n3.4 Contextual awareness \n\n\n4 HII is information interaction, not data interaction \n5 Communication of simple and complex information \n6 HII for decision-making \n\n6.1 Decision-making strategies \n\n6.1.1 Experience-based \n6.1.2 Knowledge-based \n\n\n\n\n7 Conclusions \n8 Conflict of interests \n9 References \n10 Notes \n\n\n\nAbstract \nHuman\u2013information interaction (HII) for simple information and for complex information is different because people's goals and information needs differ between the two cases. With complex information, comprehension comes from understanding the relationships and interactions within the information and factors outside of a design team's control. Yet, a design team must consider all these within an HII design in order to maximize the communication potential. This paper considers how simple and complex information requires different design strategies and how those strategies differ.\nKeywords: human\u2013information interaction, decision-making, complex communication, information design\n\nIntroduction \nA primary goal of human\u2013information interaction (HII) for complex information is to communicate concepts and ideas to be used for decisions. That is, information that a reader uses to develop an understanding of a situation and to make decisions about a complex situation.[1][2][3]\nIn any complex communication situation, the proper amount of information should result in maximum communication; of course, too little, too much, poor organization, or inappropriate content reduces the communication. Stating that the communication needs to provide the proper amount of information could considered tautological. It is easy to claim \u2014 and adds little to a discussion \u2014 a text should have the proper information (I use text within the article in a generic sense: paper, web pages, etc). However, defining and measuring the communication value of a document (and the information it contains) proves to be a much more difficult task.\nThe interaction and presentation needs of complex information are very different from those of simple information, where a person looks for single information elements. For example, consider the difference in looking up the score of yesterday\u2019s game (simple information) and considering house remodeling to help care for a chronically sick parent (complex information). When we move from a communication viewpoint to an HII viewpoint, we also bring into play issues of how a person can manipulate the information.[2][4][5] The base text may contain the \u201cproper information\u201d but for maximum communication, it may need to be reordered or otherwise manipulated to meet individual needs.\nA large percentage of the HII literature either looks at simple information search and interaction or conflates simple and complex information. It also tends to take a web-based focus with the person having an essentially infinite number of sources to explore.[6] While this is true in many cases, within this article, I take a much narrower focus on HII and look at the construction of content within a single set of information that is under a content development team\u2019s control, such as the production of corporate reports or public information for a specific goal (an example presented later will be on communicating hurricane information). A view focused on the design team\u2019s task needs to consider what information to create and how people will interact with it. The more commonly discussed view of people searching the Internet (or other sources) for existing information[7] will not be directly considered since a design team has no control over how that search proceeds or what information the person may view.\n\nExplanation of terms \nIn the introduction, I used the terms simple and complex information. Here I provide a working definition for each. \n\nSimple information \nSimple information can be characterized by existing as a single information element. A high percentage of the \u201clook it up on Google\u201d amounts to simple information. For example, a person can look up when a movie was released or a recipe for a carrot cake. Both fit the definition of simple information. Factors that distinguish simple information include:\n\n Single path. One path can be fully defined that will result in the answer.\n Right\/wrong answer. The correctness can be tested and the information declared correct or not correct.\n Complete information. The completeness can be tested and the information declared complete or not complete.\n Closed system. All of the factors that might influence the answer can be defined and accounted for.\nComplex information \nComplex information exists at the opposite end from simple answers. There is no single \u201canswer\u201d. Addressing complex questions requires using and integrating multiple information elements, which often conflict. People have a complex web of information needs and interactions to fill those needs.[8] For example, analyzing monthly business reports and making decisions about this month\u2019s production, or making health care choices. Factors that distinguish complex information include:\n\n Multiple paths. There is no single path to an answer. A person can take many different paths and all will work. The effectiveness of the paths may, of course, vary.\n Open-ended. The idea of \u201ccomplete\u201d is undefined. A person can continue to collect information and refine their understanding with an essentially infinite amount of information. Instead, a person has to pick a stopping point and make a decision.\n Needs cannot be predefined. The information that a person needs cannot be predefined. Of course the major or essential information can be predefined, but the many smaller information elements that can exert a strong influence vary too much between individuals.\n History. The information exists within a continuum and that history influences how it gets interpreted and used.\n Non-linear. The overall situation shows a non-linear response with small differences in some information elements resulting in very large differences in appropriate decisions.\n Open system. All of the factors that might influence the decision cannot be defined. There are too many and the information is dynamic, changing on time scales relevant to the decision situation.\nMost of the simple information that gets communicated tends to be single facts or procedural type of information. Complex information, on the other hand, deals with a broader scope.\n\n Complex information communicates concepts and ideas.\n Complex information communicates an understanding of a situation.\n Complex information communicates relationships and interactions.\nClearly, the HII required to maximize the communication of complex information is very different between the two. Most writing and UX guidelines deal with communicating simple information. But to really address people\u2019s information needs, we need to consider why and how they interact with complex information. Consider the case of electronic health records \u2014 they clearly represent complex information \u2014 and represent an example where the design team has control over a significant part of how the HII occurs. The design has to consider many potentially conflicting sets of priorities (development time, maintenance cost, hospital administration goals, government regulations, medical needs, etc.), which, in turn, affect how well the users can interact with the information. The input methods must be efficient for data entry, but presenting the information the same way to a nurse or physician may not be the best for integrating it into a clear view of the patient\u2019s condition. The relationships of information (connecting the results of various medical tests, etc.) should reflect the initial diagnosis and should also reflect the changing patient status as treatment progresses.\nIn previous work, I have defined a situation as \u201cthe current world state which the user needs to understand. An underlying assumption is that the user needs to interact with a system to gain the necessary information in order to understand the situation\u201d[1] (p. 11). The information relevant to the HII exists both inside and outside any circle drawn to enclose \u201cthe situation\u201d (Figure 1). Honestly, the difficulty of drawing that enclosing circle is a characteristic of a complex situation.\n\n\n\nFigure 1. Complex situation with information both inside and outside of a design team\u2019s control.\r\nHuman\u2013information interaction (HII) only directly applies to the information a design team controls.\nOnce an upfront analysis has defined the people\u2019s goals and information needs, then the HII factors which support manipulating the information and revealing the relationships can be considered and entered into the design.[2][9]\n\nTypically, the failure of these technical documents [used here in a generic sense for any information source] comes not from a lack of information; the text probably contains an excess of information. Post hoc studies of communication failures find many sources to blame: poor information architecture, poor organization, wrong grade level or writing style, or poor presentation. But instead of seeing these problems as a root cause, let\u2019s consider them as symptoms of a more fundamental problem: a problem stemming from the underlying complexity of the situational context and a failure of the information presentation to match that complexity.[10](p. 110)\nThe task assigned to an HII design team is to ensure the underlying complexity is not overly simplified and that the information presentation and manipulation meets the needs of the people and the demands of the situation. As others have discussed, the HII problem for maximizing the communication for complex information is not a tools question, but one of understanding what people need and how they come to understand the information.[11]\n\nInformation relationships \nUnderstanding complex information depends on understanding relationships, but the potential relationships are essentially infinite. Comprehension of a complex situation occurs when people can mentally integrate those relationships into their view of the situation.\n\nInformation integration lies, not in a text element itself, but in the relationships between those elements. A reader needs to figure out what information is relevant and how to connect it to the current problem. Without proper information relationships, the reader does not gain an integrated understanding of information, but instead gains a collection of facts. Without relationships, information exists as a bunch of interesting factoids which do not help a person form an adequate mental picture of the situation. Collections of facts are less than useful for understanding and working with the open-ended problems that people encounter in complex situations.[12][13] Without the relationships, a person learns about X and Y, but not how X and Y relate to each other or to Z in terms of their current problem or situation.[14] The text fails to communicate because the reader can\u2019t form the necessary information relationships.[10](p. 111)\nA patient trying to understand a medical problem often has difficulty understanding the situation because they don\u2019t understand the relationships. They can look at a set of lab results and know the numbers, but lack the medical practitioner\u2019s knowledge of what the values should be for their condition. The patient may be worried because some values are high, but the physician is satisfied with them because the specific condition often has even higher values. Likewise, a junior manager needs to develop the knowledge of how to interpret multiple values that may appear on a collection of monthly reports. Good HII can help by providing a more integrated view and providing help with the integration, without interfering with more experienced people\u2019s information interaction.\nBuilding relationships means developing a deep-level understanding of the text rather than just a surface-level knowledge. Deep-level knowledge involves seeing the macrostructure of the text and being able to apply prior knowledge to it and fit the text within the reader\u2019s prior knowledge framework. Surface-level knowledge involves knowing the basic text. People with surface-level knowledge can quote the text and if asked recall-type questions would respond with answers very close to the text language, but they would not be able to elaborate on the answer or connect it to other information. People with deep-level understanding would be able to place the text\u2019s information into their own words. A major element in the difference between people being able to develop deep-level understanding versus a surface-level understanding is their prior knowledge.\nCommunicating complex information for decision-making can be viewed as working to help build relationships within the information.[8] Unfortunately, building relationships is a great theoretical concept, but one that does not lend itself to a direct operational definition. One way to define judging or qualifying relationships can be how they reduce the uncertainty a person has about the situation.[15] This reshapes an HII design team\u2019s goal to one of focusing content creation in terms of what information does the reader need to reduce their uncertainty and, consequently, to build a web of relationships between the information elements. \n\nContextual awareness \nExcept for training material, the readers of technical information typically understand the basics but need to know specific information about the current situation (as opposed to the general situation) in order to make decisions. The understanding of a complex situation needed to make informed decisions comes when people can distinguish the information structure, grasp the relationships within it[16][17][18], and make inferences on the future evolution of the situation.\nBuilding on Endsley\u2019s (1995)[19] situation awareness work, we can call this contextual awareness. Contextual awareness is the understanding of the information within an informational situation which forms the basis for how to interpret new information and how to make decisions for interacting with that situation. With poor contextual awareness, people can know something is occurring or that a particular piece of information exists, but they cannot easily find relevant related information or they have the information but do not understand how it relates to the overall situation. On the other hand, good contextual awareness does not guarantee a person will form the proper intention or make the proper decisions; the error analysis literature is filled with cases where people understood a situation, but still made incorrect choices. Unfortunately for design teams, the concept of relevance itself is highly nuanced and multifaceted[17], creating a complex interplay that must be understood to engineer high quality HII.\nElements of good contextual awareness are (Figure 2): (1) Understands how the information fits within the current situation. (2) Understands the information relationships.[10] Information comprehension requires knowing how information relates to other information. (3) Understands the future development of the situation and can make predictions about the ripple effects of any decision across the entire situation. HII that supports complex information needs to provide the interactions that support people developing high quality contextual awareness. \nKain, de Jong, and Smith (2010)[20] study into how to communicate hurricane risks and warnings highlighted the issues of how people interpret information and make decisions about how they will react. The hurricane experts had their view of what information was needed and how it should be presented, but the research showed the people wanted\/needed a different presentation. Their mental methods of forming relationships and of interpreting complex information differed from what the experts thought. The process of building contextual awareness differs and a design team\u2019s analysis must capture those differences.\nA design team working on developing the HII for a country\u2019s hurricane awareness plan needs to balance both the expert\u2019s \u201chere is what the people must hear\u201d against the more pragmatical \u201chere\u2019s what I want to know\u201d as well as the local people\u2019s opinion of how they react to hurricane warnings. Many of them have been through multiple hurricanes and have strongly held views that often conflict with the authorities. Notice how in this case, the design team has control of the information. A general search on hurricane warnings will have sentences such as \u201cconsult your local authorities for evacuation routes\u201d but, here, the design team will be tasked with providing that evacuation route information. At the same time, they need to ensure the relationships between evacuation route, getting ready for evacuation, and planning on returning are all clearly laid out and connected. \n\n\n\nFigure 2. Stages of developing contextual awareness\nHII is information interaction, not data interaction \nPeople who are engaged with decision-making and complex information need to be presented with information, not data (Figure 3). Quick definition of my terms here, which I discuss in more detail elsewhere.[1]\n\n Data: Raw numbers, facts, and figures.\n Information: Information is data in context. It relates to the situation and contains the relationships that connect the information to the situation.\n Knowledge: Interconnected web of the relevant information and the relationships linking the information within the situation. \n\n\n\nFigure 3. Data, information, and knowledge hierarchy. The higher the HII works in the hierarchy,\r\nthe better it fits building an understanding of the situation.\nAs an example, I heard a presentation that looked at the effect of sea level rise on the Norfolk, VA area.[21] A software program had nice manipulation that let a person dynamically see the effects of different amounts of sea level rise from global warming and how it would affect the city. The problem was not with the implementation, but with the underlying assumptions of a design team. They were assuming that by providing a tool and letting people see how different sea level changes would affect the area that it would bring about understanding and increase long-term preparedness. But the tool was presenting content at the data level. Yes, I can change the sliders and see how the sea level changes, but it was devoid of supporting content (in the metaphor of this paper, the tool was a single puzzle piece). It not only didn\u2019t connect with other pieces, those other pieces were not provided. As a result, a global warming denier could play with the model, agree that a four foot rise would be a catastrophic problem, but then reject it as something that would never happen. One of the software\u2019s goals was to help people prepare, but without giving them the content and relationships to build their contextual awareness; it was a single data point and not part of a coherent presentation of information. \nThe transition from information to knowledge is important since it involves comprehending the relationships within the data and placing them within the context of the situation. Moving to knowledge is essential to building contextual awareness and must be the goal of an HII design team.\n\nCommunication of simple and complex information \nMany discussions on communication start with Shannon information theory (1948)[22] (Figure 4), which works for analyzing simple information. With clearly defined information to be conveyed, a design team can concentrate on efficient communication methods. Unfortunately, Shannon was really discussing the minimum information to convey a message, not how to communicate in a natural language. It works well for computer-computer communication, but is not as applicable to human-human communication, especially for complex information, with or without a computer mediator. \n\n\n\nFigure 4. Basic block diagram of Shannon information theory. Information moves from the source\r\nto the final destination with the overall goal to be minimizing both the noise and total amount of required content.\nDriven in part by a Shannon-based desire to efficiently communication information, design teams try to break the communication into individual components. This fits with our reductionist nature to break any problem into individual components. Standard approaches for communicating complex information (and analyzing any complex system) are to assume it is assembled from smaller systems[3][13][23][24]. Thus, it is hardly a surprising statement that we see a complex system as the sum of its components: 1 + 1 + 1 + 1 + \u2026.. = THE SYSTEM.\nUnfortunately, assuming a complex system is the sum of its components at best redefines it as a simple system. The complexity that people need to understand and the HII must support is not just a sum of individual components, but the interactions and relationships between components. The redefinition to a sum of individual components ignores the complex interactions that give rise to more than the sum of the parts: 1 + 1 + 1 + 1 + \u2026..+ in >> \u01a9n. \nThe communication problem, and consequently the HII problem, is that redefining the problem to be a simple problem fundamentally changes the problem itself. As a result, the communication itself changes and the relationships and interactions that are essential to understanding get lost. The communication-related literature seems to generally miss this important concept, although the computer science literature consistently contains quotes such as \u201cthe fundamental reason today\u2019s software engineering cannot effectively manage inherent complexity is that its basis is in developing individual programs rather than in interacting systems.\u201d[24] (p. 73). In designs that strive to communicate complex information, an over-privileging of developing and\/or organizing content while discounting the interactions and relationships within and between content leads to results similar to those critiqued by Sommerville. \nIn the introduction, I described most simple information as being single information elements that can be looked up (Figure 5). Complex information on the other hand, has a multitude of information elements that a person must understand, some of which are outside of the bounds of a HII system. Factors such as asking a person down the hall, having heard of a bad experience with product X, etc.[10] (p. 115) \n\n\n\nFigure 5. Relationships and understanding a situation\nFigure 5 shows content as nice rectangles. But they could be better viewed as jigsaw pieces with the interlocks on the pieces representing the relationships (Figure 6), how the pieces interconnect with each other into a coherent understanding. \n\n\n\nFigure 6. Relationships and the complexities of assembling information into a coherent understanding.\r\nThe understanding comes from both the content and knowing how it fits together.\nUnfortunately, the pieces in Figure 6 are not static jigsaw shapes. They are better viewed as a collection of puzzle pieces that change shape as they move away\/toward other pieces, and as they merge together (Figure 7). As a result, changes to one piece can ripple through the entire information web. A change to a piece in the upper left corner results in many of the pieces changing in some relevant manner. The information and HII appropriate and relevant for each point of a situation can be different. Situations are not static; they have a beginning, middle, and end. The puzzle pieces of Figure 7 are not just dynamic from a person\u2019s interaction, but also change over time and over the situation\u2019s evolution. \n\n\n\nFigure 7. Dynamic evolution of information. The relationships between information elements\r\ncan change over both time and by the act of connecting them.\nLet\u2019s reexamine the six characteristics of complex information with respect to the puzzle metaphor.\n\nMultiple paths: People have multiple paths through the information. The order in which they move between the puzzle pieces cannot be predefined. Each HII with a piece can change how it interconnects; thus, different paths through the information change how people build the relationships and, consequently, how the pieces fit together. \nOpen-ended: No clear point of enough information. The HII cannot move toward a predefined point of \u201cnow you have all the information.\u201d With a goal of communicating concepts and ideas, how many pieces a person needs, cannot be defined. Of course, the issue of information needs versus information wants also comes into play. Coupled with this is the fact that information search and problem-solving are sufficing processing.[25][26] People stop once they are comfortable with their understanding of the found information. Unfortunately, people are poor judges of knowing they have found adequate information.\nCannot be predefined: How many pieces a person will interact with remains unknowable to a design team. Interestingly, the size (content) of individual pieces is dynamic; some people need smaller\/larger pieces to effectively comprehend the overall situation and tend to make choices in terms of immediate, rather than long-term, efficiency and effort of the HII.[27][28]\nHistory: Past history of a situation affects how the pieces will evolve and the past history of the people interacting with the information affects how they interpret it. Two situations may appear to be the same but the past strongly influences how the piece will change. For example, six months ago sales in the southeast region were down by 30%. A set of decisions was made and sales have steadily increased since them. They are still 10% below desired levels, but are increasing at an acceptable rate. This situation is much different than if sales are 10% below the desired level and not conforming to predictions.\nNon-linear: The relationships between pieces can show non-linear response to changes. Minor differences in past history or the HII path can result in widely different \u2014 but appropriate \u2014 end points. In individual piece can morph into very different final shapes, even though they started from similar initial conditions.\nOpen system: As the system evolves, the overall content within the system changes. Some new information gets introduced and some information drops out. The number of pieces, their content, and their shape can change.\nWhen the information is viewed as dynamic puzzle pieces, how those pieces change shape helps to emphasize how many solutions to problems introduce other problems, sometimes worse than the one they were intended to solve. For example, many environmental solutions inevitably end up doing more harm than good. The future changes to relationships are not understood when decisions are made, so the evolution of the situation is poorly predicted (poor contextual awareness). Worse, too many people make assumptions that they can change one piece with nothing else changing. In complex ecosystems \u2014 biological, educational or industrial \u2014 actions always have consequences that are hard to predict; a change to one piece ripples through all the others. The \u201chard to predict\u201d is a hallmark of a complex system. HII that supports interacting with the information must consider those ramifications.\nBeyond the need for the HII to handle dynamic puzzle pieces, it must also allow for easy trimming of the information space. All of the available information is not relevant to any specific situation. Instead, a person needs a select collection of information (Figure 8).\n\n\n\nFigure 8. Information relevant to a situation. Each situation (or class of situations) only needs\r\na subset of the information. High quality HII helps to keep the focus on only the relevant subset.\nHII for decision-making \nThe overall scope of the information flow for the HII of complex information moves to prominence when the interaction goal is decision-making. People need to collect information, analyze it, and make a decision; a process that depends on the flow of information both within and outside the control of an HII design team.\nDecision-making occurs as a result of comparing what is perceived in the environment and what is known by the decision-maker.[29] Essentially all of our current literature discusses considering the needs of people when designing information. However, current practice does not typically consider how people\u2019s questions depend on how they make decisions and interact with the system.[30][31] HII attempts to take the entire cycle of information interpretation and decision-making and place it within people\u2019s current situation.\nMirel (1998)[12] follows the same line of thinking as Conklin\u2019s (2003)[32] wicked problems when she points out that analyzing complex tasks requires seeing more than a single path:\n\nThis broader view is necessary to capture the following traits of complex tasks: paths of action that are unpredictable, paths that are never completely visible from any one vantage point, and nuance judgments and interpretations that involve multiple factors and that yield many solutions.[12] (p. 14)\nMirel may have focused on the analysis aspects, but that is an early and vital step in any information interaction, especially for HII of complex information. Unless a design team clearly understands the people\u2019s goals and information needs, there is minimal chance for the HII to support it.\nThe issue of wicked problems and unpredictable paths that both Conklin and Mirel discuss brings to light an interesting point. As design teams work toward HII of complex information, they encounter an interesting and confounding recursion: the HII of complex information is itself a complex problem and, as such, does not lend itself to easy answers. Yet, many design teams try to address those issues with the same process that works for simple information, which leads to an easy-answer mentality. As a result, they reduce the scope and avoid confronting the complex information HII issues head-on. Design teams who try to reduce all problems to a collection of individual parts or who assume they should consider only the information they control fall into this trap. They have redefined the complex HII problem into a simple problem. \n\nDecision-making strategies \nPeople use the available information to make a decision about how to proceed in their current situation; information use is always based upon the context within which it is used.[16] This hard connection between use and context greatly complicates a design team\u2019s task since they must understand both factors. And, depending on audience, the base information may exist within multiple contexts.\nBased on the results of the decision, people take action and then need to re-analyze the situation. Closely related to decision-making is monitoring the situation to ensure it continues along the predicted path. If the information fits the expected progress, then a decision of \u201cdo nothing\u201d is correct. However, if the information indicates that it is deviating, then people cycle back into decision-making.\nThere are two broad decision-making strategies that people use. \n\nExperience-based \nExperience-based decision-making is used the most and is a person\u2019s default strategy. It is a rule-based strategy: \u201cif X happens, then do Y.\u201d Most decision-making, at least in the corporate world, is\nexperienced-based.[12] The reader is skilled in the overall area (such as a CFO making financial decisions), but does need specific information for the current situation. They need to know how this month\u2019s production or sales numbers differ from last month\u2019s or between different regions, but they do have a set of mental rules on how to interpret those numbers.\nBased on past experience, people develop a rule-based strategy so they can reliably expect that \u201cwhen X happens, then do Y.\u201d Rule-based decisions reduce cognitive effort and work well within normal situations because the rules themselves evolved from past experience. However, that same rule-based strategy can cause problems if a person jumps to a decision based on incomplete information. X can happen in multiple situations but other factors differ\u2014the relationships of what\/how X is interconnected within the situation\u2014which cause solution Y to only be an appropriate response in some situations.\nKlein\u2019s (1999)[25] recognition-primed model of decision-making posits that experienced people do little problem solving, especially in routine situations. Instead, once they recognize a situation, they form a possible intention very rapidly, mentally evaluate it, and, if no major problems are evident, take action on it. Alternatives are not considered. In other words, they tend to use their prior knowledge and initial assessment to immediately pick a solution, and if it seems workable, then that is considered the answer. (Concerns about whether or not the solution is optimal are not part of the mental evaluation.)\nPost-failure analysis tends to highlight that the decision-makers had the proper information, but improperly interpreted it. In many cases, it is a result of using experience-based decision-making and not realizing the \u201cif x, then y\u201d rule did not apply to the situation. \n\nKnowledge-based \nExperience-based decision-making can break down, typically when the situation contains unexpected factors or relationships. When people recognize their rules no longer apply, they have to shift to knowledge-based decision-making. Unfortunately, the shift to knowledge-based decision-making often happens after the monitoring phase of an experienced-based strategy reveals the situation is not developing as predicted.\nIn knowledge-based decision, a person knows their experience-based rule set has broken down and they are no longer working within a normal situation. Something is not right in the current situation and they are trying to figure out what to change to fix the problem. In other words, the expected relationships are not being found and a person needs to understand why. It tends to happen when people do not understand what caused the base problem. Unlike decision-making based on rules of \u201cwhen X happens, then do Y,\u201d a person has to fully develop their contextual awareness so they can make a decision.\nThe HII of complex information applies more for knowledge-based strategies than it does for experience-based ones. Knowledge-based strategies require people to engage in a systematic interaction with the information and work to build up their contextual awareness. \nSomeone working on a global warming problem is working at the knowledge-based and not experienced-based level. Allowing people to manipulate data as if it was a simple information \u2014 a collection of individual components \u2014 can misguide them into thinking it\u2019s an experience-based problem and they can draw on their prior knowledge. Worse, defining the problem as \u201cif they can work this sea level rise model, then they will understand the problem\u201d redefines the problem from a complex problem to a simple problem. Unfortunately, but inevitable with the history component of complex information, prior knowledge and biases strongly influence how information gets interpreted. Of course, with global warming\u2014and many other situations\u2014the political aspects, factors outside of an HII design team\u2019s control, come to the forefront. \n\nConclusions \nProviding people with any collection of information, especially complex information, can easily result in information overload. With the large amounts of information and its dynamic nature, high quality HII is essential for people to comprehend that information. \nRather than a lack of information, the failure to anticipate people\u2019s needs forms the basis of most information problems and poor decision-making; an issue design teams must explicitly address. Managers typically suffer from a shortage of decision-relevant information and a simultaneous overabundance of irrelevant information.[33] People are forced to make decisions based on an unfiltered avalanche of information that may or may not be relevant to their needs.\nUnderstanding a complex situation comes from understanding the relationships within the information. Building that understanding results not in knowing the individual pieces of information but in knowing how those pieces of information interconnect. That, in turn, requires looking at the system as not just a large collection of information, but as a highly dynamic integration of information and relationships between information. As a design team works on developing the system HII, they must ensure they capture those relationships and interconnections so they can be explicitly made to the people using the system. As such, a design team must:\n\n Define the situations that must be understood\n Determine the information people need to understand the situation\n Determine how that information is connected and how people see those connections as they build their contextual awareness\n Understand how the information and relationships change as the situation evolves\n Understand the biases, interaction expectations, and decision-making styles of the audience\n Determine the best HII for presenting the information to achieve the best information communication while allowing for those biases, interaction expectations, and decision-making styles\nHII plays a fundamental part in ensuring the overall information content is usable and that people can form the relationships needed to build their contextual awareness. As that understanding develops, people develop their contextual awareness of the situation and can make informed decisions. \n\nConflict of interests \nThe author declares there is no conflict of interest.\n\nReferences \n\n\u2191 1.0 1.1 1.2 Albers, M. (2004). Communication of Complex Information: User Goals and Information Needs for Dynamic Web Information. Erlbaum.   \n\n\u2191 2.0 2.1 2.2 Albers, M. (2012). Human-Information Interaction and Technical Communication: Concepts and Frameworks. IGI Global.   \n\n\u2191 3.0 3.1 Mirel, B. (2003). Interaction Design for Complex Problem Solving: Developing Useful and Usable Software. Morgan Kaufmann.   \n\n\u2191 Albers, M. (July 2011). \"Human-information interactions with complex software\". Proceedings of the HCI International.   \n\n\u2191 Sedig, K.; Parsons, P. (2013). \"Interaction design for complex cognitive activities with visual representations: A pattern-based approach\". AIS Transactions on Human-Computer Interaction 5: 84\u2013133.   \n\n\u2191 Cole, C. (2014). \"Google, tear down this wall to exploratory search!\". Bulletin of the Association for Information Science and Technology 40: 50\u201354.   \n\n\u2191 Ingwersen, P.; Jarvelin, K. (2005). The Turn: Integration of Information Seeking and Retrieval in Context. Springer.   \n\n\u2191 8.0 8.1 Cole, C. (2012). Information Need: A Theory Connecting Information Search to Knowledge Formation. Information Today.   \n\n\u2191 Parsons, P.; Sedig, K. (2014). \"Adjustable properties of visual representations: Improving the quality of human-information interaction\". Journal of the Association for Information Science and Technology 65: 455\u2013482.   \n\n\u2191 10.0 10.1 10.2 10.3 Albers, M.; Still, B. (Eds.) (2010). \"Usability and information relationships: Considering content relationships when testing complex information\". Usability of Complex Information Systems: Evaluation of User Interaction. CRC Press. pp. 109\u2013132.   \n\n\u2191 Taylor, R.S. (1968). \"Question-negotiation and information seeking in libraries\". College & Research Libraries 29: 178\u2013194.   \n\n\u2191 12.0 12.1 12.2 12.3 \"Applied constructivism for user documentation\". Journal of Business and Technical Communication 12: 7\u201349. 1998.   \n\n\u2191 13.0 13.1 Mirel, B.; Albers, M. (Ed.); Mazur, B. (Ed.) (2003). \"Dynamic usability: Designing usefulness into systems for complex tasks\". Content and Complexity: Information Design in Software Development and Documentation. Erlbaum. pp. 233\u2013261.   \n\n\u2191 \"Can we ever escape from data overload? A cognitive systems diagnosis\". Cognition, Technology & Work 4: 22\u201336. 2002.   \n\n\u2191 Albers, M. (October 2009). \"Information relationships: The source of useful and usable content\". Proceedings of the 29th Annual International Conference on Computer Documentation.   \n\n\u2191 16.0 16.1 Fidel, R. (2012). Human Information Interaction: An Ecological Approach to Information Behavior. MIT.   \n\n\u2191 17.0 17.1 Saracevic, T. (2007). \"Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II. Nature and manifestations of relevance\". Journal of the Association for Information Science and Technology 58: 1915\u20131933.   \n\n\u2191 Thuring, M.; Hannemann, J.; Haake, J. (1995). \"Hypermedia and cognition: Designing for comprehension\". Communications of the ACM 38: 57\u201366.   \n\n\u2191 Endsley, M. (1995). \"Toward a theory of situation awareness in dynamic systems\". Human Factors 37: 32\u201364.   \n\n\u2191 Kain, K. de Jong, M.; Smith, C.; Albers, M. (Ed.); Still, B. (Ed.) (2010). \"Information Usability testing as audience and context analysis for risk communication\". Usability of Complex Information Systems: Evaluation of User Interaction. CRC Press. pp. 305\u2013332.   \n\n\u2191 Richards, D. (2015). \"Testing the waters: Local Users, sea level rise, and the productive usability of interactive geovisualizations\". Communication Design Quarterly 3: 24\u201332.   \n\n\u2191 Shannon, C. (1948). \"A mathematical theory of communication\". Bell System Technical Journal 27: 379\u2013423.   \n\n\u2191 Redish, J. (2007). \"Expanding usability testing to evaluate complex systems\". The Journal of Usability Studies 2: 102\u2013111.   \n\n\u2191 24.0 24.1 Sommerville, O.; Cliff, D.; Calinescu, R.; Keen, J.; Kelly, T.; Kwiatkowska, M.; Mcdermid, J.; Paige, R. (2012). \"Large scale complex IT systems\". Communications of the ACM 55: 71\u201377.   \n\n\u2191 25.0 25.1 Klein, G. (1999). Sources of Power: How People make Decisions. MIT.   \n\n\u2191 Simon, H. (1979). Models of Thought. Yale University Press.   \n\n\u2191 Fennema, M.; Kleinmuntz, D. (1995). \"Anticipation of effort and accuracy in multi-attribute choice\". Organizational Behavior and Human Decision Processes 63: 21\u201332.   \n\n\u2191 Payne, S.; Howes. A.; Reader, W. (2001). \"Adaptively distributing cognition: A decision making perspective on human-computer interaction\". Behaviour & Information Technology 20: 339\u2013346.   \n\n\u2191 Mason, R.; Mitroff, I. (1973). \"A program for research on management information systems\". Management Science 19: 475\u2013487.   \n\n\u2191 Gerlach, K.; Kuo, F. (1991). \"Understanding human-computer interaction for information system design\". MIS Quarterly 15: 527\u2013550.   \n\n\u2191 Woods, D.; Roth, E. (1988). \"Cognitive engineering: Human problem solving with tools\". Human Factors 30: 415\u2013430.   \n\n\u2191 Conklin, J.. \"Wicked problems and fragmentation\". http:\/\/www.cognexus.org\/id26.htm . Retrieved 24 April 2015 .   \n\n\u2191 Ackoff, R. (1967). \"Management misinformation systems\". Management Science 14: 147\u2013156.   \n\n\nNotes \nThis presentation is faithful to the original, with only a few minor changes to presentation and content. The two most notable changes involve citations. Albers lists the following citation in his references as citation 20:\n\"20. Albers, M. Contextual awareness as measure of human-information interaction in usability and design. In Proceedings of the HCI International, Orlando, FL, USA, 12\u201314 July 2011.\" \nHowever, he never referenced that citation number in the paper. We have thus omitted it. Additionally, under the subsection \"Experience-based\", he references \"Klein\u2019s (1999)\" but seemingly uses the incorrect citation number of 12, for Mirel 1998. We've corrected the citation number.\n\n\n\n\n\n\nSource: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\">https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making<\/a>\n\t\t\t\t\tCategories: LIMSwiki journal articles (added in 2015)LIMSwiki journal articles (all)LIMSwiki journal articles on informaticsLIMSwiki journal articles on software\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\tNavigation menu\n\t\t\t\t\t\n\t\t\tViews\n\n\t\t\t\n\t\t\t\t\n\t\t\t\tJournal\n\t\t\t\tDiscussion\n\t\t\t\tView source\n\t\t\t\tHistory\n\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\t\n\t\t\t\tPersonal tools\n\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\tLog in\n\t\t\t\t\t\t\t\t\t\t\t\t\tRequest account\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\n\t\t\t\t\n\t\tNavigation\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tMain page\n\t\t\t\t\t\t\t\t\t\t\tRecent changes\n\t\t\t\t\t\t\t\t\t\t\tRandom page\n\t\t\t\t\t\t\t\t\t\t\tHelp\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tSearch\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t \n\t\t\t\t\t\t\n\t\t\t\t\n\n\t\t\t\t\t\t\t\n\t\t\n\t\t\t\n\t\t\tTools\n\n\t\t\t\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tWhat links here\n\t\t\t\t\t\t\t\t\t\t\tRelated changes\n\t\t\t\t\t\t\t\t\t\t\tSpecial pages\n\t\t\t\t\t\t\t\t\t\t\tPermanent link\n\t\t\t\t\t\t\t\t\t\t\tPage information\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\tPrint\/export\n\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\tCreate a book\n\t\t\t\t\t\t\t\t\t\t\tDownload as PDF\n\t\t\t\t\t\t\t\t\t\t\tDownload as Plain text\n\t\t\t\t\t\t\t\t\t\t\tPrintable version\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\n\t\t\n\t\tSponsors\n\t\t\n\t\t\t \r\n\n\t\r\n\n\t\r\n\n\t\r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n \r\n\n\t\n\t\r\n\n\t\n\t\r\n\n\t\r\n\n\t\r\n\n\t\r\n\t\t\n\t\t\n\t\t\t\n\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t This page was last modified on 28 December 2015, at 20:55.\n\t\t\t\t\t\t\t\t\tThis page has been accessed 2,714 times.\n\t\t\t\t\t\t\t\t\tContent is available under a Creative Commons Attribution-ShareAlike 4.0 International License unless otherwise noted.\n\t\t\t\t\t\t\t\t\tPrivacy policy\n\t\t\t\t\t\t\t\t\tAbout LIMSWiki\n\t\t\t\t\t\t\t\t\tDisclaimers\n\t\t\t\t\t\t\t\n\t\t\n\t\t\n\t\t\n\n","be30bf8a40599ea6d6b707ba00d0baa1_html":"<body class=\"mediawiki ltr sitedir-ltr ns-206 ns-subject page-Journal_Human\u2013information_interaction_with_complex_information_for_decision-making skin-monobook action-view\">\n<div id=\"rdp-ebb-globalWrapper\">\n\t\t<div id=\"rdp-ebb-column-content\">\n\t\t\t<div id=\"rdp-ebb-content\" class=\"mw-body\" role=\"main\">\n\t\t\t\t<a id=\"rdp-ebb-top\"><\/a>\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t<h1 id=\"rdp-ebb-firstHeading\" class=\"firstHeading\" lang=\"en\">Journal:Human\u2013information interaction with complex information for decision-making<\/h1>\n\t\t\t\t\n\t\t\t\t<div id=\"rdp-ebb-bodyContent\" class=\"mw-body-content\">\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t<!-- start content -->\n\t\t\t\t\t<div id=\"rdp-ebb-mw-content-text\" lang=\"en\" dir=\"ltr\" class=\"mw-content-ltr\">\n\n\n<h2><span class=\"mw-headline\" id=\"Abstract\">Abstract<\/span><\/h2>\n<p>Human\u2013information interaction (HII) for simple <a href=\"https:\/\/www.limswiki.org\/index.php\/Information\" title=\"Information\" target=\"_blank\" class=\"wiki-link\" data-key=\"6300a14d9c2776dcca0999b5ed940e7d\">information<\/a> and for complex information is different because people's goals and information needs differ between the two cases. With complex information, comprehension comes from understanding the relationships and interactions within the information and factors outside of a design team's control. Yet, a design team must consider all these within an HII design in order to maximize the communication potential. This paper considers how simple and complex information requires different design strategies and how those strategies differ.\n<\/p><p><b>Keywords<\/b>: human\u2013information interaction, decision-making, complex communication, information design\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Introduction\">Introduction<\/span><\/h2>\n<p>A primary goal of human\u2013information interaction (HII) for complex information is to communicate concepts and ideas to be used for decisions. That is, information that a reader uses to develop an understanding of a situation and to make decisions about a complex situation.<sup id=\"rdp-ebb-cite_ref-AlbersCCI04_1-0\" class=\"reference\"><a href=\"#cite_note-AlbersCCI04-1\" rel=\"external_link\">[1]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlbersHII12_2-0\" class=\"reference\"><a href=\"#cite_note-AlbersHII12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MirelCPS03_3-0\" class=\"reference\"><a href=\"#cite_note-MirelCPS03-3\" rel=\"external_link\">[3]<\/a><\/sup>\n<\/p><p>In any complex communication situation, the proper amount of information should result in maximum communication; of course, too little, too much, poor organization, or inappropriate content reduces the communication. Stating that the communication needs to provide the proper amount of information could considered tautological. It is easy to claim \u2014 and adds little to a discussion \u2014 a text should have the proper information (I use text within the article in a generic sense: paper, web pages, <i>etc<\/i>). However, defining and measuring the communication value of a document (and the information it contains) proves to be a much more difficult task.\n<\/p><p>The interaction and presentation needs of complex information are very different from those of simple information, where a person looks for single information elements. For example, consider the difference in looking up the score of yesterday\u2019s game (simple information) and considering house remodeling to help care for a chronically sick parent (complex information). When we move from a communication viewpoint to an HII viewpoint, we also bring into play issues of how a person can manipulate the information.<sup id=\"rdp-ebb-cite_ref-AlbersHII12_2-1\" class=\"reference\"><a href=\"#cite_note-AlbersHII12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-AlbersHII11_4-0\" class=\"reference\"><a href=\"#cite_note-AlbersHII11-4\" rel=\"external_link\">[4]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SedigID13_5-0\" class=\"reference\"><a href=\"#cite_note-SedigID13-5\" rel=\"external_link\">[5]<\/a><\/sup> The base text may contain the \u201cproper information\u201d but for maximum communication, it may need to be reordered or otherwise manipulated to meet individual needs.\n<\/p><p>A large percentage of the HII literature either looks at simple information search and interaction or conflates simple and complex information. It also tends to take a web-based focus with the person having an essentially infinite number of sources to explore.<sup id=\"rdp-ebb-cite_ref-6\" class=\"reference\"><a href=\"#cite_note-6\" rel=\"external_link\">[6]<\/a><\/sup> While this is true in many cases, within this article, I take a much narrower focus on HII and look at the <i>construction of content<\/i> within a single set of information that is under a content development team\u2019s control, such as the production of corporate reports or public information for a specific goal (an example presented later will be on communicating hurricane information). A view focused on the design team\u2019s task needs to consider what information to create and how people will interact with it. The more commonly discussed view of people searching the Internet (or other sources) for existing information<sup id=\"rdp-ebb-cite_ref-IngwersenTurn05_7-0\" class=\"reference\"><a href=\"#cite_note-IngwersenTurn05-7\" rel=\"external_link\">[7]<\/a><\/sup> will not be directly considered since a design team has no control over how that search proceeds or what information the person may view.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Explanation_of_terms\">Explanation of terms<\/span><\/h2>\n<p>In the introduction, I used the terms simple and complex information. Here I provide a working definition for each. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Simple_information\">Simple information<\/span><\/h3>\n<p>Simple information can be characterized by existing as a single information element. A high percentage of the \u201clook it up on Google\u201d amounts to simple information. For example, a person can look up when a movie was released or a recipe for a carrot cake. Both fit the definition of simple information. Factors that distinguish simple information include:\n<\/p>\n<ul><li> Single path. One path can be fully defined that will result in the answer.<\/li>\n<li> Right\/wrong answer. The correctness can be tested and the information declared correct or not correct.<\/li>\n<li> Complete information. The completeness can be tested and the information declared complete or not complete.<\/li>\n<li> Closed system. All of the factors that might influence the answer can be defined and accounted for.<\/li><\/ul>\n<h3><span class=\"mw-headline\" id=\"Complex_information\">Complex information<\/span><\/h3>\n<p>Complex information exists at the opposite end from simple answers. There is no single \u201canswer\u201d. Addressing complex questions requires using and integrating multiple information elements, which often conflict. People have a complex web of information needs and interactions to fill those needs.<sup id=\"rdp-ebb-cite_ref-ColeIN12_8-0\" class=\"reference\"><a href=\"#cite_note-ColeIN12-8\" rel=\"external_link\">[8]<\/a><\/sup> For example, analyzing monthly business reports and making decisions about this month\u2019s production, or making health care choices. Factors that distinguish complex information include:\n<\/p>\n<ul><li> Multiple paths. There is no single path to an answer. A person can take many different paths and all will work. The effectiveness of the paths may, of course, vary.<\/li>\n<li> Open-ended. The idea of \u201ccomplete\u201d is undefined. A person can continue to collect information and refine their understanding with an essentially infinite amount of information. Instead, a person has to pick a stopping point and make a decision.<\/li>\n<li> Needs cannot be predefined. The information that a person needs cannot be predefined. Of course the major or essential information can be predefined, but the many smaller information elements that can exert a strong influence vary too much between individuals.<\/li>\n<li> History. The information exists within a continuum and that history influences how it gets interpreted and used.<\/li>\n<li> Non-linear. The overall situation shows a non-linear response with small differences in some information elements resulting in very large differences in appropriate decisions.<\/li>\n<li> Open system. All of the factors that might influence the decision cannot be defined. There are too many and the information is dynamic, changing on time scales relevant to the decision situation.<\/li><\/ul>\n<p>Most of the simple information that gets communicated tends to be single facts or procedural type of information. Complex information, on the other hand, deals with a broader scope.\n<\/p>\n<ul><li> Complex information communicates concepts and ideas.<\/li>\n<li> Complex information communicates an understanding of a situation.<\/li>\n<li> Complex information communicates relationships and interactions.<\/li><\/ul>\n<p>Clearly, the HII required to maximize the communication of complex information is very different between the two. Most writing and UX guidelines deal with communicating simple information. But to really address people\u2019s information needs, we need to consider why and how they interact with complex information. Consider the case of <a href=\"https:\/\/www.limswiki.org\/index.php\/Electronic_health_record\" title=\"Electronic health record\" target=\"_blank\" class=\"wiki-link\" data-key=\"f2e31a73217185bb01389404c1fd5255\">electronic health records<\/a> \u2014 they clearly represent complex information \u2014 and represent an example where the design team has control over a significant part of how the HII occurs. The design has to consider many potentially conflicting sets of priorities (development time, maintenance cost, hospital administration goals, government regulations, medical needs, <i>etc.<\/i>), which, in turn, affect how well the users can interact with the information. The input methods must be efficient for data entry, but presenting the information the same way to a nurse or physician may not be the best for integrating it into a clear view of the patient\u2019s condition. The relationships of information (connecting the results of various medical tests, <i>etc.<\/i>) should reflect the initial diagnosis and should also reflect the changing patient status as treatment progresses.\n<\/p><p>In previous work, I have defined a situation as \u201cthe current world state which the user needs to understand. An underlying assumption is that the user needs to interact with a <a href=\"https:\/\/www.limswiki.org\/index.php\/Data_system\" title=\"Data system\" target=\"_blank\" class=\"wiki-link\" data-key=\"686de8e8c43733586ac2ff962f3838a3\">system<\/a> to gain the necessary information in order to understand the situation\u201d<sup id=\"rdp-ebb-cite_ref-AlbersCCI04_1-1\" class=\"reference\"><a href=\"#cite_note-AlbersCCI04-1\" rel=\"external_link\">[1]<\/a><\/sup> (p. 11). The information relevant to the HII exists both inside and outside any circle drawn to enclose \u201cthe situation\u201d (Figure 1). Honestly, the difficulty of drawing that enclosing circle is a characteristic of a complex situation.\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig1_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"32620c019e2eeded96f213d31c31afa0\"><img alt=\"Fig1 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/f\/f8\/Fig1_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 1.<\/b> Complex situation with information both inside and outside of a design team\u2019s control.<br \/>Human\u2013information interaction (HII) only directly applies to the information a design team controls.<\/blockquote>\n<p>Once an upfront analysis has defined the people\u2019s goals and information needs, then the HII factors which support manipulating the information and revealing the relationships can be considered and entered into the design.<sup id=\"rdp-ebb-cite_ref-AlbersHII12_2-2\" class=\"reference\"><a href=\"#cite_note-AlbersHII12-2\" rel=\"external_link\">[2]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ParsonsAdj14_9-0\" class=\"reference\"><a href=\"#cite_note-ParsonsAdj14-9\" rel=\"external_link\">[9]<\/a><\/sup>\n<\/p>\n<blockquote>Typically, the failure of these technical documents [used here in a generic sense for any information source] comes not from a lack of information; the text probably contains an excess of information. Post hoc studies of communication failures find many sources to blame: poor information architecture, poor organization, wrong grade level or writing style, or poor presentation. But instead of seeing these problems as a root cause, let\u2019s consider them as symptoms of a more fundamental problem: a problem stemming from the underlying complexity of the situational context and a failure of the information presentation to match that complexity.<sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-0\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup>(p. 110)<\/blockquote>\n<p>The task assigned to an HII design team is to ensure the underlying complexity is not overly simplified and that the information presentation and manipulation meets the needs of the people and the demands of the situation. As others have discussed, the HII problem for maximizing the communication for complex information is not a tools question, but one of understanding what people need and how they come to understand the information.<sup id=\"rdp-ebb-cite_ref-TaylorQN68_11-0\" class=\"reference\"><a href=\"#cite_note-TaylorQN68-11\" rel=\"external_link\">[11]<\/a><\/sup>\n<\/p>\n<h3><span class=\"mw-headline\" id=\"Information_relationships\">Information relationships<\/span><\/h3>\n<p>Understanding complex information depends on understanding relationships, but the potential relationships are essentially infinite. Comprehension of a complex situation occurs when people can mentally integrate those relationships into their view of the situation.\n<\/p>\n<blockquote>Information integration lies, not in a text element itself, but in the relationships between those elements. A reader needs to figure out what information is relevant and how to connect it to the current problem. Without proper information relationships, the reader does not gain an integrated understanding of information, but instead gains a collection of facts. Without relationships, information exists as a bunch of interesting factoids which do not help a person form an adequate mental picture of the situation. Collections of facts are less than useful for understanding and working with the open-ended problems that people encounter in complex situations.<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-0\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MirelDes03_13-0\" class=\"reference\"><a href=\"#cite_note-MirelDes03-13\" rel=\"external_link\">[13]<\/a><\/sup> Without the relationships, a person learns about <i>X<\/i> and <i>Y<\/i>, but not how <i>X<\/i> and <i>Y<\/i> relate to each other or to <i>Z<\/i> in terms of their current problem or situation.<sup id=\"rdp-ebb-cite_ref-WoodsCan02_14-0\" class=\"reference\"><a href=\"#cite_note-WoodsCan02-14\" rel=\"external_link\">[14]<\/a><\/sup> The text fails to communicate because the reader can\u2019t form the necessary information relationships.<sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-1\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup>(p. 111)<\/blockquote>\n<p>A patient trying to understand a medical problem often has difficulty understanding the situation because they don\u2019t understand the relationships. They can look at a set of lab results and know the numbers, but lack the medical practitioner\u2019s knowledge of what the values should be for their condition. The patient may be worried because some values are high, but the physician is satisfied with them because the specific condition often has even higher values. Likewise, a junior manager needs to develop the knowledge of how to interpret multiple values that may appear on a collection of monthly reports. Good HII can help by providing a more integrated view and providing help with the integration, without interfering with more experienced people\u2019s information interaction.\n<\/p><p>Building relationships means developing a deep-level understanding of the text rather than just a surface-level knowledge. Deep-level knowledge involves seeing the macrostructure of the text and being able to apply prior knowledge to it and fit the text within the reader\u2019s prior knowledge framework. Surface-level knowledge involves knowing the basic text. People with surface-level knowledge can quote the text and if asked recall-type questions would respond with answers very close to the text language, but they would not be able to elaborate on the answer or connect it to other information. People with deep-level understanding would be able to place the text\u2019s information into their own words. A major element in the difference between people being able to develop deep-level understanding versus a surface-level understanding is their prior knowledge.\n<\/p><p>Communicating complex information for decision-making can be viewed as working to help build relationships within the information.<sup id=\"rdp-ebb-cite_ref-ColeIN12_8-1\" class=\"reference\"><a href=\"#cite_note-ColeIN12-8\" rel=\"external_link\">[8]<\/a><\/sup> Unfortunately, building relationships is a great theoretical concept, but one that does not lend itself to a direct operational definition. One way to define judging or qualifying relationships can be how they reduce the uncertainty a person has about the situation.<sup id=\"rdp-ebb-cite_ref-AlbersIR09_15-0\" class=\"reference\"><a href=\"#cite_note-AlbersIR09-15\" rel=\"external_link\">[15]<\/a><\/sup> This reshapes an HII design team\u2019s goal to one of focusing content creation in terms of what information does the reader need to reduce their uncertainty and, consequently, to build a web of relationships between the information elements. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Contextual_awareness\">Contextual awareness<\/span><\/h3>\n<p>Except for training material, the readers of technical information typically understand the basics but need to know specific information about the current situation (as opposed to the general situation) in order to make decisions. The understanding of a complex situation needed to make informed decisions comes when people can distinguish the information structure, grasp the relationships within it<sup id=\"rdp-ebb-cite_ref-FodelHII12_16-0\" class=\"reference\"><a href=\"#cite_note-FodelHII12-16\" rel=\"external_link\">[16]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SaracevicRel07_17-0\" class=\"reference\"><a href=\"#cite_note-SaracevicRel07-17\" rel=\"external_link\">[17]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-ThuringHyp95_18-0\" class=\"reference\"><a href=\"#cite_note-ThuringHyp95-18\" rel=\"external_link\">[18]<\/a><\/sup>, and make inferences on the future evolution of the situation.\n<\/p><p>Building on Endsley\u2019s (1995)<sup id=\"rdp-ebb-cite_ref-EndsleyTow95_19-0\" class=\"reference\"><a href=\"#cite_note-EndsleyTow95-19\" rel=\"external_link\">[19]<\/a><\/sup> situation awareness work, we can call this contextual awareness. Contextual awareness is the understanding of the information within an informational situation which forms the basis for how to interpret new information and how to make decisions for interacting with that situation. With poor contextual awareness, people can know something is occurring or that a particular piece of information exists, but they cannot easily find relevant related information or they have the information but do not understand how it relates to the overall situation. On the other hand, good contextual awareness does not guarantee a person will form the proper intention or make the proper decisions; the error analysis literature is filled with cases where people understood a situation, but still made incorrect choices. Unfortunately for design teams, the concept of relevance itself is highly nuanced and multifaceted<sup id=\"rdp-ebb-cite_ref-SaracevicRel07_17-1\" class=\"reference\"><a href=\"#cite_note-SaracevicRel07-17\" rel=\"external_link\">[17]<\/a><\/sup>, creating a complex interplay that must be understood to engineer high quality HII.\n<\/p><p>Elements of good contextual awareness are (Figure 2): (1) Understands how the information fits within the current situation. (2) Understands the information relationships.<sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-2\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup> Information comprehension requires knowing how information relates to other information. (3) Understands the future development of the situation and can make predictions about the ripple effects of any decision across the entire situation. HII that supports complex information needs to provide the interactions that support people developing high quality contextual awareness. \n<\/p><p>Kain, de Jong, and Smith (2010)<sup id=\"rdp-ebb-cite_ref-KainIU10_20-0\" class=\"reference\"><a href=\"#cite_note-KainIU10-20\" rel=\"external_link\">[20]<\/a><\/sup> study into how to communicate hurricane risks and warnings highlighted the issues of how people interpret information and make decisions about how they will react. The hurricane experts had their view of what information was needed and how it should be presented, but the research showed the people wanted\/needed a different presentation. Their mental methods of forming relationships and of interpreting complex information differed from what the experts thought. The process of building contextual awareness differs and a design team\u2019s analysis must capture those differences.\n<\/p><p>A design team working on developing the HII for a country\u2019s hurricane awareness plan needs to balance both the expert\u2019s \u201chere is what the people must hear\u201d against the more pragmatical \u201chere\u2019s what I want to know\u201d as well as the local people\u2019s opinion of how they react to hurricane warnings. Many of them have been through multiple hurricanes and have strongly held views that often conflict with the authorities. Notice how in this case, the design team has control of the information. A general search on hurricane warnings will have sentences such as \u201cconsult your local authorities for evacuation routes\u201d but, here, the design team will be tasked with providing that evacuation route information. At the same time, they need to ensure the relationships between evacuation route, getting ready for evacuation, and planning on returning are all clearly laid out and connected. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig2_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"4cb8c42b6e866a5cb861e21402624426\"><img alt=\"Fig2 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/7\/71\/Fig2_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 2.<\/b> Stages of developing contextual awareness<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"HII_is_information_interaction.2C_not_data_interaction\">HII is information interaction, not data interaction<\/span><\/h2>\n<p>People who are engaged with decision-making and complex information need to be presented with information, not data (Figure 3). Quick definition of my terms here, which I discuss in more detail elsewhere.<sup id=\"rdp-ebb-cite_ref-AlbersCCI04_1-2\" class=\"reference\"><a href=\"#cite_note-AlbersCCI04-1\" rel=\"external_link\">[1]<\/a><\/sup>\n<\/p>\n<ul><li> Data: Raw numbers, facts, and figures.<\/li>\n<li> Information: Information is data in context. It relates to the situation and contains the relationships that connect the information to the situation.<\/li>\n<li> Knowledge: Interconnected web of the relevant information and the relationships linking the information within the situation. <\/li><\/ul>\n<p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig3_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"a0702e0c4b311eec5fe513fe33a4bcde\"><img alt=\"Fig3 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/8\/83\/Fig3_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 3.<\/b> Data, information, and knowledge hierarchy. The higher the HII works in the hierarchy,<br \/>the better it fits building an understanding of the situation.<\/blockquote>\n<p>As an example, I heard a presentation that looked at the effect of sea level rise on the Norfolk, VA area.<sup id=\"rdp-ebb-cite_ref-RichardsUsa10_21-0\" class=\"reference\"><a href=\"#cite_note-RichardsUsa10-21\" rel=\"external_link\">[21]<\/a><\/sup> A software program had nice manipulation that let a person dynamically see the effects of different amounts of sea level rise from global warming and how it would affect the city. The problem was not with the implementation, but with the underlying assumptions of a design team. They were assuming that by providing a tool and letting people see how different sea level changes would affect the area that it would bring about understanding and increase long-term preparedness. But the tool was presenting content at the data level. Yes, I can change the sliders and see how the sea level changes, but it was devoid of supporting content (in the metaphor of this paper, the tool was a single puzzle piece). It not only didn\u2019t connect with other pieces, those other pieces were not provided. As a result, a global warming denier could play with the model, agree that a four foot rise would be a catastrophic problem, but then reject it as something that would never happen. One of the software\u2019s goals was to help people prepare, but without giving them the content and relationships to build their contextual awareness; it was a single data point and not part of a coherent presentation of information. \n<\/p><p>The transition from information to knowledge is important since it involves comprehending the relationships within the data and placing them within the context of the situation. Moving to knowledge is essential to building contextual awareness and must be the goal of an HII design team.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"Communication_of_simple_and_complex_information\">Communication of simple and complex information<\/span><\/h2>\n<p>Many discussions on communication start with Shannon information theory (1948)<sup id=\"rdp-ebb-cite_ref-ShannonMath48_22-0\" class=\"reference\"><a href=\"#cite_note-ShannonMath48-22\" rel=\"external_link\">[22]<\/a><\/sup> (Figure 4), which works for analyzing simple information. With clearly defined information to be conveyed, a design team can concentrate on efficient communication methods. Unfortunately, Shannon was really discussing the minimum information to convey a message, not how to communicate in a natural language. It works well for computer-computer communication, but is not as applicable to human-human communication, especially for complex information, with or without a computer mediator. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig4_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"5d31159b79e6cb60a6709b135d431527\"><img alt=\"Fig4 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/b\/be\/Fig4_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 4.<\/b> Basic block diagram of Shannon information theory. Information moves from the source<br \/>to the final destination with the overall goal to be minimizing both the noise and total amount of required content.<\/blockquote>\n<p>Driven in part by a Shannon-based desire to efficiently communication information, design teams try to break the communication into individual components. This fits with our reductionist nature to break any problem into individual components. Standard approaches for communicating complex information (and analyzing any complex system) are to assume it is assembled from smaller systems<sup id=\"rdp-ebb-cite_ref-MirelCPS03_3-1\" class=\"reference\"><a href=\"#cite_note-MirelCPS03-3\" rel=\"external_link\">[3]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-MirelDes03_13-1\" class=\"reference\"><a href=\"#cite_note-MirelDes03-13\" rel=\"external_link\">[13]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-RedishExp07_23-0\" class=\"reference\"><a href=\"#cite_note-RedishExp07-23\" rel=\"external_link\">[23]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SommervilleLar12_24-0\" class=\"reference\"><a href=\"#cite_note-SommervilleLar12-24\" rel=\"external_link\">[24]<\/a><\/sup>. Thus, it is hardly a surprising statement that we see a complex system as the sum of its components: 1 + 1 + 1 + 1 + \u2026.. = THE SYSTEM.\n<\/p><p>Unfortunately, assuming a complex system is the sum of its components at best redefines it as a simple system. The complexity that people need to understand and the HII must support is not just a sum of individual components, but the interactions and relationships between components. The redefinition to a sum of individual components ignores the complex interactions that give rise to more than the sum of the parts: 1 + 1 + 1 + 1 + \u2026..+ <i>i<\/i><sub>n<\/sub> >> \u01a9<sub>n<\/sub>. \n<\/p><p>The communication problem, and consequently the HII problem, is that redefining the problem to be a simple problem fundamentally changes the problem itself. As a result, the communication itself changes and the relationships and interactions that are essential to understanding get lost. The communication-related literature seems to generally miss this important concept, although the computer science literature consistently contains quotes such as \u201cthe fundamental reason today\u2019s software engineering cannot effectively manage inherent complexity is that its basis is in developing individual programs rather than in interacting systems.\u201d<sup id=\"rdp-ebb-cite_ref-SommervilleLar12_24-1\" class=\"reference\"><a href=\"#cite_note-SommervilleLar12-24\" rel=\"external_link\">[24]<\/a><\/sup> (p. 73). In designs that strive to communicate complex information, an over-privileging of developing and\/or organizing content while discounting the interactions and relationships within and between content leads to results similar to those critiqued by Sommerville. \n<\/p><p>In the introduction, I described most simple information as being single information elements that can be looked up (Figure 5). Complex information on the other hand, has a multitude of information elements that a person must understand, some of which are outside of the bounds of a HII system. Factors such as asking a person down the hall, having heard of a bad experience with product <i>X<\/i>, <i>etc.<\/i><sup id=\"rdp-ebb-cite_ref-AlbersUsa10_10-3\" class=\"reference\"><a href=\"#cite_note-AlbersUsa10-10\" rel=\"external_link\">[10]<\/a><\/sup> (p. 115) \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig5_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"3e15fa47463973960e327cfff064f68e\"><img alt=\"Fig5 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/d\/d4\/Fig5_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 5.<\/b> Relationships and understanding a situation<\/blockquote>\n<p>Figure 5 shows content as nice rectangles. But they could be better viewed as jigsaw pieces with the interlocks on the pieces representing the relationships (Figure 6), how the pieces interconnect with each other into a coherent understanding. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig6_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"2e367319ac7c5bae39a5bc8a6e19c47e\"><img alt=\"Fig6 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/7\/7b\/Fig6_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 6.<\/b> Relationships and the complexities of assembling information into a coherent understanding.<br \/>The understanding comes from both the content and knowing how it fits together.<\/blockquote>\n<p>Unfortunately, the pieces in Figure 6 are not static jigsaw shapes. They are better viewed as a collection of puzzle pieces that change shape as they move away\/toward other pieces, and as they merge together (Figure 7). As a result, changes to one piece can ripple through the entire information web. A change to a piece in the upper left corner results in many of the pieces changing in some relevant manner. The information and HII appropriate and relevant for each point of a situation can be different. Situations are not static; they have a beginning, middle, and end. The puzzle pieces of Figure 7 are not just dynamic from a person\u2019s interaction, but also change over time and over the situation\u2019s evolution. \n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig7_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"db7345555aef81a75e400460d660210f\"><img alt=\"Fig7 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/9\/9f\/Fig7_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 7.<\/b> Dynamic evolution of information. The relationships between information elements<br \/>can change over both time and by the act of connecting them.<\/blockquote>\n<p>Let\u2019s reexamine the six characteristics of complex information with respect to the puzzle metaphor.\n<\/p>\n<dl><dd><b>Multiple paths<\/b>: People have multiple paths through the information. The order in which they move between the puzzle pieces cannot be predefined. Each HII with a piece can change how it interconnects; thus, different paths through the information change how people build the relationships and, consequently, how the pieces fit together. <\/dd><\/dl>\n<dl><dd><b>Open-ended<\/b>: No clear point of enough information. The HII cannot move toward a predefined point of \u201cnow you have all the information.\u201d With a goal of communicating concepts and ideas, how many pieces a person needs, cannot be defined. Of course, the issue of information needs versus information wants also comes into play. Coupled with this is the fact that information search and problem-solving are sufficing processing.<sup id=\"rdp-ebb-cite_ref-KleinSou99_25-0\" class=\"reference\"><a href=\"#cite_note-KleinSou99-25\" rel=\"external_link\">[25]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-SimonMod79_26-0\" class=\"reference\"><a href=\"#cite_note-SimonMod79-26\" rel=\"external_link\">[26]<\/a><\/sup> People stop once they are comfortable with their understanding of the found information. Unfortunately, people are poor judges of knowing they have found adequate information.<\/dd><\/dl>\n<dl><dd><b>Cannot be predefined<\/b>: How many pieces a person will interact with remains unknowable to a design team. Interestingly, the size (content) of individual pieces is dynamic; some people need smaller\/larger pieces to effectively comprehend the overall situation and tend to make choices in terms of immediate, rather than long-term, efficiency and effort of the HII.<sup id=\"rdp-ebb-cite_ref-FennemaAnt95_27-0\" class=\"reference\"><a href=\"#cite_note-FennemaAnt95-27\" rel=\"external_link\">[27]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-PayneAda01_28-0\" class=\"reference\"><a href=\"#cite_note-PayneAda01-28\" rel=\"external_link\">[28]<\/a><\/sup><\/dd><\/dl>\n<dl><dd><b>History<\/b>: Past history of a situation affects how the pieces will evolve and the past history of the people interacting with the information affects how they interpret it. Two situations may appear to be the same but the past strongly influences how the piece will change. For example, six months ago sales in the southeast region were down by 30%. A set of decisions was made and sales have steadily increased since them. They are still 10% below desired levels, but are increasing at an acceptable rate. This situation is much different than if sales are 10% below the desired level and not conforming to predictions.<\/dd><\/dl>\n<dl><dd><b>Non-linear<\/b>: The relationships between pieces can show non-linear response to changes. Minor differences in past history or the HII path can result in widely different \u2014 but appropriate \u2014 end points. In individual piece can morph into very different final shapes, even though they started from similar initial conditions.<\/dd><\/dl>\n<dl><dd><b>Open system<\/b>: As the system evolves, the overall content within the system changes. Some new information gets introduced and some information drops out. The number of pieces, their content, and their shape can change.<\/dd><\/dl>\n<p>When the information is viewed as dynamic puzzle pieces, how those pieces change shape helps to emphasize how many solutions to problems introduce other problems, sometimes worse than the one they were intended to solve. For example, many environmental solutions inevitably end up doing more harm than good. The future changes to relationships are not understood when decisions are made, so the evolution of the situation is poorly predicted (poor contextual awareness). Worse, too many people make assumptions that they can change one piece with nothing else changing. In complex ecosystems \u2014 biological, educational or industrial \u2014 actions always have consequences that are hard to predict; a change to one piece ripples through all the others. The \u201chard to predict\u201d is a hallmark of a complex system. HII that supports interacting with the information must consider those ramifications.\n<\/p><p>Beyond the need for the HII to handle dynamic puzzle pieces, it must also allow for easy trimming of the information space. All of the available information is not relevant to any specific situation. Instead, a person needs a select collection of information (Figure 8).\n<\/p><p><a href=\"https:\/\/www.limswiki.org\/index.php\/File:Fig8_Albers_Informatics2015_2-2.jpg\" class=\"image wiki-link\" target=\"_blank\" data-key=\"79fadafb2918308d2d80aa78f461f323\"><img alt=\"Fig8 Albers Informatics2015 2-2.jpg\" src=\"https:\/\/www.limswiki.org\/images\/e\/e5\/Fig8_Albers_Informatics2015_2-2.jpg\" style=\"width: 100%;max-width: 400px;height: auto;\" \/><\/a>\n<\/p>\n<div style=\"clear:both;\"><\/div>\n<blockquote><b>Figure 8.<\/b> Information relevant to a situation. Each situation (or class of situations) only needs<br \/>a subset of the information. High quality HII helps to keep the focus on only the relevant subset.<\/blockquote>\n<h2><span class=\"mw-headline\" id=\"HII_for_decision-making\">HII for decision-making<\/span><\/h2>\n<p>The overall scope of the information flow for the HII of complex information moves to prominence when the interaction goal is decision-making. People need to collect information, analyze it, and make a decision; a process that depends on the flow of information both within and outside the control of an HII design team.\n<\/p><p>Decision-making occurs as a result of comparing what is perceived in the environment and what is known by the decision-maker.<sup id=\"rdp-ebb-cite_ref-MasonProg73_29-0\" class=\"reference\"><a href=\"#cite_note-MasonProg73-29\" rel=\"external_link\">[29]<\/a><\/sup> Essentially all of our current literature discusses considering the needs of people when designing information. However, current practice does not typically consider how people\u2019s questions depend on how they make decisions and interact with the system.<sup id=\"rdp-ebb-cite_ref-GerlachUnd91_30-0\" class=\"reference\"><a href=\"#cite_note-GerlachUnd91-30\" rel=\"external_link\">[30]<\/a><\/sup><sup id=\"rdp-ebb-cite_ref-WoodsGoc88_31-0\" class=\"reference\"><a href=\"#cite_note-WoodsGoc88-31\" rel=\"external_link\">[31]<\/a><\/sup> HII attempts to take the entire cycle of information interpretation and decision-making and place it within people\u2019s current situation.\n<\/p><p>Mirel (1998)<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-1\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup> follows the same line of thinking as Conklin\u2019s (2003)<sup id=\"rdp-ebb-cite_ref-ConklinWicked_32-0\" class=\"reference\"><a href=\"#cite_note-ConklinWicked-32\" rel=\"external_link\">[32]<\/a><\/sup> wicked problems when she points out that analyzing complex tasks requires seeing more than a single path:\n<\/p>\n<blockquote>This broader view is necessary to capture the following traits of complex tasks: paths of action that are unpredictable, paths that are never completely visible from any one vantage point, and nuance judgments and interpretations that involve multiple factors and that yield many solutions.<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-2\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup> (p. 14)<\/blockquote>\n<p>Mirel may have focused on the analysis aspects, but that is an early and vital step in any information interaction, especially for HII of complex information. Unless a design team clearly understands the people\u2019s goals and information needs, there is minimal chance for the HII to support it.\n<\/p><p>The issue of wicked problems and unpredictable paths that both Conklin and Mirel discuss brings to light an interesting point. As design teams work toward HII of complex information, they encounter an interesting and confounding recursion: the HII of complex information is itself a complex problem and, as such, does not lend itself to easy answers. Yet, many design teams try to address those issues with the same process that works for simple information, which leads to an easy-answer mentality. As a result, they reduce the scope and avoid confronting the complex information HII issues head-on. Design teams who try to reduce all problems to a collection of individual parts or who assume they should consider only the information they control fall into this trap. They have redefined the complex HII problem into a simple problem. \n<\/p>\n<h3><span class=\"mw-headline\" id=\"Decision-making_strategies\">Decision-making strategies<\/span><\/h3>\n<p>People use the available information to make a decision about how to proceed in their current situation; information use is always based upon the context within which it is used.<sup id=\"rdp-ebb-cite_ref-FodelHII12_16-1\" class=\"reference\"><a href=\"#cite_note-FodelHII12-16\" rel=\"external_link\">[16]<\/a><\/sup> This hard connection between use and context greatly complicates a design team\u2019s task since they must understand both factors. And, depending on audience, the base information may exist within multiple contexts.\n<\/p><p>Based on the results of the decision, people take action and then need to re-analyze the situation. Closely related to decision-making is monitoring the situation to ensure it continues along the predicted path. If the information fits the expected progress, then a decision of \u201cdo nothing\u201d is correct. However, if the information indicates that it is deviating, then people cycle back into decision-making.\n<\/p><p>There are two broad decision-making strategies that people use. \n<\/p>\n<h4><span class=\"mw-headline\" id=\"Experience-based\">Experience-based<\/span><\/h4>\n<p>Experience-based decision-making is used the most and is a person\u2019s default strategy. It is a rule-based strategy: \u201cif <i>X<\/i> happens, then do <i>Y<\/i>.\u201d Most decision-making, at least in the corporate world, is\nexperienced-based.<sup id=\"rdp-ebb-cite_ref-MirelApp98_12-3\" class=\"reference\"><a href=\"#cite_note-MirelApp98-12\" rel=\"external_link\">[12]<\/a><\/sup> The reader is skilled in the overall area (such as a CFO making financial decisions), but does need specific information for the current situation. They need to know how this month\u2019s production or sales numbers differ from last month\u2019s or between different regions, but they do have a set of mental rules on how to interpret those numbers.\n<\/p><p>Based on past experience, people develop a rule-based strategy so they can reliably expect that \u201cwhen <i>X<\/i> happens, then do <i>Y<\/i>.\u201d Rule-based decisions reduce cognitive effort and work well within normal situations because the rules themselves evolved from past experience. However, that same rule-based strategy can cause problems if a person jumps to a decision based on incomplete information. <i>X<\/i> can happen in multiple situations but other factors differ\u2014the relationships of what\/how <i>X<\/i> is interconnected within the situation\u2014which cause solution <i>Y<\/i> to only be an appropriate response in some situations.\n<\/p><p>Klein\u2019s (1999)<sup id=\"rdp-ebb-cite_ref-KleinSou99_25-1\" class=\"reference\"><a href=\"#cite_note-KleinSou99-25\" rel=\"external_link\">[25]<\/a><\/sup> recognition-primed model of decision-making posits that experienced people do little problem solving, especially in routine situations. Instead, once they recognize a situation, they form a possible intention very rapidly, mentally evaluate it, and, if no major problems are evident, take action on it. Alternatives are not considered. In other words, they tend to use their prior knowledge and initial assessment to immediately pick a solution, and if it seems workable, then that is considered the answer. (Concerns about whether or not the solution is optimal are not part of the mental evaluation.)\n<\/p><p>Post-failure analysis tends to highlight that the decision-makers had the proper information, but improperly interpreted it. In many cases, it is a result of using experience-based decision-making and not realizing the \u201cif x, then y\u201d rule did not apply to the situation. \n<\/p>\n<h4><span class=\"mw-headline\" id=\"Knowledge-based\">Knowledge-based<\/span><\/h4>\n<p>Experience-based decision-making can break down, typically when the situation contains unexpected factors or relationships. When people recognize their rules no longer apply, they have to shift to knowledge-based decision-making. Unfortunately, the shift to knowledge-based decision-making often happens after the monitoring phase of an experienced-based strategy reveals the situation is not developing as predicted.\n<\/p><p>In knowledge-based decision, a person knows their experience-based rule set has broken down and they are no longer working within a normal situation. Something is not right in the current situation and they are trying to figure out what to change to fix the problem. In other words, the expected relationships are not being found and a person needs to understand why. It tends to happen when people do not understand what caused the base problem. Unlike decision-making based on rules of \u201cwhen <i>X<\/i> happens, then do <i>Y<\/i>,\u201d a person has to fully develop their contextual awareness so they can make a decision.\n<\/p><p>The HII of complex information applies more for knowledge-based strategies than it does for experience-based ones. Knowledge-based strategies require people to engage in a systematic interaction with the information and work to build up their contextual awareness. \n<\/p><p>Someone working on a global warming problem is working at the knowledge-based and not experienced-based level. Allowing people to manipulate data as if it was a simple information \u2014 a collection of individual components \u2014 can misguide them into thinking it\u2019s an experience-based problem and they can draw on their prior knowledge. Worse, defining the problem as \u201cif they can work this sea level rise model, then they will understand the problem\u201d redefines the problem from a complex problem to a simple problem. Unfortunately, but inevitable with the history component of complex information, prior knowledge and biases strongly influence how information gets interpreted. Of course, with global warming\u2014and many other situations\u2014the political aspects, factors outside of an HII design team\u2019s control, come to the forefront. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conclusions\">Conclusions<\/span><\/h2>\n<p>Providing people with any collection of information, especially complex information, can easily result in information overload. With the large amounts of information and its dynamic nature, high quality HII is essential for people to comprehend that information. \n<\/p><p>Rather than a lack of information, the failure to anticipate people\u2019s needs forms the basis of most information problems and poor decision-making; an issue design teams must explicitly address. Managers typically suffer from a shortage of decision-relevant information and a simultaneous overabundance of irrelevant information.<sup id=\"rdp-ebb-cite_ref-AckoffMan67_33-0\" class=\"reference\"><a href=\"#cite_note-AckoffMan67-33\" rel=\"external_link\">[33]<\/a><\/sup> People are forced to make decisions based on an unfiltered avalanche of information that may or may not be relevant to their needs.\n<\/p><p>Understanding a complex situation comes from understanding the relationships within the information. Building that understanding results not in knowing the individual pieces of information but in knowing how those pieces of information interconnect. That, in turn, requires looking at the system as not just a large collection of information, but as a highly dynamic integration of information and relationships between information. As a design team works on developing the system HII, they must ensure they capture those relationships and interconnections so they can be explicitly made to the people using the system. As such, a design team must:\n<\/p>\n<ul><li> Define the situations that must be understood<\/li>\n<li> Determine the information people need to understand the situation<\/li>\n<li> Determine how that information is connected and how people see those connections as they build their contextual awareness<\/li>\n<li> Understand how the information and relationships change as the situation evolves<\/li>\n<li> Understand the biases, interaction expectations, and decision-making styles of the audience<\/li>\n<li> Determine the best HII for presenting the information to achieve the best information communication while allowing for those biases, interaction expectations, and decision-making styles<\/li><\/ul>\n<p>HII plays a fundamental part in ensuring the overall information content is usable and that people can form the relationships needed to build their contextual awareness. As that understanding develops, people develop their contextual awareness of the situation and can make informed decisions. \n<\/p>\n<h2><span class=\"mw-headline\" id=\"Conflict_of_interests\">Conflict of interests<\/span><\/h2>\n<p>The author declares there is no conflict of interest.\n<\/p>\n<h2><span class=\"mw-headline\" id=\"References\">References<\/span><\/h2>\n<ol class=\"references\">\n<li id=\"cite_note-AlbersCCI04-1\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlbersCCI04_1-0\" rel=\"external_link\">1.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersCCI04_1-1\" rel=\"external_link\">1.1<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersCCI04_1-2\" rel=\"external_link\">1.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Albers, M. (2004). <i>Communication of Complex Information: User Goals and Information Needs for Dynamic Web Information<\/i>. Erlbaum.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Communication+of+Complex+Information%3A+User+Goals+and+Information+Needs+for+Dynamic+Web+Information&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=2004&rft.pub=Erlbaum&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersHII12-2\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlbersHII12_2-0\" rel=\"external_link\">2.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersHII12_2-1\" rel=\"external_link\">2.1<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersHII12_2-2\" rel=\"external_link\">2.2<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Albers, M. (2012). <i>Human-Information Interaction and Technical Communication: Concepts and Frameworks<\/i>. IGI Global.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Human-Information+Interaction+and+Technical+Communication%3A+Concepts+and+Frameworks&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=2012&rft.pub=IGI+Global&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MirelCPS03-3\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MirelCPS03_3-0\" rel=\"external_link\">3.0<\/a><\/sup> <sup><a href=\"#cite_ref-MirelCPS03_3-1\" rel=\"external_link\">3.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Mirel, B. (2003). <i>Interaction Design for Complex Problem Solving: Developing Useful and Usable Software<\/i>. Morgan Kaufmann.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Interaction+Design+for+Complex+Problem+Solving%3A+Developing+Useful+and+Usable+Software&rft.aulast=Mirel%2C+B.&rft.au=Mirel%2C+B.&rft.date=2003&rft.pub=Morgan+Kaufmann&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersHII11-4\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlbersHII11_4-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Albers, M. (July 2011). \"Human-information interactions with complex software\". <i>Proceedings of the HCI International<\/i>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Human-information+interactions+with+complex+software&rft.jtitle=Proceedings+of+the+HCI+International&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=July+2011&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SedigID13-5\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SedigID13_5-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sedig, K.; Parsons, P. (2013). \"Interaction design for complex cognitive activities with visual representations: A pattern-based approach\". <i>AIS Transactions on Human-Computer Interaction<\/i> <b>5<\/b>: 84\u2013133.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Interaction+design+for+complex+cognitive+activities+with+visual+representations%3A+A+pattern-based+approach&rft.jtitle=AIS+Transactions+on+Human-Computer+Interaction&rft.aulast=Sedig%2C+K.%3B+Parsons%2C+P.&rft.au=Sedig%2C+K.%3B+Parsons%2C+P.&rft.date=2013&rft.volume=5&rft.pages=84%E2%80%93133&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-6\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-6\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Cole, C. (2014). \"Google, tear down this wall to exploratory search!\". <i>Bulletin of the Association for Information Science and Technology<\/i> <b>40<\/b>: 50\u201354.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Google%2C+tear+down+this+wall+to+exploratory+search%21&rft.jtitle=Bulletin+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Cole%2C+C.&rft.au=Cole%2C+C.&rft.date=2014&rft.volume=40&rft.pages=50%E2%80%9354&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-IngwersenTurn05-7\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-IngwersenTurn05_7-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Ingwersen, P.; Jarvelin, K. (2005). <i>The Turn: Integration of Information Seeking and Retrieval in Context<\/i>. Springer.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=The+Turn%3A+Integration+of+Information+Seeking+and+Retrieval+in+Context&rft.aulast=Ingwersen%2C+P.%3B+Jarvelin%2C+K.&rft.au=Ingwersen%2C+P.%3B+Jarvelin%2C+K.&rft.date=2005&rft.pub=Springer&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ColeIN12-8\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-ColeIN12_8-0\" rel=\"external_link\">8.0<\/a><\/sup> <sup><a href=\"#cite_ref-ColeIN12_8-1\" rel=\"external_link\">8.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Cole, C. (2012). <i>Information Need: A Theory Connecting Information Search to Knowledge Formation<\/i>. Information Today.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Information+Need%3A+A+Theory+Connecting+Information+Search+to+Knowledge+Formation&rft.aulast=Cole%2C+C.&rft.au=Cole%2C+C.&rft.date=2012&rft.pub=Information+Today&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ParsonsAdj14-9\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ParsonsAdj14_9-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Parsons, P.; Sedig, K. (2014). \"Adjustable properties of visual representations: Improving the quality of human-information interaction\". <i>Journal of the Association for Information Science and Technology<\/i> <b>65<\/b>: 455\u2013482.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adjustable+properties+of+visual+representations%3A+Improving+the+quality+of+human-information+interaction&rft.jtitle=Journal+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Parsons%2C+P.%3B+Sedig%2C+K.&rft.au=Parsons%2C+P.%3B+Sedig%2C+K.&rft.date=2014&rft.volume=65&rft.pages=455%E2%80%93482&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersUsa10-10\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-AlbersUsa10_10-0\" rel=\"external_link\">10.0<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersUsa10_10-1\" rel=\"external_link\">10.1<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersUsa10_10-2\" rel=\"external_link\">10.2<\/a><\/sup> <sup><a href=\"#cite_ref-AlbersUsa10_10-3\" rel=\"external_link\">10.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Albers, M.; Still, B. (Eds.) (2010). \"Usability and information relationships: Considering content relationships when testing complex information\". <i>Usability of Complex Information Systems: Evaluation of User Interaction<\/i>. CRC Press. pp. 109\u2013132.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Usability+and+information+relationships%3A+Considering+content+relationships+when+testing+complex+information&rft.atitle=Usability+of+Complex+Information+Systems%3A+Evaluation+of+User+Interaction&rft.aulast=Albers%2C+M.%3B+Still%2C+B.+%28Eds.%29&rft.au=Albers%2C+M.%3B+Still%2C+B.+%28Eds.%29&rft.date=2010&rft.pages=pp.%26nbsp%3B109%E2%80%93132&rft.pub=CRC+Press&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-TaylorQN68-11\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-TaylorQN68_11-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Taylor, R.S. (1968). \"Question-negotiation and information seeking in libraries\". <i>College & Research Libraries<\/i> <b>29<\/b>: 178\u2013194.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Question-negotiation+and+information+seeking+in+libraries&rft.jtitle=College+%26+Research+Libraries&rft.aulast=Taylor%2C+R.S.&rft.au=Taylor%2C+R.S.&rft.date=1968&rft.volume=29&rft.pages=178%E2%80%93194&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MirelApp98-12\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MirelApp98_12-0\" rel=\"external_link\">12.0<\/a><\/sup> <sup><a href=\"#cite_ref-MirelApp98_12-1\" rel=\"external_link\">12.1<\/a><\/sup> <sup><a href=\"#cite_ref-MirelApp98_12-2\" rel=\"external_link\">12.2<\/a><\/sup> <sup><a href=\"#cite_ref-MirelApp98_12-3\" rel=\"external_link\">12.3<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">\"Applied constructivism for user documentation\". <i>Journal of Business and Technical Communication<\/i> <b>12<\/b>: 7\u201349. 1998.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Applied+constructivism+for+user+documentation&rft.jtitle=Journal+of+Business+and+Technical+Communication&rft.date=1998&rft.volume=12&rft.pages=7%E2%80%9349&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MirelDes03-13\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-MirelDes03_13-0\" rel=\"external_link\">13.0<\/a><\/sup> <sup><a href=\"#cite_ref-MirelDes03_13-1\" rel=\"external_link\">13.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Mirel, B.; Albers, M. (Ed.); Mazur, B. (Ed.) (2003). \"Dynamic usability: Designing usefulness into systems for complex tasks\". <i>Content and Complexity: Information Design in Software Development and Documentation<\/i>. Erlbaum. pp. 233\u2013261.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Dynamic+usability%3A+Designing+usefulness+into+systems+for+complex+tasks&rft.atitle=Content+and+Complexity%3A+Information+Design+in+Software+Development+and+Documentation&rft.aulast=Mirel%2C+B.%3B+Albers%2C+M.+%28Ed.%29%3B+Mazur%2C+B.+%28Ed.%29&rft.au=Mirel%2C+B.%3B+Albers%2C+M.+%28Ed.%29%3B+Mazur%2C+B.+%28Ed.%29&rft.date=2003&rft.pages=pp.%26nbsp%3B233%E2%80%93261&rft.pub=Erlbaum&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoodsCan02-14\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WoodsCan02_14-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">\"Can we ever escape from data overload? A cognitive systems diagnosis\". <i>Cognition, Technology & Work<\/i> <b>4<\/b>: 22\u201336. 2002.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Can+we+ever+escape+from+data+overload%3F+A+cognitive+systems+diagnosis&rft.jtitle=Cognition%2C+Technology+%26+Work&rft.date=2002&rft.volume=4&rft.pages=22%E2%80%9336&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AlbersIR09-15\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AlbersIR09_15-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Albers, M. (October 2009). \"Information relationships: The source of useful and usable content\". <i>Proceedings of the 29th Annual International Conference on Computer Documentation<\/i>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Information+relationships%3A+The+source+of+useful+and+usable+content&rft.jtitle=Proceedings+of+the+29th+Annual+International+Conference+on+Computer+Documentation&rft.aulast=Albers%2C+M.&rft.au=Albers%2C+M.&rft.date=October+2009&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FodelHII12-16\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-FodelHII12_16-0\" rel=\"external_link\">16.0<\/a><\/sup> <sup><a href=\"#cite_ref-FodelHII12_16-1\" rel=\"external_link\">16.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Fidel, R. (2012). <i>Human Information Interaction: An Ecological Approach to Information Behavior<\/i>. MIT.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Human+Information+Interaction%3A+An+Ecological+Approach+to+Information+Behavior&rft.aulast=Fidel%2C+R.&rft.au=Fidel%2C+R.&rft.date=2012&rft.pub=MIT&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SaracevicRel07-17\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SaracevicRel07_17-0\" rel=\"external_link\">17.0<\/a><\/sup> <sup><a href=\"#cite_ref-SaracevicRel07_17-1\" rel=\"external_link\">17.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Saracevic, T. (2007). \"Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II. Nature and manifestations of relevance\". <i>Journal of the Association for Information Science and Technology<\/i> <b>58<\/b>: 1915\u20131933.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Relevance%3A+A+review+of+the+literature+and+a+framework+for+thinking+on+the+notion+in+information+science.+Part+II.+Nature+and+manifestations+of+relevance&rft.jtitle=Journal+of+the+Association+for+Information+Science+and+Technology&rft.aulast=Saracevic%2C+T.&rft.au=Saracevic%2C+T.&rft.date=2007&rft.volume=58&rft.pages=1915%E2%80%931933&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ThuringHyp95-18\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ThuringHyp95_18-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Thuring, M.; Hannemann, J.; Haake, J. (1995). \"Hypermedia and cognition: Designing for comprehension\". <i>Communications of the ACM<\/i> <b>38<\/b>: 57\u201366.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Hypermedia+and+cognition%3A+Designing+for+comprehension&rft.jtitle=Communications+of+the+ACM&rft.aulast=Thuring%2C+M.%3B+Hannemann%2C+J.%3B+Haake%2C+J.&rft.au=Thuring%2C+M.%3B+Hannemann%2C+J.%3B+Haake%2C+J.&rft.date=1995&rft.volume=38&rft.pages=57%E2%80%9366&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-EndsleyTow95-19\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-EndsleyTow95_19-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Endsley, M. (1995). \"Toward a theory of situation awareness in dynamic systems\". <i>Human Factors<\/i> <b>37<\/b>: 32\u201364.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Toward+a+theory+of+situation+awareness+in+dynamic+systems&rft.jtitle=Human+Factors&rft.aulast=Endsley%2C+M.&rft.au=Endsley%2C+M.&rft.date=1995&rft.volume=37&rft.pages=32%E2%80%9364&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KainIU10-20\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-KainIU10_20-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Kain, K. de Jong, M.; Smith, C.; Albers, M. (Ed.); Still, B. (Ed.) (2010). \"Information Usability testing as audience and context analysis for risk communication\". <i>Usability of Complex Information Systems: Evaluation of User Interaction<\/i>. CRC Press. pp. 305\u2013332.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Information+Usability+testing+as+audience+and+context+analysis+for+risk+communication&rft.atitle=Usability+of+Complex+Information+Systems%3A+Evaluation+of+User+Interaction&rft.aulast=Kain%2C+K.+de+Jong%2C+M.%3B+Smith%2C+C.%3B+Albers%2C+M.+%28Ed.%29%3B+Still%2C+B.+%28Ed.%29&rft.au=Kain%2C+K.+de+Jong%2C+M.%3B+Smith%2C+C.%3B+Albers%2C+M.+%28Ed.%29%3B+Still%2C+B.+%28Ed.%29&rft.date=2010&rft.pages=pp.%26nbsp%3B305%E2%80%93332&rft.pub=CRC+Press&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RichardsUsa10-21\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RichardsUsa10_21-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Richards, D. (2015). \"Testing the waters: Local Users, sea level rise, and the productive usability of interactive geovisualizations\". <i>Communication Design Quarterly<\/i> <b>3<\/b>: 24\u201332.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Testing+the+waters%3A+Local+Users%2C+sea+level+rise%2C+and+the+productive+usability+of+interactive+geovisualizations&rft.jtitle=Communication+Design+Quarterly&rft.aulast=Richards%2C+D.&rft.au=Richards%2C+D.&rft.date=2015&rft.volume=3&rft.pages=24%E2%80%9332&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ShannonMath48-22\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ShannonMath48_22-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Shannon, C. (1948). \"A mathematical theory of communication\". <i>Bell System Technical Journal<\/i> <b>27<\/b>: 379\u2013423.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+mathematical+theory+of+communication&rft.jtitle=Bell+System+Technical+Journal&rft.aulast=Shannon%2C+C.&rft.au=Shannon%2C+C.&rft.date=1948&rft.volume=27&rft.pages=379%E2%80%93423&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-RedishExp07-23\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-RedishExp07_23-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Redish, J. (2007). \"Expanding usability testing to evaluate complex systems\". <i>The Journal of Usability Studies<\/i> <b>2<\/b>: 102\u2013111.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Expanding+usability+testing+to+evaluate+complex+systems&rft.jtitle=The+Journal+of+Usability+Studies&rft.aulast=Redish%2C+J.&rft.au=Redish%2C+J.&rft.date=2007&rft.volume=2&rft.pages=102%E2%80%93111&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SommervilleLar12-24\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-SommervilleLar12_24-0\" rel=\"external_link\">24.0<\/a><\/sup> <sup><a href=\"#cite_ref-SommervilleLar12_24-1\" rel=\"external_link\">24.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Sommerville, O.; Cliff, D.; Calinescu, R.; Keen, J.; Kelly, T.; Kwiatkowska, M.; Mcdermid, J.; Paige, R. (2012). \"Large scale complex IT systems\". <i>Communications of the ACM<\/i> <b>55<\/b>: 71\u201377.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Large+scale+complex+IT+systems&rft.jtitle=Communications+of+the+ACM&rft.aulast=Sommerville%2C+O.%3B+Cliff%2C+D.%3B+Calinescu%2C+R.%3B+Keen%2C+J.%3B+Kelly%2C+T.%3B+Kwiatkowska%2C+M.%3B+Mcdermid%2C+J.%3B+Paige%2C+R.&rft.au=Sommerville%2C+O.%3B+Cliff%2C+D.%3B+Calinescu%2C+R.%3B+Keen%2C+J.%3B+Kelly%2C+T.%3B+Kwiatkowska%2C+M.%3B+Mcdermid%2C+J.%3B+Paige%2C+R.&rft.date=2012&rft.volume=55&rft.pages=71%E2%80%9377&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-KleinSou99-25\"><span class=\"mw-cite-backlink\">\u2191 <sup><a href=\"#cite_ref-KleinSou99_25-0\" rel=\"external_link\">25.0<\/a><\/sup> <sup><a href=\"#cite_ref-KleinSou99_25-1\" rel=\"external_link\">25.1<\/a><\/sup><\/span> <span class=\"reference-text\"><span class=\"citation book\">Klein, G. (1999). <i>Sources of Power: How People make Decisions<\/i>. MIT.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Sources+of+Power%3A+How+People+make+Decisions&rft.aulast=Klein%2C+G.&rft.au=Klein%2C+G.&rft.date=1999&rft.pub=MIT&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-SimonMod79-26\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-SimonMod79_26-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation book\">Simon, H. (1979). <i>Models of Thought<\/i>. Yale University Press.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=book&rft.btitle=Models+of+Thought&rft.aulast=Simon%2C+H.&rft.au=Simon%2C+H.&rft.date=1979&rft.pub=Yale+University+Press&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-FennemaAnt95-27\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-FennemaAnt95_27-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Fennema, M.; Kleinmuntz, D. (1995). \"Anticipation of effort and accuracy in multi-attribute choice\". <i>Organizational Behavior and Human Decision Processes<\/i> <b>63<\/b>: 21\u201332.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Anticipation+of+effort+and+accuracy+in+multi-attribute+choice&rft.jtitle=Organizational+Behavior+and+Human+Decision+Processes&rft.aulast=Fennema%2C+M.%3B+Kleinmuntz%2C+D.&rft.au=Fennema%2C+M.%3B+Kleinmuntz%2C+D.&rft.date=1995&rft.volume=63&rft.pages=21%E2%80%9332&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-PayneAda01-28\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-PayneAda01_28-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Payne, S.; Howes. A.; Reader, W. (2001). \"Adaptively distributing cognition: A decision making perspective on human-computer interaction\". <i>Behaviour & Information Technology<\/i> <b>20<\/b>: 339\u2013346.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adaptively+distributing+cognition%3A+A+decision+making+perspective+on+human-computer+interaction&rft.jtitle=Behaviour+%26+Information+Technology&rft.aulast=Payne%2C+S.%3B+Howes.+A.%3B+Reader%2C+W.&rft.au=Payne%2C+S.%3B+Howes.+A.%3B+Reader%2C+W.&rft.date=2001&rft.volume=20&rft.pages=339%E2%80%93346&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-MasonProg73-29\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-MasonProg73_29-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Mason, R.; Mitroff, I. (1973). \"A program for research on management information systems\". <i>Management Science<\/i> <b>19<\/b>: 475\u2013487.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A+program+for+research+on+management+information+systems&rft.jtitle=Management+Science&rft.aulast=Mason%2C+R.%3B+Mitroff%2C+I.&rft.au=Mason%2C+R.%3B+Mitroff%2C+I.&rft.date=1973&rft.volume=19&rft.pages=475%E2%80%93487&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-GerlachUnd91-30\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-GerlachUnd91_30-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Gerlach, K.; Kuo, F. (1991). \"Understanding human-computer interaction for information system design\". <i>MIS Quarterly<\/i> <b>15<\/b>: 527\u2013550.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+human-computer+interaction+for+information+system+design&rft.jtitle=MIS+Quarterly&rft.aulast=Gerlach%2C+K.%3B+Kuo%2C+F.&rft.au=Gerlach%2C+K.%3B+Kuo%2C+F.&rft.date=1991&rft.volume=15&rft.pages=527%E2%80%93550&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-WoodsGoc88-31\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-WoodsGoc88_31-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Woods, D.; Roth, E. (1988). \"Cognitive engineering: Human problem solving with tools\". <i>Human Factors<\/i> <b>30<\/b>: 415\u2013430.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Cognitive+engineering%3A+Human+problem+solving+with+tools&rft.jtitle=Human+Factors&rft.aulast=Woods%2C+D.%3B+Roth%2C+E.&rft.au=Woods%2C+D.%3B+Roth%2C+E.&rft.date=1988&rft.volume=30&rft.pages=415%E2%80%93430&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-ConklinWicked-32\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-ConklinWicked_32-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation web\">Conklin, J.. <a rel=\"external_link\" class=\"external text\" href=\"http:\/\/www.cognexus.org\/id26.htm\" target=\"_blank\">\"Wicked problems and fragmentation\"<\/a><span class=\"printonly\">. <a rel=\"external_link\" class=\"external free\" href=\"http:\/\/www.cognexus.org\/id26.htm\" target=\"_blank\">http:\/\/www.cognexus.org\/id26.htm<\/a><\/span><span class=\"reference-accessdate\">. Retrieved 24 April 2015<\/span>.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=bookitem&rft.btitle=Wicked+problems+and+fragmentation&rft.atitle=&rft.aulast=Conklin%2C+J.&rft.au=Conklin%2C+J.&rft_id=http%3A%2F%2Fwww.cognexus.org%2Fid26.htm&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<li id=\"cite_note-AckoffMan67-33\"><span class=\"mw-cite-backlink\"><a href=\"#cite_ref-AckoffMan67_33-0\" rel=\"external_link\">\u2191<\/a><\/span> <span class=\"reference-text\"><span class=\"citation Journal\">Ackoff, R. (1967). \"Management misinformation systems\". <i>Management Science<\/i> <b>14<\/b>: 147\u2013156.<\/span><span class=\"Z3988\" title=\"ctx_ver=Z39.88-2004&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Management+misinformation+systems&rft.jtitle=Management+Science&rft.aulast=Ackoff%2C+R.&rft.au=Ackoff%2C+R.&rft.date=1967&rft.volume=14&rft.pages=147%E2%80%93156&rfr_id=info:sid\/en.wikipedia.org:Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\"><span style=\"display: none;\"> <\/span><\/span><\/span>\n<\/li>\n<\/ol>\n<h2><span class=\"mw-headline\" id=\"Notes\">Notes<\/span><\/h2>\n<p>This presentation is faithful to the original, with only a few minor changes to presentation and content. The two most notable changes involve citations. Albers lists the following citation in his references as citation 20:\n<\/p><p>\"20. Albers, M. Contextual awareness as measure of human-information interaction in usability and design. In Proceedings of the HCI International, Orlando, FL, USA, 12\u201314 July 2011.\" \n<\/p><p>However, he never referenced that citation number in the paper. We have thus omitted it. Additionally, under the subsection \"Experience-based\", he references \"Klein\u2019s (1999)\" but seemingly uses the incorrect citation number of 12, for Mirel 1998. We've corrected the citation number.\n<\/p>\n<!-- \nNewPP limit report\nCached time: 20181213210241\nCache expiry: 86400\nDynamic content: false\nCPU time usage: 0.694 seconds\nReal time usage: 0.738 seconds\nPreprocessor visited node count: 23416\/1000000\nPreprocessor generated node count: 35444\/1000000\nPost\u2010expand include size: 112730\/2097152 bytes\nTemplate argument size: 35500\/2097152 bytes\nHighest expansion depth: 14\/40\nExpensive parser function count: 0\/100\n-->\n\n<!-- \nTransclusion expansion time report (%,ms,calls,template)\n100.00% 645.442 1 - -total\n 75.14% 484.986 33 - Template:Citation\/core\n 48.64% 313.930 21 - Template:Cite_journal\n 29.02% 187.328 11 - Template:Cite_book\n 10.31% 66.526 1 - Template:Infobox_journal_article\n 9.89% 63.842 1 - Template:Infobox\n 5.92% 38.236 80 - Template:Infobox\/row\n 4.04% 26.061 36 - Template:Citation\/make_link\n 2.79% 18.005 1 - Template:Cite_web\n 0.32% 2.077 8 - Template:Clear\n-->\n\n<!-- Saved in parser cache with key limswiki:pcache:idhash:7605-0!*!0!!en!5!* and timestamp 20181213210240 and revision id 23410\n -->\n<\/div><div class=\"printfooter\">Source: <a rel=\"external_link\" class=\"external\" href=\"https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making\">https:\/\/www.limswiki.org\/index.php\/Journal:Human%E2%80%93information_interaction_with_complex_information_for_decision-making<\/a><\/div>\n\t\t\t\t\t\t\t\t\t\t<!-- end content -->\n\t\t\t\t\t\t\t\t\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t<\/div>\n\t\t<!-- end of the left (by default at least) column -->\n\t\t<div class=\"visualClear\"><\/div>\n\t\t\t\t\t\n\t\t<\/div>\n\t\t\n\n<\/body>","be30bf8a40599ea6d6b707ba00d0baa1_images":["https:\/\/www.limswiki.org\/images\/f\/f8\/Fig1_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/7\/71\/Fig2_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/8\/83\/Fig3_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/b\/be\/Fig4_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/d\/d4\/Fig5_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/7\/7b\/Fig6_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/9\/9f\/Fig7_Albers_Informatics2015_2-2.jpg","https:\/\/www.limswiki.org\/images\/e\/e5\/Fig8_Albers_Informatics2015_2-2.jpg"],"be30bf8a40599ea6d6b707ba00d0baa1_timestamp":1544734960,"13b3b6c47f60a3b75f007d8cd017d140_type":"article","13b3b6c47f60a3b75f007d8cd017d140_title":"Generalized procedure for screening free software and open-source software applications (Joyce 2015)","13b3b6c47f60a3b75f007d8cd017d140_url":"https:\/\/www.limswiki.org\/index.php\/Journal:Generalized_procedure_for_screening_free_software_and_open-source_software_applications\/Print_version","13b3b6c47f60a3b75f007d8cd017d140_plaintext":"\n\n\t\t\n\t\t\t\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n\n\t\t\t\tJournal:Generalized procedure for screening free software and open-source software applications\/Print version\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\tFrom LIMSWiki\n\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\tJump to: navigation, search\n\n\t\t\t\t\t\n\t\t\t\t\tFull article title\n \nGeneralized Procedure for Screening Free Software and Open Source Software ApplicationsAuthor(s)\n \nJoyce, JohnAuthor affiliation(s)\n \nArcana Informatica; Scientific ComputingPrimary contact\n \nEmail: jrjoyce@gmail.comYear published\n \n2015Distribution license\n \nCreative Commons Attribution-ShareAlike 4.0 InternationalWebsite\n \nOriginal versionDownload\n \nPDF (Note: Inline references fail to load in PDF version)\n\nContents\n\n1 Abstract \n2 Introduction \n3 Results \n\n3.1 Literature review \n3.2 Initial evaluation and selection recommendations \n\n3.2.1 Identify needs \n3.2.2 Identify candidates \n\n\n3.3 In-depth evaluation \n\n3.3.1 System functionality \n3.3.2 Community \n3.3.3 System cost \n3.3.4 Popularity \n3.3.5 Product support \n3.3.6 Maintenance \n3.3.7 Reliability \n3.3.8 Performance \n3.3.9 Scalability \n3.3.10 Usability \n3.3.11 Security \n3.3.12 Flexibility\/Customizability \n3.3.13 Interoperability\/Integration \n3.3.14 Legal\/License issues \n\n\n3.4 Completing the evaluation \n\n3.4.1 Caveats \n3.4.2 Screening tabulation \n3.4.3 Demonstration surveys \n\n\n\n\n4 Summary \n5 Glossary \n6 References \n7 Notes \n\n\n\nAbstract \nFree software and open-source software projects have become a popular alternative tool in both scientific research and other fields. However, selecting the optimal application for use in a project can be a major task in itself, as the list of potential applications must first be identified and screened to determine promising candidates before an in-depth analysis of systems can be performed. To simplify this process, we have initiated a project to generate a library of in-depth reviews of free software and open-source software applications. Preliminary to beginning this project, a review of evaluation methods available in the literature was performed. As we found no one method that stood out, we synthesized a general procedure using a variety of available sources for screening a designated class of applications to determine which ones to evaluate in more depth. In this paper, we examine a number of currently published processes to identify their strengths and weaknesses. By selecting from these processes we synthesize a proposed screening procedure to triage available systems and identify those most promising of pursuit. To illustrate the functionality of this technique, this screening procedure is executed against a selected class of applications.\n\nIntroduction \nThere is much confusion regarding free software and open-source software, and many people use these terms interchangeably. However, the connotations associated with the terms are highly significant. So perhaps we should start with an examination of the terms to clarify what we are attempting to screen. While there are many groups and organizations involved with open-source software, two of the main ones are the Free Software Foundation (FSF) and the Open Source Initiative (OSI).\nWhen discussing free software, we are not explicitly discussing software for which no fee is charged; rather, we are referring to \"free\" in terms of liberty. To quote the Free Software Foundation (FSF)[1]:\n\nA program is free software if the program's users have the four essential freedoms:\n The freedom to run the program as you wish, for any purpose (freedom 0).\n The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.\n The freedom to redistribute copies so you can help your neighbor (freedom 2).\n The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.<\/blockquote>\nThis does not mean that a program is provided at no cost, or gratis, though some of these rights imply that it would be. In the FSF's analysis, any application that does not conform to these freedoms is unethical. While there is also \"free software\" or \"freeware\" that is given away at no charge but without the source code, this would not be considered free software under the FSF definition.\nThe Open Source Initiative (OSI), originally formed to promote free software, refers to it as open-source software (OSS) to make it sound more business friendly. The OSI defines open-source software as any application that meets the following 10 criteria, which they based on the Debian Free Software Guidelines[2]:\n\n Free redistribution\n Source code included\n Must allow derived works\n Must preserve the integrity of the author's source code\n License must not discriminate against persons or groups\n License must not discriminate against fields of endeavor\n Distribution of licenses\n License must not be specific to a production\n License must not restrict other software\n License must be technology-neutral\nOpen-source software adherents take what they consider the more pragmatic view of looking more at the license requirements and put significant effort into convincing commercial enterprises of the practical benefits of open source, meaning the free availability of application source code.\nIn an attempt to placate both groups when discussing the same software application, the term free\/open-source software (F\/OSS) was developed. Since the term \"free\" was still tending to confuse some people, the term \"libre,\" which connotes freedom, was added resulting in the term free\/libre open-source software (FLOSS). If you perform a detailed analysis on the full specifications, you will find that all free software fits the open-source software definition, while not all open-source software fits the free software definition. However, any open-source software that is not also free software is the exception rather than the rule. As a result, you will find these acronyms used almost interchangeably, but there are subtle differences in meaning, so stay alert. In the final analysis, the software license that accompanies the software is what you legally have to follow.\nThe reality is that since both groups trace their history back to the same origins, the practical differences between an application being free software or open-source are generally negligible. Keep in mind that the above descriptions are to some degree generalizations, as both organizations are involved in multiple activities. There are many additional groups interested in open source for a wide variety of reasons. However, this diversity is also a strong point, resulting in a vibrant and dynamic community. You should not allow the difference in terminology to be divisive. The fact that all of these terms can be traced back to the same origin should unite us.[3] In practice, many of the organization members will use the terms interchangeably, depending on the point that they are trying to get across. With an excess of 300,000 FLOSS applications currently registered in SourceForge.net[4] and over 10 million repositories on GitHub[5], there are generally multiple options accessible for any class of application, be it a laboratory information management system (LIMS), an office suite, a data base, or a document management system. Presumably you have gone through the assessment of the various challenges to using an open-source application[6] and have decided to move ahead with selecting an appropriate application. The difficulty now becomes selecting which application to use. While there are multiple indexes of FOSS projects, these are normally just listings of the applications with a brief description provided by the developers, with no indication of the vitality or independent evaluation of the project.\nWhat is missing is a catalog of in-depth reviews of these applications, eliminating the need for each group to go through the process of developing a list of potential applications, screening all available applications, and performing in-depth reviews of the most promising candidates. While true that once an organization has made a tentative selection it will need to perform its own testing to confirm that the selected application meets its specific needs, there is no reason for everyone to go through the tedious process of identifying projects and weeding out the untenable ones.\n\n\n\n\n\n\n\n\n\n Illustration 1.: This diagram, originally by Chao-Kuei and updated by several others since, explains the different categories of software. It's available as a scalable vector graphic and as an XFig document, under the terms of any of the GNU GPL v2 or later, the GNU FDL v1.2 or later, or the Creative Commons Attribution-Share Alike v2.0 or later\n\n\n\nThe primary goal of this document is to describe a general procedure capable of being used to screen any selected class of software applications. The immediate concern is with screening FLOSS applications, though allowances can be made to the process to allow at least rough cross-comparison of both FOSS and commercial applications. To that end, we start with an examination of published survey procedures. We then combine a subset of standard software evaluation procedures with recommendations for evaluating FLOSS applications. Because it is designed to screen such a diverse range of applications, the procedure is by necessity very general. However, as we move through the steps of the procedure, we will describe how to tune the process for the class of software that you are interested in.\nYou can also ignore any arguments regarding selecting between FLOSS and commercial applications. In this context, \"commercial\" refers to the marketing approach, not to the quality of the software. Many FLOSS applications have comparable, if not superior, quality to products that are traditionally marketed and licensed. Wheeler discusses this issue in more detail, showing that by many definitions FLOSS is commercial software.[7]\nThe final objective of this process is to document a procedure that can then be applied to any class of FOSS applications to determine which projects in the class are the most promising to pursue, allowing us to expend our limited resources most effectively. As the information available for evaluating FOSS projects is generally quite different from that available for commercially licensed applications, this evaluation procedure has been optimized to best take advantage of this additional information.\n\nResults \nLiterature review \nA search of the literature returns thousands of papers related to open-source software, but most are of limited value in regards to the scope of this project. The need for a process to assist in selecting between open-source projects is mentioned in a number of these papers, and there appear to be over a score of different published procedures. Regrettably, none of these methodologies appear to have gained large-scale support in the industry. \nStol and Babar have published a framework for comparing evaluation methods targeting open-source software and include a comparison of 20 of them.[8] They noted that web sites that simply consisted of a suggestion list for selecting an open-source application were not included in this comparison. This selection difficulty is nothing new with FLOSS applications. In their 1994 paper, Fritz and Carter review over a dozen existing selection methodologies, covering their strengths, weaknesses, the mathematics used, as well as other factors involved.[9]\n\n\n\n\n\n\n\n No.\n\n Name\n\n Year\n\n Orig\n\n Method\n\n\n1\n\nCapgemini Open Source Maturity Model\n\n2003\n\nI\n\nYes\n\n\n2\n\nEvaluation Framework for Open Source Software\n\n2004\n\nR\n\nNo\n\n\n3\n\nA Model for Comparative Assessment of Open Source Products\n\n2004\n\nR\n\nYes\n\n\n4\n\nNavica Open Source Maturity Model\n\n2004\n\nI\n\nYes\n\n\n5\n\nWoods and Guliani's OSMM\n\n2005\n\nI\n\nNo\n\n\n6\n\nOpen Business Readiness Rating (OpenBRR)[10][11]\n\n2005\n\nR\/I\n\nYes\n\n\n7\n\nAtos Origin Method for Qualification and Selection of Open Source Software (QSOS)\n\n2006\n\nI\n\nYes\n\n\n8\n\nEvaluation Criteria for Free\/Open Source Software Products\n\n2006\n\nR\n\nNo\n\n\n9\n\nA Quality Model for OSS Selection\n\n2007\n\nR\n\nNo\n\n\n10\n\nSelection Process of Open Source Software\n\n2007\n\nR\n\nYes\n\n\n11\n\nObservatory for Innovation and Technological transfer on Open Source software (OITOS)\n\n2007\n\nR\n\nYes\n\n\n12\n\nFramework for OS Critical Systems Evaluation (FOCSE)\n\n2007\n\nR\n\nNo\n\n\n13\n\nBalanced Scorecards for OSS\n\n2007\n\nR\n\nNo\n\n\n14\n\nOpen Business Quality Rating (OpenBQR)\n\n2007\n\nR\n\nYes\n\n\n15\n\nEvaluating OSS through Prototyping\n\n2007\n\nR\n\nYes\n\n\n16\n\nA Comprehensive Approach for Assessing Open Source Projects\n\n2008\n\nR\n\nNo\n\n\n17\n\nSoftware Quality Observatory for Open Source Software (SQO-OSS)\n\n2008\n\nR\n\nYes\n\n\n18\n\nAn operational approach for selecting open source components in a software development project[12]\n\n2008\n\nR\n\nNo\n\n\n19\n\nQualiPSo trustworthiness model\n\n2008\n\nR\n\nNo\n\n\n20\n\nOpenSource Maturity Model (OMM)[13]\n\n2009\n\nR\n\nNo\n\n\n\nTable 1.: Comparison frameworks and methodologies for examination of FLOSS applications extracted from Stol and Babar.[8] The selection\r\nprocedure is described in Stol's and Barbar's paper, however, 'Year' indicates the date of publication, 'Orig.' indicates whether the described\r\n process originated in industry (I) or research (R), while 'Method' indicates whether the paper describes a formal analysis method and procedure (Yes)\r\nor just a list of evaluation criteria (No).\nExtensive comparisons between some of these methods have also been published, such as Deprez's and Alexandre's comparative assessment of the OpenBRR and QSOS techniques.[14] Wasserman and Pal have also published a paper under the title of \"Evaluating Open Source Software,\" which appears to be more of an updated announcement and in-depth description of the Business Readiness Rating (BRR) framework.[15] Jadhav and Sonar have also examined the issue of both evaluating and selecting software packages. They include a helpful analysis of the strengths and weaknesses of the various techniques.[16] Perhaps more importantly, they clearly point out that there is no common list of evaluation criteria. While the majority of the articles they reviewed listed the criteria used, Jadhav and Sonar indicated that these criteria frequently did not include a detailed definition, which required each evaluator to use their own, sometimes conflicting, interpretation.\nSince the publication of Stol and Babar's paper, additional evaluation methods have been published. Of particular interest are a series of papers by Pani et al. describing their proposed FAME (Filter, Analyze, Measure and Evaluate) methodology.[17][18][19][20] In their \"Transferring FAME\" paper, they emphasized that all of the evaluation frameworks previously described in the published literature were frequently not easy to apply to real environments, as they were developed using an analytic research approach which incorporated a multitude of factors.[20]\nTheir stated design objective with FAME is to reduce the complexity of performing the application evaluation, particularly for small organizations. They specify \"[t]he goals of FAME methodology are to aid the choice of high-quality F\/OSS products, with high probability to be sustainable in the long term, and to be as simple and user friendly as possible.\" They further state that \"[t]he main idea behind FAME is that the users should evaluate which solution amongst those available is more suitable to their needs by comparing technical and economical factors, and also taking into account the total cost of individual solutions and cash outflows. It is necessary to consider the investment in its totality and not in separate parts that are independent of one another.\"[20]\nThis paper breaks the FAME methodology into four activities:\n\n Identify the constraints and risks of the projects\n Identify user requirements and rank\n Identify and rank all key objectives of the project\n Generate a priority framework to allow comparison of needs and features\nTheir paper includes a formula for generating a score from the information collected. The evaluated system with the highest \"major score,\" Pjtot, indicates the system selected. While it is a common practice to define an analysis process which condenses all of the information gathered into a single score, I highly caution against blindly accepting such a score. FAME, as well as a number or the other assessment methodologies, is designed for iterative use. The logical purpose of this is to allow the addition of factors initially overlooked into your assessment, as well as to change the weighting of existing factors as you reevaluate their importance. However, this feature means that it is also very easy to unconsciously, or consciously, skew the results of the evaluation to select any system you wish. Condensing everything down into a single value also strips out much of the information that you have worked so hard to gather. Note that you can generate the same result score using significantly different input values. While of value, selecting a system based on just the highest score could potentially leave you with a totally unworkable system.\nPani et al. also describe a FAMEtool to assist in this data gathering and evaluation.[19] However, a general web search as well as a review of their FAME papers revealed no indication of how to obtain this resource. While this paper includes additional comparisons with other FLOSS analysis methodologies, and there are some hints suggesting that the FAMEtool is being provided as a web service, I have found no URL specified for it. As of now, I have received no responses from the research team via either e-mail or Skype regarding FAME, the FAMEtool, or feedback on its use.\nDuring this same time frame Soto and Ciolkowski also published papers describing the QualOSS Open Source Assessment Model and compared it to a number of the procedures in Stol and Barbar's table.[21][22] Their focus was primarily on three process perspectives: product quality, process maturity, and sustainability of the development community. Due to the lack of anything more than a rudimentary process perspective examination, they felt that the following OSS project assessment models were unsatisfactory: QSOS, CapGemni OSMM, Navica OSMM, and OpenBRR. They position QualOSS as an extension of the tralatitious CMMI and SPICE process maturity models. While there are multiple items in the second paper that are worth incorporating into an in-depth evaluation process, they do not seem suitable for what is intended as a quick survey.\nAnother paper, published by Haaland and Groven, also compared a number of open source quality models. To this paper's credit, the authors devoted a significant amount of space to discussing the different definitions of quality and how the target audience of a tool might affect which definition was used.[23] Like Stohl and Babar, they listed a number of the quality assessment models to choose from, including OSMM, QSOS, OpenBRR, and others. For their comparison, they selected OpenBRR and QualOSS. They appear to have classified OpenBRR as a first generation tool with a \"[us]ser view on quality\" and QualOSS as a second generation tool with a \"business point of view.\" An additional variation is that OpenBRR is primarily a manual tool, while QualOSS is primarily an automated tool. Their analysis in this article clearly demonstrates the steps involve in using these tools and in highlighting where they are objective and subjective. While they were unable to answer their original question as to whether the first- or second-generation tools did a better job of evaluation, to me they answered the following even more important but unasked question. As they proceeded through their evaluation, it became apparent as to how much the questions defined in the methods could affect the results of the evaluations. Even though the authors might have considered the questions to be objective, I could readily see how some of these questions could be interpreted in alternate ways. My takeaway is an awareness of the potential danger of using rigid tools, as they can skew the accuracy of the evaluation results depending on exactly what you want the evaluated application to do and how you plan to use it. These models can be very useful guides, but they should not be used to replace a carefully considered evaluation as there will always be factors influencing the selection decision which did not occur to anyone when the specifications were being written.\nHauge et al. have noted that despite the development of several normative methods of assessment, empirical studies have failed to show wide spread adoption of these methods.[24] From their survey of a number of Norwegian software companies, they have noticed a tendency for selectors to skip the in-depth search for what they call the \"best fit\" application and fall back on what they refer to as a '\"first fit.\" This is an iterative procedure with the knowledge gained from the failure of one set of component tests being incorporated into the evaluation of the next one. Their recommendation is for researchers to stop attempting to develop either general evaluation schemas or normative selection methods which would be applicable to any software application and instead focus on identifying situationally sensitive factors which can be used as evaluation criteria. This is a very rational approach as all situations, even if evaluating the same set of applications is going to be different, as each user's needs are different.\nAyalal et al. have performed a study to try to more accurately determine why more people don't take advantage of the various published selection methodologies.[25] While they looked at a number of factors and identified several possible problems, one of the biggest factors was the difficulty in obtaining the needed information for the evaluation. Based on the projects they studied, many did not provide a number of the basic pieces of information required for the evaluation, or perhaps worse, required extensive examination of the project web site and documentation to retrieve the required information. From her paper, it sounds as if this issue was more of a communication breakdown than an attempt to hide any of the information, not that this had any impact on the inaccessibility of the information.\nIn addition to the low engagement rates for the various published evaluation methods, another concern is the viability of the sponsoring organizations. One of the assessment papers indicated that the published methods with the smallest footprint, or the easiest to use, appeared to be FAME and the OpenBRR. I have already mentioned my difficulty obtaining additional information regarding FAME, and OpenBRR appears to be even more problematic. BRR was first registered on SourceForge in September of 2005[26], and an extensive Request For Comments from the founding members of the BRR consortium (SpikeSource, the Center for Open Source Investigation at Carnegie Mellon West, and Intel Corporation) was released.[10] In 2006, in contrast to typical Open Source development groups, the OpenBRR group announced the formation of an OpenBRR Corporate Community group. Peter Galli's story indicates that \"the current plan is that membership will not be open to all.\"[27] He quotes Murugan Pal saying \"membership will be on an invitation-only basis to ensure that only trusted participants are coming into the system.\" However, for some reason, at least some in the group \"expressed concern and unhappiness about the idea of the information discussed not being shared with the broader open-source community.\"[27]\nWhile the original Business Readiness Rating web site still exists, it is currently little more than a static web page.[28] It appears that some of the original information posted on the site is still there, you just have to know what its URL is to access it, as the original links on the web site have been removed. Otherwise, you may have to turn to the Internet Archive to retrieve some of their documentation. The lack of any visible activity regarding OpenBRR prompted a blog post from one graduate student in 2012 asking \"What happened to OpenBRR (Business Readiness Rating for Open Source)?\"[29]\nIt appears that at some point any development activity regarding OpenBRR was morphed into OSSpal.[30] However, background information on this project is sparse as well. While the site briefly mentions that OSSpal incorporates a number of lessons learned from BRR, there is very little additional information regarding the group or the methods procedures. Their \"All Projects\" tab provides a list of over 30 open-source projects, but the majority simply show \"No votes yet\" under the various headings. In fact, as of now, the only projects showing any input at all are for Ubuntu and Mozilla Firefox.\n\nInitial evaluation and selection recommendations \nAt this point, we'll take a step back from the evaluation methodologies papers and examine some of the more general recommendations regarding evaluating and selecting FLOSS applications. The consistency of their recommendations may provide a more useful guide for an initial survey of FLOSS applications.\nIn TechRepublic, de Silva recommends 10 questions to ask when selecting a FLOSS application.[31] While he provides a brief discourse on each question in his paper to ensure you understand the point of his question, I've collected the 10 questions from his article into the following list. Once we see what overlap, if any, are amongst our general recommendations, we'll address some of the consolidated questions in more detail.\n\n Are the open source license terms compatible with my business requirements?\n What is the strength of the community?\n How well is the product adopted by users?\n Can I get a warranty or commercial support if I need it?\n What quality assurance processes exist?\n How good is the documentation?\n How easily can the system be customized to my exact requirements?\n How is this project governed and how easily can I influence the road map?\n Will the product scale to my enterprise's requirements?\n Are there regular security patches?\nSimilarly, in InfoWorld Phipps lists seven questions you should have answered before even starting to select a software package.[32] His list of questions, pulled directly from his article are:\n\n Am I granted copyright permission?\n Am I free to use my chosen business model?\n Am I unlikely to suffer patent attack?\n Am I free to compete with other community members?\n Am I free to contribute my improvements?\n Am I treated as a development peer?\n Am I inclusive of all people and skills?\nThis list of questions shows a moderately different point of view, as it is not only just about someone selecting an open-source system, but also it's about getting involved in its direct development. Padin, of 8th Light, Inc., takes the viewpoint of a developer who might incorporate open-source software into their projects.[33] The list of criteria pulled directly from his blog includes:\n\n Does it do what I need it to do?\n How much more do I need it to do?\n Documentation\n Easy to review source code\n Popularity\n Tests and specs\n Licensing\n Community\nMetcalfe of OSS Watch lists his top tips as[34]:\n\n Reputation\n Ongoing effort\n Standards and interoperability\n Support (Community)\n Support (Commercial)\n Version\n Version 1.0\n Documentation\n Skill setting\n Project Development Development Model\n License\nIn his LIMSexpert blog, Joel Limardo of ForwardPhase Technologies, LLC lists the following as components to check when evaluating an open-source application[35]:\n\n Check licensing\n Check code quality\n Test setup time\n Verify extensibility\n Check for separation of concerns\n Check for last updated date\n Check for dependence on outdated toolkits\/frameworks\nPerhaps the most referenced of the general articles on selecting FLOSS applications is David Wheeler's \"How to Evaluate Open Source Software \/ Free Software (OSS\/FS) Programs.\"[36] The detailed functionality to consider will vary with the types of applications being compared, but there are a number of general features that are relevant to almost any type of application. While we will cover them in more detail later, Wheeler categorizes the features to consider as the following:\n\n System functionality\n System cost \u2013 direct and in-direct\n Popularity of application, i.e. its market share for that type of application\n Varieties of product support available\n Maintenance of application, i.e, is development still taking place\n Reliability of application\n Performance of application\n Scalability of application\n Usability of application\n Security of application\n Adaptability\/customizability of application\n Interoperability of application\n Licensing and other legal issues\nWhile a hurried glance might suggest a lot of diversity in the features these various resources suggest, a closer look at the meaning of what they are saying shows a repetitive series of concerns. The primary significant differences between the functionality lists suggested is actually due more to how wide a breadth of the analysis process the authors are considering, as well as the underlying features that they are concerned with.\nWith a few additions, the high-level screening template described in the rest of this communication is based on Wheeler's previously mentioned document describing his recommended process for evaluating open-source software and free software programs. Structuring the items thus will make it easier to locate the corresponding sections in his document, which includes many useful specific recommendations as well as a great deal of background information to help you understand the why of the topic. I highly recommend reading it and following up on some of the links he provides. I will also include evaluation suggestions from several of the previously mentioned procedures where appropriate.\nWheeler defines four basic steps to this evaluation process, as listed below:\n\n Identify candidate applications.\n Read existing product reviews.\n Compare attributes of these applications to your needs.\n Analyze the applications best matching your needs in more depth.\nWheeler categorizes this process with the acronym IRCA. In this paper we will be focusing on the IRC components of this process. To confirm the efficacy of this protocol we will later apply it to several classes of open-source applications and examine the output of the protocol.\n\nIdentify needs \nRealistically, before you can perform a survey of applications to determine which ones best match your needs, you must determine what your needs actually are. The product of determining these needs is frequently referred to as the user requirements specification (URS).[37][38] This document can be generated in several ways, including having all of the potential users submit a list of the functions and capabilities that they feel is important. While the requirements document can be created by a single person, it is generally best to make it a group effort with multiple reviews of the draft document, including all of the users who will be working with the application. The reason for this is to ensure that an important requirement is not missed. When a requirement is missed, it is frequently due to the requirement being so basic that it never occurs to anyone that it specifically needed to be included in the requirements document. Admittedly, a detailed URS is not required at the survey level, but it is worth having if only to identify, by their implications, other features that might be significant.\nNeeds will, of course, vary with the type of application you are looking for and what you are planning to do with it. Keep in mind that the URS is a living document, subject to change through this whole process. Developing a URS is generally an iterative process, since as you explore systems, you may well see features that you hadn't considered that you find desirable. This process will also be impacted by whether the application to be selected will be used in a regulated environment. If it is, there will be existing documents that describe the minimum functionality that must be present in the system. Even if it is not to be used in a regulated environment, documents exist for many types of systems that describe the recommended functional requirements that would be expected for that type of system.\nFor a clarifying example, if you were attempting to select a laboratory information management system (LIMS), you can download checklists and standards of typical system requirements from a variety of sources.[39][40][41][42] These will provide you with examples of the questions to ask, but you will have to determine which ones are important to, or required for, your particular effort.\nDepending on the use to which this application is to be applied, you may be subject to other specific regulatory requirements as well. Which regulations may vary, since the same types of analysis performed for different industries fall under different regulatory organizations. This aspect is further complicated by the fact that you may be affected by more than one country's regulations if your analysis is applicable to products being shipped to other countries. While some specific regulations may have changed since its publication, an excellent resource to orient you to the diverse factors that must be considered is Siri Segalstad's book International IT Regulations and Compliance.[43] My understanding is that an updated version of this book is currently in preparation. Keep in mind that while regulatory requirements that you must meet will vary, these regulations by and large also describe best practices, or at least the minimal allowed practices. These requirements are not put in place arbitrarily (generally) or to make things difficult for you but to ensure the quality of the data produced. As such, any deviations should be carefully considered, whether working in a regulated environment or not. Proper due diligence would be to determine which regulations and standards would apply to your operation.\nFor a LIMS, an example of following best practices is to ensure that the application has a full and detailed audit trail. An audit trail allows you to follow the processing of items through your system, determining who did what and when. In any field where it might become important to identify the actions taken during a processing step, an audit trail should be mandatory. While your organization's operations may not fall under the FDA's 21 CFR Part 11 regulations, which address data access and security (including audit trails), it is still extremely prudent that the application you select complies with them. If it does not, then almost anyone could walk up to your system and modify data, either deliberately or accidentally, and you would have no idea of who made the changes or what changes they made. For that matter, you might not even be able to tell a change was made at all, which likely will raise concerns both inside and outside of your organization. This would obviously cause major problems if they became a hinge issue for any type of liability law suit.\nFor this screening procedure, you do not have to have a fully detailed URS, but it is expedient to have a list of your make-or-break issues. This list will be used later for comparing systems and determining which ones justify a more in-depth evaluation.\n\nIdentify candidates \nTo evaluate potential applications against your functional criteria, you must initially generate a list of potential systems. While this might sound easy, generating a comprehensive list frequently proves to be a challenge. When initiating the process, you must first determine the type of system that you are looking for, be it a LIMS, a hospital management system, a database, etc. At this point, you should be fairly open in building your list of candidates. By that, I mean that you should be careful not to select applications based solely on the utilization label applied to them. The same piece of software can frequently be applied to solve multiple problems, so you should cast a wide net and not automatically reject a system because the label you were looking for hadn't been applied to it. While the label may give you a convenient place to start searching, it is much more important to look for the functionality that you need, not what the system is called. In any case, many times the applied labels are vague and mean very different things to different people.\nThere are a variety of ways to generate your candidate list. A good place to start is simply talking with colleagues in your field. Have they heard of or used a FLOSS application of the appropriate type that they like? Another way is to just flip through journals and trade magazines that cover your field. Any sufficiently promising applications are likely to be mentioned there. Many of the trade magazines will have a special annual issue that covers equipment and software applicable to their field. It is difficult to generate a list of all potential resources, as many of these trade publications are little-known outside of their field. Also keep in mind that with the continued evolution of the World Wide Web, many of these trade publications also have associated web sites that you can scan or search. The table below includes just a minor fraction of these sites that are available. (We would welcome the suggestion of any additional resource sites that you are aware of. Please e-mail the fields covered, the resource name, and either its general URL or the URL of the specific resource section to the corresponding editor.)\n\n\n\n\n\n\n\n Field\n\n Resource Name\n\n URL\n\n\nAstronomy\n\nTech Support Alert\n\nhttp:\/\/www.techsupportalert.com\/best-free-astronomy-software.htm\n\n\nBusiness intelligence reporting\n\nTechnology Innovation Management Review\n\nhttp:\/\/timreview.ca\/article\/288\n\n\nCommunity radio\n\nPrometheus Radio Project\n\nhttp:\/\/prometheusradio.org\/Free_Open_Source_Tools_C_R\n\n\nComprehensive range of apps\n\nBlack Duck KnowledgeBase\n\nhttps:\/\/www.blackducksoftware.com\/products\/knowledgebase\n\n\nComprehensive range of apps\n\nThe Directory of Open Access Repositories - OpenDOAR\n\nhttp:\/\/www.opendoar.org\/\n\n\nData storage\n\nInfoStor\n\nhttp:\/\/www.infostor.com\/nas\/58-top-open-source-storage-project-1.html\n\n\nDigital audio editing\n\n25 Free Digital Audio Editors You Should Know\n\nhttp:\/\/www.hongkiat.com\/blog\/25-free-digital-audio-editors\/\n\n\nDigital video editing\n\nBest free video editing software: 20 top programs in 2015\n\nhttp:\/\/www.techradar.com\/us\/news\/software\/applications\/best-free-video-editing-software-9-top-programs-you-should-download-1136264\n\n\nDiverse range of Android apps\n\nWikipedia \u2013 Android applications\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_Android_applications\n\n\nDiverse range of apps\n\nLaunchpad, The Canonical Group\n\nhttp:\/\/launchpad.net\/\n\n\nDiverse range of apps\n\nMicrosoft CodePlex\n\nhttps:\/\/www.codeplex.com\/\n\n\nDiverse range of apps\n\nOpen Hub, Black Duck Software, Inc.\n\nhttps:\/\/www.openhub.net\/\n\n\nDiverse range of apps (but site is deprecated)\n\nGoogle Code\n\nhttps:\/\/code.google.com\/p\/support\/\n\n\nDiverse range of apps (focus on infrastructure)\n\nOpen Source Guide by Smile\n\nhttp:\/\/www.open-source-guide.com\/en\n\n\nDiverse range of apps (primarily targeting Samsung products)\n\nSamsung Open Source Release Center\n\nhttp:\/\/opensource.samsung.com\/reception.do\n\n\nDiverse range of commercial open source apps and services\n\nWikipedia \u2013 Commercial open source applications\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_commercial_open-source_applications_and_services\n\n\nDiverse range of GNU apps\n\nSavannah GNU\n\nhttp:\/\/savannah.gnu.org\/\n\n\nDiverse range of non-GNU apps\n\nSavannah Non-GNU\n\nhttp:\/\/savannah.nongnu.org\n\n\nDiverse range of top apps\n\nProjects and Applications\n\nhttp:\/\/opensource.com\/resources\/projects-and-applications\n\n\nDrug discovery\n\nDrug Discovery Today\n\nhttp:\/\/www.sciencedirect.com\/science\/article\/pii\/S1359644605036925\n\n\nElectronic engineering\n\nEducation Engineering (EDUCON), 2010 IEEE\n\nhttp:\/\/ieeexplore.ieee.org\/xpl\/articleDetails.jsp?arnumber=5492430\n\n\nEmbroidery\n\nMachine Embroidery Portal\n\nhttp:\/\/www.k2g2.org\/portal:machine_embroidery\n\n\nEnterprise resource planning (ERP)\n\nA Comparison of Open Source ERP Systems\n\nhttp:\/\/www.big.tuwien.ac.at\/system\/theses\/20\/papers.pdf?1298476232\n\n\nEnterprise resource planning (ERP)\n\nOpen Source ERP Site\n\nhttp:\/\/www.open-source-erp-site.com\/list-of-open-source-erps.html\n\n\nFashion\n\nEthical Fashion Forum - 9 Open-Source & Low Cost Digital Fashion Business Tools\n\nhttp:\/\/source.ethicalfashionforum.com\/article\/9-open-source-low-cost-digital-fashion-business-tools\n\n\nGeographic information systems (GIS)\n\nOpen Source GIS\n\nhttp:\/\/opensourcegis.org\/\n\n\nGeophysics\n\nGeopsy project\n\nhttp:\/\/www.geopsy.org\/\n\n\nGeophysics\n\nWikipedia - Geophysics software\n\nhttps:\/\/en.wikipedia.org\/wiki\/Comparison_of_free_geophysics_software\n\n\nGIS for libraries\n\nLibrary Hi Tech\n\nhttp:\/\/www.emeraldinsight.com\/doi\/abs\/10.1108\/07378831011026742\n\n\nHelp desk\n\nCIO\n\nhttp:\/\/www.cio.com.au\/article\/320110\/5_open_source_help_desk_apps_watch\/\n\n\nHighly diverse range of apps\n\nCode NASA\n\nhttps:\/\/code.nasa.gov\/#\/\n\n\nHighly diverse range of apps\n\nNASA Open Source Software\n\nhttp:\/\/ti.arc.nasa.gov\/opensource\/\n\n\nLaboratory informatics\n\nLIMSwiki - Laboratory informatics software (open source)\n\nhttp:\/\/www.limswiki.org\/index.php\/Category:Laboratory_informatics_software_%28open_source%29\n\n\nLearning management systems (LMS)\n\nOpen Source LMS \u2013 10 Alternatives to Moodle\n\nhttp:\/\/barrysampson.com\/2009\/04\/08\/open-source-lms-10-alternatives-to-moodle\/\n\n\nLibrary science\n\nFOSS4LIB\n\nhttps:\/\/foss4lib.org\/packages\n\n\nLimited range of iOS apps\n\nWikipedia - Open-source iOS applications\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_free_and_open-source_iOS_applications\n\n\nMedical and laboratory informatics\n\nHealthcare Freeware\n\nhttp:\/\/www.healthcarefreeware.com\/lim.htm\n\n\nMedical office records\n\nZDNet\n\nhttp:\/\/www.zdnet.com\/article\/free-and-open-source-healthcare-software-for-your-practice\/\n\n\nMedicine\n\n'Open' Health IT Solutions for the Public & Private Sectors\n\nhttps:\/\/drive.google.com\/file\/d\/0B_TuX9zE68eWcVM1dkdDQTVCdnM\/edit\n\n\nMedicine\n\n50 Successful Open Source Projects That Are Changing Medicine\n\nhttp:\/\/nursingassistantguides.com\/2009\/50-successful-open-source-projects-that-are-changing-medicine\/\n\n\nMedicine\n\nDatamation\n\nhttp:\/\/www.datamation.com\/open-source\/50-open-source-replacements-for-health-care-software-1.html\n\n\nMedicine\n\nMedfloss.org\n\nhttp:\/\/www.medfloss.org\/\n\n\nMIDI controllers\n\nOpen Source MIDI Controllers\n\nhttp:\/\/punkmanufacturing.com\/wiki\/open-source-midi-controllers\n\n\nMusic\n\n20 great free and open source music making programs\n\nhttp:\/\/www.musicradar.com\/tuition\/tech\/20-great-free-and-open-source-music-making-programs-582934\n\n\nPattern management for textiles\n\nTools for computer generated patterns\n\nhttps:\/\/xxxclairewilliamsxxx.wordpress.com\/tools-to-create-and-explore-digital-patterns\/\n\n\nPersonal information management\n\nWikipedia - Personal information managers\n\nhttps:\/\/en.wikipedia.org\/wiki\/List_of_personal_information_managers#Open_source_applications\n\n\nPhotography\n\nOpen Source Photography\n\nhttp:\/\/opensourcephotography.org\/software-list\n\n\nProject management\n\nOpen Source Photography\n\nhttp:\/\/opensourcephotography.org\/software-list\n\n\nRepository management\n\nFree and open-source repository software\n\nhttp:\/\/oad.simmons.edu\/oadwiki\/Free_and_open-source_repository_software\n\n\nSoftware test management\n\nSoftware Testing Help - 15 Best Test Management Tools for Software Testers\n\nhttp:\/\/www.softwaretestinghelp.com\/15-best-test-management-tools-for-software-testers\/\n\n\nSurvey and feedback\n\nCapterra - Top survey software products\n\nhttp:\/\/www.capterra.com\/survey-software\/\n\n\n\nTable 2.: Examples of focused FLOSS resource sites available on the web\n Illustration 2.: Result page displayed on Black Duck Software's Open Hub site for the Bika Open Source LIMS 3 project.\nI also recommend checking some of the general open source project lists, such as the ones generated by Cynthia Harvey at Datamation[44], which has been covering the computer and data-processing industry since 1957. In particular, you might find their article \"Open Source Software List: 2015 Ultimate List\" useful. It itemizes over 1,200 open source applications, including some in categories that I didn't even know existed.[45]\nIt would also be prudent to search the major open source repositories such as SourceForge[46] and GitHub.[47] Wikipedia includes a comparison of source code hosting facilities that would be worth reviewing as well.[48] Keep in mind that you will need to be flexible with your search terms, as the developers might be looking at the application differently than you are. While they were created for a different purpose, an examination of the books in The Architecture of Open Source Applications might prove useful as well.[49] Other sites where you might find interesting information regarding new open-source applications, are the various OpenSource award sites, such as the InfoWorld Best of Open Source Software Awards, colloquially known as the Bossies.[50] \nWhen searching the web, don't rely on just Google or Bing. Don't forget to checkout all of the journal web sites such as SpringerLink, Wiley, ScienceDirect, PubMed, and others as they contain a surprising amount of information on FLOSS. If you don't wish to search each of them individually, there are other search engines out there which can give you an alternate view of the research resources available. To name just two, be sure to try both Google Scholar[51] and Microsoft Academic Search.[52] These tools can also be used to search for masters theses and doctoral dissertations, which likewise contain a significant amount of information regarding open source.\nWhile working on creating your candidate list, be sure to pull any application reviews that you come across.[53] If done well, these reviews can save you a significant amount of time in screening a potential system. However, unless you're familiar with the credentials of the author, be cautious of relying on them for all of your information. While not common, people have been known to post fake reviews online, sometimes when it is not even April 1! Another great resource, both for identifying projects and obtaining information about them, is Open HUB[54], a web site dedicated to open source and maintained by Black Duck Software, Inc. Open Hub allows you to search by project, person, organization, forums, and code. For example, if I searched for Bika LIMS, it would currently return the results for Bika Open Source LIMS 3 along with some basic information regarding the system. If I were to click on the project's name, a much more detailed page regarding this project is displayed. Moving your mouse cursor over the graphs displays the corresponding information for that date.\nOnce a list of candidate applications has been generated, the list of entries must be compared. Some of this comparison can be performed objectively, but it also requires subjective analysis of some components. As Stol and Babar have shown, there is no single recognized procedure for either the survey or detailed comparison of FLOSS applications that has shown a marked popularity above the others.\nThe importance of any one specific aspect of the evaluation will vary with the needs of the organization. General system functionality will be an important consideration, but specific aspects of the contained functionality will have different values to different groups. For instance, interoperability may be very important to some groups, while others may be using this application as their only data system and they have no interest in exchanging data files with others, so interoperability is not a concern to them. While you can develop a weighting system for different aspects of the system, this can easily skew selections, resulting in a system that has a very good rating yet is unable to perform the required function. Keep in mind that though this is a high-level survey, we are asking broad critical questions, not attempting to compare detailed minutia. Also keep in mind that a particular requirement might potentially fall under multiple headings. For example, compliance with 21 CFR Part 11 regulations might be included under functionality or security.\n\nIn-depth evaluation \nWhile in-depth analysis of the screened systems will require a more detailed examination and comparison, for the purpose of this initial survey a much simpler assessment protocol will serve. While there is no single \"correct\" evaluation protocol, something in the nature of the three-leaf scoring criteria described for QSOS[14] should be suitable. Keep in mind that for this quick assessment we are using broad criteria, so both the criteria and the scoring will both be more ambiguous than that required for an in-depth assessment. Do not be afraid to split any of these criteria up into multiple finer classes of criteria if the survey requires it. This need would be most likely appear under \"system functionality,\" as that is where most people's requirements greatly diverge.\nIn this process, we will assign one of three numeric values to each of the listed criteria. A score of zero indicates that the system does not meet the specified criteria. A score of one indicates that the system marginally meets the specified criteria. You can look on this as the feature is present and workable, but not to the degree you'd like it. Finally, a score of two indicates that the system fully meets or exceeds the specified requirement. In the sections below I will list some possible criteria for this table. However, you can adjust these descriptions or add a weighting factor, as many other protocols do, to adjust for the criticality of a given requirement.\nRealistically, when it comes down to some of your potential evaluation criteria, the actuality is that for some of them, you can compensate for the missing factor in some other way. For other criteria, their presence or absence can be a drop-dead issue. That is, if the particular criteria or feature isn't present, then it doesn't matter how well any of the other criteria are ranked: that particular system is out of consideration. Deciding which, if any, criteria are drop-dead items should ideally be determined before you start your survey. This will not only be more efficient, in that it will allow you to cut off data collection at any failure point, but it will also help dampen the psychological temptation to fudge your criteria, retroactively deciding that a given criteria was not that important after all.\nAt this stage we are just wanting to reduce the number of systems for in-depth evaluation from potentially dozens, to perhaps three or four. As such, we will be refining our review criteria later, so if something isn't really a drop-dead criteria, don't mark it so. It's amazing the variety of feature tradeoffs people tend to make further down the line.\n\nSystem functionality \nWhile functionality is a key aspect of selecting a system, its assessment must be used with care. Depending on how a system is designed, a key function that you are looking for might not have been implemented, but in one system it can easily be added, while in another it would take a complete redesign of the application. Also consider the possibility of whether this function must be intrinsic to the application or if you can pair the application being evaluated with another application to cover the gap.\nIn most cases, you can obtain much of the functionality information from the project's web site or, occasionally, web sites. Some projects have multiple web sites, usually with one focused on project development and another targeting general users or application support. There are two different types of functionality to be tested. The first might be termed \"general functionality\" that would apply to almost any system. Examples of this could include the following:\n\n User authentication\n Audit trail (I'm big on detailed audit trails, as they can make the difference between between being in regulatory compliance and having a plant shut down. Even if you aren't required to have them, they are generally a good idea, as the records they maintain may be the only way for you to identify and correct a problem.)\n Sufficient system status display that the user can understand the state of the system\n Ability to store data in a secure fashion\nWe might term the second as \"application functionality.\" This is functionality specifically required to be able to perform your job. As the subject matter expert, you will be the one to create this list. Items might be as diverse as the following:\nFor a laboratory information management system...\n\n Can it print bar coded labels?\n Can it track the location of samples through the laboratory, as well as maintain the chain-of-custody for the sample?\n Can it track the certification of analysts for different types of equipment, as well as monitor the preventive maintenance on the instruments?\n Can it generate and track aliquots of the original sample?\nFor a geographic information system (GIS)[55]...\n\n What range of map projections is supported?\n Does the system allow you to create custom parameters?\n Does it allow import of alternate system formats?\n Can it directly interface with geographic positioning devices (GPS)?\nFor a library management software system (LMSS) (or integrated library system [ILS], if you prefer, or even library information management system [LIMS]; have you ever noticed how scientists love to reuse acronyms, even within the same field?)...\n\n Can you identify the location of any item in the collection at a given time?\n Can you identify any items that were sent out for repair and when?\n Can it identify between multiple copies of an item and between same named items on different types of media?\n Can it handle input from RFID tags?\n Can it handle and differentiate different clients residing at the same address?\n If needed, can it correlate the clients age with the particular item they are requesting, in case you have to deal with any type of age appropriate restrictions? \n If so, can it be overridden where necessary (maintaining the appropriate records in the audit trail as to why the rule was overridden)?\nFor an archival record manager (This classification can cover a lot of ground due to all of the different ways that \"archival\" and \"record\" are interpreted.)...\n\n In some operations, a record can be any information about a sample, including the sample itself. By regulation, some types of information must be maintained essentially for ever. In others, you might have to keep information for five years, while you have to maintain data for another type for 50 years.\n Can the application handle tracking records for different amounts of time?\n Does the system automatically delete these records at the end of the retention period or does it ask for confirmation from a person?\n Can overrides on a particular sample be applied so that records are not allowed to be deleted, either manually of because they are past their holding date, such as those that might be related to any litigation, while again maintaining all information about the override and who applied the override in the audit trail?\n In other operations, an archival record manager may actually refer to the management of archival records, be these business plans, architectural plans, memos, art work, etc.\n Does the system keep track of the type of each record?\n Does the system support appropriate meta data on different types of records?\n Does it just record the items location and who is responsible for it, such as a work of art?\n If a document, does it just maintain an electronic representation of a document, such as a PDF file, or does it record the location of the original physical document, or can it do both?\n Can it manage both permanently archived items, such as a work of art or a historically significant document, and more transitory items, where your record retention rules say to save it for five, 10, 15 years, etc., and then destroy it?\n In the latter case, does the system require human approval before destroying any electronic documents or flagging a physical item for disposal? Does it require a single human sign-off or must a chain of people sign-off on it, just to confirm that it is something to be discarded by business rules and not an attempt to hide anything?\nFor medical
