Historical Data and the New LIMS
Historical Data and the New LIMS
By Robert Benz, Sales and Marketing Director, at Khemia Software, Inc.
I recently received a call from a very nice municipal laboratory director and the conversation regarding LIMS, LIMS implementation, good LIMS implementers turned towards historical data. I laughed and thanked her for the idea of this article. I really should have thought of it. With virtually every LIMS sale whether it be to a municipality, commercial laboratory or university, the issue of historical data comes up.
Most laboratories purchasing a LIMS, if they are replacing an old LIMS or are buying their first, have been in operation for some time. As we are all aware, historical data generally must be kept on record for some period of time based on various regulations and is often wanted for general trending information. In many cases, that period of time is five years. So, now what?
There are a few options for handling historical data. The most common are leave it as is, make a historical data repository or fully integrate the old data in the new LIMS. We’ll discuss all of these in some detail below.
Leave the old data as is:
This is often the answer a laboratory does not want to hear, however, it is often the best answer. A laboratory looks for a LIMS when the old way of record keeping, whether within LIMS or not, does not meet its requirements. Newer LIMS have advanced in capabilities a great deal in a short time, far surpassing the capabilities of LIMS five to seven years old or older. Often, to make old data fit neatly into the new LIMS, a laboratory must hobble the new LIMS to meet the old conventions and criteria. One commonly faced example is methodology. In older LIMS, even older versions of currently sold LIMS, if there was a slight change to a method then there had to be a second method created (i.e. for EPA Method 8260B it is common to have 8260B, 8260BClientA, 8260BClientAsub, 8260BClientB, etc.). I’ve often seen laboratories with 10+ copies of the same method. Multiply that by the number of variations on many methods and it is not too rare to find a laboratory that runs a total 80 methods having 350+ listed in their LIMS.
In newer LIMS, changes to method do not require a new method to be created. Instead the method is modified at the client or project level, allowing a laboratory to have many variations of the method itself with only one method listed. This not only makes method selection much easier for log in and analysts, but also for quality control. Updating one method’s MDL study is one thing, updating 10 plus is another.
Attempting to squeeze all historic data into a new LIMS often results in a loss of much of the capability of the new LIMS. You would never purchase a new car and then ask for the gas mileage to be made worse, request clogged fuel injectors and to affect a high pitched whine in the back speaker…so why would you essentially do that to your LIMS? It is often best to keep things simple. Buy a new LIMS and start over fresh. Then simply keep a copy of the old LIMS running along smoothly on one or two computers in the laboratory until the data is no longer needed, then retire it. More often than not, old data must be kept, but is rarely used.
Make a historical data repository:
Historical data repositories are a halfway step in between the leave it as is scenario and fully integrating old data. In a historical data repository, finalized data is essentially moved into a flat table where it may be accessed by the new LIMS. Often, this finalized data must be combined in some ways to make it fit the new LIMS. Using the example above, if a specific method went from being listed 10 times down to two or three times with variations the data from the 10 various methods must be “squashed down” in a logical fashion to fit within those two or three methods now listed. Also, part of the rationale behind only finalized data coming over is that no two commercially available LIMS are identical. You simply cannot bring over pieces and parts; you can only modify it all and bring it over or finalized data must be made to fit.
Fully integrate historical data in the new LIMS:
Full integration of historical data within a new LIMS is frankly no easy feat. As no two LIMS are identical, it requires the breakdown of data into its most granular form and then re-entering the data into the new LIMS. Is it possible? As with many things in the IT world, well, yes, it is. How long will it take? Probably three times longer than originally quoted, which was already inflated “just in case”. Aside from the time it takes, the cost is often most prohibitive. While it is common for laboratories to ask for this upfront, when the realities of time and money come to light, they rarely choose this route. In situations where it does occur, it is often as a joint project between the LIMS company and the laboratory, done over a much longer period of time. In theory it seems simple. The reality though is that every result reported is the result of an analytical run; That analytical run potentially had a prep batch, has taken place on a calibrated, verified instrument (initial verification, MDL study, etc.) by an approved analyst after being sampled (all field data) and logged in (date/time stamped, cooler temperature verified, etc.). Each one result can have 10 to 50 associated details. When that is multiplied out over all of the individual results for five years, one can image the immense task at hand. True, some tables may be directly imported and some details easily brought over, many eventually have to be done one by one and matched.
How to decide on handling historical data?
Cost is a major consideration in relation to historical data; the cost of how to handle historical data in terms of ease of use of data, in terms of converting the data and in terms of financial requirements. While keeping old data as is or in a historical data repository may work perfectly for many, if not most laboratories, some will truly need a full integration of historical data and not a workaround. A laboratory should carefully evaluate what they need and speak to their prospected LIMS vendor. Be aware that without carefully looking and having access to the back and front end of the data, a LIMS vendor may have to give gross estimations in terms of data migration or may outright refuse. The terms of use of the historic LIMS need also be evaluated to make sure a stable, unedited copy of the existing program may be kept for use for the required time.
Indeed, it is true, this conversation almost always happens to some degree during the selling process of a new LIMS. Sometimes a laboratory is well versed on the options, other times, not so much. This is a conversation that should occur early in the process though so no one, laboratory or LIMS vendor, deals with an unpleasant surprise based on unrealistic assumptions and expectations.
For more information contact:
Robert Benz, rbenz@khemia.com, 843-810-2075
