1. Managing Scanning Failures: Wristband Update 2014
And everything you ever wanted to know about “keyed entries”

2. What is “Managing Scanning Failures?”
Managing Scanning Failures in BCMA (MSF) software was released in January 2009 as patch PSB*3*28
Software developed because there was no way to track when and why bar code scanning was not used.
Provide users with an automated reporting method of bar code scanning failures of both patient wristbands and medication bar code labels without disrupting their normal routine.
Also allows quantification of system issues so that performance improvement teams can target areas needing improvement and compare before and after results to determine if solutions were effective.
Purpose is to encourage bar code scanning over manual entry, capture information on why scanning was bypassed, create reports that notify Pharmacy, Nursing and Performance Improvement Services of system issues that cause nurses to bypass scanning and to create problem alerts so they can be resolved in a timely manner.

3. FileMan BCMA Reports Unable to Scan (Detailed)
Local level reports (by facility)
Reports by ward
Locally developed FileMan reports (not supported by BCRO)
Unable to Scan (Detailed)
Unable to Scan (Summary)
FileMan

4. VSSC Reports

5. VSSC Reports Continued
National aggregate reports
Local level reports
Various report types available

6. BCMA vs. VSSC BCMA VSSC Local reports only Available back to 2009
Uses report requester profile information to filter data
National or local reports
Aggregate data
Only last 11 months data available
Reports come from raw data

7. Goals
The Managing Scanning Failures feature of BCMA was designed to allow end-users to communicate to local BCMA support staff when items were unable to be scanned at the point-of-care without significant interruption to his or her workflow.
Primary goal of MSF = allow users to report non-scanning items in real time to local BCMA support staff
BCMA support staff to fix reported problems

8. What MSF is not Not designed to “force” user to report problems
Not designed to identify “non-compliance”
Indicator targets are not national goals
MSF data should NOT be used as a performance measure related to behavior or compliance
MSF data SHOULD be used as an indicator of SYSTEM performance
Never designed to “force” a user to report non-scanning items
Not designed to identify “non-compliant” behaviors
“Indicator targets” in data definitions document are not “national goals”
MSF data should NOT be used as a “performance measure” related to behavior/compliance
MSF data SHOULD be used to fix reported problems and may be used as an indicator of system performance

9. BCRO and MSF Scanned and scannable are two different concepts
BCRO changed internal (office) measure in 2012
Removed keyed-entries from internal BCRO calculation
Goal is to achieve 100% scannability of medications (i.e. no reports of items “unable to scan”)
Going back to the original goal of MSF. Scanned and scannable are two different concepts. BCRO is not really concerned with whether or not something was actually scanned. What we are concerned with is whether or not the item was scannable at the bedside. If a user reports an item as unable to scan, then that decreases the number of items that were scannable at the bedside. However, the scannability can only be determined if we put that number in a context. That is where the number of scanned meds comes in to play. In order to come up with a number that makes sense to people, we put it in the context of a percentage and to do that, we have to have a denominator. Originally, we were using total wristband transaction and total medication transactions as the denominator for our scanning rates and this is still true on the VSSC site. However, if a user chooses not to scan an item (such as keying an item in), that is a personal or behavioral choice and because they chose not to scan the item, we have no idea whether the item was or wasn’t scannable. Therefore, in 2012, we changed our internal performance measure by removing keyed-entries from the calculation. Therefore, instead of reporting generic overall scan rates, we are focusing more accurately on what we actually have some control over, which is scannability “are items able to be scanned at the point of care?”.
Our goal in BCRO for 2014 was to reach 99.25% scannability for medications (which we are currently exceeding at 99.37%) and although we are no longer reporting wristband scannability up to management; our current wristband scannability rate is 98.83% based on our updated calculation. So, our goal in BCRO is to achieve 100% scannability, in other words, we want to see no (zero) reports of items that were “unable to scan”.
So, you’re thinking locally “ok, we’ve already gotten our numbers of ‘unable to scan’ items down to nearly zero, but our problem now is that we still have some keyed entries that are bringing our numbers down. We want our numbers to be 100%. How can we achieve that?”
I respond that you are likely looking at this the wrong way. We will discuss keyed entries at length in just a few moments, but for now…

10. Control Chart http://vaww.ncps.med.va.gov/Tools/ControlCharts/
Aside from addressing actual reports of items that are unable to scan, your main concern should not be a specific number or target (such as 100%), but rather ensuring that the process is “in control”.
A brief description of a control chart is a chart that has a centerline indicated by the mean (average), an upper control limit and a lower control limit defined as +/- 3 standard deviations from the mean. Each data point (for example, your monthly MSF numbers) is plotted on the chart like you would a line chart and then evaluated that data point in relation to previous data points and the control limits according to some rules. The rules may vary as in the difference between the example on the screen and what is in my speakers notes, because they came from different sources.
Out of control situations identified from the control charts:
A point outside the 3-sigma control limits.
At least two out of three successive values fall on the same side of, and more than two sigma units away from, the center line.
At least four out of five successive values fall on the same side of, and more than one sigma unit away from, the center line.
At least eight successive values fall on the same side of the center line.
You may define your own rules. For example, you may set your upper and lower control limits at +/- 2 standard deviations instead of 3.
The National Center for Patient Safety has a page on their website dedicated to control charts.
Again, your focus should really be on keeping the process in control and not worrying about keyed entries (unless those keyed entries make up more than 2% of your total transactions).

11. What is a keyed entry? PS2 “Y” adapter or “keyboard wedge” USB
Now, I know I’ve just said not to worry about keyed entries unless they make up more than 2% of your total transactions, but I also know that you are a tenacious group and don’t just do something because we’ve said so. You all want to know the rationale, the evidence, and see some hard data. So, for those who really want to know all about keyed entries, I’m going to hopefully satisfy your thirst for knowledge.
So what is a keyed entry?
A keyed entry is really a misnomer. A keyed entry is not always a keyed entry. In order to really understand this, you have to have a good understanding of BCMA including a bit of history. BCMA was designed before the Universal Serial Bus (USB) port was commonly used to connect input devices (such as keyboards, mice, and scanners) to the CPU. As a result, most of the computers used what was called a “keyboard wedge” connector. This is basically a “Y” adapter that on one end plugs into the CPU and has two input ports on the other end. The keyboard was plugged into one input port and the scanner was plugged into the other. Since the scanner and keyboard then shared the same input port, the computer had no way to distinguish if data was being input from the scanner or if data was being input by the keyboard.
USB

12. Prompts removed from view
Pre-MSF
Post-MSF
Before MSF, there were visible input prompts for the wristband and medications. Users had been allowed to manually type human readable numbers because there were so many medications that did not have manufacturers bar codes on them and wristband printers and print media weren’t always reliable. However, since the FDA requirement to bar code all unit dose prescription medications went into effect in 2006, the need to manually enter the data had decreased significantly. Also, if the user was able to just type in the number, there was nothing to encourage them to report when something didn’t scan. To encourage staff to scan more consistently and report when an item wouldn’t scan in BCMA, the visible input prompts were removed from view.
Keep in mind that as recently as 2011, there were still facilities using keyboard wedge type connectors on their scanners. Therefore, we could only remove the input prompts from view. If we had removed the input prompts completely, the user would have no way to input data at all (not even through the scanner). If we designed BCMA to only accept input from a USB port, those facilities using PS2 connected scanners would no longer have been able to use BCMA.
What this means is that the users still have the option to type information in to BCMA at the “scan patient wristband” prompt and when the user is looking at the VDL. The difference is that the user can no longer see what he or she is typing. So then you are asking, if BCMA can’t tell if something is scanned or keyed in, how is data even being classified as a keyed entry or a scan?

13. Input Method Determination
Uses a time limit setting
Time from 1st character of input string to [CR] command
Must be < input parameter setting to appear as “scanned”
Input data string 1 2 3 4 5 6 7 8 CR
Timer Starts
Timer Stops
Input time
0ms
400ms
200ms
The answer is in the input timing mechanism. There is a timer built in to BCMA that measures the time it takes from the first character of an input data string to the receipt of the carriage return. It does not usually matter how long the user sits at a particular prompt, what does matter is how long it takes from time the first character is input in the field to the time the [Enter] or [CR] command is received. This is what BCMA uses to classify an item as keyed or scanned. And this is where I attempt to redefine the term “keyed entry”. The term keyed entry (which I said is really a misnomer) should really be called “An entry that exceeded the 'input method determination' time limit setting in BCMA”. Of course, “keyed entry” is much easier to say, even though it really isn’t an accurate description, so no matter how hard I try, I don’t think people are going to stop calling them keyed entries. But at least I know that I have shared the factual information with you.
The time limit setting in BCMA is set at a default of 400ms (4/10 of 1 second). This parameter can be adjusted from 250 to 1500ms or ¼ of a second to 1.5 seconds to accommodate for a variety of scanner input speeds. The instructions for adjusting the timing mechanism can be found in the BCMA Managers Manual.

14. Input Timer Mechanism Scanned Entry Timing threshold Keyed Entry
Input data string 1 2 3 4 5 6 7 8 CR
Timer Starts
Timer Stops
Input time
400
200
Timing threshold
500msThreshold
Keyed Entry
On the left is an example of a site that has adjusted their threshold from the default of 400ms to 500ms. The /K= parameter indicates the timer setting. If there is no /K= parameter in the target line, then that instance of BCMA is using the default setting. Depending on how BCMA launches at your site, you may have different machines with different settings. That’s not really anything to be concerned about unless you are finding a lot of problems (such as a suspicious number of keyed entries) in your MSF data. Leaving the parameter discussion aside, here is an example of how the mechanism works.
Now with an understanding of the mechanism BCMA uses to classify an entry, you can hopefully understand how this could result in a variety of classifications, including some misclassifications.
Input data string 1 2 3 4 5 6 7 8 CR
Timer Starts
Timer Stops
Input time
800
400
500msThreshold

15. Potential MSF Results table
Actual
Scanned (+)
Keyed-in (-)
Reported
True Positive
False Positive
False Negative
True Negative
Because BCMA has to evaluate the input time of every entry, there is the potential that the entry may be incorrectly classified. Here you see a table with the actual occurrence across the top and BCMA’s classification of that entry along the left. (describe each box in table).
True positive and true negative results are expected “normal” occurrences. Since we know most items are scanned and are classified correctly, based on the aggregate reporting across all sites, we’ll say these true positives make up about 98% of all entries. Keyed entries across the organization make up less than 1% of total transactions, so even if every keyed entry were classified as a keyed entry incorrectly, we are only talking about less than a 1% difference. So, unless keyed entries are really out of control at your facility, the amount of effort required to identify and eradicate them may not be worth it. That being said, I know there are people who just want to know where they are coming from, so I will continue.
False positive and false negative results (as infrequent as they may be) are “abnormal” and occur at higher or lower rates depending on the timer mechanism threshold setting and other variables. Next, I’m going to talk about the impact of changing the timing threshold as it applies to the frequency of these abnormal events on medications and wristbands as well as the outcomes from each.

16. False Positives
When an item is actually keyed in but classified as scanned
Medications
Usually a very small number (only short IENs)
Shorter threshold may increase false positives
Wristbands
Never a false positive for a wristband (even at maximum threshold of 1500ms)
Takes too long for a human to type in 9 digit SSN
A false positive will occur when an item is actually keyed-in but BCMA records the entry as scanned. In order for this to occur, the item must be keyed-in faster than the threshold limit. In testing the timing mechanism when MSF was under development, it was possible for a human to type very short (one, two, or three digit) internal entry numbers (IENs) of medications within the 400ms threshold. However, with the exception of the 1 and 2 digit long IENs, it required an active desire to “try to beat the system” exhibited by practicing the keystroke sequence, pre-aligning fingers on the necessary keys so as to maximize typing speed, etc. Our office is not concerned with false positives of medication entries because there are so few that they are not of any real consequence. Also, they tend to be a matter of personnel behavior and not indicative of problems or issues with the system.
Increasing the timing threshold decreases the effort needed to actively deceive the system and at the uppermost limit of 1500ms was shown to significantly increase the number of false positive results for medications with IENs up to four digits in length. However, this only applies to medications. Because no human can type a typical 9 digit SSN accurately in less than 1500ms, increasing the threshold to the maximum of 1500ms will not increase the number of false positive reports for wristbands. Moreover, it may actually decrease the number of false negative classifications for wristbands, therefore making your classifications for wristbands more accurate.
Actual
Keyed-in (-)
Reported
Scanned (+)
False Positive

17. False Negatives
When an item is actually scanned but classified as a keyed entry
Medications
Occasional (Meds with very long NDCs)
Wristbands
Increasing threshold may decrease number of false negatives
Actual
Scanned (+)
Reported
Keyed-in (-)
False Negative
A false negative occurs when an item is actually scanned but BCMA records the entry as keyed-in. To revisit my last statement in another way, if your facility has a high number of wristband keyed entries. One way to determine if some of these wristband keyed entries are erroneous, is to increase the input timing mechanism threshold to the maximum of 1500ms and review the data after a few weeks. If the number of wristband keyed entries drops significantly, then many of the wristband entries may have been erroneously classified in the past. Again, increasing the threshold cannot negatively impact your wristband keyed entries. If your medication keyed entries are already at or near 0, then increasing your threshold will also have little or no impact on your medication keyed entries.
Unlike a false positive which generally requires an active attempt to deceive the system, several scenarios or actions that may be unknown or out of the user’s control can lead to false negatives. Any time the threshold is exceeded BCMA records a keyed-entry. Testing before MSF was released proved that medications with long bar coded numbers (12 or more digits) sometimes caused false negative results with the default threshold of 400ms. This means a user’s transaction may be erroneously classified as keyed-in when it was in fact scanned. Increasing the threshold would decrease the number of this particular type of false negative by recording these items as scanned, thus converting these false negatives into true positives.

18. Factors Known to Cause False Negatives
Touching a key on a keyboard before scanning
Overloaded demands of the central processing unit (CPU) in the computer
Goofing off
Scanning QR codes or other 2D bar codes with long strings of data
Here are some things that we know of that can cause “false” keyed entries:
If a user touches a key on the keyboard (usually accidentally) and then scans an item, that will generate a keyed entry because the initial keystroke starts the timer and the timer will run until the carriage return character is processed. There is no way to know if this happened unless you witness it and I have never known a nurse to remember that this action occurred; therefore, the detectability for this issue is very low and because it occurs accidentally, it is virtually impossible to eliminate. Note: We have seen instances where non-password-protected screen savers were used and when the staff touched a key on the keyboard to “wake-up” the monitor. This keystroke is captured by BCMA and begins the timing mechanism and the subsequent scan is classified as a keyed-entry. Obviously, this instance is controllable because you can disable the screen saver. However, in that situation, the keyed entries are usually much greater because it happens very frequently since all the computers in the hospital are configured with the same screen saver. In any of these situations, where an extra keystroke is accidentally pressed, the scan following the keystroke is also going to result in an error message, because the keystroke is now part of the input string. The user would either scan the item again (at which time it BCMA would accept it) or the user would go on to use the Unable to Scan feature to report the item didn’t scan.
The timing mechanism can be affected by competing CPU demands. If the computer is in the middle of a software update, multitasking or any other task that uses a great deal of CPU memory at any given time, the competing CPU demand can skew the timing mechanism and result in a false keyed entry. Detectability for this instance is also low because unless you are standing at the computer to be able to evaluate the processes occurring simultaneously, it may be difficult to identify. Note: This one was discovered because the antenna from a mobile workstation was broken and almost all the keyed entries on that unit came from that one cart. When the cart was repaired, the keyed entries disappeared. If this were the case at most facilities, the number of keyed entries would be much higher on a single unit. However, it did solidify the case that a CPU which is overburdened (in this case, by using additional resources to maintain the wireless connection) can contribute to false keyed entries.
Goofing off – if a user walks up to the computer and types a series of random characters and then presses the enter key, this may result in a keyed entry. This is a true keyed entry, not a false keyed entry and you have no way of knowing unless the user admits to doing it because the characters are not captured. However, because the user just typed some random characters and clicked enter, they did not open a patient record or administer a medication, so there is no patient safety issue here either.
Scanning QR codes or other 2D bar codes with very long strings of data. The Indian Health Service recently started using our BCMA application. When they designed their wristbands, they decided to include a DataMatrix bar code on the wristband. This DataMatrix bar code is used by another application to print specimen labels at their facility and is comprised of a string of about 75 characters. However, some staff were scanning the DataMatrix bar code into BCMA instead of the linear bar code. The number of encoded characters took longer to process than the maximum limit setting in BCMA and resulted in a keyed-entry. Detectability for this specific scenario was high because the DataMatrix was introduced to the user on the wristband, so it was a known risk. However, unless you have provided these types of bar codes to the staff, you may not be able to detect if this is the problem. 2D bar codes are becoming ubiquitous and some are appearing on medications or other supplies, so staff must be made aware of which bar code to scan if there are multiple bar codes available on an item. Staff may become bored or curious and just start scanning random 2D bar codes (from magazines or posters) to see what happens. The bad news is that they appear as keyed entries on these reports. The good news is, there is little cause for concern over this because BCMA would generate an error when the bar code was scanned and no patient harm would result.
While all these issues may produce keyed-entries in the Unable to Scan log file, none are of any real concern and do not indicate systems problems or present patient safety risks. We feel strongly that resources should be reserved for fixing the identified issues of meds or wristbands that were reported as “unable to scan” (actual known systems problems) and/or those that pose patient safety risks.

19. Additional Info on Keyed-Entries
Not all keyed-entries are bad
Users are not informed of whether an entry is assessed as keyed or scanned (therefore they may truly not know they produced a keyed entry)
False negatives do exist (and are not easily controlled)
False negatives can account for up to 2% of all transactions
BCRO does not recommend following up on keyed entries unless they account for more than 2% of the facility’s overall transactions
Not all keyed entries are bad. As I have shown, many are accidental and inconsequential. Although we put a great deal of faith into the bar codes and scanning, the exact mechanism of data entry (scanned or keyed) is not that important. While studies have shown how bar code scanning is more reliable than typing, these studies do not take into account that every action taken by a user is verified by the system. If a user mistakes a keystroke, is BCMA going to pull up the wrong patient or administer the wrong med? Not likely, BCMA still forces the user to confirm the patient’s identity even after the patient information is entered and if a user types in the number of a medication incorrectly, BCMA will tell the user it is not the correct medication. For all intents and purposes, bar code scanning is just a practical method of data entry. It speeds up the process by not forcing the user to key everything in.
I would dare to say that BCMA could provide the same error reduction rate even without bar codes or scanning (although if nurses had to type in every single NDC of every medication, they would likely throw the computer out the window). What is important is how the users are using the system. Are they using the system as intended (by allowing the computer to match the patient and drug information)? Is the user entering the information from the wristband which is attached to the patient and are they entering the identifying information from the medication in their hand at the bedside?
The problem arises when the user is entering information away from the patient’s bedside or from memory (like when we used to type in the IEN from the blood glucose strips). This is the one thing that cannot be captured in any report. There is no perfect solution for assessing this. Looking at whether items are scanned or keyed can only point you in the general direction of someone or an area that might not be using BCMA appropriately, but as detailed above, the simple determination of scanned versus keyed is not directly related to improved safety. Therefore, scanning rates are not an appropriate measure for overall success.
The Bar Code Resource Office has never supported using the MSF data as a performance measure (especially for keyed-entries). If a user reports something as “Unable to Scan”, they are (generally) doing so as a way to inform you (the BCMA Coordinator) that there is a problem with the system that needs to be fixed…like a driver reporting a problem with a traffic signal. They are reporting it because it benefits them (and their peers) when you fix an issue that is reported. Their information is captured so you (or the police officer in the traffic signal analogy) can contact them to get more information if it is needed. Not every person who comes upon a broken stoplight is going to take the time to report it. Likewise, when someone types something in, the information is captured, but this person has chosen not to report the problem and therefore, BCMA does not provide any reporting capability for this.  
MSF was not designed to be a traffic camera to catch people going through red lights; it was designed only as a way for the user to quickly report when there is a problem with the system. The fact that MSF captures data when a user manually bypasses the system (i.e. keys in entries without reporting scanning problems) is only a byproduct of the original system; the keyed entry reports are NOT 100% accurate and should NOT be used as any sort of performance measure. The classification of an entry as either “keyed” or “scanned” is performed by an algorithm and occurs in a “black box” with no feedback to the user. The user has NO IDEA how his or her entry was classified by the system (unless they are a habitual abuser of the workaround). That is part of the reason why the data is not available from a BCMA report…if you (or someone else) had to create a FileMan routine in VistA to tease out the data (with regard to BCMA), then you really should be asking yourself about the validity of the data and why BCMA doesn’t already provide the ability to get the data directly.
If your facility’s keyed entries make up greater than 2% of the overall transactions at your facility or if you strongly suspect there is a pattern to your keyed entries and you would like us to investigate, you may contact us in the BCRO and we can take a look at your data with you. We have tools to help determine if something is a systems issue or potentially a behavioral issue and we have 5 years of experience evaluating analyzing this data.

20. Systems problem?
This is an example of one of the reports we have access to that can help us determine if something is a systems issue. Each bar on the graph represents a single user’s transactions. As you can see, there are no names on the report, but we do not need names to tell that this is widespread across the unit. We have other tools we would look at in more detail, such as historical patterns and individual user’s patterns to see if this may me an actual systems problem, a temporary “blip”, or if this is a cultural problem that grew over time.

21. Behavioral/Personnel Issue?
This is an example of what appears to be a behavioral or personnel issue. However, this is only one point in time. We need to look more deeply into this data to make this determination. All too often we find sites that try to get this information on their own using a FileMan routine they wrote themselves or received from another site only to find out that they are not getting the right data or they are not looking at the big picture. It is important to have the right data and to be able to look at the big picture to really understand what is going on and we are willing to help with that if there is truly a need for it.

22. Summary BCMA and VSSC report results may be different
VSSC and BCRO results differ because BCRO has removed keyed entries from its internal measure
Goal of MSF is to report and fix items that are unable to scan (improve scannability; 0 reports of UAS)
Sites should focus on maintaining control of the process, not reaching 100% total scans
There are known uncontrollable variances that can cause keyed entries
Keyed entries aren’t always keyed entries
BCRO may help if keyed entries are causing process to be out of control (keyed entries > 2%)