Epic fail for DST fallback in hospital health records
Sydney Lupkin reports in today's Kaiser Health News that Epic Systems, a leading system used for hospital health records, cannot handle DST fallbacks such as Sunday's fallback in the US. Lupkin writes: 'Carol Hawthorne-Johnson, an ICU nurse in California, said her hospital doesn’t shut down the Epic system during the fall time change. But she’s come to expect that the vitals she enters into the system from 1 a.m. to 2 a.m. will be deleted when the clock falls back to 1 a.m. One hour’s worth of electronic record-keeping “is gone,” she said.' Apparently competing systems have similar problems. Lupkin writes that many hospitals plan for Cerner (another major system) to be down after a DST fallback as well. Dr. Steven Stack, past president of the American Medical Association, said these DST glitches are "unacceptable", considering that Apple and Google seem to have dealt with seasonal time changes long ago. Although Lupkin didn't say so, credit should be given to the design for the Unix operating system that is the basis for these successes of Apple and Google, as Unix standardized on UTC in the early 1970s; apparently Epic and Cerner (both founded 1979) did not get the message. The moral of this story seems to be: arrange for your medical emergencies to occur some time other than early Sunday in the US. Source: Lupkin S. Like Clockwork: How Daylight Saving Time Stumps Hospital Record Keeping. Kaiser Health News. 2018-11-03. https://khn.org/news/like-clockwork-how-daylight-saving-time-stumps-hospital...
I'm not sure what to make of this (query: how many people a year does it have to kill to be an "epic fail"?), but in many ways this graf sticks out: Not all hospitals turn Epic off. At the Johns Hopkins Hospital in Baltimore, providers who need to check patients periodically through the night use a workaround: They enter vitals at 1 a.m. and then when the clock falls back an hour later and they have to enter new vitals, they list them at 1:01 a.m. They leave a note that it’s an hour later, not a minute later. That’s how the Cleveland Clinic does it, too. This is confusing, since it suggests the problem is just about not entering multiple datasets at the same HH:MM, which is a little differnet from actual deletion (although also a bit more plausible); or maybe about elapsed time measurement. On the other hand, read strictly, it also implies not entering anything from 01:00:59 EDT through 01:59:59 EDT all the way through 01:01:00 EST; but that strict interpretation doesn't seem too likely. Be afraid when drug delivery systems (e.g. IV pumps, etc.) decide they know what time it is and mess up dosages across time changes. Fortunately we're not there [yet]? I suppose since dosages are relatively continuous (so many millileters per second?) rather than coarsely discrete (1 bag per hour?), this sort of failure might be less likely. Maybe.
The moral of this story seems to be: arrange for your medical emergencies to occur some time other than early Sunday in the US.
I dunno. A different takeaway might be that hospitals have scheduled an annual drill for computer downtime at a regular predictable time. I'm sure there are plenty of other times of the year when electronic systems go down, and that those failures happen with less predictability and at more inconvenient times then 2:00am in the morning on a Sunday in November. So in a certain perverse way, it's nice to know they're forced to deal with it? --jhawk@mit.edu John Hawkinson
John Hawkinson wrote:
This is confusing
Yes, unfortunately there's not much technical detail in Lupkin's article.
Be afraid when drug delivery systems (e.g. IV pumps, etc.) decide they know what time it is and mess up dosages across time changes
I found a recent scholarly article by Ehlers et al (cited below) that covers DST problems in clinical laboratories. Although these labs don't do drug delivery, the article is illustrative nonetheless. Ehlers et al mention the following topics (I am in some cases making educated guesses about the real problems, as the authors wrote for an medical audience): * The electronic health records (EHR) system may be from a different vendor than the laboratory information system (LIS), and the vendors may have different ways of dealing with DST issues and DST rule updates. * Many hospitals and labs run older systems such as MS-Windows XP where the DST change rules may be wrong and the systems may be no longer supported. * When a time server disagrees with the time settings on the local computer, the wrong side might win. * Some devices update automatically and some don't, leading to time mismatches when output from one is fed as input to the other. * The automatic clock-change procedure sometimes fails on some devices, and the manual process for fixing this can be quite involved and can require IT support. * Some instruments require a pause in operation, then a manual time change, then a restart. * Staffing issues for the missing or extra hour. * Staff might not know manual clock-change procedures done only twice yearly. * These problems surface on the third shift which has lower staffing; conversely, there tends to be less work during the third shift so staff has more time to fix DST-related problems. Ehelers et al. give examples of extended downtime due to some of these issues and recommend having an IT expert available during DST changes. They summarize by saying that fallback is more of a pain than spring-forward, that the switchover isn't much of a problem for labs due to low volumes early Sunday, but that people should watch out for the rare problems that do occur. Ehlers A, Dyson RL, Hodgson CK, Davis SR, Krasowski MD. Impact of Daylight Saving Time on the Clinical Laboratory. Acad Pathol. 2018 Jul 11;5:2374289518784222. doi:10.1177/2374289518784222 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6047237/
On a much more minor note, but a similar problem, Fitbit's sleep tracking still doesn't work with the daylight saving time change. It somehow manages to both think that it lost an hour of data in the middle of the night (represented by a grey bar hour with no pulse tracking) and think the night was an hour shorter than it actually was (the final data point, which was actually measured at 5:54am on the new clock and 6:54am on the old clock, is recorded as 4:54am). I'm not quite sure how they managed to get time handling that wrong, but it amuses me twice a year. Thankfully, the sleep tracking is basically a toy with no real medical significance and useful mostly for personal amusement. But Fitbit probably has more funding and engineering resources to fix problems like this than a lot of medical device companies, which is terrifying. -- Russ Allbery (eagle@eyrie.org) <http://www.eyrie.org/~eagle/>
"Russ" == Russ Allbery <eagle@eyrie.org> writes:
Russ> I'm not quite sure how they managed to get time handling that wrong, but Russ> it amuses me twice a year. It also fails pretty bad if you change timezones... a US bicoastal trip almost always messes up a few hours worth of data. Why they just don't record all the data internally as UTC, I do not know. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 <merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/> Perl/Unix/Dart consulting, Technical writing, Comedy, etc. etc. Still trying to think of something clever for the fourth line of this .sig
On Nov 3, 2018, at 10:54 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
Dr. Steven Stack, past president of the American Medical Association, said these DST glitches are "unacceptable", considering that Apple and Google seem to have dealt with seasonal time changes long ago. Although Lupkin didn't say so, credit should be given to the design for the Unix operating system that is the basis for these successes of Apple and Google, as Unix standardized on UTC in the early 1970s; apparently Epic and Cerner (both founded 1979) did not get the message.
Apple doesn't always get it right: https://www.cultofmac.com/67254/how-to-avoid-the-iphone-daylight-savings-bug... I don't know if there have been any Android glitches, but it's not inconceivable. "XXX runs atop Unix" isn't sufficient; something running atop Unix could still get it wrong. I don't know what the lowest layers of Windows (NT) use (I don't have any "NT Native API" documentation handy), but the Windows API offers GetSystemTimeAsFileTime(), which supplies tenths-of-microseconds-since-the-FILETIME-epoch ("January 1, 1601 (UTC)", using the proleptic Gregorian calendar and some probably-proleptic form of UTC), so it can do UTC as well. According to https://en.wikipedia.org/wiki/Epic_Systems Epic "offers an integrated suite of healthcare software centered on a Caché database provided by InterSystems"; according to https://en.wikipedia.org/wiki/InterSystems_Caché Caché runs atop Windows, assorted UN*Xes, and OpenVMS. SYS$GETTIME on VMS supplies "the number of 100-nanosecond intervals since November 17, 1858." (I guess Cutler likes .1 microseconds as a unit), according to http://h30266.www3.hpe.com/odl/axpos/opsys/vmsos84/4527/4527pro_066.html There's also SYS$GETUTC, which "returns the current system time in 128-bit UTC format", so I guess if you have a version of VMS that has SYS$GETUTC, you can work in UTC. So it's probably just that Epic never got DST handling right.
On Nov 4, 2018, at 5:59 PM, Guy Harris <guy@alum.mit.edu> wrote:
So it's probably just that Epic never got DST handling right.
Back in 1979, UNIX wasn't as big a platform that sort of application as it is now; if they started on VMS (I don't know whether it had a "get the time in UTC" API back then) or on MVS (I'm not sure what it offers), maybe using local time was a more obvious choice, but you'd still think they'd know that stuff does happen at hospitals between 1 AM and 3 AM on Sunday, so....
Guy Harris wrote:
Back in 1979, UNIX wasn't as big a platform that sort of application as it is now; if they started on VMS (I don't know whether it had a "get the time in UTC" API back then) or on MVS (I'm not sure what it offers)
As I understand it, Epic's core suite was written in MUMPS (also known as M), which stores each timestamp as an integer count of days since 1840-12-31, paired with an integer count of seconds since local midnight. Although there are facilities to get UTC and timezone-adjusted timestamps, my guess is that they were added later and that most code assumes local time, which could help to explain the DST problems with that Epic and other EHR companies have (and why they can't or won't fix the problems). The MUMPS zero point 1840-12-31 was chosen to be the end of a leap year for convenience in calculations. The leap year 1840 was chosen so that it could safely represent the birthday of the oldest person conceivably needing healthcare in the US when the timestamp format was effectively standardized for MUMPS in 1969. This is according to a letter from James M. Poitras published in the "Just Ask!" column of the September 1993 issue of "M Computing"; see: http://www.faqs.org/faqs/m-technology-faq/part1/
On Nov 4, 2018, at 8:53 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
As I understand it, Epic's core suite was written in MUMPS (also known as M),
Could be: https://www.forbes.com/sites/zinamoukheiber/2013/03/04/behind-epic-systems-a... "Phillip Ragon, known as Terry, has quietly built Cambridge, Mass-based InterSystems into a $443 million (2012 revenues) company, selling the guts of electronic health records: databases that can easily ramp up and allow doctors to quickly retrieve patient information. Forbes estimates Ragon is worth $1.5 billion, thanks in part to booming sales of electronic health records. He joins two other health IT entrepreneurs on the Forbes billionaires list--Cerner’s Neal Patterson, and Epic Systems founder Judy Faulkner. Remarkably, he is the sole owner of InterSystems. ... InterSystems’ trajectory has been closely tied to the fortunes of Epic, a leading electronic health records vendor, which generated $1.5 billion in revenues last year. Epic is InterSystems’ single biggest customer. “When we met Epic, all the people from Epic and InterSystems could fit around a conference table. We’ve grown up together,” says Grabscheid. In several ways, the two companies followed similar paths. Both are rooted in an older programming language called MUMPS, developed at Mass General Hospital in the 1960s. Their founders coded the first products, and turned down outside investors, preferring to maintain control."
On 2018-11-04 19:11, Guy Harris wrote:
On Nov 4, 2018, at 5:59 PM, Guy Harris wrote:
So it's probably just that Epic never got DST handling right. Back in 1979, UNIX wasn't as big a platform that sort of application as it is now; if they started on VMS (I don't know whether it had a "get the time in UTC" API back then) or on MVS (I'm not sure what it offers), maybe using local time was a more obvious choice, but you'd still think they'd know that stuff does happen at hospitals between 1 AM and 3 AM on Sunday, so...
[Open]VMS provides a 64 bit time with 100ns resolution from the MJD epoch of 1858-11-17, but the update frequency and accuracy depends on interval timer frequency, which may be as low as 100Hz/10ms. IBM S/390 (System z) STCK stores a 64 bit time with a resolution of 1/4096us from the NTP epoch 1900-01-01, updated with a frequency equivalent to incrementing bit 51 (big endian) every us. Most TZ and DST issues result from app designers and/or implementors thinking they understand what systems should do during changeovers, and trying to implement it themselves, possibly undoing the correct behaviour of a tested time/zone/DST handling library. Regulators ensure that the electrical power industry has had a good handle on dealing with local times and 23-25 hour days for decades. OTOH my Motorola PVR broadcast schedule/cable program guide properly showed two periods from 01.00-02.00; but a European sports series broadcast after the switch skipped recording the first hour Sunday morning early-ish local time! -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised.
Guy Harris <guy@alum.mit.edu> writes:
I don't know what the lowest layers of Windows (NT) use (I don't have any "NT Native API" documentation handy), but the Windows API offers GetSystemTimeAsFileTime(), which supplies tenths-of-microseconds-since-the-FILETIME-epoch ("January 1, 1601 (UTC)", using the proleptic Gregorian calendar and some probably-proleptic form of UTC), so it can do UTC as well. ... Caché runs atop Windows, assorted UN*Xes, and OpenVMS. SYS$GETTIME on VMS supplies "the number of 100-nanosecond intervals since November 17, 1858."
In the department of weird epoch reference dates: I have a very distinct recollection that HYDRA/C.mmp[1] ran their system clock in microseconds since Charles Babbage's birthday (26 December 1791). Unfortunately I can't find any evidence other than 40-plus-year-old memory to support that. The C.mmp book cited by Wikipedia describes the clock as being 56 bits wide, which would be plenty for that timespan, but there's nothing in it about the timekeeping convention. regards, tom lane [1] https://en.wikipedia.org/wiki/C.mmp
Guy Harris wrote:
the Windows API offers GetSystemTimeAsFileTime(), which supplies tenths-of-microseconds-since-the-FILETIME-epoch ("January 1, 1601 (UTC)"
As I understand it, it is not so much the API as the operating procedure underneath. One problem is that when an MS-Windows machine is powered off during a DST transition, it does not update the time properly when it is powered back on. In a How-To Geek article published earlier today, Chris Hoffman says the problem occurred with MS-Windows 10 this weekend, and I recall hearing about similar issues years ago. See: Hoffman C. Some Windows 10 Clocks Didn’t “Fall Back,” Here’s How to Fix It. How-To Geek. 2018-11-04 15:23 EDT. https://www.howtogeek.com/fyi/some-windows-10-clocks-didn%e2%80%99t-%e2%80%9... Kuhn M. IBM PC Real Time Clock should run in UT. 2016-10-29. https://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html PS. I don't know whether the "EDT" in Hoffman's article's timestamp reflects the MS-Windows 10 bug or is a subtle joke; it could be both.
On Mon, 5 Nov 2018 at 00:08, Paul Eggert <eggert@cs.ucla.edu> wrote:
PS. I don't know whether the "EDT" in Hoffman's article's timestamp reflects the MS-Windows 10 bug or is a subtle joke; it could be both.
Based on what I can tell, neither. It's currently 10:30 EST, and this article <https://www.howtogeek.com/393423/how-to-change-windows-10-updates-download-f...> is listed from the homepage with a relative timestamp of "6m ago", but the timestamp given on the page itself reads "10:24 EDT". It seems "EDT" has been used for all articles there since the time change, at least. -- Tim Parenti
On 2018-11-05 08:30, Tim Parenti wrote:
On Mon, 5 Nov 2018 at 00:08, Paul Eggert wrote: PS. I don't know whether the "EDT" in Hoffman's article's timestamp reflects the MS-Windows 10 bug or is a subtle joke; it could be both. Based on what I can tell, neither. It's currently 10:30 EST, and this article <https://www.howtogeek.com/393423/how-to-change-windows-10-updates-download-f...> is listed from the homepage with a relative timestamp of "6m ago", but the timestamp given on the page itself reads "10:24 EDT". It seems "EDT" has been used for all articles there since the time change, at least. The HTML time tags include classes which may use JavaScript to generate the text from the attribute and possibly some format which could have tzabbr EDT hardcoded. Hoffman: <time class="dt-updated post-date updated" datetime="2018-11-04 15:23:30">November 4, 2018, 3:23pm EDT</time> Gavin: <time class="dt-updated post-date updated" datetime="2018-11-05 10:24:00">November 5, 2018, 10:24am EDT</time>
-- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised.
Le 05/11/2018 à 02:59, Guy Harris a écrit :
SYS$GETTIME on VMS supplies "the number of 100-nanosecond intervals since November 17, 1858." (I guess Cutler likes .1 microseconds as a unit)
FWIW, VMS also has at least one parameter (namely how long it waits for the user to enter the system time during boot if the time-of-year clock is invalid) whose value is measured in microfortnights. -- John
On 2018-11-05 01:04, John Wilcock wrote:
Le 05/11/2018 à 02:59, Guy Harris a écrit :
SYS$GETTIME on VMS supplies "the number of 100-nanosecond intervals since November 17, 1858." (I guess Cutler likes .1 microseconds as a unit)
FWIW, VMS also has at least one parameter (namely how long it waits for the user to enter the system time during boot if the time-of-year clock is invalid) whose value is measured in microfortnights.
$ units microfortnight 1e-6 fortnight = 1.2096 s -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised.
"John" == John Wilcock <john@tradoc.fr> writes:
John> FWIW, VMS also has at least one parameter (namely how long it waits for the John> user to enter the system time during boot if the time-of-year clock is John> invalid) whose value is measured in microfortnights. The explanation I heard for that is that it's a delay implemented by a fast busy loop, because it's before all the clock mechanisms are set up. And when they timed it out, it was "about a second", hence the obscure time unit so nobody would complain. Or rather, people would just chuckle. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 <merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/> Perl/Unix/Dart consulting, Technical writing, Comedy, etc. etc. Still trying to think of something clever for the fourth line of this .sig
The Epic Systems daylight saving time failure I mentioned three days ago is also mentioned in a well-written article about important electronic health records problems that appears in the next issue of The New Yorker. The article reports the following for last fall's daylight saving transition in a Partners HealthCare hospital running the Epic software system: "The only solution was to shut down the lab systems during the repeated hour. Data from integrated biomedical devices (such as monitoring equipment for patients’ vital signs) would be unavailable and would have to be recorded by hand. Fetal monitors in the obstetrics unit would have to be manually switched off and on at the top of the repeated hour." Gawande A. Why doctors hate their computers. The New Yorker. 2018-11-12. https://www.newyorker.com/magazine/2018/11/12/why-doctors-hate-their-compute...
participants (9)
-
Brian Inglis -
Guy Harris -
John Hawkinson -
John Wilcock -
merlyn＠stonehenge.com -
Paul Eggert -
Russ Allbery -
Tim Parenti -
Tom Lane