Turning Point?

Image courtesy thisfabtrek.com

There was a meeting involving everyone with a stake in our PACS problem. This included radiologists, IT folks, administrators, and representatives from the vendor. The purpose of the meeting was to outline where we were, where we are, and where we are going. I left the meeting feeling cautiously optimistic for the first time in quite a while.

I’ll not delve into specifics much, as they probably won’t help you with your PACS problems, nor will they help you help me with mine. The generalities however, should prove more valuable.

The bottom line is quite simple, really. Our problems stem in greatest part from lack of communication. This gap occurred primarily between IT and the vendor, but there was also a rather large chasm between the radiologist end-users and the other two entities. Let’s talk about that one first.

As an electrical engineer, I’m well-versed in the concept of the feedback loop, particularly the need in a circuit for negative feedback:

You’ve all heard the screeching “feedback” from a public-address system going out of control. Most amplifiers have a negative feedback loop, illustrated by the wire above using some of the output of the amplifier to “tone down” the input. So it is with life in general. If you think you are doing everything properly, and you receive no negative modulation, you will keep doing the same thing whether or not your actions are correct.

We rads had (or more honestly, exercised) only limited feedback options when it came to PACS. We the rads never really put our heads together to create a grievance list. Yes, one rad here or there would get upset with slow speeds, workstation crashes, slow scrolling, etc. Sometimes these could be fixed by the PACS administrators, sometimes only marginally improved. Someone would get a personal profile remade, although we never learned why this would help, someone would get his worklists redone. And so it went. In this way, little problems propagated into big problems.

I’ll take some level of personal responsibility. I was so, well, jaded may not be the word…disheartened? Complacent? Resigned? Well, I had more or less decided that nothing would change until we had a major upgrade, whenever that might be, and I found ways to tolerate the glitches. It took two of my former partners, now bosses, who recently started working nights, to say “ENOUGH!” The minor slow-downs become quite major when you’re trying to pump out dozens and dozens of studies through the wee hours of the morning. To be fair, there was further deterioration of the system during their initiation into the dark side (I mean dark hours), to the point that the workstations crashed, and ultimately there were a number of system outages, which certainly brought the whole situation to a head. The newly-minted night-stalkers began the campaign that has brought us to the brink of…a solution.

Communication is paramount as always, and we’ve made some great strides in that realm. First, we insisted on having a call-team from both IT and the vendor. We were briefly relegated to the “Help” desk; when they actually answered, they were little more than a rather slow answering service for IT. Second, we streamlined the process for rads to report problems directly. This was prompted by a response to a complaint implying that no one had ever mentioned the problem before. When Donald Trump’s cell phone number was published, instead of having a little hissy-fit as we saw with a certain Senator, Trump simply put a campaign ad on his line. This inspired us to create an email thread wherein any and all PACS complaints could be reported directly to the right people. And report we did. Initially, there were tens of emails per day; this has tapered off to only one or two. We have definitely made progress.

While we physicians can be quite problematic, the deeper institutional snafus lie with IT, the vendor, and their somewhat dysfunctional relationship. There will be yet another meeting to more precisely define just how that relationship is to be defined, and while I detest meetings, that is one that should have been held years ago. You see, there really wasn’t a single point of failure, but there were quite a few, shall we say, lost opportunities for improvement.

It turns out that one of our major outages was a network problem, caused by an update push that got out of hand. Another slow-down was the result of the EMR grabbing too much bandwidth. There was a bug in a NetApp image server that took us down. OK, we assume these things happen and can be fixed.

But it took a village full of angry radiologists to bring to light that yearly service on some of the servers might not do the trick, particularly when a couple of the critical servers running SunOS/Solaris weren’t touched at all. The latter had been running on an elderly version of the OS, and had a bug that was fixed umpteen versions ago. Update the OS, kill the bug. And here is where we had trouble. Putting it simply, everyone assumed the other entity was going to take care of stuff like this, if they assumed anything at all about it. And so nothing happened until the recent unpleasantness.

We found that hardware was sometimes purchased without consulting the vendor, and then retrofitted with the vendor’s help when it wasn’t quite right. Perhaps both parties could be a little more proactive here, so we can all ask for permission instead of forgiveness.

Computers are made by imperfect human beings, and are thus imperfect themselves. To assume otherwise is naive at best. And so in a mission-critical area such as PACS, one must be ready for the inevitable glitch. There has to be a downtime plan, and an out-and-out disaster recovery solution. Guess what? We have neither. To my knowledge, the downtime plan hasn’t been changed since I spoke at RANZCR in Perth in 2010: after four hours of outage, we start printing to film. Unfortunately, we no longer have any film printers. The next best thing, which we have been having to do, is to read directly from the modality’s monitor. It isn’t optimal, but it works, sort of. As for a full-fledged disaster, data is stored offsite as required. But it’s on tape and recovering might take a very long time. If we could muster the resources. Let’s hope it doesn’t happen.

PACS, as it turns out, is the only Tier One service that does NOT have a proper downtime solution. Why did we get left out? Money. It was hard to justify a complete, mirrored, automatic fail-over that would only be used a small fraction of the time. Unless you happen to be a trauma patient in the Emergency Department, where life-and-death decisions are put on hold while someone fiddles with the server. Then it seems perfectly justified.

In the end, we all serve one customer, and that is YOU, the patient. Everything we do in this business, every decision we make, every scrap of hardware and line of code we purchase and use is meant to promote your health and well-being. It was said by some that we radiologists were “paying the price” for the various challenges I’ve outlined. That’s true to some extent, but the real victims, at least potentially, are the patients, and that CANNOT be allowed to happen.

I’ve been blogging about PACS for almost 11 years, and my basic message hasn’t really changed much. PACS IS the Radiology Department, and the hospital cannot function without it. Making this all work, and work properly, is in huge part a matter of communication. You have seen what happens when the discourse fails or doesn’t happen at all. The downtime plan, or lack thereof, illustrates what happens when one of the groups involved in PACS, the rads, becomes disenfranchised with respect to the decision process. One of us could have very easily convinced the powers that be that we cannot tolerate a four-hour gap in service. We weren’t asked to do so; we didn’t even know the question had been posed. Now we do. And I am cautiously optimistic that this will improve, as will the rest of our experience.

I would be remiss if I didn’t take the opportunity to excoriate the majority of PACS and EMR vendors while I’m on this particular rant. You are still not making user-friendly software. We all know it. PACS is bad enough, but our EMR and its CPOE (Computerized Physician Order Entry) is so very poorly written and implemented as to drive a good number of physicians into early retirement. Seriously. This garbage is served up as caviar to, and purchased by, those who DON’T HAVE TO USE IT, and again the physicians are disenfranchised. This too will negatively impact patient care, and it CANNOT, well, it SHOULD NOT be allowed to happen. But it is.

Let’s have a meeting about THAT, shall we?

via Blogger http://ift.tt/1U6IQ2h August 26, 2015 at 01:24PM

And Another New Pres For TR

Looks like TeraRecon has a new CEO:

(from radiologybusiness.com)

TeraRecon, a global medical image management company, announced that Jeff Sorenson is the company’s new president.
Sorenson has been with TeraRecon since 2004. His most recent position with the company was senior vice president of sales and marketing.
“TeraRecon is unique because it offers a complete and truly vendor-neutral 3D advanced visualization suite which can be extended to serve the medical image viewing needs of the entire health enterprise,” Sorenson said in a statement. “I am excited to serve our valued customers and to drive innovation in this new capacity as president.”
Meanwhile, Venkatraman Lakshminarayan has stepped down as CEO and CFO. Lakshminarayan had been TeraRecon’s CFO since 2005 and its CEO since 2014.
TeraRecon also reported that a successful August has led to an increase in iNteract+ interoperability and enterprise image-enablement projects.

As a (hopefully) valued customer, I’m looking forward to working with Mr. Sorenson as well as Fred and the rest of the gang. Congrats!

via Blogger http://ift.tt/1KNNckq August 26, 2015 at 12:09PM

A Post For Larry

My friend and colleague Larry tells me he has become an avid reader of the blog, so this one’s for you!
For the record, I’m discussing our situation in this public manner in hopes of eliciting help from those who are smarter than I am, with more experience than I have, and who might actually have a clue about what is going on with our PACS and how to fix it. I believe this supersedes any institutional thought of silencing this discussion about a situation that is IMPAIRING PATIENT CARE. I have tremendous respect for privacy and security, and I will do nothing to compromise it. But as Mr. Spock might have put it, the needs of the many outweigh the needs of the few, and the need of the patients, and those of the physicians to care for their patients, outweigh everything. That is why the hospital exists, it is why the Radiology department exists, it is why IT and the vendors exist. Period. 
But back to Larry. He and other colleagues have been gathering interesting images from our PACS, some of which are reproduced below:

If we cannot show the patient, I guess we can all go have a three-martini lunch, since there’s nothing else for us to do. Ha ha. 

This would all be funny if PACS was the conduit for Netflix or something equally frivolous. As our episodes of outage and electronic impairment (no, we didn’t have a three-martini lunch!) demonstrate, there is a severe and immediate NEGATIVE impact on patient care, and we have to get a handle on it NOW. The good news is that we are really starting to see improvement. It’s sad that we had to have a total meltdown of personnel and machinery for that to happen, but that’s the way it is.

From the outside, and from rather minimal information often fed to me third-hand, I am not really seeing a concentrated, orchestrated effort to get to the bottom of this, but rather a lot of shot-gunning that has had little impact overall. Yes, we’ve seen some improvements, but clearly, we are far from fixed. (There are those out there who would like to get ME fixed, but we won’t talk about that.)

I’m seeing three distinct but likely related problems.

1. General slowness of the system, which varies by site. Transmission tests I have managed to see show this variation. This must have something to do with the network. It was suggested early on to see if the client behaves properly on a station connected directly to the server. Has this been done? Connecting via the Internet outside of the enterprise yields good service. That’s a big clue!

2. Client software and hardware crashes. Is this a software problem? If so, is it a problem with the local client, the server, both? Could it relate to the network? Is it a Windows 7 problem? The clues are all there.

3. Slow searches. This seems to relate to corrupt profiles. I discussed this with PACS people, and I’m told that the user profile is simply an XML file, just a list of parameters that tells the client how the particular user wants things arranged, etc. Supposedly, logging on to two or more workstations simultaneously corrupts the profile. So, rather than spend hours creating a new but still fragile profile, how about someone figure out why IMPAX is even capable of corrupting the old one? We simply don’t have time to rebuild profiles during a hectic workday, and I’m not going to waste everyone’s time doing so when we don’t even know why it helps or whether the change will stick. And if logging on twice can create a patient-care-impacting situation, perhaps a hot-fix needs to be implemented to prevent this practice, and the FDA notified of such.

There is an interesting, illustrative little side show to tell you about. I heard third-hand that some were questioning if my little AutoHotKey script for keeping the RIS window active might be the source of some or all of our problems. In fact, my AutoHotKey macro apparently became quite the topic of discussion. IT security got very upset at its very presence, calling it all sorts of nasty names, and apparently a few were directed at me personally. The punch-line, though, is that the use of the macro was APPROVED by that department in 2010, with the only reservation being that someone could conceivably use AutoHotKeys to create a script which would bypass the need for a password. Given that I’m the only one who knows how to do it, and I’m not about to create something like that, we should be OK. Fortunately, most of my colleagues have made it past the days of inscribing the password on the bezel of the monitor with a wax crayon.

Being a cantankerous, semi-retired, cranky old lout, I will simply note that this episode demonstrates how things don’t get accomplished.  Accusations fly, fingers are pointed, and some of the real issues don’t get the attention they deserve. It would have taken literally 2 minutes to go to a station where this complied script is running, and activate Task-Manager to see exactly how many resources are being utilized. But I saved everyone the trouble. I’ve run task-manager (on those machines which don’t have it and right-click disabled) and found minimal impact. (In addition, it shows that IMPAX is in a non-responsive state during some of those hour-glass generating waits. There’s another clue for you. Anyone notice this before?)

I had to use Microsoft’s Process Explorer, as it was felt too dangerous to allow radiologists access to Task Manager on some machines. Here’s the result for my little script NewSigner3.exe:

The compiled script NewSigner3.exe utilizes 0.04% of CPU time on the IMPAX client workstation

Yup. Wasting 0.04% of CPU time could have devastating effects.

Once I made these results available, I was told: “Oh, no one was thinking that the script itself was the problem; but it keeps the  RIS window open, and that’s what’s wrong!” But like a bunch of Greek philosophers of old, sitting around the Parthenon contemplating how many teeth a horse should have, rather than actually finding a horse and counting its teeth, no one checked this. So I did:

Citrix window executables for RIS window use 0.15% of CPU time

Oh, but it turns out that wasn’t what was being considered either. No, it seems that the fact that the RIS window was being clicked automatically every 20 minutes or so to keep it active and to remind us to sign our reports “creates downstream problems in that it allows for people to be logged in on multiple stations.” (See discussion about profiles, multiple logons, and XML files above.) Someone please help me understand how keeping the RIS window open has any effect on the PACS. It seems pretty clear that some folks in the loop haven’t tried to understand just what this little script accomplishes, and maybe somehow think it tweaks the PACS client. But it doesn’t, as would have been discovered with 30 seconds of effort. Or a 30-second phone call to me.

You would think I had attempted to unleash the worst virus in existence upon the network.

Anyway. I did say above that some progress had been made, which is true, and some of the problems have been identified. Eventually, some of those will be fixed, too. One server had not been updated in a long time and it was delaying arrival of prior studies. No, I don’t know why it wasn’t updated. Very large individual disks had been used for storage which might have been beyond the recommended size. I don’t know how that happened either.

You might recall me saying something about image skipping or rotten scrolling performance if the annotations were on. Apparently, this relates to the five-year-old version of software we are using, and should resolve once we upgrade.

Progress is progress, however slowly it might, well, progress. Perhaps we can Git ‘R Fixed sometime soon. Larry? 

via Blogger http://ift.tt/1DUtF58 August 15, 2015 at 11:33PM

MGH Still Uses Old AMICAS!

Look familiar? This is a screen-shot from AMICAS PACS, circa 2005. That would be about 10 years ago.

But I just had a flash of deja vu…While watching tonight’s episode of Save My Life on ABC, filmed at Mass General, I very briefly spotted a PACS client of this vintage…no, I don’t have a screen shot, but I’ll work on it.

I knew AMICAS PACS had been used as an overlay to IMPAX at MGH, put in place back when IMPAX couldn’t handle multi-slice scans.

Apparently, a 10-year-old AMICAS system is good enough for Mass General. Maybe that’s because after 10 years, it still works. Unlike, well, need I say it?

Here’s the promised screen shot, although it’s from last week:

via Blogger http://ift.tt/1Tck5RH August 09, 2015 at 10:24PM

And We’re STILL Not There Yet

Despite rosy, glowingly positive reports from IT and Agfa, implying that all is well and the rads are once again fat and happy, we continue to have difficulties.

This morning, my poor, suffering bosses (formerly partners) who were on Saturday call report this amusing scenario:

Now at hospital for over one hour. Still unable to read studies. Worklist now opens in the upper left screen about the size of a postage stamp. I am able to expand it to the size of the screen. Worklist however does NOT open studies to be viewed; in another words, we see no images, just a worklist. And I have done the usual reboot four times on four different computers.

You’ll be glad to know that this glitch, which affected all three guys on call, was finally fixed by bouncing all three production servers. How long that will keep things going, I haven’t a clue. 

To be fair, much has improved, with speed improvements at most sites. I find I no longer have to turn off annotations to scroll through a CT series, which is huge. But we are clearly not out of the electronic woods as yet.
I am told that the improvements have come because of various maintenance procedures, cleaning out this, redoing that, etc., etc.  You might ask, why weren’t these things done before the radiologists started acting out? That, dear reader, is a VERY good question…

via Blogger http://ift.tt/1IzGBK6 August 08, 2015 at 03:34PM

IBM BUYS MERGE!!!Watson, Come Here! I Need You!”

NEWS FLASH!!!

I just received the notice (as did several thousand others) from Dustin Dearborn, Merge CEO, that IBM has purchased Merge, and will incorporate it into the IBM Watson Health Unit. Here is the press release:

Chicago, IL, 06 Aug 2015

Armonk, NY and CHICAGO — [August 6, 2015]: IBM (NYSE: IBM) today announced that Watson will gain the ability to “see” by bringing together Watson’s advanced image analytics and cognitive capabilities with data and images obtained from Merge Healthcare Incorporated’s (NASDAQ: MRGE) medical imaging management platform. IBM plans to acquire Merge, a leading provider of medical image handling and processing, interoperability and clinical systems designed to advance healthcare quality and efficiency, in an effort to unlock the value of medical images to help physicians make better patient care decisions.

Merge’s technology platforms are used at more than 7,500 U.S. healthcare sites, as well as most of the world’s leading clinical research institutes and pharmaceutical firms to manage a growing body of medical images. The vision is that these organizations could use the Watson Health Cloud to surface new insights from a consolidated, patient-centric view of current and historical images, electronic health records, data from wearable devices and other related medical data, in a HIPAA-enabled environment.

Under terms of the transaction, Merge shareholders would receive $7.13 per share in cash, for a total transaction value of $1 billion. The closing of the transaction is subject to regulatory review, Merge shareholder approval, and other customary closing conditions, and is anticipated to occur later this year. It is IBM’s third major health-related acquisition – and the largest – since launching its Watson Health unit in April, following Phytel (population health) and Explorys (cloud based healthcare intelligence).

“As a proven leader in delivering healthcare solutions for over 20 years, Merge is a tremendous addition to the Watson Health platform. Healthcare will be one of IBM’s biggest growth areas over the next 10 years, which is why we are making a major investment to drive industry transformation and to facilitate a higher quality of care,” said John Kelly, senior vice president, IBM Research and Solutions Portfolio. “Watson’s powerful cognitive and analytic capabilities, coupled with those from Merge and our other major strategic acquisitions, position IBM to partner with healthcare providers, research institutions, biomedical companies, insurers and other organizations committed to changing the very nature of health and healthcare in the 21st century. Giving Watson ‘eyes’ on medical images unlocks entirely new possibilities for the industry.”

Teaching Watson to “See” Medical Images
The planned acquisition bolsters IBM’s strategy to add rich image analytics with deep learning to the Watson Health platform – in effect, advancing Watson beyond natural language and giving it the ability to “see.” Medical images are by far the largest and fastest-growing data source in the healthcare industry and perhaps the world – IBM researchers estimate that they account for at least 90% of all medical data today – but they also present challenges that need to be addressed:

The volume of medical images can be overwhelming to even the most sophisticated specialists – radiologists in some hospital emergency rooms are presented with as many as 100,000 images a day.

Tools to help clinicians extract insights from medical images remain very limited, requiring most analysis to be done manually.

At a time when the most powerful insights come at the intersection of diverse data sets (medical records, lab tests, genomics, etc.), medical images remain largely disconnected from mainstream health information.

IBM plans to leverage the Watson Health Cloud to analyze and cross-reference medical images against a deep trove of lab results, electronic health records, genomic tests, clinical studies and other health-related data sources, already representing 315 billion data points and 90 million unique records. Merge’s clients could compare new medical images with a patient’s image history as well as populations of similar patients to detect changes and anomalies. Insights generated by Watson could then help healthcare providers in fields including radiology, cardiology, orthopedics and ophthalmology to pursue more personalized approaches to diagnosis, treatment and monitoring of patients.

Cutting-edge image analytics projects underway in IBM Research’s global labs suggest additional areas where progress can be made. They include teaching Watson to filter clinical and diagnostic imaging information to help clinicians identify anomalies and form recommendations, which could help reduce physician viewing loads and increase physician effectiveness.

“As Watson evolves, we are tackling more complex and meaningful problems by constantly evaluating bigger and more challenging data sets,” Kelly said. “Medical images are some of the most complicated data sets imaginable, and there is perhaps no more important area in which researchers can apply machine learning and cognitive computing. That’s the real promise of cognitive computing and its artificial intelligence components – helping to make us healthier and to improve the quality of our lives.”

Watson Health and Merge Capabilities Will Benefit Researchers, Clinicians and Individuals

IBM’s Watson Health unit plans to bring together Merge’s product and solution offerings with existing expertise in cognitive computing, population health, and cloud-based healthcare intelligence offerings to:

Offer researchers insights to aid clinical trial design, monitoring and evaluation;

Help clinicians to efficiently identify options for the diagnosis, treatment and monitoring a broad array of health conditions such as cancer, stroke and heart disease;
Enable providers and payers to integrate and optimize patient engagement in alignment with meaningful use and value-based care guidelines;

Support researchers and healthcare professionals as they advance the emerging discipline of population health, which aims to optimize an individual’s care by identifying trends in large numbers of people with similar health status.

“Merge is widely recognized for delivering market leading imaging workflow and electronic data capture solutions,” said Justin Dearborn, chief executive officer, Merge. “Today’s announcement is an exciting step forward for our employees and clients. Becoming a part of IBM will allow us to expand our global scale and deliver added value and insight to our clients through Watson’s advanced analytic and cognitive computing capabilities.”

“Combining Merge’s leading medical imaging solutions with the world-class machine learning and analytics capabilities of IBM’s Watson Health is the future of healthcare technology,” said Michael W. Ferro, Jr., Merge’s chairman. “Merge’s leading technology and proven expertise represent a unique combination of assets that will deliver unparalleled value to Watson Health clients. Together, we will unlock unprecedented new opportunities to improve patient diagnostics and deliver enhanced care.”

Interesting. Justin goes on to add a personal note:

We are very pleased to share some exciting news with you. Earlier today we announced that we have entered into a definitive agreement under which IBM will acquire Merge Healthcare. Through this transaction, Merge will become part of the IBM Watson Health unit. The plan is for the Merge management team to remain in place following closing. You may read the full press release here, but allow me to take this opportunity to tell you about the news directly and what it means for you.

Combining our strengths as a leader in healthcare imaging with IBM’s powerful Watson Health Cloud cognitive and analytic capabilities will enable us to expand the reach and effectiveness of our solutions. The vision is that healthcare organizations could use the Watson Health Cloud to surface new insights from a consolidated, patient-centric view of current and historical images, electronic health records, data from wearable devices and other related medical data, in a HIPAA-enabled environment.

Additionally, we expect to benefit from the ample resources IBM can offer to support the continued growth and development of our business. In short, as a result of this transaction, Merge products will only become better and you will benefit from continued innovation to support your medical imaging needs.

I want to assure you that you can continue to expect the same level of service that you have come to rely on from Merge. Importantly, IBM will continue to support the Merge platform, and will continue to honor all existing contracts and agreements.

While we are very excited about today’s news, this announcement is just the first step in the process. The transaction is subject to regulatory review and shareholder approval. Until the transaction closes, which we expect will be later this year, we will remain an independent company, and it is business as usual. We remain focused, as we always have, on execution and results, and will continue to deliver the innovation and support that you have come to expect from us.

We’ll stay in touch as future developments take place, and we look forward to continuing to serve you.

Please do not hesitate to contact your Merge account manager with any questions.

What does this mean for us end users? Probably not much difference in service for the immediate future. But this does boost Merge significantly, now putting it up there with the other “big companies”. Moreover, IBM does not have the previous stake that those others have with existing PACS software, etc., that has to be taken into consideration when moving forward.

This will prove interesting. I do recall Mike Ferro declaring that Merge would become the premier HIT company, and would eventually be worth $1 Billion. Looks like they made it.

Of course, now we have to worry that Watson will put us out of our jobs, but I’m not expecting that to happen for quite a while.

I wonder when Apple will take the plunge into the HIT pool. Between you and me, I was hoping Apple would be the suitor…

via Blogger http://ift.tt/1M8fr1i August 06, 2015 at 10:41AM