Fairhaven, The River

Medical Market "failure"

I had to change my regular medications last week.  I delayed the conversation with the nurse while I did some checking of prices.  From both Costco and Walmart I could get a 3-month supply for $10, cash, no insurance.  The nurse commented "we don't price compare".  My response was "I do."  (It's actually quite unreasonable to expect them to help price compare.  Insurance plan complexities are different for every patient and finding a cost is hard.)  Since my co-pay is $25, the $10 is a good price.

I've been through this before, but just for my amusement I checked again:

• At both local chain pharmacies they will not quote a price.  Prices are secret.  All they will disclose is co-pay and only 1 month supply salable if insured.  They are not willing to sell on other terms, even for cash, because they know I have insurance.
• I didn't waste time with the PBM that used to nag me.  Last time they were not willing to quote a price.  All they would disclose was the copay, for a 3 month supply, delivered by mail sometime in a two week window.

The only way to find a price is to buy the drug and look at the receipt details.

I find the nonsense about healthcare demonstrating "market failure" to be highly annoying.  You can't call it a market when prices are kept secret from potential customers.  Price and quality comparison are fundamental to anything called a market.

The increasing penetration of Costco and Walmart may partially reflect customer intolerance for the mandated inconvenient delivery and secret pricing.  The cost conscious public clearly want a real marketplace with public prices.

Actual failure experience (re ATNA-Syslog)

This has to be somewhat vague for trade secret reasons, but during the mid-90's Polaroid gathered extensive data on network problems as part of a total system reliability tracking effort.  The systems were medical printing systems, and data was gathered for video capture units, print servers, and printers.  The networks were hospital networks, all with wired ethernet connections.  There were several hundred devices involved and measurements taken at a couple hundred hospitals over an extended period at these hospitals.  Some of the units were stationary, and some of the units were mobile.

The network traffic is noticably different from ATNA-Syslog, so these results would need some adjustment for that purpose.  The network traffic for printing was DICOM, "lpr", and "ftp" traffic.  The observations were made over a period of several years.  In excess of 10 million transactions were recorded.  Transactions were almost entirely print transactions, with a very few print status queries.  The observations were:

• Network problems were entirely insignificant.
  
  • Stationary systems reported no network problems.  TCP/IP handled whatever happened just fine.  (I should note that a down server or printer was not considered a network problem if this was detected before the print data was sent.  The sending systems would queue their prints and wait for the server  or printer to be restored to service. Down servers or printers were considered a different kind of reliability problem.)
  • Mobile systems detected network problems at about one per 250,000 transactions.  These were all the kind of failure that I described earlier.  The application acks did help identify problems.  At a problem rate of 4 ppm we decided it was reasonable to deal with these problems by just re-printing whenever a possible problem was observed.  Four extra prints per million is an insignificant problem and not worth extensive engineering work.

ATNA Syslog is a different network use pattern, and the motivation for directed attacks is different.   Hospital network characteristics may have changed since the 1990's, and this data is not a measurement of cross enterprise data traffic.  This data is relevant in characterizing the internal characteristics of the hospital environment, and can be taken into consideration when doing reliabiliity design, FMEA, and similar analysis for ATNA product design and installation design.

Explanation of TCP/TLS vulnerability

This explains the vulnerability mentioned previously. The vulnerability is an uncertainty by a sending application regarding whether messages have been received or not.  This is not directly a security vulnerability.  It's inherent in the TCP/TLS design, and normally is only an issue with mobile devices.

The real world sequence of events is this:

1. Application write/sends X bytes to other side of TCP connection.
2. Network is disconnected.
3. Application gets a "Reset" or similar error from read/write/close operation.

Details of the system I/O functions don't matter.  In all these cases, the application is left uncertain whether the X bytes were received or not.  The "Reset" and related errors are an indication that something went wrong.  Previously acknowledged messages might have still been in buffers, in transit, etc.  Some of them might have been lost.  TCP itself cannot know, because it is possible that the network disconnect prevented a TCP acknowledgement from arriving.

The application mest decide whether it cares.  Possible situations range from:

• Everything made it just fine and was acknowledged.  This is quite common in situations where there was a long delay between the last send and a close().  Odds are that the network disconnect is indirectly related to the close and occurred long after the TCP transmissions were completed.
• Everything in the TCP window could have been lost.  This can happen when the network is lost during a large transmission.  TCP has all this data buffered and is trying to get it through.  It's using timeouts, retries, etc.  If the network failure is just a transient, it will succeed.  But, in this example, a "reset" indicates that it cannot succeed and has failed.  There might be a complete TCP window's worth of buffers that were previously accepted but did not get transmitted.

There are also a variety of intermediate states possible that the application could determine. 

This is where application ack is sometimes proposed as a solution.  It's initially appealing, but it's still imperfect.  The ack could be the only part of the transmission that was lost.  Application acks do make sense in a number of application contexts.  They don't automatically solve the underlying problem.  More protocol design will be needed, with idempotency or transaction rollback dealing with parts of the problem.

This kind of timing occurs naturally with mobile devices.  The form that I see most often is a real world situation where the user finished a task with a mobile device, disconnects it, and moves on the next task.  Sometimes the mobile device is still transmitting results over the network when this happens.  The disconnect can be a cable disconnect or simply moving the device out of WiFi range.

Asking users to wait does not work.  They may understand the issue and try, but in practice it's like asking people not to stub their toe or spill their coffee.  They understand and try, but it still happens.  It's more realistic to design a device that survives these occasional problems.

 

Should Syslog use application acks?

This started as an ATNA-Syslog question, that I will consider in the context of some DICOM history. 

Should Syslog have an appliction ack to deal with some known vulnerabilities?  I call them vulnerabilities because they are extremely unlikely except when combined with intentional external actions.  These vulnerabilities become apparent with mobile equipment and with skilled attackers.

DICOM has used application ack since it's introduction.  But, this was for application reasons, not to deal with vulnerabilities in the underlying TCP and TLS services.

• In C-MOVE there is a C-MOVE-REQ that says "please accept object X".  The corresponding C-MOVE-RSP says either "yes, got it" or "no, this is why not."  Most often the no is because the responder is out of storage space, but it can also be due other problems.

• In C-FIND the pair is more complex.  C-FIND-REQ says "please provide information on objects that match these criteria".  The C-FIND-RSP normally conveys the information requested.  It can also deal with "and there is more to come later" or "this is everything".  The "no, this is why not" deals with invalid requests and other problems.

The use of application ack to deal with network problems as part of failure management was a side effect.  These are just another kind of failure to be reported.  The state machine for the application layer has to deal with network failures as part of completeness, not as a reason for application ack.

DICOM also tolerates a certain level of indeterminacy.  The designers basically said "close enough", much like the designers of CRC and FEC codes.  These cover most of the possible errors, and accept that a few will sneak through and be dealt with elsewhere.

The "elsewhere" in DICOM leads back towards the isues with Syslog and ATNA.  DICOM makes transactions idempotent to the maximum extent possible.

The C-MOV transaction is idempotent.  These is no end state difference between "C-MOV object X" once and "C-MOV object X" one hundred times.  The only difference is the time it takes.  So when in doubt or uncertain, DICOM applications just do it again.  The second time probably reaches a determined state.  Similarly the C-FIND is idempotent.  A look at operational DICOM logs shows well over 99.9% of the transactions performed are idempotent.

There are some necessary exceptions, like "print".  Sending "print" once is different from sending "print" a hundred times.  You get a lot more copies printed.  DICOM tries to take the non-idempotent applications and split them into an idempotent part and non-idempotent part.  DICOM puts as much of the application into the idempotent part as it can.  In the case of "print", everything except the N-ACTION-PRINT is idempotent.  This minimizes the window of vulnerability.  DICOM then took the attitude, "close enough, maybe there is the occasional duplicate print when something goes wrong.  We'll accept that.  It won't happen often enough to be a problem."
    
Syslog and ATNA have the issue that:

• There is no application level ack, and

• There is a delivery uncertainty in the face of some kinds of errors that occur with mobile devices and skilled attackers.

Can idempotency deal with this?  It can if syslog messages are designed properly.  This would allow a gradual transition to reliability without requiring changes to the underlying syslog protocols.  The application change would be to send extras when uncertain about delivery.  This is probably also a much smaller network impact also.  There is extra traffic only in those error situations, not during normal traffic.

This takes messages that are universally unique over all time and sources, and that are idempotent.  The idempotency allows sending duplicates.  Duplicates can be recognized and discarded by recipients.  (This also simplifies some of the multiple database and dispersed database issues for log processing.)

Unique IDs deal with uniqueness.  DICOM uses unique IDs for all sorts of things.  The hardest problem with unique IDs is persuading all the hot shot programmers that they really should read the recommendations on best practices for creating unique IDs. The home grown unique ID algorithms all seem to fall into one or another of the well known traps that result in non-unique ID generation.

If the lead-in to every syslog message body includes that unique ID for that message, you may be done.  The rest is ensuring that the generating application doesn't generate multiple messages for the same event.  That's bad design in any case.  You also want to avoid non-idempotent content.  This should be easy for ATNA and syslog.  The message is describing an event.  It shouldn't be that hard to make these messages idempotent.

The one idempotency trap that I've seen with log messages is the use of incremental messages.  These need sequence integrity.  The idempotence of meaning is lost if the messages are processed in the wrong order.  Time tags and other tools can be used to preserve sequence integrity despite repeats and out of order arrival.  But it helps a lot if the syslog messages are designed to be a self-contained complete description of the event of interest.

How well did RFC-3881 and DICOM do when defining audit messages?  So-so.  The event identification mandates identifying a source, date-time, etc.  It does not require that these be message unique. The messages are fully self-contained.  What they need is a unique ID.

I also checked the MITRE CEE effort.  They don't mandate idempotent messages either.

So, should I propose a change to add an optional unique ID?  Something to think about.

The good, bad, and the ugly

Summary: Examples of good climate investment and a bad investment.  Unfortunately, the environmentalist hucksters are pushing the bad investments constantly.  The crony capitalists are doing worse.

The good

This summer the Tower 55 project got funding.  It's a $90 million construction job fixing a major railroad bottleneck, eliminating mutual traffic interference with local highways, and improving traffic flow for rail and roads.  It will be $40 million of government funds (for the highway work) and $50 million in private funds (for the rail, environmental, legal, admin, etc.).  Esitmated carbon reduction is 180 million lbs, giving a cost of $1000/ton for carbon reductions.

There are many projects like this, both large and small: CREATE, Heartland Corridor, TIGER,...

The bad

I reviewed the financial proposals for a local solar power park.  This will be a $12 million investment, paid for primarily by tax credits on the contruction (3 million) and green energy credits for electricity production (9 million).  The unadjusted revenue from power generation basically covers the operation and maintenance costs.  It will eliminate 3 million lbs of carbon, giving a cost of $8000/ton for carbon reductions.

There are far too many projects like this being pushed.  In terms of both economic impact and climate impact, this money should go the projects like Tower 55.  But this one will probably be funded.  The environmentalist hucksters want publicity, shiny toys, etc.  Actual climate impact comparisons are not part of their creed.

The ugly

The extensive corruption and mismanagement of government agencies and funds is surfacing with new stories weekly from the US, India, and elsewhere.  Crony capitalism is well entrenched and has found many ways to extract money from environmental concerns.

Part of dealing with this is to always ask "what will the cost per ton of carbon reductions be?"  Cost must include construction costs, subsidies, tax credits, incentives, etc.  Look at the total social cost.  Then compare projects.  For organizations with a track record, also compare their original estimates with the actual results upon project completion.  It's a modest step that is easy to explain and easy to justify.  It's enough to make a big shift away from bad projects to good projects.

Modern Human, Neanderthal, and Denisovan cross-breeding

I went to talk by David Reich, Harvard Med School, last Thursday on genetic evidence measuring interbreeding among modern humans, Neanderthals, and Denisovans.  His talk also included significant background information.

First, he covered the techniques and difficulty of getting DNA samples from old bones.  Typically, the body has been completely contaminated with bacteria, fungi, plants, and insects.  Even after all their efforts at selecting protected interior portions of protected bones, less than 3% of the recovered DNA is mammalian.  This also explains the early emphasis on mitochondrial DNA.  There are thousands of mitochondria per copy of cell DNA.  This makes it much easier to get an acceptable sample of mitochondrial DNA.

Then he explained significant differences between Neanderthals and Denisovans.  For those who missed the recent news, the Denisovans are another form of human.  Remains were found in the Denisovan caves in southern Siberia in 2010.  Based on DNA clustering, the Denisovans, Neanderthals, and Modern Humans are each distinct. Within these three the various DNA samples cluster tightly and overlap in variations.  There is a substantial and statistically significant separation between the three. 

There are multiple samples of DNA for all these human variations.  These are sufficient to obtain reasonably complete genomes, despite the limited samples.

Neanderthal range is Europe and to an unknown extent eastward into Central and Southern Asia.  So far, the only Denisovan source is the caves in southern Siberia.

He then explained his terminology.  When comparing different animals, like man vs chimpanzees, they look at gene overlap.  When measuring interbreeding, they look at base-pair matching.  The basic measure is to find single base-pair changes in a gene, then determine how often that change is found in Modern vs Neandertal vs Denisovan.  This is used to derive percent sources.

Results:

• Africans have the largest genetic diversity of modern humans and no measurable contribution of DNA from Neanderthal or Denisovan.
• Non-Africans have about 2.5% Neanderthal DNA.  This level is relatively uniform across all non-Africans.
• There is a cluster of Denisovan contribution in Papua, New Guinea, Australia and neighboring islands.  This level is not uniform.  Outside this region there is no Denisovan contribution.  Within these clusters there is one group at about 2.5% Denisovan, plus two other groups with different substitutions but both at about 5% Denisovan.

Based on change rates for base-pair substitutions, these interbreedings took place about 50K ya for both Neanderthals and Denisovans.

 Speculations

The simplest explanation for the Neanderthal mix is the "out of Africa" theory, with the interbreeding taking place in the Levant.  There is paleontological evidence for both Neanderthal and Modern humans living in the same hills at the same time, about 35-50K ya.  This is consistent with migration from Africa.  The lack of any Neanderthal contribution to Africans makes other mixing unlikely.

More speculatively, it also supports the hypothesis that the southern route (arabia, india, southeast asia, to australia) was first for modern human expansion.  This would explain multiple interbreeding events with Denisovans that affect only southeast asia and australia.  There was a second wave of modern humans from china later.  This shows up in the genetics, and shows that this was a separate event from the earlier wave.

Unrelated Comment

There is increasing evidence that interbreeding events are the norm, not unusual.  Genetics show events where Europeans substantially contributed to India.  There is presently a major interbreeding event between Africans and Amerindians in south america.  This is a change from previous theories that interbreeding events were rare.

Audience  Questions:

What percent of interbreeding couples does this indicate?  At most 2%, probably less than 1%, of children would be from interbreeding.  The genetic match is close enough that this must have been a social effect, not a viability effect.  There should have been no biological problems with interbreeding.  If the percent was above 2% the overlap the genetic overlap would be higher.

What about the "hobbits" of Indonesia?  No usable DNA could be recovered.  Hot climates degrade DNA too fast.  There are no usable DNA samples from any prehistoric bones of humans in hot climates.

Standards and drug costs

In honor of Dr. Meier's recent death, a comment on how standards choices affect drug costs.  Dr. Mieir was the major advocate for the use of randomized trials.  Standards can reduce the costs of these trials when used properly.  Some of the results are counterintuitive, but that's common when using statistics.

The major cost of developing a new drug is the Stage III trials.  These involve many patients as trial subjects, plus all the associated costs.  These costs are huge.  The two major approaches used to reduce these costs are:

1. better early screening to reject candidates.  This reduces the number of Stage III failures.
2. reduce the cost of individual trials.  Standards can help directly with this.

The primary cost driver for the Stage III trial is the number of subjects needed.  This number results from using randomized trials to remove the effects of variance.  This variance is composed of:

1. Subject variance, the inherent variability of the disease and patients.  There is little that can or should be done to reduce this.
2. Observer variance, the variability of the observation methods, calibration, recording accuracy, etc.  In an ideal world this would be zero, and standards tackle this problem.

The techniques covered by standards include:

1. recording all the relevant details about observation methods.  DICOM enables capturing the important observational parameters.  It defines the terminology, measurement units, etc.  As observational equipment evolves and is better understood, DICOM extends these definitions.
2. recording calibration information and making it available for subsequent use.  This was a low hanging fruit for DICOM, requiring only a minor clarification and extension to the patient identification module to incorporate calibration phantom information.  See CP-613 and CP-764 (http://www.dclunie.com/dicom-status/status.html#CorrectionProposalsByNumber) This enables subsequent trial specific calibrations for the patient results.
3. recording and providing measurement methods.  There is significant variance that results from differences in setup, patient preparation, etc.  This often involves machine specific information for individual model types.  DICOM is working on methods to capture this and communicate it.  See Supplement 121 (http://www.dclunie.com/dicom-status/status.html) The hope is that this will enable all of the sites involved with a particular clinical trial to use the same measurement methods, and reduce variance this way.
4. definition and distribution of standard codes and terminology.  Clear definitions of measurement meaning reduce variation between observers when reporting.  IHE has contributed a Shared Value Set (SVS) facility so that it is easy to distribute these terms and their definitions to all the staff and other participants.  Other standards efforts like SNOMED and RADLEX try to define universally useful clinical terminologies.

I emphasize variance reduction because that is where there are potentially huge savings.  The number of patients needed in a trial is proportional to the square of the variance.  If we can cut variance in half, it would cut the cost of Stage III by as much as 75%.  More realistic goal is a 10% reduction in variance that generates a 20% reduction in Stage III costs.  That would be a multi-billion dollar per year savings.  There is much naive discussion about using the network to find more subjects.  This does help, but in almost all cases the real cost driver is the large number of subjects participating in the trial.

Now for the statistical subtlety.  One potentially large source of variance is changing methods or terminology in the middle of a study.  So healthcare faces an ethical dilema.  Changing methods and terminology is needed to make improvements in care, but interferes with ongoing clinical trials.  Making a change should involve the patient and the clinical trials organization as well as the healthcare provider.  The clinical trials organization needs to inform the decision makers about the impact that this change will have.  How much does it affect the trial?  The patient and provider need to assess what the effect of the change will be on the patient's prognosis and goals.

An example of a change is a new patient prep procedure that reduces CT prep time by 5 minutes.  For any patient not involved in a trial involving CT, the answer is obvious.  They should get the change.

But for a patient in a trial using CT data you need to consider whether this change will affect the trial.  If this will increase trial variance by 1%, it may increase the trial costs by 2%, reduce trial quality, or delay trial results.  The patient may well prefer to take the extra 5 minutes rather than interfere with the trial.

To make this work you need:

• a system in place to keep track of which patients are in trials, and what procedures are affected by these trials.
• a system to provide trial specific information for everyone conducting those procedures
• a system of people who are prepared for the operational variety that this creates.  It's like conservation of mass.  The variations have been removed from the clinical trial and put into the day to day operation of the healthcare provider.

The IHE SVS profile helps by enabling the use of date, version, and trial tags to identify value sets that support particular trials.  DICOM Supplement 121 helps by easing distribution and implementation of consistent imaging protocols.  These are a small part of the overall effort, but the bulk of the systems described above are internal to each healthcare provider.

A note on energy efficiency

I haven't ranted much lately about energy efficiency for the simple reason that it's becoming widely adopted and its saving a lot of energy.  The eco-activists are still unhappy.  I've even read some VC complaining bitterly that people are spending their money on energy efficiency rather than the VC's new alternative energy thing.

Press coverage is rare because the buyers are doing this to save money, so why waste money on PR efforts?  Sometimes you do get energy control companies that think the PR is worthwhile, especially for famous buildings.  But most of the saving is from small renovations and replacement of facilities and equipment.  Those don't make the news.

So it was nice to see the net effect is large enough that it now sometimes makes the news.

For a more sane view of world economy

I recommend the lecture "Grasshoppers, Ants, and Locusts" to those who want to understand more about what is happening financially.  The recent US kabuki dances and the obsession with beltway trivium has become more annoying to me than usual.

Make sure to download the slides before starting to listen.  There is a lot of good material there too.

Internet trust lecture

Summary: A lecture by Miriam Meckel reminded me of the importance of reciprocity in healthcare relationships.

The lecture by Miriam Meckel presented results of a study on building trust on internet.  They picked ten realistic things that are part of establishing trust.  Then they examined surveys over 1 year for a variety of B2B and consumer organizations.  These were studied for primacy component analysis to see what drives trust the most.

Biggest factor is "reciprocity".  This is the agreement by both sides that their expected actions make sense and are appropriate for the nature of the relationship.  A very simple example is that customers expect to pay for goods delivered.  This reciprocity is a factor independent of contract or other terms.  It applies to later discoveries, undisclosed activity, etc. 

Reciprocity was 1/3 of the determination of trust.

Also noted was the penalty for violating reciprocity expectations.  A slimey crook who presents as a crook is not trusted to begin with.  A trusted relationship that then has a reciprocity failure is treated as a major betrayal.  The betrayer becomes much worse than an untrusted crook.  They are an enemy to society.

Three factors are responsible for the next 1/3 of establishing trust:

• Technical reliability.  This means all aspects of the relationship work smoothly and without problems.  It's much more than just web site stabiltiy.
• Customer control.  The more that the customer determines the relationship and activities, the greater the trust.
• 3rd party recommendations.

So, with four reasonably well defined areas you get 2/3 of the trust establishment, and reciprocity is the dominating factor. 

This has some relevance to healthcare and its security.  The trust relationship is important to most aspects of healthcare. 

One result is that the risk assessment priorities for security analysis need reconsideration.  It's true that inappropriate disclosure is a risk. I would consider that a technical reliability problem. But, reciprocity, patient control, and 3rd party recommendations are also assets to be protected. 

This also points to a flaw in an argument that I hear often regarding data losses.  The many disclosures due to stolen laptops are discounted because the data is rarely actually disclosed.  In practice the laptop is wiped, because the thief stole it for resale.  Wiped laptops are easier to sell.  That's an argument dealing with the 10% factor of technical reliability failure.  This argument ignores the reciprocity failure, and leaves the vendor open to enemy of society treatment.  That's a big loss of an important asset.

The solution to the reciprocity failure is some mix of

• have the customer accept that the loss is reasonable.  This is the rather unpopular "there is no privacy any more" argument.
• make sure the customer knows that you cannot be trusted with their data.  This downgrades you from enemy of society to merely not trustworthy.
• don't lose data on laptops

We need to add the assets of reciprocity, reliability, customer control, and 3rd party assessment to the risk analysis mix.  It's more than loss of data and data disclosure.

I've seen two other related problems in healthcare.

1.  "Consent"s, which are important but generally bungled.  Reciprocity does not mean that you told me something would happen.  Reciprocity means that when I later learn about it I agree that it was appropriate.  Consent is only part of reciprocity to the extent that it ensures that the customer understands the other side and knows who not to trust. 
2. The "patient control" implementations that I've seen have generally asked the patient to do the impossible.  The patient is expected to make an agreement while under extreme stress, inadequately informed, and with no time to get proper advice or more information.  Then the agreement is used to rationalize all kinds of reciprocity failures.  They would do better to deal with the reciprocity failures in most cases, and concentrate patient control on situations where the patient is not under stress, has adequate information, and the time to make an informed decision (including getting 3rd party advice).