Fairhaven, The River

Medical Market "failure"

I had to change my regular medications last week.  I delayed the conversation with the nurse while I did some checking of prices.  From both Costco and Walmart I could get a 3-month supply for $10, cash, no insurance.  The nurse commented "we don't price compare".  My response was "I do."  (It's actually quite unreasonable to expect them to help price compare.  Insurance plan complexities are different for every patient and finding a cost is hard.)  Since my co-pay is $25, the $10 is a good price.

I've been through this before, but just for my amusement I checked again:

• At both local chain pharmacies they will not quote a price.  Prices are secret.  All they will disclose is co-pay and only 1 month supply salable if insured.  They are not willing to sell on other terms, even for cash, because they know I have insurance.
• I didn't waste time with the PBM that used to nag me.  Last time they were not willing to quote a price.  All they would disclose was the copay, for a 3 month supply, delivered by mail sometime in a two week window.

The only way to find a price is to buy the drug and look at the receipt details.

I find the nonsense about healthcare demonstrating "market failure" to be highly annoying.  You can't call it a market when prices are kept secret from potential customers.  Price and quality comparison are fundamental to anything called a market.

The increasing penetration of Costco and Walmart may partially reflect customer intolerance for the mandated inconvenient delivery and secret pricing.  The cost conscious public clearly want a real marketplace with public prices.

Actual failure experience (re ATNA-Syslog)

This has to be somewhat vague for trade secret reasons, but during the mid-90's Polaroid gathered extensive data on network problems as part of a total system reliability tracking effort.  The systems were medical printing systems, and data was gathered for video capture units, print servers, and printers.  The networks were hospital networks, all with wired ethernet connections.  There were several hundred devices involved and measurements taken at a couple hundred hospitals over an extended period at these hospitals.  Some of the units were stationary, and some of the units were mobile.

The network traffic is noticably different from ATNA-Syslog, so these results would need some adjustment for that purpose.  The network traffic for printing was DICOM, "lpr", and "ftp" traffic.  The observations were made over a period of several years.  In excess of 10 million transactions were recorded.  Transactions were almost entirely print transactions, with a very few print status queries.  The observations were:

• Network problems were entirely insignificant.
  
  • Stationary systems reported no network problems.  TCP/IP handled whatever happened just fine.  (I should note that a down server or printer was not considered a network problem if this was detected before the print data was sent.  The sending systems would queue their prints and wait for the server  or printer to be restored to service. Down servers or printers were considered a different kind of reliability problem.)
  • Mobile systems detected network problems at about one per 250,000 transactions.  These were all the kind of failure that I described earlier.  The application acks did help identify problems.  At a problem rate of 4 ppm we decided it was reasonable to deal with these problems by just re-printing whenever a possible problem was observed.  Four extra prints per million is an insignificant problem and not worth extensive engineering work.

ATNA Syslog is a different network use pattern, and the motivation for directed attacks is different.   Hospital network characteristics may have changed since the 1990's, and this data is not a measurement of cross enterprise data traffic.  This data is relevant in characterizing the internal characteristics of the hospital environment, and can be taken into consideration when doing reliabiliity design, FMEA, and similar analysis for ATNA product design and installation design.

Explanation of TCP/TLS vulnerability

This explains the vulnerability mentioned previously. The vulnerability is an uncertainty by a sending application regarding whether messages have been received or not.  This is not directly a security vulnerability.  It's inherent in the TCP/TLS design, and normally is only an issue with mobile devices.

The real world sequence of events is this:

1. Application write/sends X bytes to other side of TCP connection.
2. Network is disconnected.
3. Application gets a "Reset" or similar error from read/write/close operation.

Details of the system I/O functions don't matter.  In all these cases, the application is left uncertain whether the X bytes were received or not.  The "Reset" and related errors are an indication that something went wrong.  Previously acknowledged messages might have still been in buffers, in transit, etc.  Some of them might have been lost.  TCP itself cannot know, because it is possible that the network disconnect prevented a TCP acknowledgement from arriving.

The application mest decide whether it cares.  Possible situations range from:

• Everything made it just fine and was acknowledged.  This is quite common in situations where there was a long delay between the last send and a close().  Odds are that the network disconnect is indirectly related to the close and occurred long after the TCP transmissions were completed.
• Everything in the TCP window could have been lost.  This can happen when the network is lost during a large transmission.  TCP has all this data buffered and is trying to get it through.  It's using timeouts, retries, etc.  If the network failure is just a transient, it will succeed.  But, in this example, a "reset" indicates that it cannot succeed and has failed.  There might be a complete TCP window's worth of buffers that were previously accepted but did not get transmitted.

There are also a variety of intermediate states possible that the application could determine. 

This is where application ack is sometimes proposed as a solution.  It's initially appealing, but it's still imperfect.  The ack could be the only part of the transmission that was lost.  Application acks do make sense in a number of application contexts.  They don't automatically solve the underlying problem.  More protocol design will be needed, with idempotency or transaction rollback dealing with parts of the problem.

This kind of timing occurs naturally with mobile devices.  The form that I see most often is a real world situation where the user finished a task with a mobile device, disconnects it, and moves on the next task.  Sometimes the mobile device is still transmitting results over the network when this happens.  The disconnect can be a cable disconnect or simply moving the device out of WiFi range.

Asking users to wait does not work.  They may understand the issue and try, but in practice it's like asking people not to stub their toe or spill their coffee.  They understand and try, but it still happens.  It's more realistic to design a device that survives these occasional problems.

 

Should Syslog use application acks?

This started as an ATNA-Syslog question, that I will consider in the context of some DICOM history. 

Should Syslog have an appliction ack to deal with some known vulnerabilities?  I call them vulnerabilities because they are extremely unlikely except when combined with intentional external actions.  These vulnerabilities become apparent with mobile equipment and with skilled attackers.

DICOM has used application ack since it's introduction.  But, this was for application reasons, not to deal with vulnerabilities in the underlying TCP and TLS services.

• In C-MOVE there is a C-MOVE-REQ that says "please accept object X".  The corresponding C-MOVE-RSP says either "yes, got it" or "no, this is why not."  Most often the no is because the responder is out of storage space, but it can also be due other problems.

• In C-FIND the pair is more complex.  C-FIND-REQ says "please provide information on objects that match these criteria".  The C-FIND-RSP normally conveys the information requested.  It can also deal with "and there is more to come later" or "this is everything".  The "no, this is why not" deals with invalid requests and other problems.

The use of application ack to deal with network problems as part of failure management was a side effect.  These are just another kind of failure to be reported.  The state machine for the application layer has to deal with network failures as part of completeness, not as a reason for application ack.

DICOM also tolerates a certain level of indeterminacy.  The designers basically said "close enough", much like the designers of CRC and FEC codes.  These cover most of the possible errors, and accept that a few will sneak through and be dealt with elsewhere.

The "elsewhere" in DICOM leads back towards the isues with Syslog and ATNA.  DICOM makes transactions idempotent to the maximum extent possible.

The C-MOV transaction is idempotent.  These is no end state difference between "C-MOV object X" once and "C-MOV object X" one hundred times.  The only difference is the time it takes.  So when in doubt or uncertain, DICOM applications just do it again.  The second time probably reaches a determined state.  Similarly the C-FIND is idempotent.  A look at operational DICOM logs shows well over 99.9% of the transactions performed are idempotent.

There are some necessary exceptions, like "print".  Sending "print" once is different from sending "print" a hundred times.  You get a lot more copies printed.  DICOM tries to take the non-idempotent applications and split them into an idempotent part and non-idempotent part.  DICOM puts as much of the application into the idempotent part as it can.  In the case of "print", everything except the N-ACTION-PRINT is idempotent.  This minimizes the window of vulnerability.  DICOM then took the attitude, "close enough, maybe there is the occasional duplicate print when something goes wrong.  We'll accept that.  It won't happen often enough to be a problem."
    
Syslog and ATNA have the issue that:

• There is no application level ack, and

• There is a delivery uncertainty in the face of some kinds of errors that occur with mobile devices and skilled attackers.

Can idempotency deal with this?  It can if syslog messages are designed properly.  This would allow a gradual transition to reliability without requiring changes to the underlying syslog protocols.  The application change would be to send extras when uncertain about delivery.  This is probably also a much smaller network impact also.  There is extra traffic only in those error situations, not during normal traffic.

This takes messages that are universally unique over all time and sources, and that are idempotent.  The idempotency allows sending duplicates.  Duplicates can be recognized and discarded by recipients.  (This also simplifies some of the multiple database and dispersed database issues for log processing.)

Unique IDs deal with uniqueness.  DICOM uses unique IDs for all sorts of things.  The hardest problem with unique IDs is persuading all the hot shot programmers that they really should read the recommendations on best practices for creating unique IDs. The home grown unique ID algorithms all seem to fall into one or another of the well known traps that result in non-unique ID generation.

If the lead-in to every syslog message body includes that unique ID for that message, you may be done.  The rest is ensuring that the generating application doesn't generate multiple messages for the same event.  That's bad design in any case.  You also want to avoid non-idempotent content.  This should be easy for ATNA and syslog.  The message is describing an event.  It shouldn't be that hard to make these messages idempotent.

The one idempotency trap that I've seen with log messages is the use of incremental messages.  These need sequence integrity.  The idempotence of meaning is lost if the messages are processed in the wrong order.  Time tags and other tools can be used to preserve sequence integrity despite repeats and out of order arrival.  But it helps a lot if the syslog messages are designed to be a self-contained complete description of the event of interest.

How well did RFC-3881 and DICOM do when defining audit messages?  So-so.  The event identification mandates identifying a source, date-time, etc.  It does not require that these be message unique. The messages are fully self-contained.  What they need is a unique ID.

I also checked the MITRE CEE effort.  They don't mandate idempotent messages either.

So, should I propose a change to add an optional unique ID?  Something to think about.