Fairhaven, The River

Software Risk (and the end of the world)

When I was in college I made some book shelves from boards and cider blocks rescued from a dumpster.  In my first job after college, I worked with a guy who made custom crafted bookshelves that sold for tens of thousands of dollars.  We would occasionally discuss the difficulty that software has with user recognition of the difference between home built and custom crafted, and the difficulty that users have in deciding what level is appropriate for the job.

Now we have discussion of how bad excel spreadsheets made the financial crisis much worse, how this is due in part to home built vs custom crafted, and how custom crafted is no assurance of quality.

Everyone is still struggling with the problem.

Chromebook C7 Experience (Initial Review)

Summary: The Acer Chromebook C7 works as an ultralight Linux system and I will continue using it.  There are plenty of annoyances and it is not for everyone.  It is for people who can tolerate annoyances and who like to tinker with things.  The $200 price compensates for a lot of annoyances.  If you can't handle annoyances or don't want to tinker with things, get an Ultrabook ($800+) or Mac Air ($1300+).

I got a Chromebook C7 and have replaced the ChromeOS with Ubuntu linux.  It will replace my aging Mac Pro laptop.  This began as an experiment.  I was willing to risk $200 on this.  It's working and I'm now committing to it.

I evaluated my needs by examining all the applications on my Mac Pro and on my corporate laptop.  This established what I want from a travel machine.

• I rejected the tablet plus keyboard alternative.  An Android tablet plus keyboard could do about two thirds of what I want.  I'm not willing to give up the other third.  One thing that I really want is the ability to browse the web, refer to PDF, Word, and other documents, all as part of writing another document (using either emacs or an office system).  That's the one thing that tablets can't do easily.  They are really limited to showing one application at a time.
• The Mac Air and the PC Ultrabooks can do everything that I want.  These cost about $1400 or $800 respectively.  If the Chromebook experiment failed, I would have gone to a PC Ultrabook.
• The Chromebooks have the option of replacing ChromeOS with Linux.  The $200 model is the better choice for this than the $250 model.  The two deciding factors were storage space and CPU.  The $200 model has 320GB disk and an x86 CPU.  The $250 model has 16GB static RAM and an ARM CPU.  For the non-tinkering user, the $250 is a much nicer machine.  But if you want to install Linux on it you need to put linux onto a separate 16GB microSD, and you're very storage limited.  The ARM means a lot of cross-compiling of software because there are relatively few Linux packages compiled and configured for the ARM family.  There are many different incompatible instruction sets among the ARM family, so you need to target machine types, not just ARM.

Physically the C7 looks like a Mac Air designed and built by a PC vendor.  It's clunky, boxy, a bit heavier, a bit thicker, slower, and much shorter battery life.  On the plus side, it has intelligent security protections, 320GB storage, and many useful connectors (like VGA and wired Ethernet).  It's flimsy, which does lead to some annoyances.  I haven't broken anything, but it feels fragile.  The battery life is only 3-4 hours, but you can buy additional batteries and swap batteries if you want.

Installing the software is straightforward but requires following a lot of instructions carefully.  (Instructions with pictures, and the real instructions). You can't just boot from a CDROM or Flash stick.  You need to get the machine into developer mode, download and run ChrUbuntu as an experimental OS, repartition the disk, sign it, and get Linux stored onto disk.  The first download is 1.5GB which takes a long time even with a fast Internet connection.  Then, the first update to bring it up to current versions will download another 400 MB.  And forever you must accept warnings at boot time that you are running an experimental OS in developer mode.

I do expect other distributions to follow Ubuntu's path and configure versions that can be installed in this way, but they haven't done it yet.  For other versions you are much more on your own.  You get to go read the ChromeOS developer documentation and figure it out yourself.

I've installed another 500MB of software downloads for

• emacs
• truecrypt
• Skype, and
• XCFE4

These first three are because they are core applications that I want to use.  They were the highest risk of failure on this system.  The other missing applications are going to work if Linux works.  (Most of the 400MB is dependencies pulled in by Skype.  Skype is only available in 32-bit mode on Linux.  ChrUbuntu is 64-bit mode.  So the Skype package pulls in a mass of 32-bit mode library and compatibility support.)

I installed XCFE4 because this system needs a lightweight windowing system.  I tried Canonical's preferred Unity system that comes with ChrUbuntu.  I replaced it because:

• It places too many demands on the 1.1Ghz Celeron.  The clickpad was highly erratic.  Other features were slow or erratic.  Unity really needs a beefier CPU and GPU.  It's full of demanding eye candy.
• It interferes with my doing multiple things at once.  It's like the tablets.  It's set up to show one application at a time.  If you fight hard you can have multiple applications at once, but it was easier to switch than fight.

An example of the kinds of annoyances you must face and fix is a conflict between Unity login manager and XCFE4 over access to the power system controls.  These are known bugs.  They leave it unpredictable whether power status will be shown and whether lid closure will trigger standby or not.  I dealt with the first by installed "xosview" to show me system status.  Further examination of comments on the bug reports showed how to configure XCFE4 to not use the power control daemon.  Now power management works much better.  But there are still occasional glitches where the system powers up from standby despite the lid being closed.  I haven't figured out how to stop this yet.

I've got plenty of software installing and customizing to go, but I'm confident it will work.  The high risk stuff is working and indicates that the rest will work.  It's just going to take time to do.

It's got lots of annoyances and limitations, but for $200 I can accept them.  (Extra:  I've just added 8GB of RAM ($40) and it's doing fine.  Instructions are here.)

On LinkedIn Passwords

The LinkedIn password disclosure might not have also released account names.  We went over it at a security lunch today.  If they used a system similar to Radius servers, there are two separate databases, one that maps username to account number and one that maps account number to password hash.  It is plausible that LinkedIn used this structure for the same reasons that Radius does.  It improves performance in some respects and reduces the harm from partial breaches of security.

I had not considered it likely that LinkIn would do this, given their silence on security methods and the available information on their database breach.  But copying the Radius approach (or perhaps using a Radius or Radius derived system) is plausible to me.

Dell U2711 and getting full resolution

The documentation from Dell and others is utterly atrocious, so people like me write posts like this to help others with their setup.  First, although Dell never documents this, these are the limits for various kinds of input:

• DisplayPort can do the full 2560x2048
• DVI-I Dual Link can do the full 2560x2048
• VGA can only do up to 2048x1152.  (Interpolation makes text a bit fuzzy.)
• DVI Single Link can do up to 2048x1152
• HDMI I can't test properly.

Then there are the variations of support by various systems.  Vendors do not document their hardware well enough to know what will work without trying it.  I have found:

• MacBook Pro with MacOS, DVI-I Dual Link is supported at full resolution
• Intel D94GZIS with Windows 7, DVI-single link at 1600x1200, DVI-VGA Adapter at 2048x1152
• Intel DH67CL with Linux and Intel Driver 2011Q3, VGA adapter to DVI works at 2048x1152, DVI only up to 1600x1200
• Dell Latitude 630 in Docking Station: DisplayPort at full resolution

The documentation and connector for the DH67CL claim dual link capability but after various experiments I've found no way to configure the driver to use it properly.  Documentation of the Intel driver provides no information about dual link setup.  The default configuration did not set it up automatically. 

Good podcast video from MIT Libraries

This video (1.5hr) stood out from the crowd that I usuallyscan.  It's got good insights into the learning processes of high-end digital native kids (MIT undergrads), the needs of a modern library, what MIT's educational life is like, and where MIT libraries are heading.  Some of it's obvious and some was surprising.  For example, the changing role of books on paper was not what I expected.

Org-mode, documents, and XDS

I've created an org-mode date-tree for organizing my various documents. This leads to some thoughts on XDS, etc.

Org-mode stuff (skip this if you don't care much about org-mode)

• I've created a date-tree for these documents. 
• Each set of documents gets a headline, with whatever description, tags, and metadata I feel like adding.  This date-tree lives in my regular org files.  So I can now search for tags, concepts, dates, etc. to find headlines.  Then I can read the headlines, their descriptions, and their metadata in order to decide which documents I want to read.
• The attachments are in their own git archive.  I can make this work by having a different attachment directory in each of the .emacs files on each machine.  My first thought of using links in the file system failed because Windows XP does not support links.
• I still need to do extra manual work, but it's now tolerable.  I've created a temporary save area.  I detach the documents from an email into that save area.  I create a headline in the date-tree. The headline text includes key aspects from the email, if any, and a short description of the documents.  I then attach the documents from the temporary save area (which copies them into the org attachment directory) to the headline.  Finally, from time to time I empty the save area.
• The org attachments directory is outside the org tree so that my git sharing of org files doesn't pick up all the attachments too.  Those are managed separately, also by git sharing.  This lets me do things like put the attachments directory on a removable drive.  (This makes the org file vaguely like the XDS Registry, and the attachments kind of like an XDS Repository.)
• Attachments management is adjustable by machine.  On the Mac the nicest way to deal with them is to open the attachments directory for an entry, and then use Mac tools.  On the Linux and Windows, the easiest is to do directory management within emacs and open files from there.  (XDS folks can think of the attachments for a headline as being like a submission set.)
• I had to fix the ~/.mailcap on my Linux system to point the PDF relationship at "evince".  Emacs was opening the PDF files within an emacs buffer.  This works, but it's slow and has a lot fewer features.  It works by converting the PDF into a PNG and displaying that with image display tools.  That's not as good as a full function PDF viewer.  The Mac and PC already have PDFs pointed to the appropriate viewer.

Resulting thoughts on organizers and XDS

This raises some thoughts on the state of information organizers.  Most people still use something like Sharepoint with all of it's preconfiguration restrictions and rules.  Some use wikis, which don't have the tags, properties, and other metadata facilities of org.  (Most wikis also have a lot more problems with editing the content.  The good content editors are platform specific.)  There are various more sophisticated metadata management tools that do a better job with document management.

Org still has too many manual operations.  Properties, tags, etc. are good, but don't integrate well with ontology tools.  (On the other hand, org has no problems with multiple ontologies.)  Tags, etc. are at the level of headlines, so there can be inheritence and the like.  But this is all stuff that you need to set up for yourself.  So it's a tool for tool builders, not a tool for general users.

The potential is there to do all kinds of things since this is build upon a structurally aware lisp engine.  I think I could make this into an integrated XDS Registry/Repository if I felt like it.  The really hard part would be implementing all the SOAP crap and enforcing metadata rules.  Org's attitude toward metadata is "do whatever you want." Org tries to adapt as best it can.

My org approach doesn't have the XDS single registry viewpoint.  But that's because I'm accustomed to systems  and system designs that are normally "broken".  They are inconsistent in contents, etc. in that each different viewpoint may reflect a different set of updates and changes.  The processes exist to reconcile the changes, but there is no requirement to present a consistent or "accurate" viewpoint.

This kind of "it depends on where and when you look" is an inherent necessity for many operational environments.  Presenting a single view requires 100% connectivity, 100% functional, error free network and systems.  In some enterprise environments it is practical to achieve this, or close enough that people don't notice the failures. 

In telecoms management this is impossible.  Networks will have failures that partition the network.  The various parts must continue to operate and be managed based on what information is still available, and they must adapt to mergers, including reconciliation of conflicting information when merging.  In the largest of the networks that I was involved with we never had fully functional networks.  There was always a portion of the network that had failed.  There were always a few islands operating on their own while problems were being fixed.

Healthcare IT folks (like those involved with XDS) are still having real problems coming to grips with accepting a system design that is always "broken".  They don't realize that in healthcare  everything is always "broken" in that sense.  Patients don't provide full or consistent  information, so patient information is "broken".  Measurements are incomplete.  Diagnoses are really just working hypotheses to be tested.  Disease and health are constantly changing, so the recorded state is never correct.  It's like large networks.  You must proceed with incomplete, inconsistent, changing information.

With time they should come to grips with this.

 

A new definition for privacy

Discussion with Gila and a podcast have led me to a new definition for privacy.

Privacy is the control over information to ensure that

• No physical harm results for the person, their family, loved ones, friends, or property.
• No financial, emotional or other harm is done to the person, their family, loved ones, or friends.
• No harm is done to any social relationships of the person, their family, loved ones, or friends.

My new definition is driven partially be a story from a Google security expert at an LSE panel session.  This was during one of the publicity bursts for computer security.  The press was putting about stories about how the public behavior showed that they didn't really care.  She had gone to a morning focus session with some members of the public, and then attended a computer security conference.  The public was actually very concerned about the potential for the use of private information by stalkers.  When she mentioned this to the computer security experts, they laughed, said this shows how little the public understands, and continued on their chosen path.

I think the public was right, although they got some of the details wrong.  So I've rephrased this as a harm prevention goal.  That's the root requirement. 

Gila argued that the requirement includes personal consent and control.  I still disagree.  I consider personal consent and control to be part of the mitigation strategy.

Consider an imaginary world where there is magic pixie dust that can be sprinkled on information.  The magic pixie dust can read minds, understand social relationships, and predict the future.  It ensures that the information is never revealed or used for anything that will cause any of the harms listed above.  In that imaginary world I do not think that any person would complain that they were not controlling or consenting to data releases.  The magic pixie dust will do a much better job, and with much less burden on the person.  People have no ability to read minds, very limited ability to predict and future, and even find much simpler tasks like understanding the complex implications of consenting to a particular policy to be very difficult.

But there is no magic pixie dust.  Current technology is unable to meet most these goals.  Centrally designed policies are a very limited mitigation.  Personal consent and control are another mitigation to deal with the many details that a centrally designed policy will not have.  Considering consent and control as a technology, the inability to read minds, predict the future, etc. still leave many potentials for harm.  Further mitigations are needed, e.g., audit controls and process feedback loops.

By considering consent and control to be a technology for mitigation we have the established framework for recognizing that more mitigations are needed.  Considering consent and control to be the driving requirements closes this path, since the system will have fulfilled its requirements.

There are other social requirements.  The right of personhood is a requirement that every person be able to make the decisions that affect the future path of their life.  In the absence of the magic pixie dust and the inadequacy of privacy mitigations, this right of personhood means that the person should have the control to decide what risks of harm they are willing to accept.

The privacy requirements are also subject to societal restrictions.  I will add the statement:

• Society may choose to inflict harm on a person rather than suffer societal harm.  For example, the privacy of criminals will be reduced rather than accept unrestricted social harm from crime.  There are also more positive motivations, such as the infliction of privacy harm by public health organizations so that other people will have less harm from disease.

 

A good ftp proxy software

I've been using "jwftp" as an ftp proxy for about six months now, and it works very well.  I recommend it.  FTP is a very stable protocol, so "jwftp" is also stable.  It just gets a very occasional tweak to reflect changing operating system environmentts.

This is a bit of a niche interest, but as a security guru I inflict my recommendations onto myself.  Therefore, my home network lives behind some application proxies.  I've had issues getting a good ftp proxy that supports all the various ftp clients and not just the browsers.  This proxy has worked well with not just the browsers.  It's also happy with ftp, sftp, wget, and lftp.  (I personally use "lftp" as a method for intelligent mirroring of ftp sites up and down.  Wget is fine for mirroring downloads, but doesn't do intelligent mirroring uploads.  "lftp" does.)

The audit logs are reasonable to parse, and so far just confirm that the only ftp traffic is authorized traffic that I have initiated.

 

Transition to org - Part 7 of n

(In late March 2010 I installed the latest and greatest version of git.  It has some new rules about configuration, use of bare git repositories, warnings and errors, etc.  The intent is to avoid mistaken destruction of working code by forcing the master repository to be in a different directory than the working copies.  So, I needed to create a bare git repository on my master Linux system.  Then all the machines would have working "org" directories that push and pull to the master.

[2010-03-30 Tue 11:04] Git rebasing

I created an "org-master" that contains the master.  I made sure everything was checked in, then renamed "org" to "org-master".  I then deleted the org directories and recreated them by cloning.  Also, updated the .emacs on the master machine to point back to org as the working directory.  Only git knows that there is an org-master.

The git commands built into emacs have some sort of bug.  The update to latest generated an error rather than updating a file.  I should investigate this at some point.  Right now, emacs automation does not matter.

[2010-03-31 Wed 09:23] Git rebasing

I did the rebasing of org onto a new directory.  It went reasonably well.  The process was:

• I made sure that all the computers were fully synched and consistent with old git archive.
• I renamed the master directory to be "org-master"
• I deleted the old org directories and cloned them from org-master.  Nice thing about git is that the whole history survives just fine.

Everything was still there just fine.

[2010-04-05 Mon 10:56] Org - spreadsheet mode (emacs calc)

I tried this today.  It reminded me of the old supercalc and other predecessors to the modern spreadsheets.  It's not as easy to use as a modern sheet, but it integrates rather well into emacs and org.  So I created a little table of computations rather easily.  I did make one mistake by using "=" rather than ":=" for the formulas.  "=" is a column formula that applies to the entire column.  I didn't notice until later when I went to edit something.  The error was minor and now I'll remember.

[2010-04-20 Tue 07:52] Journal Conversion

(I discovered the "date-tree" facility while looking for something in the new manual.  It's a rather nice addition that organizes notes by date into a hierarchy.  I modified the .emacs to make it easier to gather notes.  My .emacs now has remember config lines of:

(setq org-remember-templates
      '((?t "* TODO %?\n  %i\n  %a" "~/org/TODO.org")
        (?j "* %U %?\n\n  %i\n  %a" "~/org/JOURNAL.org" date-tree)
        (?i "* %U %?\n\n  %i\n  %a" "~/org/ihe.org" date-tree)
        (?d "* %U %?\n\n  %i\n  %a" "~/org/dicom.org" date-tree)
        (?n "* %U %^{Title}\n %?%i\n  %a" "~/org/idea.org" date-tree)))

It would be nice if I could have multiple date trees in one file.  I would like to have a file for one organization, e.g., DICOM, and then have a date tree for my meeting and tcon notes for each working group.  For now, it's one tree for all of DICOM.  I looked at the code and it's moderately complex modifications to have another level of hierarchy like that.  I don't have time to make that kind of changes.)

The date tree stuff seems to be working OK.  Converting from the old form to the new form by hand is not that bad.  The writing of lisp to do it is more work than doing it by hand.  This manual re-organizing is also a good review of past ideas.

Transition to org - Part 6 of N

(Later:  At this point I started using C-c C-! to tag entries with a date.  This now turns into a more sporadic discovery of features and changes to how I use org.  By this point I was comfortably using org as a replacement for planner.  So here's the rest of 2009.)

[2009-03-30 Mon 10:12] Use of remind

I took a quick look at org2remind.  It seems to just move the scheduled and deadline tasks into the remind file with the proper format.  It does not appear to get the other calendar items.

This is not what I am after.  I want something closer to the agenda view.  Remind's interesting scheduling flex would have been nice, but it's not worth the problems.

[2009-04-21 Tue 09:55] Experiment with recipes

I tried making up a test recipe.  Two possible approaches for ingredients:

• I can use a Properties drawer.  This has some search advantages because I can limit searches to properties.  By default the drawer will be closed.  I have to open it manually.  There is a minor disadvantage that an ingredient could only occur once.
• I can use a table.  This is easy to edit, nicely visible.  There is not a search restricted to tables that I noticed.
• I could use a table, and then a property that lists the ingredients.  That would be more data entry, but get both the property search advantage and the nice table.  I could even use tags for this rather than invent a property.  Just tag each recipe with ingredients.  I had originally thought to tag with main dish, dessert, etc.

All look pretty easy.  It's just time and energy doing data entry. A later step is to try to integrate the links with the Firefox scrapbook.  Make a file of recipes and let it be synchronized as part of org.

[2009-04-29 Wed 08:40] Org (outline) search

I just noticed that the org mode incremental search selectively opens and closes outline view as appropriate.  It's probably been doing this all along without my noticing.  I must have been doing searches, although I may have done a bulk outline open first just to see things.

[2009-04-29 Wed 08:46] Recipes

I put in a few recipes.  Right now the process is mostly manual. I've got them in a scrapbook.  As I finish, I move a recipe from one folder to another.  So far it's easier to retype than to use cut and paste.  This could change for more complex instructions.

Next thing to consider is how to organize the linkages back to the originals.  I could continue to use a scrapbook form.  Or I could just use the HTML pages with a file link.  The scrapbook form has more opaque file names.  Pages are tagged by date and time.  It also has the metadata contents, although I've found it to be a little quirky.  The HTML export form uses readable names and has no metadata.

[2009-05-28 Thu 09:28] Search stuff

I finally set up a TAGS file.  Now I have a choice:

• use M-x grep to search for a pattern, and remember to limit it to searching *.org.  This gives the list of all occurrences.
• use tags-search to incrementally search through the files in the order that they occur in the TAGS file.

The etags program makes semi-random guesses when used on org files. I then edited that TAGS file to just list the filenames and have no tags.   

[2009-06-08 Mon 09:17] Categorization

I'm finding it hard to get categorization right at first.  Also, tagging continues to be tricky.  I don't have a good mental categorization of work yet.

[2009-06-08 Mon 09:19] More Recipes

I'm getting recipes slowly converted over.  The manual process of conversion adapts well to the highly variable format of recipes. I've not dealt with setting up a proper storage area for the HTML form.  It needs to be somewhere reasonably stable that I can refer to.  I think it should be HTML format.  The surrounding file structure and naming makes HTML export format more sensible than scrapbook format.

[2009-07-20 Mon 07:58] Parallel editing

I've found that I'm not worrying as much about maintaining proper synchronization.  I'll let the org files get more substantially out of sync.  Repairing the collisions is not that hard.

The typical situation I find is this:

1. I do a "push" or "commit" and git reports a conflict that cannot be resolved automatically.
2. I use "pull" or "git gui" plus commit to get a simple local conflict collision.
3. The conflicts are marked in the conflicting files.  I just edit as necessary to fix.  The biggest early problem was realizing that this cannot be done in org-mode.  I need to switch the buffer manually to text-mode.  The fixes are generally just a matter of deleting or moving lines.  I rarely edit the exact same text.  It is usually a matter of deciding which lines to keep.
4. "commit" and distribute the fixed up version.

Sometimes the changes look complex.  In those cases I'll branch, work on the branch, and merge it back in.  Most of the time it makes more sense to just treat each different computer as if it were a different branch, but without the actual branch tagging.   

[2009-08-16 Sun 14:57] Emacs 23.1 on Mac  

Seems to work just fine.  One keyboard glitch noted so far is with the mapping of the ALT key.  The Carbon port of Emacs 22 remapped this to the Apple-Clover key, which made it much more consistent with other Apple applications.  I actually prefer the new mode of mapping ALT to ALT, since I switch back and forth between Linux, Mac, and Windows regularly.  Cross OS consistency helps me more than within MacOS consistency.