Fairhaven, The River

Transition to org - Part 7 of n

(In late March 2010 I installed the latest and greatest version of git.  It has some new rules about configuration, use of bare git repositories, warnings and errors, etc.  The intent is to avoid mistaken destruction of working code by forcing the master repository to be in a different directory than the working copies.  So, I needed to create a bare git repository on my master Linux system.  Then all the machines would have working "org" directories that push and pull to the master.

[2010-03-30 Tue 11:04] Git rebasing

I created an "org-master" that contains the master.  I made sure everything was checked in, then renamed "org" to "org-master".  I then deleted the org directories and recreated them by cloning.  Also, updated the .emacs on the master machine to point back to org as the working directory.  Only git knows that there is an org-master.

The git commands built into emacs have some sort of bug.  The update to latest generated an error rather than updating a file.  I should investigate this at some point.  Right now, emacs automation does not matter.

[2010-03-31 Wed 09:23] Git rebasing

I did the rebasing of org onto a new directory.  It went reasonably well.  The process was:

• I made sure that all the computers were fully synched and consistent with old git archive.
• I renamed the master directory to be "org-master"
• I deleted the old org directories and cloned them from org-master.  Nice thing about git is that the whole history survives just fine.

Everything was still there just fine.

[2010-04-05 Mon 10:56] Org - spreadsheet mode (emacs calc)

I tried this today.  It reminded me of the old supercalc and other predecessors to the modern spreadsheets.  It's not as easy to use as a modern sheet, but it integrates rather well into emacs and org.  So I created a little table of computations rather easily.  I did make one mistake by using "=" rather than ":=" for the formulas.  "=" is a column formula that applies to the entire column.  I didn't notice until later when I went to edit something.  The error was minor and now I'll remember.

[2010-04-20 Tue 07:52] Journal Conversion

(I discovered the "date-tree" facility while looking for something in the new manual.  It's a rather nice addition that organizes notes by date into a hierarchy.  I modified the .emacs to make it easier to gather notes.  My .emacs now has remember config lines of:

(setq org-remember-templates
      '((?t "* TODO %?\n  %i\n  %a" "~/org/TODO.org")
        (?j "* %U %?\n\n  %i\n  %a" "~/org/JOURNAL.org" date-tree)
        (?i "* %U %?\n\n  %i\n  %a" "~/org/ihe.org" date-tree)
        (?d "* %U %?\n\n  %i\n  %a" "~/org/dicom.org" date-tree)
        (?n "* %U %^{Title}\n %?%i\n  %a" "~/org/idea.org" date-tree)))

It would be nice if I could have multiple date trees in one file.  I would like to have a file for one organization, e.g., DICOM, and then have a date tree for my meeting and tcon notes for each working group.  For now, it's one tree for all of DICOM.  I looked at the code and it's moderately complex modifications to have another level of hierarchy like that.  I don't have time to make that kind of changes.)

The date tree stuff seems to be working OK.  Converting from the old form to the new form by hand is not that bad.  The writing of lisp to do it is more work than doing it by hand.  This manual re-organizing is also a good review of past ideas.

Transition to org - Part 6 of N

(Later:  At this point I started using C-c C-! to tag entries with a date.  This now turns into a more sporadic discovery of features and changes to how I use org.  By this point I was comfortably using org as a replacement for planner.  So here's the rest of 2009.)

[2009-03-30 Mon 10:12] Use of remind

I took a quick look at org2remind.  It seems to just move the scheduled and deadline tasks into the remind file with the proper format.  It does not appear to get the other calendar items.

This is not what I am after.  I want something closer to the agenda view.  Remind's interesting scheduling flex would have been nice, but it's not worth the problems.

[2009-04-21 Tue 09:55] Experiment with recipes

I tried making up a test recipe.  Two possible approaches for ingredients:

• I can use a Properties drawer.  This has some search advantages because I can limit searches to properties.  By default the drawer will be closed.  I have to open it manually.  There is a minor disadvantage that an ingredient could only occur once.
• I can use a table.  This is easy to edit, nicely visible.  There is not a search restricted to tables that I noticed.
• I could use a table, and then a property that lists the ingredients.  That would be more data entry, but get both the property search advantage and the nice table.  I could even use tags for this rather than invent a property.  Just tag each recipe with ingredients.  I had originally thought to tag with main dish, dessert, etc.

All look pretty easy.  It's just time and energy doing data entry. A later step is to try to integrate the links with the Firefox scrapbook.  Make a file of recipes and let it be synchronized as part of org.

[2009-04-29 Wed 08:40] Org (outline) search

I just noticed that the org mode incremental search selectively opens and closes outline view as appropriate.  It's probably been doing this all along without my noticing.  I must have been doing searches, although I may have done a bulk outline open first just to see things.

[2009-04-29 Wed 08:46] Recipes

I put in a few recipes.  Right now the process is mostly manual. I've got them in a scrapbook.  As I finish, I move a recipe from one folder to another.  So far it's easier to retype than to use cut and paste.  This could change for more complex instructions.

Next thing to consider is how to organize the linkages back to the originals.  I could continue to use a scrapbook form.  Or I could just use the HTML pages with a file link.  The scrapbook form has more opaque file names.  Pages are tagged by date and time.  It also has the metadata contents, although I've found it to be a little quirky.  The HTML export form uses readable names and has no metadata.

[2009-05-28 Thu 09:28] Search stuff

I finally set up a TAGS file.  Now I have a choice:

• use M-x grep to search for a pattern, and remember to limit it to searching *.org.  This gives the list of all occurrences.
• use tags-search to incrementally search through the files in the order that they occur in the TAGS file.

The etags program makes semi-random guesses when used on org files. I then edited that TAGS file to just list the filenames and have no tags.   

[2009-06-08 Mon 09:17] Categorization

I'm finding it hard to get categorization right at first.  Also, tagging continues to be tricky.  I don't have a good mental categorization of work yet.

[2009-06-08 Mon 09:19] More Recipes

I'm getting recipes slowly converted over.  The manual process of conversion adapts well to the highly variable format of recipes. I've not dealt with setting up a proper storage area for the HTML form.  It needs to be somewhere reasonably stable that I can refer to.  I think it should be HTML format.  The surrounding file structure and naming makes HTML export format more sensible than scrapbook format.

[2009-07-20 Mon 07:58] Parallel editing

I've found that I'm not worrying as much about maintaining proper synchronization.  I'll let the org files get more substantially out of sync.  Repairing the collisions is not that hard.

The typical situation I find is this:

1. I do a "push" or "commit" and git reports a conflict that cannot be resolved automatically.
2. I use "pull" or "git gui" plus commit to get a simple local conflict collision.
3. The conflicts are marked in the conflicting files.  I just edit as necessary to fix.  The biggest early problem was realizing that this cannot be done in org-mode.  I need to switch the buffer manually to text-mode.  The fixes are generally just a matter of deleting or moving lines.  I rarely edit the exact same text.  It is usually a matter of deciding which lines to keep.
4. "commit" and distribute the fixed up version.

Sometimes the changes look complex.  In those cases I'll branch, work on the branch, and merge it back in.  Most of the time it makes more sense to just treat each different computer as if it were a different branch, but without the actual branch tagging.   

[2009-08-16 Sun 14:57] Emacs 23.1 on Mac  

Seems to work just fine.  One keyboard glitch noted so far is with the mapping of the ALT key.  The Carbon port of Emacs 22 remapped this to the Apple-Clover key, which made it much more consistent with other Apple applications.  I actually prefer the new mode of mapping ALT to ALT, since I switch back and forth between Linux, Mac, and Windows regularly.  Cross OS consistency helps me more than within MacOS consistency.

Transition to org - Part 5 of n

Todo Templates

I've found that making a template for various common tasks is easy and useful.  I've created three checklist templates: International travel; long domestic travel (3+ days); and short domestic travel.  It's easy to make a copy and insert it along with the trip specifics whenever I schedule a trip.

This isn't a template for remember to use as a todo.  It's more a pre-packaged mini project plan.

Unicode

I did a cut and paste from a Nature article into Emacs.  It worked fine, but included some non-ASCII characters.  Emacs gave me the option of editing the file or converting the characterset when I went to save it.  I picked convert characterset.  As far as GIT is concerned, I just did a complete file replacement.  This may prove a nuisance later.  I need to check how Mac and Linux respond to this.  The oddball characters are the degree symbol, the plus/minus symbol, and one other.

Checking with Notepad it looks the same.  Checking with Wordpad it looks like there is an extra character in there.  (Wordpad seems to have chosen the angstrom character.)

Scheduled vs not

I figured out that the org set up for dates is a bit different than my usual interpretation.  I had been making appointments scheduled to get them in the Agenda., and then marking them done. But this introduced notification entries a few days early.

Sometimes I want this reminder, sometimes not.  It depends upon whether I need to think about the appointment or do something to get ready.  If it's just the reminder whether this is the recycling week or not, I don't want the reminder.

That is resolved by just putting in a date tag with no scheduled, deadline, closed, etc.  If the date tag has angle brackets it shows up in the agenda.  If it has square brackets it does not.

You see scheduled means "plan to start work on a project" not "this activity is scheduled to take place."  So now I have to remember "C-c ." for appointments that need no prep work.  "C-cC-s" for appointments that I must think about.

Attaching files

I noticed the ability to attach files to tasks.  I need to dig into this more.  My first thought issue is dealing with GUI vs keyboard.  Does this have a reasonable GUI process for finding the file to attach?  It's a pain to have to remember the relationship between the directory structure as seen by the GUI and as seen by the keyboard.

As I read into it, there is a relationship with git, existing git repositories, and existing directories.  This has potential.  I need to read it over carefully.

A new kind of task/note

I find myself more often creating a journal entry describing something and embedding a task within it.  Then I put the journal entry into the generic task list for a project.  This reflects realizing that there are things to do because of whatever it is that I am documented.

Merge conflict resolution

I just finished doing some merge conflict resolution because I swapped between two laptops without synchronizing.  The resolution is easy.  Git had it all nicely marked.  But, org-mode interacted poorly with the markup. It concealed a lot of it.

The resolution is simple enough.  I switched to text mode so that I could see the merge markup. But, it is a potential for errors.

(Several weeks of just use later)

It seems to be working out well.  I've abandoned the notion of using the task stuff for shopping lists.  I just have some sections for the different routine grocery and supply stores. Things get put on a list, then when I go shopping it's copied to paper.  That works better.

EMR Polling, a communications disconnect

My initial reaction to the link to a recent poll analysis showing how little the public understands the value of EMRs was their intended "ignorant public" reaction.  I then shift to an "ignorant pollsters"  reaction, with a bit of wondering about whether it was ignorance or deliberate.

I think that the public answered a different question than the pollsters thought they had asked.

It seems likely that the public answered the question "who will benefit from the US government spending and regulation on EMRs", not the polling question about the "benefits of EMRs".  For obvious reasons my mind turned to GOSIP.  A similar question mismatch about "benefits of an Internet" versus "who will benefit from US government spending and regulation of GOSIP" would have a similar result.  GOSIP was a multi-billion dollar waste that caused delay and harm to the acceptance and growth of the Internet.  The US spending on EMRs could go down the same path.

As with GOSIP, the problem with the present EMR push is not with the technical concept, the problem is with the government spending and regulation approach.  I think that is what the public answers reflect.  They do not think that this government spending and regulation will have the claimed benefits.

The advertised functional goals for the EMR are twofold:

• Reduce costs, and
• Improve patient outcomes.

I will believe that a project will reduce costs when the project plan includes a proposed layoff size.  We are past the "let's build one and see what happens" stage for EMRs.  By now there should be a reasonable plan for layoffs of staff that will result from the EMR installation, where these layoffs result in savings that exceed the cost of installing and operating the EMR.  I'm not asking that these be perfect.  I've seen how things change as projects progress.  But if layoffs are not even in the plan at the beginning, there is no credibility for claims of cost savings.

I have seen plans like this in RIS/PACS installations and some EMR installations.  The systems are justified by layoffs of couriers, elimination of staff overtime, the elimination of warehousing, equipment maintenance, etc.  The results don't always match the plan, but in general there are real savings. 

I will believe the goal of improved patient outcomes when there are clinical trials and statistical evidence that show a reason to expect improved patient outcomes.  A blind faith of "technology will save us" is not accepted elsewhere in medicine, and should not be accepted here.  There is plenty of potential, but does it work in the field?  This needs to be evidence based.

Again, in RIS/PACS/EMRs there are sometimes situations where equipment is installed because it improves patient outcomes.  There are clinical trials and there is statistical evidence gathered showing that these technologies do improve patient outcomes. 

I will believe an outcomes motivations for an EMR project when I see sound scientific evidence in the justification for project features.

Transition to org part 4 of n

(I should mention that there are already a lot of good tutorials on using org.  I found this one on using it for GTD especially good.)

Planning activities

I'm getting familiar enough that I should start writing up an interim organization for how to do weekly reviews, daily reviews, etc.  This will evolve, but it's time to start writing down the patterns that seem to be working.

With planner, visiting the files will trigger changes if it has been over a day since the last visit.  This introduces a variety of glitches from the perspective of git.  With org, I can do all sorts of examinations without triggering changes.

And a related mac problem

The Mac OS path setup mechanisms and defaults are a little different that Linux.  The git installation process gets git into the search path for interactive shells, but not for non-interactive shells.  This was not noticable except when using the mac as host for a pull.  I had to add the path file logic to the .bashrc also:

    if [ -x /usr/libexec/path_helper ]; then
    eval `/usr/libexec/path_helper -s`
    fi

The path_helper deals with the rest of usual Mac installer path management and avoids duplicating entries on the path variables.    This is probably related to my MacOS upgrade.  I think I would have noticed it before.  I might have done this before and just not noticed that the OS upgrade restored various files.

Use of priorities and agenda

I've found that using the task view in the agenda plus the priority is a good way to pull out the immediate work items for today. This is a different definition of priority.  It's really "work on it today" versus "deadline".

Git update

I had the first real case of dealing with merger.  I accidentally ended up making updates on all three computers without first synch and checkout.  The result is beyond the ability of the automatic merger to handle.  The changes came too close and overlapped in a couple cases.  So git generated a report on the merger conflict and dropped it in my lap.  I generated a minor tangle of deltas as I figured out which changes should survive. 

This is what the gitk tree looked like.  You'll also note that I do all my transactions using ssh links.

Re-structuring files

Org is a lot friendlier when I decide to split up the file structure and re-organize the layout.  But, it loses track of relationships with dates.  With planner, there can be a common bit of text that lives both in the day page and the project page.

I'll have to make better use of remember so that I preserve this kind of information.  If I have a note that is relevant to two projects I want an indication of this commonality.  I also want the linkages so that I can have it in the journal page with links out to the two projects, and in each project have links to the other project and the journal page.

Transition to org - part 3 of n

Time tags

I played around with tagging appointments and todo items.  It's got all the stuff I want, except maybe for some of the trickier scheduling.  The one  hard one is the trash schedules.  Pickups are "Every other Wednesday, except if Monday, Tuesday, or Wednesday is an official holiday.  Then it will be Thursday."  This means pulling in an official holiday list and then doing some logic.

The trash thing looks like a lot more complexity than it is worth it.  It's good enough to show it on Wednesday.  I can deal with the off by one for holidays.  I need the computer to keep track of whether this week is a recycling week.  I travel enough that I just lose track of which week has recycling and which does not.

Email Trial

I set up work laptop.  Started just be grabbing latest version from mac, and putting that as initial version on it.  Then I did a git init.  Then a checkout to email branch, did some changes, committed, switched back to main, did a merge, created an email delta, and emailed it.

The email was retrieved, applied, and seems to be OK.  There is a whitespace issue to be fixed to eliminate some whitespace warnings.

Warning days

After a while I found the variable that controls when tasks show up in the agenda view.  It's org-deadline-warning-days.  Default is 14 days.

This leads to an interesting approach change.  I think I'll make most of the tasks without deadline or schedule.  Then use tags to find the ones that I'm after.  The many items were cluttering up the agenda view.

PC Fixes

I'm dealing with variations of the Windows PC.  The biggest headache was getting the .emacs file right.  It lives in the Application Data for my profile, which is not the home directory for cygwin.  This leads to lots of mistakes.

Also, getting longlines to be the default is tricky.  Simply adding it to the auto list is not sufficient.  That turned on longlines and turned off longlines.

Git

Git integration is doing OK.  Org is a lot more like code than planner.  The pattern of changes is much better.  Planner modifies the underlying day files and the project files to reflect deadlines, schedules, current date, etc.  Org does not.  It computes delays, lead times, etc. dynamically.  With all this clutter removed it is easy to manage multiple machine updates and reconcile overlapping updates by hand if necessary.

The more code-like pattern also means that git does better at finding the deltas and fixes automatically.

One issue will be long lines.  The longlines minor mode interacts badly with org tagging.  So I have to use auto-fill mode.  This is fine for text I generate in emacs.  But it's a nuisance when I need to cut and paste into other programs, e.g., blog editors.  For those I would rather have longlines mode.  According to mailing lists there is a mode coming as part of emacs 23 that will deal with this.  Meanwhile, I have a nuisance to deal with when I copy notes into other programs.  (I wonder if the easiest path is to write an "unfill" lisp action.  I could "unfill", cut/paste, then refill.)

(The visual mode in emacs 23 does do what I want.  It just hadn't been written yet.)

Transition to "org" part 2 (of n)

(Continuing, I noted some of the immediately visible differences and annoyances with the change.  This is fairly normal.  The changes that annoy you are immediately noticed.  The improvements are more gradually observed.)

Early annoyance

Org puts the agenda description files into the .emacs file.  That interferes with sharing my files using git.  The directory structures on MacOS, Windows, and Linux are different.  I'll preserve the org directory structure and mirror it. If there were a separate file in the org directory that kept all the org stuff it would be much easier for me to synchronize. 

Early file organization

So far I've decided on on a modest list of files: work, blog, home, and three major project categories.  Within these I've got an orthogonal set of tags for specific useful sorting categories.

I've also made two sequences for TODOs: (TODO, STARTED, WAITING, DELEGATED, MAYBE | CANCELLED DONE) (BUY LATER | DONTBUY BOUGHT).

One is for tasks, the other for shopping lists.

(Later, I made some changes, but these held basically for the past year.)

Early stuff

The .emacs issue continues as an annoyance.  The org system for tracking schedules and deadlines is a lot better.  With some more thought I should be able to use git to merge parallel updates on multiple systems.  It's no worse than the usual issues with software development. 

I need to learn how repeating tasks work.  I've got some.  The repeat schedule was easy enough.  But setting one to complete has some issues.  I need to figure out how best to do that.  Time for more manual and code reading.

I also need to see whether they have work underway to move the agenda definition into a separate file so that my multi-system sharing can be easier.  (They never did, but it's been easy enough to adapt.)

Remember problems   

The [f8] was not starting remember.  Instead it complained about an autoload failure for org-remember. 

• I spent a while fussing with .emacs to no effect. 
• I tried changing (org-install) to no effect.
• I tried deleting the distribution org lisp files to no effect.
• I started reading through the details of how autoload and emacs startup work.
• While deciding which computer system to use to tear apart emacs (because it would likely be non-functional for emacs while I did it) I was doing a line by line read of the .emacs.
• I noticed that (require 'org-install) was missing. Doh. (The real problem.)
• I went to machine Duo (where I had made no modifications beyond putting in the latest remember and org) and tried adding the (require ...).  It behaved properly.
• I have just finished undoing all the changes I made on the mac laptop, where I had done most my experiments.  It now works properly.

Remember templates and archival

The surround environment issues are not as tidy as in planner.  In planner I would get prompts for subjects when I finished the notes.  It would then put a cross-linked copy of the note into the journal section of each subject.  Org does not do this.  (Cross links are still not there, but the rest can be done.)

Org puts the note into the file (subject) where I started the note, or into some other file (it asks) depending on how I finish it.  This is not as useful.  If I later go to a different but related subject I will not see the note.  With planner there will be a copy there if I recalled to add that subject to the note.

Org can easily be extended to add semantic tags to the note.  I might be able to build something out of that, but it's not quite what I want.

I want to have a journal section for each subject, and in that section have a copy of each relevant note.  These should be crosslinked to their copies, and preferably edit protected somehow.  Then one subject should also get the note somewhere in the org outline structure.

It ought to be possible to set up the remember templates to do this.  It's something to do.

I just tried it using remember to create a task linked to this.    The task is created and sitting over there.  There is no indication of that relationship in this file.

(Later: I've ended up using tags and accepting that I can't get fine grained references between one note and several related subjects.  It was a nice planner feature.  I can live without it.)

The transition to "org"

These notes date back to more than a year ago.  I'm only just now getting around to gradually blogging this.  It violates the guideline about blogging as it happens, but most of this happened when I was not connected.  Instead of blogging, I was using the org tools that I am now blogging.  I did capture this as it happened, just not online.  I captured it into my journal.  Journals work in cafe's, lounges. etc.  That's where I frequently work.  So after the fact hindsight insights will be tagged in italics.

Why

Shortly after I got started with GTD I went looking for computer tools to help me.  But I had some uncommon requirements:

1. It needed to work on Linux, Windows, and MacOS.
2. I needed to be able to exchange files seamlessly among these three systems.  Initially I expected to use rsync.
3. It would be nice if I could do parallel operation, i.e., switch machines without a prior coordinated rsync, but it is practical for me to shift from machine to machine only occasionally.  An rsync as part of the shift was not a big barrier.  (Later: It turns out I run in parallel more than I thought that I would.)
   

These requirement led me to an Emacs based approach, and within Emacs there was Planner and Org.  There are only a few other planning tools that work on all three platforms.  None of them matched the features of either Emacs package. I picked Planner because it fit my work process better.  The highly hierarchical structure of org and outlines is not a tidy fit.  Also, the integration with remember was much better in planner.  Finally, the fan boys were being obnoxiously loud about org. (I was also influenced by writeups by the writeups by planner contributors like Sacha Chua.)

(I used planner for about a year.  The functions were OK.)

Now the situation has changed.

1. The first and most significant change is that I've shifted to using git as my synchronization method.  Planner has a number of problems with the way it constantly modifies sequence numbers and has extensive minor changes to files as time advances.  This makes git interation awkward and not much better than rsync.
2. Org has gotten a lot better.  Links, remember integration, and tagging improved dramatically.  It's good enough that I can use tags and links to accomplish the multi-topic linkages that I use in planner.
3. The fan boys have moved on and a lot of practical people are driving good enhancements for org.
4. (Later: Planner also seems to have gone moribund, although at the time it was the improvements to org that I was seeing.)

So I'm trying org.  The quick early testing indicates enough improvements in various areas to justify switching from planner.  The highly hierarchical structure has relaxed enough to meet my needs.  I can use tags etc. to handle the extent to which I have many projects that meet multiple goals.

Important first steps

Make sure that you get a version 6 edition of org.  The current distributions of emacs for MacOS, Windows, and as part of SuSE all have a 5.2x version.  You want the re-factoring that was part of 6.x and the link, remember, and tagging improvements.  I'm using the 6.11b version.  (Later: At the moment I'm at 6.36, and the Emacs 23+ releases include a reasonable version of org as part of that distribution.  So the setup for a current user is much simpler.  Just use Emacs 23 or later.)

This means downloading it and then modifying the .emacs file.  That's not much of a barrier to the typical emacs user.  We emacs users do things like that all the time.

Schneier Interview

There is a nice short podcast interview with Schneier by ISN.  It's associated with a good collection of resource material.  I recommend it as a good introduction about current events for newcomers.

I found some  interesting snippets:

1. The only accomplishment from all the Times Square security cameras was to provide a lot of nice pictures for the TV shows.  The person captured turned out to be an innocent bystander unrelated to the terrorist attempt.
2. Only one out of two thousand official wiretaps report any problems from encryption.  This indicates something about privacy concerns of potential wiretap targets.  (me: it could be too much work, the taps could be targeting the wrong people, the conversations could themselves be in code, etc.)
3. Schneier mentioned that relatively few people take the effort to shred sensitive documents.  This inspired me to check marketing data.  From what I could find, the annual market for shredders is $300 million.  Of that, 2/3 are commercial shredders.  I think this means that the SOHO market will be 1-2 million shredders each year.  That makes for a substantial minority, but still a minority of the public.

I also like his replacement quote:

    It's not privacy versus security, it's liberty versus control.  Privacy is one aspect of security.