Org-mode tables may be very convenient for data entry and routine use, but the interface to R is very inefficient. What I've found is that when org-mode converts a 1,000 line table into a data table variable for R it consumes 15 seconds on my computer. Reading a 200,000 line table from a CSV file for the same purpose takes less than a second when done directly by R.
I don't plan to change my current processes. I can easily tolerate a 15 second delay for data analysis in return for the ease of editing. But when the data is gathered automatically, then it will be in the form of CSV files. The 200,000 line table is my past 2+ years of fitbit data, preserved with 5 minute data binning.
Next step is figuring what analysis and presentation is useful for the fitbit data. The simple data plot confirms that I've got all the raw data and it's all reasonable looking. But it doesn't reveal much in terms of patterns or trends. Summarization and other processing will be needed.
Comments