Using these tools for gathering data from my Fitbit I've now got a little over 2 years worth of CSV files to analyze. I've only made a little slow progress due to other time commitments. What I know so far:
- The data is kept in a directory per day per category. These CSV files lack a trailing newline. A little bit of awk took care of the newlines and combining all that data into annual files.
- Reading and plotting the two years of data (about 200,000) individual data points, took two seconds on my computer. That's plenty fast enough.
- 5 minute data isn't too useful without further processing. I'm still experimenting with what form of averaging, summarization, etc. makes sense. I'm considering two kinds of summarization as a first step. First, create a CSV of daily results, and second eliminate all the "zero" step periods. Those may reveal something useful.
- It turns out that R has all sorts of interesting time series analysis work that comes out of the financial community. I should have realized that they would be intensively involved in this.