Exploring Apple Health data with F#

Rest HR zones by day

If some doctor by luck read this, please check if the above is healthy. It is time spent in each heart rate zone per day during the last 3 months. I’ve filtered out sleep and exercise to have it include only resting heart rate.

Health Data

Apple Health App is recording my heart rate, steps, workouts, stand hours, sleep etc. All data can easily be exported as a zipped xml-file.

Since F# have a good interactive repl and some interesting data science libraries (eg. FsLab and Deedle) I decided to use it to experiment with the data.

Source code for experiments: https://github.com/henrikwallstrom/fsharp_apple_healthdata

Read data

My first failed attempt was to use F# Data XmlProvider to read the XML-file.

#r "FSharp.Data.dll"
#r "System.Xml.Linq.dll"
open FSharp.Data

type HealthData = 
  XmlProvider<"../../../export.xml", 
  Global=true>

I’m not sure if it’s the structure or size of the file but it was way too slow.

I instead went for reading XML manually which is quite pleasant in F#.

#r "System.Xml.Linq.dll"
open System

let xmlDoc path =
    let xdoc = Xml.XmlDocument()
    xdoc.Load(path : string) |> ignore
    xdoc

let doc = xmlDoc "export.xml"

The DOCTYPE declaration of the file include an internal document type definition that describes the structure of the XML document and element types that exists.

It looks like the most interesting part is the Record elements.

doc.OuterXml.Substring(0, 2548);;
...
<!ELEMENT Record (MetadataEntry*)>
<!ATTLIST Record
  type          CDATA #REQUIRED
  unit          CDATA #IMPLIED
  value         CDATA #IMPLIED
  sourceName    CDATA #REQUIRED
  sourceVersion CDATA #IMPLIED
  device        CDATA #IMPLIED
  creationDate  CDATA #IMPLIED
  startDate     CDATA #REQUIRED
  endDate       CDATA #REQUIRED
>

The document consists of a very long list of health record nodes that can be matched with the XPath expression /HealthData/Record

let healthRecordXml = 
    doc.SelectNodes "/HealthData/Record" 
        |> Seq.cast<System.Xml.XmlNode> 

Seq.length healthRecordXml //=> 729456

Almost one million records after 3 months.

Health Records

It looks like records data have type, unit, sourceName, startDate and endDate in common.

let healtRecordXmlSample = Seq.head healthRecordXml
healtRecordXmlSample.OuterXml
<Record type="HKQuantityTypeIdentifierBodyMassIndex" 
sourceName="Lifesum" sourceVersion="7.1.0" unit="count" 
creationDate="2016-11-07 19:23:05 +0100" 
startDate="2016-10-29 23:00:00 +0100" 
endDate="2016-10-29 23:00:00 +0100" 
value="26.3036" />

We can make a pass over all health records and check what types that exists.

let xmlNodeAttributeValue attributeName (elem: Xml.XmlNode)  = 
    match Option.ofObj(elem.Attributes.GetNamedItem(attributeName)) with
    | Some (v) -> Option.ofObj v.Value
    | _ -> None

let recordXmlByType = 
    healthRecordXml
    |> Seq.groupBy (xmlNodeAttributeValue "type" >> Option.get)
    |> dict
    
let recordTypeCount = 
    recordXmlByType
    |> Seq.map (fun (KeyValue(t, r)) -> (t, Seq.length r)) 
    |> Seq.toList
[("HKQuantityTypeIdentifierBodyMassIndex", 6);
   ("HKQuantityTypeIdentifierHeight", 1);
   ("HKQuantityTypeIdentifierBodyMass", 7);
   ("HKQuantityTypeIdentifierHeartRate", 88333);
   ("HKQuantityTypeIdentifierStepCount", 170738);
   ("HKQuantityTypeIdentifierDistanceWalkingRunning", 209347);
   ("HKQuantityTypeIdentifierBasalEnergyBurned", 99847);
   ("HKQuantityTypeIdentifierActiveEnergyBurned", 142656);
   ("HKQuantityTypeIdentifierFlightsClimbed", 7352);
   ("HKQuantityTypeIdentifierDietaryFatTotal", 178);
   ("HKQuantityTypeIdentifierDietaryFatSaturated", 122);
   ("HKQuantityTypeIdentifierDietaryCholesterol", 35);
   ("HKQuantityTypeIdentifierDietarySodium", 106);
   ("HKQuantityTypeIdentifierDietaryCarbohydrates", 206);
   ("HKQuantityTypeIdentifierDietaryFiber", 83);
   ("HKQuantityTypeIdentifierDietarySugar", 124);
   ("HKQuantityTypeIdentifierDietaryEnergyConsumed", 242);
   ("HKQuantityTypeIdentifierDietaryProtein", 207);
   ("HKQuantityTypeIdentifierDietaryPotassium", 87);
   ("HKQuantityTypeIdentifierAppleExerciseTime", 7085);
   ("HKCategoryTypeIdentifierSleepAnalysis", 440);
   ("HKCategoryTypeIdentifierAppleStandHour", 2232);
   ("HKCategoryTypeIdentifierMindfulSession", 22)]

It would be nice to plot that as a PIE chart.

FsLab and XPlot

FsLab is a collection of libraries for data-science. Among many things it includes XPlot. XPlot is a cross-platform data visualization package for F# powered by Google Charts and Plotly.

XPlot Documentation: http://tahahachana.github.io/XPlot/

#load "packages/FsLab/Themes/DefaultWhite.fsx"
#load "packages/FsLab/FsLab.fsx"

open XPlot.GoogleCharts

Record types as a pie chart:

recordTypeCount
    |> List.map (fun (k, v) -> 
        (k.Replace("HKQuantityTypeIdentifier", "")
            .Replace("HKCategoryTypeIdentifier", "") 
            + " " + v.ToString("N0"), v))
    |> List.sortByDescending snd
    |> Chart.Pie
    |> Chart.WithOptions (
        Options(
            width = 1200,
            height = 1200,
            legend = Legend(),
            sliceVisibilityThreshold = 0,
            pieHole = 0.4))

record types

I will focus on HKQuantityTypeIdentifierHeartRate, HKQuantityTypeIdentifierAppleExerciseTime and HKCategoryTypeIdentifierSleepAnalysis to get heart rate readings when not sleeping or exercising.

Parse records

We could parse the XML elements into a list of dictionaries

let healthRecordFields = ["type"; "unit"; "value"; 
  "sourceName"; "sourceVersion"; "device"; 
  "creationDate"; "startDate"; "endDate"]

let getStringField elem field =
    (field, xmlNodeAttributeValue field elem 
    |> function | Some (s) -> s | None -> "")

let getHealthRecord elem =
    healthRecordFields 
    |> Seq.map (getStringField elem) 
    |> dict

let getRecords ofType = 
    recordXmlByType.Item(ofType)
    |> Seq.map getHealthRecord

getRecords "HKQuantityTypeIdentifierHeartRate" 

We could then use that and plain F# to get quite far but I would like to try Deedle.

Deedle

Deedle is a library for data and time series manipulation. There are two basic concepts, Series and Frames. A Series is a vector like collection of values indexed by a key. We will mainly work with time series where the keys are time stamps. A Frame is a matrix like collection of series that share the same row key. We will use frames to e.g. group properties of health records.

#load "packages/Deedle/Deedle.fsx"
open Deedle

A frame can be constructed in a few ways. One way is using Frame.ofValues from a sequence of tuples:

Frame.ofValues;;
val it :
  arg00:seq<'a * 'b * 'c> -> Frame<'a,'b> when 'a : equality and 'b : equality

It takes tuples of row * column * value. We could turn our records into tuples like that by mapping over all Records and use the row index as row in the tuple. Then map over all entries in the dictionary in each row and use dictionary key as column and value as value.

let getFrame records =
    records
    |> Seq.mapi (fun index record -> record 
      |> Seq.map (fun (KeyValue(k, v)) -> (index, k, v)))
    |> Seq.concat
    |> Frame.ofValues
getRecords "HKQuantityTypeIdentifierHeartRate"
    |> Seq.head
    |> Seq.map (|KeyValue|) 
    |> Seq.toList 
[("type", "HKQuantityTypeIdentifierHeartRate"); ("unit", "count/min");
   ("value", "74"); ("sourceName", "Henrik’s Apple Watch");
   ("sourceVersion", "3.1");
   ("device",
    "<<HKDevice: 0x174285dc0>, name:Apple Watch, manufacturer:Apple, model:Watch, hardware:Watch2,4, software:3.1>");
   ("creationDate", "2016-11-06 09:41:19 +0100");
   ("startDate", "2016-11-06 09:41:19 +0100");
   ("endDate", "2016-11-06 09:41:19 +0100")]

Heart records

Heart Rate records/samples are collected by my Apple Watch and Chest Strap. I suspect there are some overlaps between the two sources.

Unit seems to be count/min on all records.

let hrUnits : string list =
    getRecords "HKQuantityTypeIdentifierHeartRate" 
    |> getFrame
    |> Frame.getCol "unit"
    |> Series.values
    |> Seq.distinct
    |> Seq.toList

// => hrUnits : string list = ["count/min"]

Start and end dates are the same on most records so I guess the value is for a given moment. Some HR entries seems to have a duration. I guess it is a bug.

let hasZeroDuration (row : ObjectSeries<_>) = 
    row
      .GetAs<DateTime>("endDate")
      .Subtract(row.GetAs<DateTime>("startDate"))
      .TotalMinutes = 0.0

let hrEntriesWithPeriods =
    getRecords "HKQuantityTypeIdentifierHeartRate" 
    |> getFrame
    |> Frame.filterRowValues (hasZeroDuration >> not)
         type                              unit      value sourceName sourceVersion device creationDate              startDate                 endDate
8686  -> HKQuantityTypeIdentifierHeartRate count/min 82    Fitness    20161115.1602        2016-11-19 18:35:09 +0100 2016-11-19 18:34:37 +0100 2016-11-19 18:35:02 +0100
9210  -> HKQuantityTypeIdentifierHeartRate count/min 77    Fitness    20161115.1602        2016-11-19 18:45:01 +0100 2016-11-19 18:26:33 +0100 2016-11-19 18:43:04 +0100
13744 -> HKQuantityTypeIdentifierHeartRate count/min 125   Fitness    20161115.1602        2016-11-21 18:11:59 +0100 2016-11-21 09:29:42 +0100 2016-11-21 10:24:54 +0100
14074 -> HKQuantityTypeIdentifierHeartRate count/min 137   Fitness    20161115.1602        2016-11-21 18:51:19 +0100 2016-11-21 18:11:20 +0100 2016-11-21 18:48:26 +0100
18369 -> HKQuantityTypeIdentifierHeartRate count/min 114   Fitness    20161115.1602        2016-11-28 09:29:25 +0100 2016-11-24 08:47:45 +0100 2016-11-24 10:05:28 +0100
18373 -> HKQuantityTypeIdentifierHeartRate count/min 133   Fitness    20161115.1602        2016-11-28 09:29:26 +0100 2016-11-22 18:07:56 +0100 2016-11-22 18:50:20 +0100
18375 -> HKQuantityTypeIdentifierHeartRate count/min 116   Fitness    20161115.1602        2016-11-28 09:29:27 +0100 2016-11-22 07:39:35 +0100 2016-11-22 08:46:59 +0100
18384 -> HKQuantityTypeIdentifierHeartRate count/min 113   Fitness    20161115.1602        2016-11-28 09:45:28 +0100 2016-11-28 07:56:25 +0100 2016-11-28 09:39:27 +0100
18386 -> HKQuantityTypeIdentifierHeartRate count/min 94    Fitness    20161115.1602        2016-11-28 09:49:57 +0100 2016-11-28 09:43:25 +0100 2016-11-28 09:49:03 +0100
18792 -> HKQuantityTypeIdentifierHeartRate count/min 135   Fitness    20161115.1602        2016-11-28 18:51:14 +0100 2016-11-28 18:07:29 +0100 2016-11-28 18:49:57 +0100

As I suspected there are duplicate HR readings with the same start and end date:

getRecords "HKQuantityTypeIdentifierHeartRate" 
    |> getFrame
    |> Frame.indexRowsDate "startDate"

// => System.ArgumentException: Duplicate key '11/19/2016 6:30:15 PM'. Duplicate keys are not allowed in the index.

But since I have only one heart I guess I can pick any of the readings and filter with Seq.distinctBy.

let heartRateFrame = 
    getRecords "HKQuantityTypeIdentifierHeartRate" 
    |> Seq.distinctBy (fun r -> r.Item("startDate"))
    |> getFrame
    |> Frame.filterRowValues hasZeroDuration 
    |> Frame.indexRowsDate "startDate"
    |> Frame.sortRowsByKey
                         type                              unit      value sourceName           sourceVersion device
                           creationDate              endDate
11/6/2016 9:41:19 AM  -> HKQuantityTypeIdentifierHeartRate count/min 74    Henriks Apple Watch 3.1           <<HKDevice: 0x174285dc0>, name:Apple Watch, manufacturer:Apple, model:Watch, hardware
:Watch2,4, software:3.1>   2016-11-06 09:41:19 +0100 2016-11-06 09:41:19 +0100
11/6/2016 9:41:25 AM  -> HKQuantityTypeIdentifierHeartRate count/min 68    Henriks Apple Watch 3.1           <<HKDevice: 0x174289ba0>, name:Apple Watch, manufacturer:Apple, model:Watch, hardware
:Watch2,4, software:3.1>   2016-11-06 09:41:25 +0100 2016-11-06 09:41:25 +0100
11/6/2016 9:46:17 AM  -> HKQuantityTypeIdentifierHeartRate count/min 79    Henriks Apple Watch 3.1           <<HKDevice: 0x174289740>, name:Apple Watch, manufacturer:Apple, model:Watch, hardware
:Watch2,4, software:3.1>   2016-11-06 09:46:27 +0100 2016-11-06 09:46:17 +0100
11/6/2016 9:48:47 AM  -> HKQuantityTypeIdentifierHeartRate count/min 74    Henriks Apple Watch 3.1           <<HKDevice: 0x17428a5a0>, name:Apple Watch, manufacturer:Apple, model:Watch, hardware
:Watch2,4, software:3.1>   2016-11-06 09:51:40 +0100 2016-11-06 09:48:47 +0100
11/6/2016 9:57:04 AM  -> HKQuantityTypeIdentifierHeartRate count/min 74    Henriks Apple Watch 3.1           <<HKDevice: 0x17428a640>, name:Apple Watch, manufacturer:Apple, model:Watch, hardware
:Watch2,4, software:3.1>   2016-11-06 09:57:54 +0100 2016-11-06 09:57:04 +0100
11/6/2016 10:06:07 AM -> HKQuantityTypeIdentifierHeartRate count/min 81    Henriks Apple Watch 3.1           <<HKDevice: 0x17428a6e0>, name:Apple Watch, manufacturer:Apple, model:Watch, hardware

Value is a string. We can add the parsed number value as a new column.

heartRateFrame?HR <- heartRateFrame 
    |> Frame.mapRowValues (fun row -> 
        row.GetAs<double>("value"))

Heart rate readings are read more often when eg. exercising or using the chest strap we therefore give them a weight by calculating the duration between readings.

let secondsBetweenRows field maxValue aFrame   =
    aFrame 
    |> Frame.getCol field
    |> Series.pairwise    
    |> Series.map (fun _ (v1: DateTime, v2:DateTime) -> 
        Math.Min(v2.Subtract(v1).TotalSeconds, maxValue))

The watch usually read every 5th minute so let´s assume that gaps larger than 5 minutes (300s) are periods without readings

heartRateFrame?Seconds <- 
    secondsBetweenRows "endDate" 300.0 heartRateFrame
heartRateFrame?Seconds
    |> Series.values
    |> Seq.map (fun v -> ("Seconds", v))
    |> Chart.Histogram
    |> Chart.WithOptions(
        Options(
            width = 1200,
            scaleType =  "mirrorLog",
            histogram = Histogram(bucketSize = 10)))

Most readings are done every 10th second or often since the watch read every 5th minute during rest but more often when exercising.

record types

Before playing more with HR we need to add sleep, standing and exercise to the Frame

Sleep

Information is stored when I go to bed, wake up and actually sleep.

getRecords "HKCategoryTypeIdentifierSleepAnalysis" 
    |> Seq.map (fun r -> r.Item("value")) 
    |> Seq.distinct
    |> Seq.toList
["HKCategoryValueSleepAnalysisInBed"; "HKCategoryValueSleepAnalysisAsleep";
   "HKCategoryValueSleepAnalysisAwake"]

Sleep readings from different sources can overlap.

Merge records

Metrics such as sleep, exercise, stand can overlap when collected from different sources.

We could:

  • Sort records by start date
  • Merge records with previous record if start is before the previous end

We can merge two records by mapping over the first records values, keeping values that does not match the new field and emitting a new value for the updated field.

let setField field value series =
    Series.map (fun k v -> 
        if k = field then 
            value :> obj 
        else v) series

We can now set endDate:

let testSeries = series [
    "startDate", DateTime.MinValue :> obj; 
    "endDate", DateTime.MaxValue :> obj]

setField "endDate" DateTime.Now testSeries

Now we can merge overlapping records by sorting the list and merging records with the previous record if they overlap.

let mergeOverlappingValues (aFrame: Frame<_, string>) =
    aFrame
    |> Frame.sortRows "startDate"
    |> Frame.rows
    |> Series.foldValues (fun (state: 
        ObjectSeries<string> list) nextRow  -> 
        match state, nextRow with
        | [], next -> [next]
        | current::tail, next -> 
            let currentEnd = 
                current.GetAs<DateTime>("endDate")
            let nextStart = 
                next.GetAs<DateTime>("startDate")
            if currentEnd >= nextStart  then // overlaps
                let nextEnd = 
                    next.GetAs<DateTime>("endDate")
                if currentEnd >= nextEnd then 
                    current::tail //within
                else 
                    let merged = 
                        setField "endDate" nextEnd current
                    ObjectSeries(merged)::tail // merge
            else 
                next::current::tail 
    ) [] 
    |> List.rev
    |> Frame.ofRowsOrdinal;

Example:

let testDates =  
    [
        (1, "startDate", DateTime(2013,1,1, 19, 00, 00));
        (1, "endDate", DateTime(2013,1,1, 20, 00, 00))
        (2, "startDate", DateTime(2013,1,1, 19, 30, 00));
        (2, "endDate", DateTime(2013,1,1, 19, 45, 00))
        (3, "startDate", DateTime(2013,1,1, 21, 00, 00));
        (3, "endDate", DateTime(2013,1,1, 22, 00, 00))
        (4, "startDate", DateTime(2013,1,1, 21, 30, 00));
        (4, "endDate", DateTime(2013,1,1, 22, 30, 00));
        (5, "startDate", DateTime(2013,1,1, 22, 30, 00));
        (5, "endDate", DateTime(2013,1,1, 23, 30, 00))]
    |> Frame.ofValues
mergeOverlappingValues testDates
//     startDate           endDate
// 0 -> 1/1/2013 7:00:00 PM 1/1/2013 8:00:00 PM
// 1 -> 1/1/2013 9:00:00 PM 1/1/2013 10:30:00 PM

Back to sleep

Sleep recorded by different apps and merged:

let sleepRecordsFrame = 
    getRecords "HKCategoryTypeIdentifierSleepAnalysis" 
    |> Seq.filter (fun r -> r.Item("value") =
     "HKCategoryValueSleepAnalysisAsleep")
    |> getFrame
    |> mergeOverlappingValues
    |> Frame.indexRowsDate "startDate"
    |> Frame.sortRowsByKey

Calculate sleep duration:

sleepRecordsFrame?Duration <- sleepRecordsFrame 
    |> Frame.mapRows (fun start row -> 
         row.GetAs<DateTime>("endDate").Subtract(start))

Histogram of the durations:

sleepRecordsFrame?Duration
    |> Series.values
    |> Seq.map (fun v -> ("Hours", v.TotalHours))
    |> Chart.Histogram
    |> Chart.WithOptions(
        Options(
            width = 1200,
            scaleType =  "mirrorLog",
            histogram = Histogram(bucketSize = 1)))

record types

Looks like nights for a parent of a 3-year-old.

Exercise

let exerciseRecordsFrame = 
  getRecords "HKQuantityTypeIdentifierAppleExerciseTime"  
  |> getFrame
  |> mergeOverlappingValues
  |> Frame.indexRowsDate "startDate"
  |> Frame.sortRowsByKey
exerciseRecordsFrame?Duration <- exerciseRecordsFrame 
  |> Frame.mapRows (fun k r -> 
    r.GetAs<DateTime>("endDate").Subtract(k))

Stand

let standRecordsFrame = 
  getRecords "HKCategoryTypeIdentifierAppleStandHour"
  |> getFrame
  |> Frame.filterRows (fun _ r -> 
     r.GetAs<string>("value")
       .Equals("HKCategoryValueAppleStandHourStood"))
  |> mergeOverlappingValues
  |> Frame.indexRowsDate "startDate"
  |> Frame.sortRowsByKey
standRecordsFrame?Duration <- standRecordsFrame 
  |> Frame.mapRows (fun k r -> 
       r.GetAs<DateTime>("endDate").Subtract(k))

Merge in activities

For every time stamp in HR readings I would like to know if I was awake, exercising or standing. I want a IsASleep, IsExercising and IsStanding column/flag on each HR reading row.

Since activity frames have a duration (start - end) while hr readings are momental (timestamp) we need to first transform them.

We need to split rows of durations:

["value", "HKCategoryValueSleepAnalysisInBed"; 
 "startDate", 23:30, endDate: 23:45]` 

Into timestamp rows:

[23:30, true] 
[23:45, false]`

Frame to Time Stamp Frame:

let tsFrame key aFrame  =
    aFrame 
    |> Frame.mapRows (fun startDate row -> 
        [(startDate, key, true); 
        (row.GetAs<DateTime>("endDate"), key, false)])
    |> Series.values
    |> Seq.concat
    |> Frame.ofValues
    |> Frame.sortRowsByKey

Example:

tsFrame "IsAsleep" sleepRecordsFrame
// 2/25/2017 10:51:32 AM  -> False
// 2/26/2017 2:33:37 AM   -> True
// 2/26/2017 7:44:02 AM   -> False
// 2/26/2017 7:51:59 AM   -> True
// 2/26/2017 9:19:33 AM   -> False
// 2/28/2017 12:41:51 AM  -> True
// 2/28/2017 1:37:35 AM   -> False
// 2/28/2017 1:45:33 AM   -> True
// 2/28/2017 6:24:06 AM   -> False

Deedle can merge frames. We could eg. join heartRateFrame with sleepRecordsFrame like this:

Frame.joinAlign 
    JoinKind.Right Lookup.ExactOrSmaller 
    heartRateFrame 
    (tsFrame "IsAsleep" sleepRecordsFrame)

It would give us a new frame with all HR readings and a sleep flag when there is a sleep row that exactly match the key/startDate or if there is a smaller/previous key.

Let’s say I was a sleep at 23:30 and have a HR reading 23:31 I can assume I was still a asleep

Let’s add all flags:

let mergeTsFrame f1 f2 = Frame.joinAlign JoinKind.Right Lookup.ExactOrSmaller f1 f2
let hrFrameWithActivity = 
    heartRateFrame
    |> mergeTsFrame (tsFrame "IsAsleep" sleepRecordsFrame)
    |> mergeTsFrame (tsFrame "IsStanding" standRecordsFrame)
    |> mergeTsFrame (tsFrame "IsExercising" exerciseRecordsFrame)

Looks like I’m sleep walking :)

hrFrameWithActivity
    |> Frame.filterRowsBy "IsAsleep" true
    |> Frame.filterRowsBy "IsStanding" true
    |> Frame.countRows

// => 553

Resting HR

I want my resting heart rate. We can do that by filtering out sleep and exercise.

But it looks like I have HR readings above 110 with time stamps within 10s. I assume that is exercise that is not logged correctly. I will filter them out as well.

hrFrameWithActivity?startDate <- 
  hrFrameWithActivity.RowKeys;

let restingRecordsFrame =
        hrFrameWithActivity
        |> Frame.sortRowsByKey
        |> Frame.filterRowsBy "IsAsleep" false
        |> Frame.filterRowsBy "IsExercising" false
        |> Frame.rows
        |> Series.foldValues (fun (state: 
            ObjectSeries<string> list) nextRow  -> 
            match state, nextRow with
            | [], next -> [next]
            | current::tail, next -> 
                // If HR is > 110 and readings are more 
                // often than every 10s then probably 
                // exercising but not logged
                if 
                    current.GetAs<float>("HR") > 110.0 
                    && next.GetAs<float>("HR") > 110.0 
                    &&
                    current.GetAs<DateTime>("startDate")
                      .Subtract(next
                        .GetAs<DateTime>("startDate")
                      ).TotalSeconds < 10.0
                then next::tail
                else next::current::tail 
        ) [] 
        |> List.rev
        |> Frame.ofRowsOrdinal
        |> Frame.indexRowsDate "startDate"
        |> Frame.sortRowsByKey

Average resting HR per day:

let restingMeanByDay =
    restingRecordsFrame 
    |> Frame.getCol "HR"
    |> Series.chunkWhileInto 
      (fun d1 d2 -> d1.Date = d2.Date) 
      Stats.mean

// =>
// 2/26/2017 12:03:13 AM  -> 67.8488372093023
// 2/27/2017 9:46:26 AM   -> 66.8031496062992
// 2/28/2017 12:02:20 AM  -> 68.3812949640288

HR Zones

I want a stacked column chart with %-of-minutes-of-the-day in different HR zones per day.

I will use the same colors and HR zones as the iOS app Heart Watch. It is a great iOS app that you should buy.

let zones = [
    (100.0, "> 100", "#CC0000");
    (80.0, "80-99", "#674EA7");
    (55.0, "55-79", "#3D85C6");
    (40.0, "40-54", "#BF737A");
    (0.0, "< 40", "#804D52")]

Helpers to extract data from table:

let zoneNameByHr (hr : double) = 
    let (_, name, _) = 
      zones 
      |> List.find (fun (limit, _, _) -> 
        hr > limit)
    name

let zoneColorByName zoneName = 
    let (_, _ , color) = 
      zones 
      |> List.find (fun (_, name, _) -> 
        name = zoneName)
    color

let zoneNames = 
  zones 
  |> List.map Pair.get2Of3 
  |> Seq.ofList 
  |> Seq.rev

We can construct a Pivot Table with:

  • Dates as rows
  • HR zone name as columns
  • Total minutes in HR zone as cell values
let zoneByDay = 
    restingRecordsFrame
    |> Frame.pivotTable
        (fun k r -> k.Date)
        (fun k r -> zoneNameByHr (r.GetAs<float>("HR")))
        (fun frame -> (frame?Seconds 
            |> Series.values 
            |> Seq.sum) / 60.0)
    |> Frame.sliceCols zoneNames

And plot the table as a stepped area chart:

let zoneOptions title vAxisTitle =
    Options(
            title = title,
            isStacked = true, // percent
            connectSteps = true,
            areaOpacity = 0.7,
            chartArea = ChartArea(width = "90%"),
            colors = (zoneByDay.Columns.Keys 
              |> Seq.map zoneColorByName 
              |> Array.ofSeq), 
            width = 1280, 
            vAxis = Axis(title = vAxisTitle),
            legend = Legend(position = "bottom"))
zoneByDay
    |> Chart.SteppedArea 
    |> Chart.WithOptions(
        zoneOptions "Minutes per HR zone and day" 
        "minutes")

record types

A problem with the above chart is that I don’t have the same number of readings every day since I don’t wear it all hours and also exercise less some days.

We can get a table with the total logged resting HR minutes per day.

let zoneByDayTotal = 
    restingRecordsFrame?Seconds 
    |> Series.groupInto (fun k r -> k.Date) 
      (fun _ r -> (r |> Series.values |> Seq.sum) / 60.0)

Then apply it to our previous values to get them in % of the daily totals:

let zoneByDayPercentage = zoneByDay / zoneByDayTotal * 100

Deedle allow mathematical operations to be applied to series similar to matrix operations. Deedle automatically aligns the series and applies the operation on corresponding elements.

Zone minutes in % per day:

zoneByDayPercentage
    |> Chart.SteppedArea 
    |> Chart.WithOptions(
        zoneOptions "% Minutes per HR zone and day" 
        "% minutes")

record types

Voila! We are DONE!!!

Lets build a JS-Framework from Scratch

In 2011 we decided to build our own JS-framework, WIDGET-JS. I would probably not have done that today with all options available.

But it is not that complicated to do and it would be interesting to re-do it in ES6/ES20015. Let´s reimplement the core of Widget-JS for fun!

We want to be able to write code like this:

html.h2('Todo')
html.ul(
  todos.map(todo => html.li(
    html.checkbox({ 
      checked : todo.isDone(), 
      click: () => todo.complete() }),
    html.span(todo.description())
  )
)

My post is more or less a transcript on how to get to this result: https://gist.github.com/henrikwallstrom/12669c3865c9b90f07b3.

ES6/ES2015

ES20015 features we will use:

  • Let declarations for block scoped variables. let x = {}

  • Destructuring assignment for eg. assignments let { width, height } = getDimensions() and function parameters function fun({name, age})

  • Spread syntax for functions that take multiple arguments when we have eg. an array. fun(...someArray);

  • Rest parameters for variable number of arguments to functions. function fun(id, …args) {}

  • Arrow functions (a, b) => a + b

We will not use the new class syntax mainly because of personal taste and that it still have issues with how “this” is bound. Instead we will use factory functions and closures. If you are interested in why people consider the new classes bad go to eg. https://github.com/joshburgess/not-awesome-es6-classes

Rendering DSL

Lets start with a htmlBuilder. Maybe it should be named domBuilder since we will not generate HTML. We will append DOM elements.

const htmlBuilder = (rootElement) => {
 let that = {};
 
 that.tag = (tagName) => {
    var tag = createElement(tagName);
    rootElement.appendChild(tag);
 };
 
 let createElement = (tagName) => 
    document.createElement(tagName);
 
 return Object.freeze(that); 
};

Our builder take a rootElement as argument and appends child nodes to it from tag names. Now we can create tags like this:

let html = htmlBuilder(document.querySelector(BODY))
html.tag(HR);
html.tag(BR);

Wohoo!! We can create horizontal rulers, forced line-breaks and tags that don’t need children or attributes. Maybe not that awesome? We need to be able to nest elements:

html.tag(H1, 
  html.tag(span, Hello), 
   world
);

Let’s create a htmlTag for that. Our `htmlTag? will wrap a DOM element and expose methods to append children or mutate the element.

const htmlTag = (element) => {
 let that = {};
 
 that.element = () => element;
 
 that.appendTag = (aTag) => {
   appendToElement(aTag.element());
   return that;
 }
  
 that.appendToTag = (aTag) => aTag.appendTag(that);
 
 that.append = (children) => {
   for(var obj of children) {
     append(obj);
   } 
   return that;
 };

 let append = (object) => {
   if (typeof object === string) {
      appendString(object);
   } else if (typeof object === object &&
     object.appendToTag) {
     object.appendToTag(that); // double dispatch
   } else {
     appendToElement(object);
   } 
 };
 
 let appendString = (str) =>
   element.appendChild(document.createTextNode(str));
 
 let appendToElement = (childElement) => {
   if (element.canHaveChildren !== false) {
     element.appendChild(childElement);
   } else {
     element.text = element.text + childElement.innerHTML;
   }
 }
 
 return Object.freeze(that);
};

We now have a tag that we can append text, elements and other tags on. Also anything that implements the appendToTag interface. Let’s update our builder to use tags:

const htmlBuilder = (rootElement) => {
 let that = {};
 
 let root = htmlTag(rootElement);
 
 that.tag = (tagName, children) => {
   var tag = htmlTag(createElement(tagName));
   tag.append(children)
   root.append(tag);
   return tag;
 }
 
 let createElement = (tagName) => 
    document.createElement(tagName);
 
 return Object.freeze(that); 
};

We can now nest tags:

html.tag(H1, 
  html.tag(span, Hello), 
   world
);

It is starting to look like our rendering DSL. But we want to write html.h1() instead of html.tag(‘h1’). So lets define all the common tags:

const tags = ('a abbr acronym address area wbr' +
 ... all the other tags 
).split(' ');

And add them to our builder:

const htmlBuilder = (rootElement) => {
 ....
 tags.forEach(tagName => {
   that[tagName] = (children) => 
    that.tag(tagName, children);
 });
 ... 
};

And now we have our Rendering DSL.

html.h1(
  html.span(Hello), 
   world
);

But we need to be able to add attributes, eg. by allowing an object with key/values to be passed as an argument like this:

html.h1(
  html.span({style: background-color: yellow},Hello),
   world
);

We can do that by assuming an object without appendToTag to hold attributes:

} else if (typeof object === object) {
  that.attr(object); // assume attributes if none of above
} else {

And add a that.attr that set attributes for all key/value-pairs:

that.attr = (object) => {
  Object.keys(object).forEach(key => 
element.setAttribute(key, object[key]));
  return that;
};

Voila!

html.h1(
  html.span({style: background-color: yellow},Hello),
   world
);

Let´s attach functions!

that.attr = (object) => {
  Object.keys(object).forEach(key => {
    if(typeof object[key] === function) {
      that.on(key, object[key]);
    } else {
      element.setAttribute(key, object[key]);
    }
  });
  return that;
};

that.on = function (eventType, callback) {
  element.addEventListener(eventType, callback);
  return that;
};

Now we can listen for eg. clicks

html.h1({click: () => alert(HEY!)}, 
  html.span({style: background-color: yellow},Hello), 
   world
);

jQuery or no jQuery

It’s easy to add support for jQuery. Just add one line to htmlTag:

that.asJQuery = () => jQuery(element);

Use it like:

let name = html.tag(input);
html.button({click: () => 
    alert(HEY  + name.asJQuery().val())}, 
    Click);

But you don’t like jQuery? You might not need it http://youmightnotneedjquery.com/

Just add what you need to htmlTag

that.show = () => element.style.display = ‘’;
that.hide = () => element.style.display = none;
that.empty = () => element.innerHTML = ‘’;
that.remove = () => element.parentNode.removeChild(element);
that.html = (value) => value === undefined ?
 element.innerHTML :
 element.innerHTML = value;
that.text = (value) => value === undefined ? 
 element.textContent :
 element.textContent = value;

Stateful components aka widgets

We already have what we need to create a stateful component that keep its state and re-render itself when state change. Eg. this counter widget with two buttons to increase/decrease the count 

let counterWidget = () => {
 let that = {};
 let count = 0;
 let id = 'counter';
 
 // Render a wrapper
 that.renderOn = (html) => html.div({id: id},
   that.renderContentOn);

 // and render some content in the wrapper
 that.renderContentOn = (html) => {
   html.span(‘’ + count);
   html.button({click: () => 
    { count++; that.update();}}, +);
   html.button({click: () => 
    { count  ; that.update();}}, -)
 };
 
 that.update = () => {
   // re-render content and replace wrapper
   let rootElement = document.querySelector(‘#’ + id);
   var html = htmlBuilder(rootElement);
   html.root().empty();
   that.renderContentOn(html);
 };
 

And we have a reusable counter widget that we can create instances of like this:

// create instance of our counter
var counter = counterWidget(); 

// render on BODY
let bodyElement = document.querySelector('BODY');
counter.appendTo(htmlBuilder(element));

But we don’t want to write all of that boilerplate code for every widget. Let’s extract the update mechanism and common code into a widget class:

let widget = ({id, content, root}) => {
 id = id || widget + idGenerator()
 content = content || ((html) => {});
 
 let that = {};
 
 that.id = () => id;
 
 that.rootElement = () => 
    document.querySelector(‘#’ + that.id());
 
 that.root = () => htmlTag(that.rootElement());
 
 that.isAttached = () => that.rootElement() !== null;
 
 that.appendTo = (element) => 
    that.renderOn(htmlBuilder(element));
 
 that.appendToTag = (tag) => 
    that.renderOn(htmlBuilder(tag));
 
 that.replace = (element) => {
   var html = htmlBuilder(element);
   html.root.empty();
   renderOn(canvas);
 };
 
 that.renderRootOn = (html) => 
    html.tag(widget).attr({id: id});
 
 that.renderOn = (html) => 
    that.renderRootOn(html)
    .append(content);
 
 that.update = () => {
    if (!that.isAttached()) {
        return;
    }

    // Re-render
    var html = htmlBuilder();
    that.renderOn(html);
    return that;
  };

  return that;
};

Now we can write our counter widget as:

let counterWidget = () => {
  let that = widget({});
  let count = 0;
 
  that.renderContentOn = (html) => {
    html.span(‘’ + count);
    html.button({click: () => 
        { count++; that.update();}}, +);
    html.button({click: () => 
        { count  ; that.update();}}, -)
  };

  return Object.freeze(that);
};

Composable components

Reusing and composing widgets gives us the real power. Since widget have the method that.appendToTag widgets can be appended to tags.

let widget = ({id, content, root}) => {
  ...
 that.appendToTag = (tag) => that.renderOn(htmlBuilder(tag));
  ...
};

Eg. to make an unordered list of users we do:

html.ul(
  users.map(user => html.li(userWidget(user)))
);

ES6 or not?

I wish JavaScript would have had all of these nice ES6 features from the start. It make the code a lot more compact and clean. But I’m not sure it is worth the extra transpile step. I think we will wait until mainstream browsers support ES6 fully. Then we still need transpillation for older browsers but can develop without transpilation.

I doubt anyone will read all of this. If you did why not leave a comment as well :)

Contributed Ansible IIS modules

Ansible + IIS

We use Ansible at Waulter to automate our infrastructure. We also host our servers on Azure and use IIS on Windows as our application/web server. Unfortunately, there was no support for IIS in Ansible out of the box.

So, during my Christmas vacation I decided to fix that by writing modules to manage IIS web sites, web applications, virtual directories, application pools and web bindings.

They are now part of Ansible and can be found in the module index:

They are more or less wrappers around the Web Server (IIS) Administration Cmdlets in Windows PowerShell.