ViSiElSe Step by Step

Nastasia Fouret & Elodie Garnier

2019-10-21

Few words about ViSiElse

ViSiElse is a graphical tool designed to visualize behavioral observations over time. Based on raw time data extracted from video recorded sessions of experimental observations, ViSiElse grants a global overview of a process by combining the visualization of multiple actions timestamps for all participants in a single graph. Individuals and/or group behavior can easily be assessed. For example, ViSiElse was developed to visualize the actions performed by caregivers during simulated medical procedures in order to get insights of their behavior. Supplementary features allow users to further inspect their data by adding summary statistics (mean, standard deviation, quantile or statistical test) and/or time constraints to check the accuracy of the realized actions.

A step by step example

Let’s say we want to look at people making coffee. On the basis of this simple example that everybody could experience, we are presenting how to use ViSiElse package in order to visualize the behavior of people making coffee.

The process : Making coffee

The process underlying making coffee corresponds to a 5-steps procedure that can be defined as follows:

All those actions are defined as punctual; meaning the actions are short and don’t last long enough compared to the time scale of the process. On the opposite, long actions have measurable duration (we will give an example of a long action later).

The Dataset

In this example, let’s say we filmed 10 people making coffee and we recorded the time when they did each punctual action. We then saved all the time data in a data.frame and we added one column to identify the subjects id column.

Here is what the dataset would look like.

coffee <- c(58, 11, 5, 53, 53, 59, 24, 59, 46, 20)
fill_coffee <- c(162, 57, 103, 154, 165, 132, 74, 107, 104,  93)
fill_water <- c(66, 92, 54, 78, 74, 114, 91, 129, 71, 56)
push_B <- c(74, 99, 62, 84, 83, 120, 95, 129, 80, 63)
drink <- c(472, 176, 475, 283, 265, 207, 234, 184, 490, 520)
X <- data.frame(id = seq(1,10), coffee, fill_coffee, fill_water, push_B, drink)
head(X)
##   id coffee fill_coffee fill_water push_B drink
## 1  1     58         162         66     74   472
## 2  2     11          57         92     99   176
## 3  3      5         103         54     62   475
## 4  4     53         154         78     84   283
## 5  5     53         165         74     83   265
## 6  6     59         132        114    120   207

Each value corresponds to the time elapsed (here, in seconds) between a starting point and the moment of the action. In our coffee example, the starting point is when we began to film he experiment. The first subject took the coffee capsule at 58s to start making his coffee.

First plot

library(ViSiElse)
visi1 <- visielse(X)

On the graph, actions are represented one under the other. The execution times of punctual actions are distributed along the x-axis and are divided into intervals. A drawn rectangle means that at least one subject performed the action in this interval of time. The color intensity of the plot is proportional to the number of subjects who performed the action during the time interval.

By default :

The parameter : pixel

For punctual actions, time is divided into intervals which size is given by the parameter pixel (by default a pixel is 20 seconds). Data are aggregated into those intervals. For punctual actions, the graphic produced by ViSiElSe shows how many people have done an action in each pixel (interval) of time. When the parameter pixel is too large, the subjects are too much aggregated to allow interpretation. Otherwise, when pixel is too small, the subjects are not enough aggregated to allow interpretation.

Changing pixel parameter: smaller

visi1 <- visielse(X, pixel = 5)

Changing pixel parameter: larger

visi1 <- visielse(X, pixel = 80)

Statistics

To help understand the plotted behavior, ViSiElse offers to plot summary statistics. By default, ViSiElse displays median and quartiles but it is also possible to plot mean and standard deviation or nothing at all.

Plot mean and standard deviation

visi1 <- visielse(X, informer = "mean")

Do not plot statistics

visi1 <- visielse(X, informer = NULL)

The ViSibook

The ViSibook is an R-object containing informations on the studied behavior whereas the dataset contains its time realizations. The ViSibook is the structure of the behavioral process: list of actions composing it, time constraints, type of actions… When it is not manually defined (the ViSibook is an optional argument), the function visielse directly computes the ViSibook associated with the specified dataset. Then, we can extract the ViSibook from a visielse object.

Extraction of the ViSibook from the data

visi1 <- visielse(X, informer = NULL, doplot = FALSE) # automatically creates a ViSibook
book <- visi1@book # Exctract the ViSibook
plot(book)

The actions from the ViSibook are ordered as they appear in the dataset X.

Structure of a ViSibook

A ViSibook is a class of object, and it must follow structural rules. Basically, it is a data.frame with particular columns. The minimum structure for a ViSibook must give for each action its name (without special characters but " _ " ), its label, its type (punctual or long), its order in the behavioral process, and two other information “deb” and “fin” that are only required for long actions.

More details about when and how to use “deb” and “fin” will be given later.

book
##      vars          label         typeA showorder deb fin
## [1,] "coffee"      "coffee"      "p"   "1"       NA  NA 
## [2,] "fill_coffee" "fill_coffee" "p"   "2"       NA  NA 
## [3,] "fill_water"  "fill_water"  "p"   "3"       NA  NA 
## [4,] "push_B"      "push_B"      "p"   "4"       NA  NA 
## [5,] "drink"       "drink"       "p"   "5"       NA  NA

Changing labels

When displaying results or graphs, it is often preferred to use label instead of names of variables. By default, labels are the names of variables which are the dataset colnames, but labels can be easily changed.

book[,2] <- c("Taking the coffee",
"Fill the machine with coffee",
"Fill the tank with water",
"Push the Button",
"Drink the coffee")
book
##      vars          label                          typeA showorder deb fin
## [1,] "coffee"      "Taking the coffee"            "p"   "1"       NA  NA 
## [2,] "fill_coffee" "Fill the machine with coffee" "p"   "2"       NA  NA 
## [3,] "fill_water"  "Fill the tank with water"     "p"   "3"       NA  NA 
## [4,] "push_B"      "Push the Button"              "p"   "4"       NA  NA 
## [5,] "drink"       "Drink the coffee"             "p"   "5"       NA  NA
plot(book)

visi1 <- visielse(X, book = book, is.ViSibook = TRUE, informer = NULL)

Order of Actions in the process

As explained above, the order is given by the dataset column order. In the ViSibook, the column “showorder” save the order and can be manually changed. The preferred order is the order by which actions are supposed to happen. Changing the “showorder” column will adjust the plotted action order but will not affect calculations.

Change the order

book[,4]<- c(5,1,2,4,3)
plot(book)

visi1 <- visielse(X, book = book, is.ViSibook = TRUE)

Long Actions

Until now, we only considered punctual actions. However ViSiElse distinguishes two different types of actions: punctual and long. The actions called punctual are brief actions defined as time points, like “Push the button”, they do not last long compared to the time scale (i.e. a one-second-lasting action in an one-hour-lasting process).

The actions labeled as long (or delays) are the ones defined by a duration (i.e. a few-seconds-lasting action in a one-minute-lasting process). Delays or long actions can be helpful to understand a behavior as we can observed subjects performing the same action for different duration. Long action are delimited by a beginning punctual action and an ending one. To define a long action, there is no need to change the dataset, only to add a new line in the ViSibook. The column “typeA” in the ViSibook specifies if the action is punctual (“p”) or long (“l”).

Adding a long Action

Let’s add a long action measuring the duration between filling the machine with the coffee capsule and pushing the button. To do so, we convert the ViSibook into a data.frame and add a new line for this long action.

visi1 <- visielse( X )

book <- ConvertFromViSibook(visi1@book) # Convert book into data.frame
add_delay <- c("delay_coffee_push", "Preparation", "l", "6", "coffee", "push_B") # Create the long action (variable name, label, action type, action order, long action starting point, long action ending point)
book[6, ] <- add_delay # Add the long action at the end of the ViSibook
book
##                vars       label typeA showorder    deb    fin
## 1            coffee      coffee     p         1   <NA>   <NA>
## 2       fill_coffee fill_coffee     p         2   <NA>   <NA>
## 3        fill_water  fill_water     p         3   <NA>   <NA>
## 4            push_B      push_B     p         4   <NA>   <NA>
## 5             drink       drink     p         5   <NA>   <NA>
## 6 delay_coffee_push Preparation     l         6 coffee push_B

The columns “deb” and “fin” define the two punctual actions that delimit the long one, “deb” is the beginning (here “coffee”) and “fin” the ending (here “push_B”).

ViSiElse representation of long actions

ViSiElse displays each subject long action with an horizontal. Lines are sorted by their starting time and are proportional to the duration of the long action.

visi2 <- visielse(X = X, book = book, informer = NULL)

Green & Black zones

Green and black zone are here to help visualize time boundaries and assess if the actions are done in time. Indeed, some actions can be restricted by time guidelines. For example, if we were making tea instead of coffee and if we were following the instructions on the tea box, the infusion time should respect the instructions (let’s say between 3 and 5min) to make the perfect tea. Green zones represent time obligation i.e. when actions should be achieved while black zones set time interdiction i.e. when actions should not occur. In the tea example, the green zone is between 3 and 5 minutes and the black zones are before 3min and after 5min.

Punctual actions

For punctual actions, ViSiElse can display one green zone and two black zones. To create those zones, we only have to add the following new columns in the ViSibook:

  • Green Zone: GZDeb and GZFin.
  • Black zone 1: BZBeforeDeb and BZBeforeFin.
  • Black zone 2: BZAfterDeb and BZAfterFin.

Green zone

Let’s go back to our coffee example and let’s say “fill_coffee” in the machine should be done between 1min and 2min. So for this action, we defined the time limit of the green zone in a new column of the ViSibook; GZDeb for the beginning of the green zone and GZFin for its ending. We then use the new ViSibook to create the ViSiElse graph.

book$GZDeb <- c(NA, 60, NA, NA, NA, NA) # New column in the ViSibook, the 2nd action "fill_coffee" should be done after 60s
book$GZFin <- c(NA, 120, NA,NA, NA, NA) # New column in the ViSibook, the 2nd action "fill_coffee" should be done before 120s
visi2 <- visielse(X = X, book = book, informer = NULL)

As the coffee dataset is in seconds, all time limits must also be in seconds. If the data were in hours, the green zone should have been defined by limit values in hours.

1 black zone

For punctual actions, 1 or 2 black zones can be defined. For the example, “fill_coffee” in the machine should not be done before 30s. We set the beginning (BZBeforeDeb) and the ending (BZBeforeFin) of the back zone in new columns of the ViSibook.

book$GZDeb <- c(NA, NA, NA, NA, NA, NA) # No more green zone
book$GZFin <- c(NA, NA, NA, NA, NA, NA) # No more green zone
book$BZBeforeDeb <- c(NA, 0, NA, NA, NA, NA) # New column in the ViSibook, the 2nd action "fill_coffee" should not be done between 0 and 30s
book$BZBeforeFin <- c(NA, 30, NA, NA, NA, NA) # New column in the ViSibook, the 2nd action "fill_coffee" should not be done between 0 and 30s
visi2 <- visielse(X = X, book = book, informer = NULL)

2 black zones and 1 green zone

We can add another black zone after the green one. Let’s say “fill_coffee” in the machine should also not be done after 3min. Then we add two new columns (BZAfterDeb and BZAfterFin) to set the second black zone limits in the ViSibook.

book$GZDeb <- c(NA, 60, NA, NA, NA, NA) # The green zone is back !
book$GZFin <- c(NA, 120, NA, NA, NA, NA) # The green zone is back !
book$BZBeforeDeb <- c(NA, 0, NA, NA, NA, NA) # 1st black zone before the green zone: starting point
book$BZBeforeFin <- c(NA, 30, NA, NA, NA, NA) # 1st black zone before the green zone: ending point
book$BZAfterDeb <- c(NA, 180, NA, NA, NA, NA) # New column in the ViSibook for the 2nd black zone, "fill_coffee" should not be done after 3min
book$BZAfterFin <- c(NA, Inf, NA, NA, NA, NA) # New column in the ViSibook for the 2nd black zone, "fill_coffee" should not be done after 3min
visi2 <- visielse(X = X, book = book, informer = NULL)

The ending of the second black zone is set to “Inf” meaning it should be plotted until the end of the process.

Long Action

For long actions, ViSiElSe only allows one black zone (actually displayed by a darker blue color) but this one black zone can be of two types:

  • A deadline not to cross : “span”
  • A duration not to exceed : “time”

No green zone can be plotted for long actions.

“time” black zone

For the sake of the example, let’s say that the preparation of the coffee should be done before the first minute. We now have to create new columns in the ViSibook. The first one, BZLong, defines the time limit and the second one, BZLtype defines the type of the black zone: here a deadline not to cross (“time”).

visi1 <- visielse(X,  doplot = FALSE)
book <- ConvertFromViSibook(visi1@book) # Convert book into data.frame
add_delay <- c("delay_coffee_push", "Preparation", "l", "6", "coffee", "push_B") # We use again the same long action as before
book[6,] <- add_delay 
book$BZLong <- c(rep(NA, 5), 60) # The long action should be done before 60s
book$BZLtype <- c(rep(NA, 5), "time") # The long action time limit is a deadline
visi1 <- visielse(X, book = book, informer = NULL)

Subjects that are still preparing their coffee after 1min have a dark blue at the end of their line (from 60s to the end of the action).

“span” black zone

Now, we decide that the preparation should not last more than 30s. BZLong still is the time limit and this time, BZLtype is set to “span” as it represents a duration not to exceed.

book$BZLong <- c(rep(NA, 5), 30) # The long action should last maximum 30s
book$BZLtype <- c(rep(NA, 5), "span") # The long action time limit is a duration not to exceed
visi1 <- visielse(X, book = book, informer = NULL)

Subjects that are preparing their coffee for more than 30s have a dark blue at the end of their line (from the 31st second of the action to its end).

Group

It might be useful to distinguish groups in the graph. ViSiElse offers to plot two groups maximum. You can do it by:

ViSiElse proposes three methods to plot groups:

Method : Cut

Each groups is represented one under the other.

group <- c("group2", "group1", "group2", "group1", "group1", "group2",
           "group1", "group1", "group1", "group2") # group definition for the 10 subjects
visi1 <- visielse(X, group = group, book = book, informer = NULL, method = "cut")

Method : Join

Groups are spatially mixed but they are represented by different colors.

group <- c("group2", "group1", "group2", "group1", "group1", "group2",
           "group1", "group1", "group1", "group2") # group definition for the 10 subjects
visi1 <- visielse(X, group = group, book = book, informer = NULL, method = "join")

Method : Within

When using the method “Within”, all data are plotted together in blue and one of the groups is plotted again underneath in pink. This group is defined by the parameter grwithin.

group <- c("group2", "group1", "group2", "group1", "group1", "group2",
           "group1", "group1", "group1", "group2") # group definition for the 10 subjects
visi1 <- visielse(X, group = group, book = book, informer = NULL, method = "within", grwithin = "group1")

More info

If you have any questions, you can reach us by email at re2simlab@gmail.com

For more documentations, you can see the help in R:

help("visielse")

Or you can read the second vignette of the package: ViSiElse Paper Walkthrough.