As with many aspects of the tidyverse, its non-standard evaluation (NSE) implementation is not something entirely new, but built on top of base R. What makes this one so challenging to get your mind around, is that the Honorable Doctor Sir Lord General and friends brought concepts to the realm of the mortals that many of us had no, or only a vague, understanding of. Earlier, I gave an overview of the most common actions in tidy eval. Although appreciated by many, it left me unsatisfied, because it made clear to me I did not really understand NSE. Neither in base R, nor in tidy eval. Therefore, I bit the bullet and really studied it for a few evenings. Starting with base R NSE, and later learning what tidy eval actually adds to it. I decided to share the things I learned in this, rather lengthy, blog. I think it captures the essentials in NSE, although it surely is incomplete and might be even erronous at places. Still, I hope you find it worthwhile and it will help you understand NSE better and apply it with more confidence.

My approach was listing a number of terms and study them one by one. Mainly consulting Advanced R and the R Language Definition. For tidy eval I leaned heavily on the Programming with dplyr vignette and the function documentations. This is also how this blog post is built. We are hopping from term to term, to we see how they relate. You will find references to the sources in the text, in case you want to read more about a topic.

Note this is v3, in which the overscope gets mentioned and quosures are more elaborated on. To read earlier versions see here.

Base R, non-standard evaluation

`expression`

In standard evaluation R is like a child that receives candy from his grandmother and puts it in his mouth immediately. Every input is evaluated right away. An expression is some R code that is ready to be evaluated, but is not evaluated yet. Rather it is captured and saved for later. Think of it as the child’s father telling he can’t have the candy until they get home. Expressions break down in four categories; constants (like 4, TRUE), names (variable names referring to an object), calls (letting functions do a calculation) and pairlists. We will only look at names and calls here. Constants are equal to their value when captured in an expression, if you want to learn about pairlists see Adv-R. The act of capturing the expression instead of evaluating it, is called quoting. This is done by the quote() function.

library(tidyverse)
quote(x) %>% class()

## [1] "name"

quote(mean(1:10)) %>% class()

## [1] "call"

Confusingly, there is also the class expression, created by the parse() and expression() functions. You should think of these as lists of names and calls. Those will not be furtherly discussed here. When I use the term expression I am referring to the concept of R code that still needs to be evaluated, not objects of class expression.

`name`

When creating an object in R, you are binding the name of the object to a value. This binding of value and name is done in an environment. In the following, the name x gets associated with the value 50. Since we did not create a specific environment, this is a binding in the global environment.

x <- 50

Normally, we give R the object name to retrieve the corresponding value. However, when we save the objects name as an expression, it is not evaluated but stored as name object.

exp_x <- quote(x)
eval(exp_x)

## [1] 50

This way we can build a request for later. The variable requested for doesn’t even have to exist at creation time. (Like granny having no candy herself, but telling the kid that he can have candy when he gets back at his parent’s place).

eval_me <- quote(y)
eval(eval_me)

## Error in eval(eval_me): object 'y' not found

y <- "I am ready"
eval(eval_me)

## [1] "I am ready"

If we want to create a name from a string, for instance because you are creating on function output, you can use as.name().

as.name("x")

## x

And finally to make matters nice and unclear, a name is also called a symbol and the function as.symbol() does the same as as.name(). Perfect, we have a good idea about quoting variable names and how to retrieve their value later. Now, lets call some functions.

`call`

When we delay the evaluation of a function call, we arrive at the second subcategory of expressions: the call. The function to be called, with the names of the objects used for the arguments, are stored until further notice.

wait_for_it <- quote(x + y)
x <- 3; y <- 8
eval(wait_for_it)

## [1] 11

Note that + is a function, like every action that happens in R. If we want to return an expression as a string, we can apply deparse() on it. This allows us to do stuff like:

print_func <- function(expr){
  paste("The value of", deparse(expr), "is", eval(expr))
}
print_func(wait_for_it)

## [1] "The value of x + y is 11"

print_func(quote(log(42) %>% round(1)))

## [1] "The value of log(42) %>% round(1) is 3.7"

print_func(quote(x))

## [1] "The value of x is 3"

When the expression is a name, we print the name and the value of the object associated with the name. When it is a call, we print the function call and the evaluation of it.

`environment` and `closure`

In the last block we used a function in which we applied NSE. No coincidence, NSE and functions are a strong and natural pair. With NSE we can create powerful and user-friendly functions, like the ones in ggplot2 and dplyr. We need to elaborate on environments and closures here. I told you that an object is the binding of a name and a value in an environment. When starting an R session, you are in the global environment Adv-R. All objects created live happily in the global.

z <- 25

A function creates a new environment, objects of the same name as objects in the global can live here with different values bound to them.

z_func <- function() {
  z <- 12
  z
}
z_func()

## [1] 12

## [1] 25

The z_func did not change the global environment, but created an object in its own environment. Now functions are of a type called a closure.

typeof(z_func)

## [1] "closure"

They are called this way because they enclose their environment. At creation they have a look around in the environment in which they are created and capture all the names and values that are available there. They don’t just know the names of the objects in their own environment, but also in the environment in which they were created Adv-R.

Keep the concept of a closure in mind, we will revisit it.

`substitute` and `promise`

With the knowledge gained in the above we can start and try to write our own NSE functions. Lets make a function that adds a column to a data frame that is the square of a column it already contains.

add_squared <- function(x, col_name) {
  new_colname      <- paste0(deparse(col_name), "_sq")
  x[, new_colname] <- x[, deparse(col_name)]^2
  x
}
add_squared(mtcars, quote(cyl)) %>% head(1)

##           mpg cyl disp  hp drat   wt  qsec vs am gear carb cyl_sq
## Mazda RX4  21   6  160 110  3.9 2.62 16.46  0  1    4    4     36

You might say, “that is not too convenient, I still need to quote the col_name myself”. Well, you are very right, it would be more helpful if the function did the quoting for you. Unfortunately placing quote(col_name) inside the function body is of no use. quote() makes a literal quote of its input. So it would make the name col_name here each time it was called, no matter the value that was given to the argument, rather than quoting the value that was provided to this argument.

Here we need substitute(). This will lookup all the object names provided to it, and if it finds a value for that name, it will substitute the name for its value Adv-R. Lets do a filter function to demonstrate.

my_filt <- function(x, filt_cond) {
  filt_cond_q  <- substitute(filt_cond)
  rows_to_keep <- eval(filt_cond_q, x)
  x[rows_to_keep, ]
}
my_filt(mtcars, mpg == 21)

##               mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4      21   6  160 110  3.9 2.620 16.46  0  1    4    4
## Mazda RX4 Wag  21   6  160 110  3.9 2.875 17.02  0  1    4    4

Yeah, that works. But, wait a minute. How does eval() now know that mpg is a column in x? We provided x to the eval function, but how does this work? Well, the data frame x was provided to the envir argument of eval(). A data frame, thus, is a valid environment in which we can evaluate expressions. mpg lives in x, so the evaluation of filt_cond_q here gives the desired result.

When you think about it a little longer, NSE is only possible when function arguments are not evaluated directly. If the function was the inpatient kid that wanted to put filt_cond in its mouth right away, it would have failed to find an object with the name mpg in the global environment. When the function is called, a provided arguments is stored in a promise. The promise of the argument contains the value of the argument, but also an expression of the argument. The function does not bother about the value of the promise, until the function argument is actually used in the function. The substitute() function does only enter the expression part of the promise. In the my_filt() example, the promise associated with the x argument will have the actual data frame belonging to the object mtcars as its value, and the name mtcars as its expression. In the second and third line of the function, the value of this argument is accessed. The promise associated with the filt_cond argument, however, does not have a value. But it does have a call as its expression. As soon as we use this argument, the function would fail. But we don’t. With substitute() we only access the expression of the promise R lang.

`formula` and `overscoping`

Before we move to tidy eval there is one more concept we have to elaborate on, the formula. Probably you have used formulas a lot, but did you ever think about how odd they are? Take the following example

mod <- lm(vs ~ mpg + cyl, data = mtcars)

No R user would have trouble reading the above, but picture yourself coming from another programming language and stumbling upon it. It as an example of a domain specific language (DSL). DSLs exploit R’s NSE possibilities by giving alternative meaning to the language in specific contexts. Other examples are ggplot2 and dplyr. Just like functions, do formulas enclose the environment they are created in. Meaning that when the formula is evaluated later in a different environment, it can still access all the object that lived in its original environment.

Formulas, thus, can find variables in multiple environments. Like so:

not_in_df <- rnorm(32)
lm(disp ~ not_in_df, data = mtcars)

## 
## Call:
## lm(formula = disp ~ not_in_df, data = mtcars)
## 
## Coefficients:
## (Intercept)    not_in_df  
##      229.00        22.26

But what if the name exists in both environments, which one prevails?

cyl <- "this would throw an error"
lm(disp ~ cyl, data = mtcars)

## 
## Call:
## lm(formula = disp ~ cyl, data = mtcars)
## 
## Coefficients:
## (Intercept)          cyl  
##      -156.6         62.6

Thus, the data environment is evaluated before the enclosed environment, we say the data environment does overscope.

These are, to my understanding, the core elements of NSE in base R. If you don’t care about tidy eval you can stop reading here and try to build your own NSE functions. Thanks for making it this far.

`tidy evaluation`

There are two key additions of tidy eval to base R NSE. It uses quasiquotation and it introduces a new type of quoted object, called a quosure. Let’s find out about them one by one.

`quasiquotation`

We now know that in normal quotation the expression is captured to be evaluated later, rather than swallowed right away. Quasiquotation enables the user to swallow parts of the expression right away, while quoting the rest. We can quote the following simple function.

quote(z - x + 4)

## z - x + 4

Say we know the value of x already at the moment of quoting. How can we let the second part to be evaluated right away and quote z - the result of this evaluation? In other words how do we unquote the x + 4 part? In base R this is not going to happen, but with tidy eval this can be done.

x <- 4
rlang::expr(z - !!x + 4)

## z - (4 + 4)

rlang::expr(z - !!x + 4) %>% class()

## [1] "call"

Everything after the !! (bang bang) is unquoted. If we do not use unquoting, there is no reason to use rlang::expr() instead of quote(). They have the exact same result. There is also a tidy eval equivalent for substitute(), namely enexpr().

Now the appeal of functions that have implemented quasiquotation is that all the advantages of easy-to-use NSE interfaces remain. At the same time they enable the user to pack the functions that already quote, in custom-made wrappers. Example please! Something I do often is creating a frequency table of the values of a variable in a data frame. I want this in a function with the data frame and column name as arguments. Wrapping dplyr functions in the following way:

freq_table <- function(x, col) {
  col_q   <- rlang::enexpr(col)
  total_n <- x %>% nrow()
  x %>% group_by(!!col_q) %>% summarise(freq = n() / total_n)
}
mtcars %>% freq_table(cyl)

## # A tibble: 3 x 2
##     cyl    freq
##   <dbl>   <dbl>
## 1     4 0.34375
## 2     6 0.21875
## 3     8 0.43750

mtcars %>% freq_table(vs)

## # A tibble: 2 x 2
##      vs   freq
##   <dbl>  <dbl>
## 1     0 0.5625
## 2     1 0.4375

So the functions that use tidy eval, like those in dplyr, automatically quote their input. That is what enables you to type away and get results as quickly as you can when doing data analysis. However if you want to write programs around them you have to take care of two steps. First, quote the argument that is going to be evaluated by the functions used. If we don’t do this our wrapper function would fail because we have provided a name or call that cannot be found in the environment the function is called from. Second, since the dplyr functions quote their input themselves, we have to unquote the quoted arguments in these functions. If we don’t do this the dplyr function will quote the variable name rather than its content.

`quosure`

Very nice, that quasiquoting. Now what’s up with quosures? From their name you might guess they are hybrids of quotes and closures. We have seen that combination before when we looked at formulas. If we look at quosures, we will see that they are a subclass of the formula, besides being a class of their own.

quo(z) %>% class()

## [1] "quosure" "formula"

quo(z) %>% rlang::is_expr()

## [1] TRUE

Quosures are one-sided fomulas, capturing their environment, but not indicating a modelling relationship. By the way, we’ve seen the quo() function in action. This literally quotes its input, just like quote() and rlang::expr() do. The quosure equivalent of substitute() and enexpr() is enquo().

Just like names, calls can be converted to a quosure too.

quo(2 + 2) %>% class()

## [1] "quosure" "formula"

Note that quosures don’t make a lower level distinction between calls and names. Every expression becomes a quosure.

When is capturing of the environment by the expression actually useful? When the quosure is created in one environment and evaluated in another. This typically happens when it is created in a function and evaluated in the global environment or another function.

In base R NSE a function can evaluate a quoted argument, it can quote a bare statement, it can even return an expression. What it cannot do however, is giving the returned expression memory of the variables that were present at creation.

base_NSE_example <- function(some_arg) {
  some_var <- 10
  quote(some_var + some_arg)
}
base_NSE_example(4) %>% eval()

## Error in eval(.): object 'some_var' not found

The quosure is not memoryless, it will retrieve the values that were present at creation.

tidy_eval_example <- function(some_arg) {
  some_var <- 10
  quo(some_var + some_arg)
}
tidy_eval_example(4) %>% rlang::eval_tidy()

## [1] 14

Note that we do need to apply eval_tidy() instead of eval() to make use of the memory of the quosure.

Now you might say, if we just use substitute() instead of quote() in the base_NSE_example(), it would return 14 too. You are right, if we just capture the environment and evaluate it later, there is not really a point in saving the whole bunch to evaluate later to retrieve the values. However, we don’t have to use the value in the enclosed environment when applying eval_tidy() to a quosure. With the data argument we can provide a value for a name in the quosure, that overscopes the enclosed environment for the provided name(s). Example. Using tidy_eval_example() with some_arg = 4, will give some_arg = 4 and some_var = 10 in the enclosed environment. Now, we can overrule one or both values in eval_tidy().

tidy_eval_example(4) %>% 
  rlang::eval_tidy(list(some_var = 42))

## [1] 46

tidy_eval_example(4) %>% 
  rlang::eval_tidy(list(some_var = 42, some_arg = -1))

## [1] 41

The value in the enclosed environment is thus carried with the quosure wherever it goes, but does not have to be used. We can also evaluate (a part) of the quosure in a different environment.

So, we can provide a data argument to overscope the enclosed environment. But what about the enclosed environment versus the environment where the quosure is evaluated. Which one overscopes? Then answer is:

x <- "the enclosed env"
x_quoted <- quo(x)
eval_x <- function(q){
  x <- "the calling env"
  rlang::eval_tidy(q)
}
eval_x(x_quoted)

## [1] "the enclosed env"

How do base R NSE and tidy eval play together?

So tidy eval is build on top of base R NSE and the two can even work together. We have seen that in quasiquotation the parts to be unquoted don’t have to be quosures, we can also unquote base objects like calls and names.

using_base_r_in_tidy_eval <- function(x, col) {
  col_q <- substitute(col)
  x %>% select(!!col_q)
}
mtcars %>% using_base_r_in_tidy_eval(cyl) %>% head(1)

##           cyl
## Mazda RX4   6

If want to use the quasiquotation of tidy eval, but prefer base R quotation, you can combine the two. It does not work the other way around. Since quosures are a new kid on the block, eval() does not know how to unquote them and will throw an error. Familiar expression objects created with tidy eval can be evaluated with eval(), since the objects do not differ from the ones created with base R functions.

all.equal(quote(some_name), rlang::expr(some_name))

## [1] TRUE

all.equal(quote(x + 5), rlang::expr(x+ 5))

## [1] TRUE

The only difference between these functions is on capture, objects after capture are of base types.

Thank You

I took you along my NSE learning path, thank you for making it all the way through. If there is anything you think is incomplete or incorrect, let me know! This document is a living thing. You would do me and everybody who uses it as a reference a great favor by correcting it. The blog is maintained here, do a PR or send an email.

That's so Random

A playground for data analysis and R programming.

Non-standard evaluation, how tidy eval builds on base R

Base R, non-standard evaluation

`expression`

`name`

`call`

`environment` and `closure`

`substitute` and `promise`

`formula` and `overscoping`

`tidy evaluation`

`quasiquotation`

`quosure`

How do base R NSE and tidy eval play together?

Thank You

Base R, non-standard evaluation

expression

name

call

environment and closure

substitute and promise

formula and overscoping

tidy evaluation

quasiquotation

quosure

How do base R NSE and tidy eval play together?

Thank You

`expression`

`name`

`call`

`environment` and `closure`

`substitute` and `promise`

`formula` and `overscoping`

`tidy evaluation`

`quasiquotation`

`quosure`