# Upgrading to Varnish 5.1

## varnishd command-line options

If you have to change anything at all for version 5.1, then most likely the command-line options for varnishd in your start scripts, because we have tightened restrictions on which options may be used together. This has served mainly to clarify the use of options for testing purposes, for example using `varnishd -C` to check a VCL source for syntactic correctness. We have also added some new options.

The details are given in varnishd, but here’s a summary:

- Added `-I clifile` to run CLI commands at startup, before the worker process starts. See Startup CLI command file.
- More than one `-f` option is now permitted, to pre-load VCL instances at startup. The last of these becomes the “boot” instance that is is active at startup.
- Either `-b` or one or more `-f` options must be specified, but not both, and they cannot both be left out, unless `-d` is used to start varnishd in debugging mode. If the empty string is specified as the sole `-f` option, then varnishd starts without starting the worker process, and the management process will accept CLI commands.
- Added `-?` to print the usage message, which is only printed for this option.
- Added the `-x` option to print certain kinds of documentation and exit. When `-x` is used, it must be the only option.
- Only one of `-F` or `-d` may be used, and neither of these can be used with `-C`.
- Added the `workuser` parameter to the `-j` option.

## varnishd parameters

- The size of the shared memory log is now limited to 4G-1b (4294967295 bytes). This places upper bounds on the `-l` command-line option and on the `vsl_space` and `vsm_space` parameters.
- Added `clock_step`, `thread_pool_reserve` and `ban_cutoff` (see clock_step, thread_pool_reserve, ban_cutoff).
- `thread_pool_stack` is no longer considered experimental, and is more extensively documented, see thread_pool_stack.
- `thread_queue_limit` only applies to queued client requests, see thread_queue_limit.
- `vcl_dir` and `vmod_dir` are deprecated and will be removed from a future release, use `vcl_path` and `vmod_path` instead (see vcl_path, vmod_path).
- All parameters are defined on every platform, including those that that are not functional on every platform. Most of these involve features of the TCP stack, such as `tcp_keepalive_intvl`, `tcp_keepalive_probes`, `accept_filter` and `tcp_fastopen`. The unavailability of a parameter is documented in the output of the `param.show` command. Setting such a parameter is not an error, but has no effect.

## Changes to VCL

VCL written for Varnish 5.0 will very likely work without changes in version 5.1. We have added some new elements and capabilities to the language (which you might like to start using), clarified some matters, and deprecated some little-used language elements.

### Type conversions

We have put some thought to the interpretation of the `+` and `-` operators for various combinations of operands with differing data types, many of which count as corner cases (what does it mean, for example, to subtract a string from an IP address?). Recall that `+` denotes addition for numeric operands, and string concatenation for string operands; operands may be converted to strings and concatenated, if a string is expected and there is no sensible numeric interpretation.

The semantics have not changed in nearly all cases, but the error messages for illegal combinations of operands have improved. Most importantly, we have taken the time to review these cases, so this will be the way VCL works going forward.

To summarize:

- If both operands of `+` or `-` are one of BYTES, DURATION, INT or REAL, then the result has the same data type, with the obvious numeric interpretation. If such an expression is evaluated in a context that expects a STRING (for example for assignment to a header), then the arithmetic is done first, and the result it converted to a STRING.
- INTs and REALs can be added or subtracted to yield a REAL.
- A DURATION can be added to or subtracted from a TIME to yield a TIME.
- No other combinations of operand types are legal with `-`.
- When a `+` expression is evaluated in a STRING context, then for all other combinations of operand data types, the operands are converted to STRINGs and concatenated.
- If a STRING is not expected for the `+` expression, then no other combination of data types is legal.

Other notes on data types:

- When `bereq.backend` is set to a director, then it returns an actual backend on subsequent reads if the director resolves to a backend immediately, or the director otherwise. If `bereq.backend` was set to a director, then `beresp.backend` references the backend to which it was set for the fetch. When either of these is used in string context, it returns the name of the director or of the resolved backend.

- Comparisons between symbols of type BACKEND now work properly:

      if (bereq.backend == foo.backend()) {
          # do something specific to the foo backends
      }

- DURATION types may be used in boolean contexts, and are evaluated as false when the duration is less than or equal to zero, true otherwise.

- INT, DURATION and REAL values can now be negative.

### Response codes

Response codes 1000 or greater may now be set in VCL internally. `resp.status` is delivered modulo 1000 in client responses.

### IP address comparison

IP addresses can now be compared for equality:

    if (client.ip == remote.ip) {
      call do_if_equal;
    }

The objects are equal if they designate equal socket addresses, not including the port number. IPv6 addresses are always unequal to IPv4 addresses (the comparison cannot consider v4-mapped IPv6 addresses).

### The STEVEDORE type and storage objects

The data type STEVEDORE for storage backends is now available in VCL and for VMODs. Storage objects with names of the form `storage.SNAME` will exist in a VCL instance, where `SNAME` is the name of a storage backend provided with the `varnishd` command-line option `-s`. If no `-s` option is given, then `storage.s0` denotes the default storage. The object `storage.Transient` always exists, designating transient storage. See Storage backends, and the notes about `beresp.storage` and `req.storage` below.

### All VCL subroutines (except `vcl_fini`)

- Added `return(fail)` to immediately terminate VCL processing. In all cases but `vcl_synth`, control is directed to `vcl_synth` with `resp.status` and `resp.reason` set to 503 and “VCL failed”, respectively. `vcl_synth` is aborted on `return(fail)`. `vcl_fini` is executed when a VCL instance in unloaded (enters the COLD state) and has no failure condition.
- VCL failure is invoked on any attempt to set one of the fields in the the first line of a request or response to the empty string, such as `req.url`, `req.proto`, `resp.reason` and so forth.

### Client-side VCL subroutines

- `req.ttl` is deprecated, see VCL.

### vcl_recv

- Added `req.storage`, which tells Varnish which storage backend to use if you choose to save the request body (see BOOL cache_req_body(BYTES size)).
- `return(vcl(LABEL))` may not be called after a restart. It can only be called from the active VCL instance.

### vcl_backend_response

- Added `return(pass(DURATION))` to set an object to hit-for-pass, see Hit-For-Pass has returned.
- The object `beresp.storage` of type STEVEDORE should now be used to set a storage backend; `beresp.storage_hint` is deprecated and will be removed in a future release. Setting `beresp.storage_hint` to a valid storage will set `beresp.storage` as well. If the storage is invalid, `beresp.storage` is left untouched.

For the case where multiple storage backends have been defined with the `-s` command-line option for varnishd, but none is explicitly set in `vcl_backend_response`, storage selection and the use of the nuke limit has been reworked (see nuke_limit). Previously, a storage backend was tried first with a nuke limit of 0, and retried on failure with the limit configured as the `-p` parameter `nuke_limit`. When no storage was specified, Varnish went through every one in round-robin order with a nuke limit of 0 before retrying.

Now `beresp.storage` is initialized with a storage backend before `vcl_backend_response` executes, and the storage set in `beresp.storage` after its execution will be used. The configured nuke limit is used in all cases.

### vmod_std

- Added `std.getenv()`, see STRING getenv(STRING name).
- Added `std.late_100_continue()`, see VOID late_100_continue(BOOL late).

## Other changes

- The storage backend type umem, long in disuse, has been retired.
- `varnishstat(1)`:
  - Added the `cache_hitmiss` stat to count hits on hit-for-miss objects.
  - The `cache_hitpass` stat now only counts hits on hit-for-pass objects.
  - `fetch_failed` is incremented for any kind of fetch failure – when there is a failure after `return(deliver)` from `vcl_backend_response`, or when control is diverted to `vcl_backend_error`.
  - Added the `n_test_gunzip` stat, which is incremented when Varnish verifies a compressed response from a backend – this operation was previously counted together with `n_gunzip`.
  - Added the `bans_lurker_obj_killed_cutoff` stat to count the number of objects killed by the ban lurker to keep the number of bans below `ban_cutoff`.
- `varnishlog(1)`:
  - Hits on hit-for-miss and hit-for-pass objects are logged with the `HitMiss` and `HitPass` tags, respectively. In each case, the log payload is the VXID of the previous transaction in which the object was saved in the cache (as with `Hit`).
  - An entry with the `TTL` tag and the prefix `HFP` is logged to record the duration set for hit-for-pass objects.
  - Added `vxid` as a lefthand side token for VSL queries, allowing for queries that search for transaction IDs in the log. See vsl-query.
- `varnishncsa(1)`:
  - Clarified the meaning of the `%r` formatter, see NOTES in varnishncsa.
  - Clarified the meaning of the `%{X}i` and `%{X}o` formatters when the header X appears more than once (the last occurrence is is used).
- `varnishtest(1)`:
  - Added the `setenv` and `write_body` commands, see VTC.
  - `-reason` replaces `-msg` to set the reason string for a response (default “OK”).
  - Added `-cliexpect` to match expected CLI responses to regular expressions.
  - Added the `-match` operator for the `shell` command.
  - Added the `-hdrlen` operator to generate a header with a given name and length.
  - The `err_shell` command is deprecated, use `shell -err -expect` instead.
  - The `${bad_backend}` macro can now be used for a backend that is always down, which does not require a port definition (as does `${bad_ip}` in a backend definition).
  - `varnishtest` can be stopped with the `TERM`, `INT` of `KILL` signals, but not with `HUP`. These signals kill the process group, so that processes started by running tests are stopped.
- Added the `vtest.sh` tool to automate test builds, see Project tool improvements.