# Upgrading to Varnish 5.2

## Varnish statistics and logging

There are extensive changes under the hood with respect to statistics counters, but these should all be transparent at the user-level.

## varnishd parameters

The vsm_space and `cli_buffer` parameters are now deprecated and ignored. They will be removed in a future major release.

The updated shared memory implementation manages space automatically, so it no longer needs vsm_space. Memory for the CLI command buffer is now dynamically allocated.

We have updated the documentation for send_timeout, idle_send_timeout, timeout_idle and ban_cutoff.

Added the debug bit `vmod_so_keep`, see debug and the notes about changes for developers below.

## Changes to VCL

We have added a few new variables and clarified some matters. VCL written for Varnish 5.1 should run without changes on 5.2.

### Consistent symbol names

VCL symbols originate from various parts of Varnish: there are built-in variables, subroutines, functions, and the free-form headers. Symbols may live in a namespace denoted by the `'.'` (dot) character as in `req.http.Cache-Control`. When you create a VCL label, a new symbol becomes available, named after the label. Storage backends always have a name, even if you don’t specify one, and they can also be accessed in VCL: for example `storage.Transient`.

Because headers and VCL names could contain dashes, while subroutines or VMOD objects couldn’t, this created an inconsistency. All symbols follow the same rules now and must follow the same (case-insensitive) pattern: `[a-z][a-z0-9_-]*`.

You can now write code like:

    sub my-sub {
        new my-obj = my_vmod.my_constuctor(storage.my-store);
    }

    sub vcl_init {
        call my-sub;
    }

As you may notice in the example above, it is not possible yet to have dashes in a vmod symbol.

Long storage backend names used to be truncated due to a limitation in the VSC subsystem, this is no longer the case.

### VCL variables

#### `req.hash` and `bereq.hash`

Added `req.hash` and `bereq.hash`, which contain the hash value computed by Varnish for cache lookup in the current transaction, to be used in client or backend context, respectively. Their data type is BLOB, and they contain the raw binary hash.

You can use vmod_blob to work with the hashes:

    import blob;

    sub vcl_backend_fetch {
        # Send the transaction hash to the backend as a hex string
        set bereq.http.Hash = blob.encode(HEX, blob=bereq.hash);
    }

    sub vcl_deliver {
        # Send the hash in a response header as a base64 string
        set resp.http.Hash = blob.encode(BASE64, blob=req.hash);
    }

#### `server.identity`

If the `-i` option is not set in the invocation of `varnishd`, then `server.identity` is set to the host name (as returned by `gethostname(3)`). Previously, `server.identity` defaulted to the value of the `-n` option (or the default instance name if `-n` was not set). See varnishd.

#### `bereq.is_bgfetch`

Added `bereq.is_bgfetch`, which is readable in backend contexts, and is true if the fetch takes place in the background. That is, it is true if Varnish found a response in the cache whose TTL was expired, but was still in grace time. Varnish returns the stale cached response to the client, and initiates the background fetch to refresh the cache object.

#### `req.backend_hint`

We have clarified what happens to `req.backend_hint` on a client restart – it gets reset to the default backend. So you might want to make sure that the backend hint gets set the way you want in that situation.

### vmod_std

Added BOOL file_exists(STRING path).

### New VMODs in the standard distribution

See vmod_blob, vmod_purge and vmod_vtc. Read about them in New VMODs in the standard distribution.

### Bans

We have clarified the interpretation of a ban when a comparison in the ban expression is attempted against an unset field, see ban(STRING) in VCL.

## Other changes

- `varnishd(1)`:

  - The total size of the shared memory space for logs and counters no longer needs to be configured explicitly and therefore the second subargument to `-l` is now ignored.
  - The default value of `server.identity` when the `-i` option is not set has been changed as noted above.
  - Also, `-i` no longer determines the `ident` field used by `syslog(3)`; now Varnish is always identified by the string `varnishd` in the syslog.
  - On a system that supports `setproctitle(3)`, the Varnish management process will appear in the output of `ps(1)` as `Varnish-Mgt`, and the child process as `Varnish-Child`. If the `-i` option has been set, then these strings in the ps output are followed by `-i` and the identity string set by the option.
  - The `-f` option for a VCL source file now honors the `vcl_path` parameter if a relative file name is used, see varnishd and vcl_path.
  - The `-a` option can now take a name, for example `-a admin=127.0.0.1:88` to identify an address used for administrative requests but not regular client traffic. Otherwise, a default name is selected for the listen address (`a0`, `a1` and so forth). Endpoint names appear in the log output, as noted below, and may become accessible in VCL in the future.

- `varnishstat(1)`:

  - In curses mode, the top two lines showing uptimes for the management and child processes show the text `Not Running` if one or both of the processes are down.
  - The interpretation of multiple `-f` options in the command line has changed slightly, see varnishstat.
  - The `type` and `ident` fields have been removed from the XML and JSON output formats, see varnishstat.
  - The `MAIN.s_req` statistic has been removed, as it was identical to `MAIN.client_req`.
  - Added the counter `req_dropped`. Similar to `sess_dropped`, this is the number of times an HTTP/2 stream was refused because the internal queue is full. See varnish-counters and thread_queue_limit.

- `varnishlog(1)`:

  - The `Hit`, `HitMiss` and `HitPass` log records grew an additional field with the remaining TTL of the object at the time of the lookup. While this should greatly help troubleshooting, it might break tools relying on those records to get the VXID of the object hit during lookup.

    Instead of using `Hit`, such tools should now use `Hit[1]`, and the same applies to `HitMiss` and `HitPass`.

    The `Hit` record also grew two more fields for the grace and keep periods. This should again be useful for troubleshooting.

    See VSL.

  - The `SessOpen` log record displays the name of the listen address instead of the endpoint in its 3rd field.

    See VSL.

  - The output format of `VCL_trace` log records, which appear if you have switched on the `VCL_trace` flag in the VSL mask, has changed to include the VCL configuration name. See VSL and vsl_mask.

- `varnishtest(1)` and `vtc(7)`:

  - When varnishtest is invoked with `-L` or `-l`, Varnish instances started by a test do not clean up their copies of VMOD shared objects when they stop. See the note about `vmod_so_keep` below.
  - Added the feature switch `ignore_unknown_macro` for test cases, see VTC.

- `varnishncsa(1)`

  - Field specifiers (such as the 1 in `Hit[1]`) are now limited to to 255, see varnishncsa.

- The `-N` command-line option, which was previously available for `varnishlog(1)`, `varnishstat(1)`, `varnishncsa(1)` and `varnishhist(1)`, is not compatible with the changed internal logging API, and has been retired.

- Changes for developers:

  - The VSM and VSC APIs for shared memory and statistics have changed, and may necessitate changes in client applications, see VSM/VSC API changes.
  - Added the `$ABI` directive for VMOD vcc declarations, see $ABI [strict|vrt].
  - There have been some minor changes in the VRT API, which may be used for VMODs and client apps, see VRT API changes.
  - The VUT API (for Varnish UTilities), which facilitates the development of client apps, is now publicly available, see Added VUT API.
  - The debug bit `vmod_so_keep` instructs Varnish not to clean up its copies of VMOD shared objects when it stops. This makes it possible for VMOD authors to load their code into a debugger after a varnishd crash. See debug.

eof