# vmod_std

## Varnish Standard Module

Manual section: | 3

### SYNOPSIS

    import std [from "path"] ;

    STRING toupper(STRING s)

    STRING tolower(STRING s)

    VOID set_ip_tos(INT tos)

    REAL random(REAL lo, REAL hi)

    VOID log(STRING s)

    VOID syslog(INT priority, STRING s)

    STRING fileread(STRING)

    BOOL file_exists(STRING path)

    VOID collect(HEADER hdr, STRING sep)

    DURATION duration(STRING s, DURATION fallback)

    INT integer(STRING s, INT fallback)

    IP ip(STRING s, IP fallback, BOOL resolve, STRING p)

    REAL real(STRING s, REAL fallback)

    INT real2integer(REAL r, INT fallback)

    TIME real2time(REAL r, TIME fallback)

    INT time2integer(TIME t, INT fallback)

    REAL time2real(TIME t, REAL fallback)

    BOOL healthy(BACKEND be)

    INT port(IP ip)

    VOID rollback(HTTP h)

    VOID timestamp(STRING s)

    STRING querysort(STRING)

    BOOL cache_req_body(BYTES size)

    STRING strstr(STRING s1, STRING s2)

    TIME time(STRING s, TIME fallback)

    STRING getenv(STRING name)

    VOID late_100_continue(BOOL late)

    BOOL syntax(REAL)

    BOOL fnmatch(STRING pattern, STRING subject, BOOL pathname, BOOL noescape, BOOL period)

### CONTENTS

- BOOL cache_req_body(BYTES size)
- VOID collect(HEADER hdr, STRING sep=”, “)
- DURATION duration(STRING s, DURATION fallback)
- BOOL file_exists(STRING path)
- STRING fileread(STRING)
- fnmatch(…)
- STRING getenv(STRING name)
- BOOL healthy(BACKEND be)
- INT integer(STRING s, INT fallback)
- IP ip(STRING s, IP fallback, BOOL resolve=1, STRING p=”80”)
- VOID late_100_continue(BOOL late)
- VOID log(STRING s)
- INT port(IP ip)
- STRING querysort(STRING)
- REAL random(REAL lo, REAL hi)
- REAL real(STRING s, REAL fallback)
- INT real2integer(REAL r, INT fallback)
- TIME real2time(REAL r, TIME fallback)
- VOID rollback(HTTP h)
- VOID set_ip_tos(INT tos)
- STRING strstr(STRING s1, STRING s2)
- BOOL syntax(REAL)
- VOID syslog(INT priority, STRING s)
- TIME time(STRING s, TIME fallback)
- INT time2integer(TIME t, INT fallback)
- REAL time2real(TIME t, REAL fallback)
- VOID timestamp(STRING s)
- STRING tolower(STRING s)
- STRING toupper(STRING s)

### DESCRIPTION

vmod_std contains basic functions which are part and parcel of Varnish, but which for reasons of architecture fit better in a VMOD.

One particular class of functions in vmod_std is the conversions functions which all have the form:

    TYPE type(STRING, TYPE)

These functions attempt to convert STRING to the TYPE, and if that fails, they return the second argument, which must have the given TYPE.

#### STRING toupper(STRING s)

Description
  Converts the string s to uppercase.
Example
  set beresp.http.scream = std.toupper(“yes!”);

#### STRING tolower(STRING s)

Description
  Converts the string s to lowercase.
Example
  set beresp.http.nice = std.tolower(“VerY”);

#### VOID set_ip_tos(INT tos)

Description
  Sets the IP type-of-service (TOS) field for the current session to tos. Silently ignored if the listen address is a Unix domain socket. Please note that the TOS field is not removed by the end of the request so probably want to set it on every request should you utilize it.
Example
  if (req.url ~ “^/slow/”) {
  std.set_ip_tos(0);
  }

#### REAL random(REAL lo, REAL hi)

Description
  Returns a random real number between lo and hi. This function uses the “testable” random generator in varnishd which enables determinstic tests to be run (See m00002.vtc). This function should not be used for cryptographic applications.
Example
  set beresp.http.random-number = std.random(1, 100);

#### VOID log(STRING s)

Description
  Logs the string s to the shared memory log, using VSL tag SLT_VCL_Log.
Example
  std.log(“Something fishy is going on with the vhost ” + req.http.host);

#### VOID syslog(INT priority, STRING s)

Description

  Logs the string s to syslog tagged with priority. priority is formed by ORing the facility and level values. See your system’s syslog.h file for possible values.

  Notice: Unlike VCL and other functions in the std vmod, this function will not fail VCL processing for workspace overflows: For an out of workspace condition, the `syslog()` function has no effect.

Example

  std.syslog(9, “Something is wrong”);

  This will send a message to syslog using LOG_USER | LOG_ALERT.

#### STRING fileread(STRING)

Description
  Reads a file and returns a string with the content. The result is cached indefinitely per filename.
Example
  synthetic(“Response was served by ” + std.fileread(“/etc/hostname”));

Consider that the entire contents of the file appear in the string that is returned, including newlines that may result in invalid headers if `std.fileread()` is used to form a header. In that case, you may need to modify the string, for example with `regsub()`:

    set beresp.http.served-by = regsub(std.fileread("/etc/hostname"), "\R$", "");

#### BOOL file_exists(STRING path)

Description
  Returns true if path or the file pointed to by path exists, false otherwise.
Example
  if (std.file_exists(“/etc/return_503”)) {
  return (synth(503, “Varnish is in maintenance”));
  }

#### VOID collect(HEADER hdr, STRING sep=”, “)

Description

  Collapses multiple hdr headers into one long header. The default separator sep is the standard comma separator to use when collapsing headers, with an additional whitespace for pretty printing.

  Care should be taken when collapsing headers. In particular collapsing Set-Cookie will lead to unexpected results on the browser side.

Examples
  std.collect(req.http.accept);
  std.collect(req.http.cookie, “; “);

#### DURATION duration(STRING s, DURATION fallback)

Description
  Converts the string s to seconds. s must be quantified with ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks) or y (years) units. If conversion fails, fallback will be returned.
Example
  set beresp.ttl = std.duration(“1w”, 3600s);

#### INT integer(STRING s, INT fallback)

Description
  Converts the string s to an integer. If conversion fails, fallback will be returned.
Example
  if (std.integer(req.http.foo, 0) > 5) {
  …
  }

#### IP ip(STRING s, IP fallback, BOOL resolve=1, STRING p=”80”)

Description

  Converts the string s to the first IP number returned by the system library function getaddrinfo(3). If conversion fails, fallback will be returned or VCL failure will happen.

  The IP address includes a port number that can be found with `std.port()` that defaults to 80. The default port can be set to a different value with the p argument. It will be overriden if s contains both an IP address and a port number or service name.

  When s contains both, the syntax is either `address:port` or `address port`. If the address is a numerical IPv6 address it must be enclosed between brackets, for example `[::1] 80` or `[::1]:http`. The fallback may also contain both an address and a port, but its default port is always 80.

  If resolve is false, getaddrinfo(3) is called using `AI_NUMERICHOST` and `AI_NUMERICSERV` to avoid network lookups depending on the system’s getaddrinfo(3) or nsswitch configuration. This makes “numerical” IP strings and services cheaper to convert.

Example
  if (std.ip(req.http.X-forwarded-for, “0.0.0.0”) ~ my_acl) {
  …
  }

#### REAL real(STRING s, REAL fallback)

Description
  Converts the string s to a real. If conversion fails, fallback will be returned.
Example
  if (std.real(req.http.foo, 0.0) > 5.5) {
  …
  }

#### INT real2integer(REAL r, INT fallback)

Description
  Rounds the real r to the nearest integer, but round halfway cases away from zero (see round(3)). If conversion fails, fallback will be returned.
Example
  set req.http.integer = std.real2integer(1140618699.00, 0); set req.http.posone = real2integer( 0.5, 0); # = 1.0 set req.http.negone = real2integer(-0.5, 0); # = -1.0

#### TIME real2time(REAL r, TIME fallback)

Description
  Rounds the real r to the nearest integer (see func_real2integer) and returns the corresponding time when interpreted as a unix epoch. If conversion fails, fallback will be returned.
Example
  set req.http.time = std.real2time(1140618699.00, now);

#### INT time2integer(TIME t, INT fallback)

Description
  Converts the time t to a integer. If conversion fails, fallback will be returned.
Example
  set req.http.int = std.time2integer(now, 0);

#### REAL time2real(TIME t, REAL fallback)

Description
  Converts the time t to a real. If conversion fails, fallback will be returned.
Example
  set req.http.real = std.time2real(now, 1.0);

#### BOOL healthy(BACKEND be)

Description
  Returns true if the backend be is healthy.

#### INT port(IP ip)

Description
  Returns the port number of the IP address ip. Always returns 0 for a `*.ip` variable whose value is `0.0.0.0` because the listen address is a Unix domain socket.

#### VOID rollback(HTTP h)

Description
  Restores the h HTTP headers to their original state.
Example
  std.rollback(bereq);

#### VOID timestamp(STRING s)

Description
  Introduces a timestamp in the log with the current time, using the string s as the label. This is useful to time the execution of lengthy VCL procedures, and makes the timestamps inserted automatically by Varnish more accurate.
Example
  std.timestamp(“curl-request”);

#### STRING querysort(STRING)

Description
  Sorts the query string for cache normalization purposes.
Example
  set req.url = std.querysort(req.url);

#### BOOL cache_req_body(BYTES size)

Description

  Caches the request body if it is smaller than size. Returns true if the body was cached, false otherwise.

  Normally the request body is not available after sending it to the backend. By caching it is possible to retry pass operations, e.g. POST and PUT.

Example
  if (std.cache_req_body(1KB)) {
  …
  }

#### STRING strstr(STRING s1, STRING s2)

Description

  Returns a string beginning at the first occurrence of the string s2 in the string s1, or an empty string if s2 is not found.

  Note that the comparison is case sensitive.

Example
  if (std.strstr(req.url, req.http.restrict)) {
  …
  }

  This will check if the content of req.http.restrict occurs anywhere in req.url.

#### TIME time(STRING s, TIME fallback)

Description

  Converts the string s to a time. If conversion fails, fallback will be returned.

  Supported formats:

  “Sun, 06 Nov 1994 08:49:37 GMT”
  “Sunday, 06-Nov-94 08:49:37 GMT”
  “Sun Nov 6 08:49:37 1994”
  “1994-11-06T08:49:37”
  “784111777.00”
  “784111777”
Example
  if (std.time(resp.http.last-modified, now) < now - 1w) {
  …
  }

#### STRING getenv(STRING name)

Description

  Return environment variable name or the empty string.

  See getenv(3)

Example
  set req.http.My-Env = std.getenv(“MY_ENV”);

#### VOID late_100_continue(BOOL late)

Description

  Controls when varnish reacts to an Expect: 100-continue client request header.

  Varnish always generates a 100 Continue response if requested by the client trough the Expect: 100-continue header when waiting for request body data.

  But, by default, the 100 Continue response is already generated immediately after vcl_recv returns to reduce latencies under the assumption that the request body will be read eventually.

  Calling std.late_100_continue(true) in vcl_recv will cause the 100 Continue response to only be sent when needed. This may cause additional latencies for processing request bodies, but is the correct behavior by strict interpretation of RFC7231.

  This function has no effect outside vcl_recv and after calling std.cache_req_body() or any other function consuming the request body.

Example
  vcl_recv {
  std.late_100_continue(true);
  if (req.method == “POST”) {
  std.late_100_continue(false);
  return (pass);
  }
  …
  }

#### BOOL syntax(REAL)

Description
  Returns the true if VCL version is at least REAL.

#### fnmatch(…)

    BOOL fnmatch(
       STRING pattern,
       STRING subject,
       BOOL pathname=1,
       BOOL noescape=0,
       BOOL period=0
    )

Description

  Shell-style pattern matching; returns true if subject matches pattern, where pattern may contain wildcard characters such as * or ?.

  The match is executed by the implementation of fnmatch(3) on your system. The rules for pattern matching on most systems include the following:

  - * matches any sequence of characters
  - ? matches a single character
  - a bracket expression such as [abc] or [!0-9] is interpreted as a character class according to the rules of basic regular expressions (not PCRE regexen), except that ! is used for character class negation instead of ^.

  If pathname is true, then the forward slash character / is only matched literally, and never matches *, ? or a bracket expression. Otherwise, / may match one of those patterns. By default, pathname is true.

  If noescape is true, then the backslash character \ is matched as an ordinary character. Otherwise, \ is an escape character, and matches the character that follows it in the pattern. For example, \ matches \ when noescape is true, and \ when false. By default, noescape is false.

  If period is true, then a leading period character . only matches literally, and never matches *, ? or a bracket expression. A period is leading if it is the first character in subject; if pathname is also true, then a period that immediately follows a / is also leading (as in “/.”). By default, period is false.

  fnmatch() invokes VCL failure and returns false if either of pattern or subject is NULL – for example, if an unset header is specified.

Examples
  # Matches URLs such as /foo/bar and /foo/baz
  if (std.fnmatch(“/foo/*”, req.url)) { … }
  # Matches URLs such as /foo/bar/baz and /foo/baz/quux
  if (std.fnmatch(“/foo/*/*”, bereq.url)) { … }
  # Matches /foo/bar/quux, but not /foo/bar/baz/quux
  if (std.fnmatch(“/foo/*/quux”, req.url)) { … }
  # Matches /foo/bar/quux and /foo/bar/baz/quux
  if (std.fnmatch(“/foo/*/quux”, req.url, pathname=false)) { … }
  # Matches /foo/bar, /foo/car and /foo/far
  if (std.fnmatch(“/foo/?ar”, req.url)) { … }
  # Matches /foo/ followed by a non-digit
  if (std.fnmatch(“/foo/[!0-9]”, req.url)) { … }

### SEE ALSO

- varnishd
- VSL
- fnmatch(3)

### COPYRIGHT

    Copyright (c) 2010-2017 Varnish Software AS
    All rights reserved.

    Author: Poul-Henning Kamp <phk@FreeBSD.org>

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:
    1. Redistributions of source code must retain the above copyright
       notice, this list of conditions and the following disclaimer.
    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.

# vmod_directors

## Varnish Directors Module

Manual section: | 3

### SYNOPSIS

    import directors [from "path"] ;

    new xround_robin = directors.round_robin()

       VOID xround_robin.add_backend(BACKEND)

       VOID xround_robin.remove_backend(BACKEND)

       BACKEND xround_robin.backend()

    new xfallback = directors.fallback(BOOL sticky)

       VOID xfallback.add_backend(BACKEND)

       VOID xfallback.remove_backend(BACKEND)

       BACKEND xfallback.backend()

    new xrandom = directors.random()

       VOID xrandom.add_backend(BACKEND, REAL)

       VOID xrandom.remove_backend(BACKEND)

       BACKEND xrandom.backend()

    new xhash = directors.hash()

       VOID xhash.add_backend(BACKEND, REAL)

       VOID xhash.remove_backend(BACKEND)

       BACKEND xhash.backend(STRING)

    new xshard = directors.shard()

       VOID xshard.set_warmup(REAL probability)

       VOID xshard.set_rampup(DURATION duration)

       VOID xshard.associate(BLOB param)

       BOOL xshard.add_backend(BACKEND backend, [STRING ident], [DURATION rampup], [REAL weight])

       BOOL xshard.remove_backend([BACKEND backend], [STRING ident])

       BOOL xshard.clear()

       BOOL xshard.reconfigure(INT replicas)

       INT xshard.key(STRING)

       BACKEND xshard.backend([ENUM by], [INT key], [BLOB key_blob], [INT alt], [REAL warmup], [BOOL rampup], [ENUM healthy], [BLOB param], [ENUM resolve])

       VOID xshard.debug(INT)

    new xshard_param = directors.shard_param()

       VOID xshard_param.clear()

       VOID xshard_param.set([ENUM by], [INT key], [BLOB key_blob], [INT alt], [REAL warmup], [BOOL rampup], [ENUM healthy])

       STRING xshard_param.get_by()

       INT xshard_param.get_key()

       INT xshard_param.get_alt()

       REAL xshard_param.get_warmup()

       BOOL xshard_param.get_rampup()

       STRING xshard_param.get_healthy()

       BLOB xshard_param.use()

### CONTENTS

- new xfallback = directors.fallback(BOOL sticky=0)
- VOID xfallback.add_backend(BACKEND)
- BACKEND xfallback.backend()
- VOID xfallback.remove_backend(BACKEND)
- new xhash = directors.hash()
- VOID xhash.add_backend(BACKEND, REAL)
- BACKEND xhash.backend(STRING)
- VOID xhash.remove_backend(BACKEND)
- new xrandom = directors.random()
- VOID xrandom.add_backend(BACKEND, REAL)
- BACKEND xrandom.backend()
- VOID xrandom.remove_backend(BACKEND)
- new xround_robin = directors.round_robin()
- VOID xround_robin.add_backend(BACKEND)
- BACKEND xround_robin.backend()
- VOID xround_robin.remove_backend(BACKEND)
- new xshard = directors.shard()
- shard.add_backend(…)
- VOID xshard.associate(BLOB param=0)
- shard.backend(…)
- BOOL xshard.clear()
- VOID xshard.debug(INT)
- INT xshard.key(STRING)
- BOOL xshard.reconfigure(INT replicas=67)
- shard.remove_backend(…)
- VOID xshard.set_rampup(DURATION duration=0)
- VOID xshard.set_warmup(REAL probability=0.0)
- new xshard_param = directors.shard_param()
- VOID xshard_param.clear()
- INT xshard_param.get_alt()
- STRING xshard_param.get_by()
- STRING xshard_param.get_healthy()
- INT xshard_param.get_key()
- BOOL xshard_param.get_rampup()
- REAL xshard_param.get_warmup()
- shard_param.set(…)
- BLOB xshard_param.use()

### DESCRIPTION

vmod_directors enables backend load balancing in Varnish.

The module implements load balancing techniques, and also serves as an example on how one could extend the load balancing capabilities of Varnish.

To enable load balancing you must import this vmod (directors).

Then you define your backends. Once you have the backends declared you can add them to a director. This happens in executed VCL code. If you want to emulate the previous behavior of Varnish 3.0 you can just initialize the directors in vcl_init, like this:

    sub vcl_init {
        new vdir = directors.round_robin();
        vdir.add_backend(backend1);
        vdir.add_backend(backend2);
    }

As you can see there is nothing keeping you from manipulating the directors elsewhere in VCL. So, you could have VCL code that would add more backends to a director when a certain URL is called.

Note that directors can use other directors as backends.

#### new xround_robin = directors.round_robin()

Description

  Create a round robin director.

  This director will pick backends in a round robin fashion.

Example
  new vdir = directors.round_robin();

#### VOID xround_robin.add_backend(BACKEND)

Description
  Add a backend to the round-robin director.
Example
  vdir.add_backend(backend1);

#### VOID xround_robin.remove_backend(BACKEND)

Description
  Remove a backend from the round-robin director.
Example
  vdir.remove_backend(backend1);

#### BACKEND xround_robin.backend()

Description
  Pick a backend from the director.
Example
  set req.backend_hint = vdir.backend();

#### new xfallback = directors.fallback(BOOL sticky=0)

Description

  Create a fallback director.

  A fallback director will try each of the added backends in turn, and return the first one that is healthy.

  If `sticky` is set to true, the director will keep using the healthy backend, even if a higher-priority backend becomes available. Once the whole backend list is exhausted, it’ll start over at the beginning.

Example
  new vdir = directors.fallback();

#### VOID xfallback.add_backend(BACKEND)

Description

  Add a backend to the director.

  Note that the order in which this is done matters for the fallback director.

Example
  vdir.add_backend(backend1);

#### VOID xfallback.remove_backend(BACKEND)

Description
  Remove a backend from the director.
Example
  vdir.remove_backend(backend1);

#### BACKEND xfallback.backend()

Description
  Pick a backend from the director.
Example
  set req.backend_hint = vdir.backend();

#### new xrandom = directors.random()

Description

  Create a random backend director.

  The random director distributes load over the backends using a weighted random probability distribution. The “testable” random generator in varnishd is used, which enables deterministic tests to be run (See: d00004.vtc).

Example
  new vdir = directors.random();

#### VOID xrandom.add_backend(BACKEND, REAL)

Description

  Add a backend to the director with a given weight.

  Each backend will receive approximately 100 * (weight / (sum(all_added_weights))) per cent of the traffic sent to this director.

Example
  # 2/3 to backend1, 1/3 to backend2.
  vdir.add_backend(backend1, 10.0);
  vdir.add_backend(backend2, 5.0);

#### VOID xrandom.remove_backend(BACKEND)

Description
  Remove a backend from the director.
Example
  vdir.remove_backend(backend1);

#### BACKEND xrandom.backend()

Description
  Pick a backend from the director.
Example
  set req.backend_hint = vdir.backend();

#### new xhash = directors.hash()

Description

  Create a hashing backend director.

  The director chooses the backend server by computing a hash/digest of the string given to .backend().

  Commonly used with `client.ip` or a session cookie to get sticky sessions.

Example
  new vdir = directors.hash();

#### VOID xhash.add_backend(BACKEND, REAL)

Description

  Add a backend to the director with a certain weight.

  Weight is used as in the random director. Recommended value is 1.0 unless you have special needs.

Example
  vdir.add_backend(backend1, 1.0);

#### VOID xhash.remove_backend(BACKEND)

Description
  Remove a backend from the director.
Example
  vdir.remove_backend(backend1);

#### BACKEND xhash.backend(STRING)

Description

  Pick a backend from the backend director.

  Use the string or list of strings provided to pick the backend.

Example
  # pick a backend based on the cookie header from the client
  set req.backend_hint = vdir.backend(req.http.cookie);

#### new xshard = directors.shard()

Create a shard director.

Note that the shard director needs to be configured using at least one `shard.add_backend()` call(s) followed by a `shard.reconfigure()` call before it can hand out backends.

_Note_ that due to various restrictions (documented below), it is recommended to use the shard director on the backend side.

##### Introduction

The shard director selects backends by a key, which can be provided directly or derived from strings. For the same key, the shard director will always return the same backend, unless the backend configuration or health state changes. Conversely, for differing keys, the shard director will likely choose different backends. In the default configuration, unhealthy backends are not selected.

The shard director resembles the hash director, but its main advantage is that, when the backend configuration or health states change, the association of keys to backends remains as stable as possible.

In addition, the rampup and warmup features can help to further improve user-perceived response times.

##### Sharding

This basic technique allows for numerous applications like optimizing backend server cache efficiency, Varnish clustering or persisting sessions to servers without keeping any state, and, in particular, without the need to synchronize state between nodes of a cluster of Varnish servers:

- Many applications use caches for data objects, so, in a cluster of application servers, requesting similar objects from the same server may help to optimize efficiency of such caches.

  For example, sharding by URL or some id component of the url has been shown to drastically improve the efficiency of many content management systems.

- As special case of the previous example, in clusters of Varnish servers without additional request distribution logic, each cache will need store all hot objects, so the effective cache size is approximately the smallest cache size of any server in the cluster.

  Sharding allows to segregate objects within the cluster such that each object is only cached on one of the servers (or on one primary and one backup, on a primary for long and others for short etc…). Effectively, this will lead to a cache size in the order of the sum of all individual caches, with the potential to drastically increase efficiency (scales by the number of servers).

- Another application is to implement persistence of backend requests, such that all requests sharing a certain criterion (such as an IP address or session ID) get forwarded to the same backend server.

When used with clusters of varnish servers, the shard director will, if otherwise configured equally, make the same decision on all servers. In other words, requests sharing a common criterion used as the shard key will be balanced onto the same backend server(s) no matter which Varnish server handles the request.

The drawbacks are:

- the distribution of requests depends on the number of requests per key and the uniformity of the distribution of key values. In short, while this technique may lead to much better efficiency overall, it may also lead to less good load balancing for specific cases.
- When a backend server becomes unavailable, every persistence technique has to reselect a new backend server, but this technique will also switch back to the preferred server once it becomes healthy again, so when used for persistence, it is generally less stable compared to stateful techniques (which would continue to use a selected server for as long as possible (or dictated by a TTL)).

##### Method

When `.reconfigure()` is called, a consistent hashing circular data structure gets built from the last 32 bits of SHA256 hash values of <ident><n> (default ident being the backend name) for each backend and for a running number n from 1 to replicas. Hashing creates the seemingly random order for placement of backends on the consistent hashing ring. When `.add_backend()` is called with a weight argument, replicas is scaled by that weight to add proportionally more copies of the that backend on the ring.

When `.backend()` is called, a load balancing key gets generated unless provided. The smallest hash value in the circle is looked up that is larger than the key (searching clockwise and wrapping around as necessary). The backend for this hash value is the preferred backend for the given key.

If a healthy backend is requested, the search is continued linearly on the ring as long as backends found are unhealthy or all backends have been checked. The order of these “alternative backends” on the ring is likely to differ for different keys. Alternative backends can also be selected explicitly.

On consistent hashing see:

- http://www8.org/w8-papers/2a-webserver/caching/paper2.html
- http://www.audioscrobbler.net/development/ketama/
- svn://svn.audioscrobbler.net/misc/ketama
- http://en.wikipedia.org/wiki/Consistent_hashing

##### Error Reporting

Failing methods should report errors to VSL with the Error tag, so when configuring the shard director, you are advised to check:

    varnishlog -I Error:^shard

#### VOID xshard.set_warmup(REAL probability=0.0)

Set the default warmup probability. See the warmup parameter of `shard.backend()`. If probability is 0.0 (default), warmup is disabled.

#### VOID xshard.set_rampup(DURATION duration=0)

Set the default rampup duration. See rampup parameter of shard.backend(). If duration is 0 (default), rampup is disabled.

#### VOID xshard.associate(BLOB param=0)

Associate a default obj_shard_param object or clear an association.

The value of the param argument must be a call to the func_shard_param.use method. No argument clears the association.

The association can be changed per backend request using the param argument of func_shard.backend.

#### shard.add_backend(…)

    BOOL xshard.add_backend(
          BACKEND backend,
          [STRING ident],
          [DURATION rampup],
          [REAL weight]
    )

Add a backend backend to the director.

ident: Optionally specify an identification string for this backend, which will be hashed by shard.reconfigure() to construct the consistent hashing ring. The identification string defaults to the backend name.

ident allows to add multiple instances of the same backend.

rampup: Optionally specify a specific rampup time for this backend. Otherwise, the per-director rampup time is used (see VOID xshard.set_rampup(DURATION duration=0)).

weight: Optionally specify a weight to scale the shard.reconfigure() replicas parameter. weight is limited to at least 1. Values above 10 probably do not make much sense. The effect of weight is also capped such that the total number of replicas does not exceed UINT32_MAX.

NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.

#### shard.remove_backend(…)

    BOOL xshard.remove_backend(
          [BACKEND backend=0],
          [STRING ident=0]
    )

Remove backend(s) from the director. Either backend or ident must be specified. ident removes a specific instance. If backend is given without ident, all instances of this backend are removed.

NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.

#### BOOL xshard.clear()

Remove all backends from the director.

NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.

#### BOOL xshard.reconfigure(INT replicas=67)

Reconfigure the consistent hashing ring to reflect backend changes.

This method must be called at least once before the director can be used.

#### INT xshard.key(STRING)

Convenience method to generate a sharding key for use with the key argument to the `shard.backend()` method by hashing the given string with SHA256.

To generate sharding keys using other hashes, use a custom vmod like vmod blobdigest with the key_blob argument of the `shard.backend()` method.

#### shard.backend(…)

    BACKEND xshard.backend(
          [ENUM {HASH, URL, KEY, BLOB} by=HASH],
          [INT key],
          [BLOB key_blob],
          [INT alt=0],
          [REAL warmup=-1],
          [BOOL rampup=1],
          [ENUM {CHOSEN, IGNORE, ALL} healthy=CHOSEN],
          [BLOB param],
          [ENUM {NOW, LAZY} resolve]
    )

Lookup a backend on the consistent hashing ring.

This documentation uses the notion of an order of backends for a particular shard key. This order is deterministic but seemingly random as determined by the consistent hashing algorithm and is likely to differ for different keys, depending on the number of backends and the number of replicas. In particular, the backend order referred to here is _not_ the order given when backends are added.

- by how to determine the sharding key

  - HASH:
    - when called in backend context and in `vcl_pipe {}`: Use the varnish hash value as set by `vcl_hash{}`
    - when called in client context other than `vcl_pipe {}`: hash `req.url`
  - URL: hash req.url / bereq.url
  - KEY: use the key argument
  - BLOB: use the key_blob argument

- key lookup key with by=KEY

  the shard.key() function may come handy to generate a sharding key from custom strings.

- key_blob lookup key with by=BLOB

  Currently, this uses the first 4 bytes from the given blob in network byte order (big endian), left-padded with zeros for blobs smaller than 4 bytes.

- alt alternative backend selection

  Select the alt-th alternative backend for the given key.

  This is particularly useful for retries / restarts due to backend errors: By setting alt=req.restarts or alt=bereq.retries with healthy=ALL, another server gets selected.

  The rampup and warmup features are only active for alt==0

- rampup slow start for servers which just went healthy

  If alt==0 and the chosen backend is in its rampup period, with a probability proportional to the fraction of time since the backup became healthy to the rampup period, return the next alternative backend, unless this is also in its rampup period.

  The default rampup interval can be set per shard director using the set_rampup() method or specifically per backend with the set_backend() method.

- warmup probabilistic alternative server selection

  possible values: -1, 0..1

  -1: use the warmup probability from the director definition

  Only used for alt==0: Sets the ratio of requests (0.0 to 1.0) that goes to the next alternate backend to warm it up when the preferred backend is healthy. Not active if any of the preferred or alternative backend are in rampup.

  warmup=0.5 is a convenient way to spread the load for each key over two backends under normal operating conditions.

- healthy

  - CHOSEN: Return a healthy backend if possible.

    For alt==0, return the first healthy backend or none.

    For alt > 0, ignore the health state of backends skipped for alternative backend selection, then return the next healthy backend. If this does not exist, return the last healthy backend of those skipped or none.

  - IGNORE: Completely ignore backend health state

    Just return the first or alt-th alternative backend, ignoring health state. Ignore rampup and warmup.

  - ALL: Check health state also for alternative backend selection

    For alt > 0, return the alt-th alternative backend of all those healthy, the last healthy backend found or none.

- resolve

  default: LAZY in `vcl_init{}`, NOW otherwise

  - `NOW`: look up a backend and return it.

    Can not be used in `vcl_init{}`.

  - `LAZY`: return an instance of this director for later backend resolution.

    `LAZY` mode is required for referencing shard director instances, for example as backends for other directors (director layering).

    In `vcl_init{}` and on the client side, `LAZY` mode can not be used with any other argument.

    On the backend side and in `vcl_pipe {}`, parameters from arguments or an associated parameter set affect the shard director instance for the backend request irrespective of where it is referenced.

- param

  Use or associate a parameter set. The value of the param argument must be a call to the func_shard_param.use method.

  default: as set by func_shard.associate or unset.

  - for `resolve=NOW` take parameter defaults from the obj_shard_param parameter set

  - for `resolve=LAZY` associate the obj_shard_param parameter set for this backend request

    Implementation notes for use of parameter sets with `resolve=LAZY`:

    - A param argument remains associated and any changes to the associated parameter set affect the sharding decision once the director resolves to an actual backend.
    - If other parameter arguments are also given, they have preference and are kept even if the parameter set given by the param argument is subsequently changed within the same backend request.
    - Each call to func_shard.backend overrides any previous call.

#### VOID xshard.debug(INT)

intentionally undocumented

#### new xshard_param = directors.shard_param()

Create a shard parameter set.

A parameter set allows for re-use of func_shard.backend arguments across many shard director instances and simplifies advanced use cases (e.g. shard director with custom parameters layered below other directors).

Parameter sets have two scopes:

- per-VCL scope defined in `vcl_init{}`
- per backend request scope

The per-VCL scope defines defaults for the per backend scope. Any changes to a parameter set in backend context and in `vcl_pipe {}` only affect the respective backend request.

Parameter sets can not be used in client context except for `vcl_pipe {}`.

#### VOID xshard_param.clear()

Reset the parameter set to default values as documented for func_shard.backend.

- in `vcl_init{}`, resets the parameter set default for this VCL in
- backend context and in `vcl_pipe {}`, resets the parameter set for this backend request to the VCL defaults

This method may not be used in client context other than `vcl_pipe {}`.

#### shard_param.set(…)

    VOID xshard_param.set(
          [ENUM {HASH, URL, KEY, BLOB} by],
          [INT key],
          [BLOB key_blob],
          [INT alt],
          [REAL warmup],
          [BOOL rampup],
          [ENUM {CHOSEN, IGNORE, ALL} healthy]
    )

Change the given parameters of a parameter set as documented for func_shard.backend.

- in `vcl_init{}`, changes the parameter set default for this VCL
- in backend context and in `vcl_pipe {}`, changes the parameter set for this backend request, keeping the defaults set for this VCL for unspecified arguments.

This method may not be used in client context other than `vcl_pipe {}`.

#### STRING xshard_param.get_by()

Get a string representation of the by enum argument which denotes how a shard director using this parameter object would derive the shard key. See func_shard.backend.

#### INT xshard_param.get_key()

Get the key which a shard director using this parameter object would use. See func_shard.backend.

#### INT xshard_param.get_alt()

Get the alt parameter which a shard director using this parameter object would use. See func_shard.backend.

#### REAL xshard_param.get_warmup()

Get the warmup parameter which a shard director using this parameter object would use. See func_shard.backend.

#### BOOL xshard_param.get_rampup()

Get the rampup parameter which a shard director using this parameter object would use. See func_shard.backend.

#### STRING xshard_param.get_healthy()

Get a string representation of the healthy enum argument which a shard director using this parameter object would use. See func_shard.backend.

#### BLOB xshard_param.use()

This method may only be used in backend context and in `vcl_pipe {}`.

For use with the param argument of func_shard.backend to associate this shard parameter set with a shard director.

### ACKNOWLEDGEMENTS

Development of a previous version of the shard director was partly sponsored by Deutsche Telekom AG - Products & Innovation.

Development of a previous version of the shard director was partly sponsored by BILD GmbH & Co KG.

### COPYRIGHT

    This document is licensed under the same licence as Varnish
    itself. See LICENCE for details.

    Copyright (c) 2013-2015 Varnish Software AS
    Copyright 2009-2018 UPLEX - Nils Goroll Systemoptimierung
    All rights reserved.

    Authors: Poul-Henning Kamp <phk@FreeBSD.org>
             Julian Wiesener <jw@uplex.de>
             Nils Goroll <slink@uplex.de>
             Geoffrey Simmons <geoff@uplex.de>

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:
    1. Redistributions of source code must retain the above copyright
       notice, this list of conditions and the following disclaimer.
    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.

# vmod_vtc

## Utility module for varnishtest

Manual section: | 3

### SYNOPSIS

    import vtc [from "path"] ;

    VOID barrier_sync(STRING addr, DURATION timeout)

    BACKEND no_backend()

    STEVEDORE no_stevedore()

    IP no_ip()

    VOID panic(STRING)

    VOID sleep(DURATION)

    VOID workspace_alloc(ENUM, INT size)

    BYTES workspace_reserve(ENUM, INT size)

    INT workspace_free(ENUM)

    VOID workspace_snapshot(ENUM)

    VOID workspace_reset(ENUM)

    BOOL workspace_overflowed(ENUM)

    VOID workspace_overflow(ENUM)

    INT typesize(STRING)

### CONTENTS

- VOID barrier_sync(STRING addr, DURATION timeout=0)
- BACKEND no_backend()
- IP no_ip()
- STEVEDORE no_stevedore()
- VOID panic(STRING)
- VOID sleep(DURATION)
- INT typesize(STRING)
- VOID workspace_alloc(ENUM, INT size)
- INT workspace_free(ENUM {client, backend, session, thread})
- VOID workspace_overflow(ENUM)
- BOOL workspace_overflowed(ENUM)
- BYTES workspace_reserve(ENUM, INT size)
- VOID workspace_reset(ENUM)
- VOID workspace_snapshot(ENUM)

### DESCRIPTION

The goal for this VMOD is to provide VCL users and VMOD authors means to test corner cases or reach certain conditions with varnishtest.

#### VOID barrier_sync(STRING addr, DURATION timeout=0)

When writing test cases, the most common pattern is to start a mock server instance, a Varnish instance, and spin up a mock client. Those entities run asynchronously, and others exist like background processes (`process`) or log readers (`logexpect`). While you can synchronize with individual entities and wait for their completion, you must use a barrier if you need to synchronize two or more entities, or wait until a certain point instead of completion.

Not only is it possible to synchronize between test entities, with the `barrier_sync` function you can even synchronize VCL code:

    sub vcl_recv {
        # wait for some barrier b1 to complete
        vtc.barrier_sync("${b1_sock}");
    }

If the function fails to synchronize with the barrier for some reason, or if it reaches the optional timeout, it fails the VCL transaction.

### MISCELLANEOUS

#### BACKEND no_backend()

Fails at backend selection.

#### STEVEDORE no_stevedore()

Fails at storage selection.

#### IP no_ip()

Returns a null IP address, not even a bogo_ip.

#### VOID panic(STRING)

It can be useful to crash the child process in order to test the robustness of a VMOD.

#### VOID sleep(DURATION)

Block the current worker thread.

### WORKSPACES

It can be useful to put a workspace in a given state when testing corner cases like resource exhaustion for a transaction, especially for VMOD development. All functions available allow to pick which workspace you need to tamper with, available values are `client`, `backend`, `session` and `thread`.

#### VOID workspace_alloc(ENUM, INT size)

    VOID workspace_alloc(
       ENUM {client, backend, session, thread},
       INT size
    )

Allocate and zero out memory from a workspace. A negative size will allocate as much as needed to leave that many bytes free. The actual allocation size may be higher to comply with memory alignment requirements of the CPU architecture. A failed allocation fails the transaction.

#### BYTES workspace_reserve(ENUM, INT size)

    BYTES workspace_reserve(
       ENUM {client, backend, session, thread},
       INT size
    )

Attempt to reserve size bytes and release the reservation right away. Return the size of the reservation.

See vtc.workspace_alloc() for semantics of the size argument.

#### INT workspace_free(ENUM {client, backend, session, thread})

Find how much unallocated space there is left in a workspace.

#### VOID workspace_snapshot(ENUM)

    VOID workspace_snapshot(ENUM {client, backend, session, thread})

Snapshot a workspace. Only one snapshot may be active at a time.

#### VOID workspace_reset(ENUM)

    VOID workspace_reset(ENUM {client, backend, session, thread})

Reset to the previous snapshot of a workspace, it must be the same workspace too.

#### BOOL workspace_overflowed(ENUM)

    BOOL workspace_overflowed(ENUM {client, backend, session, thread})

Find whether the workspace overflow mark is set or not.

#### VOID workspace_overflow(ENUM)

    VOID workspace_overflow(ENUM {client, backend, session, thread})

Mark a workspace as overflowed.

#### INT typesize(STRING)

Returns the size in bytes of a collection of C-datatypes:

- `'p'`: pointer
- `'i'`: `int`
- `'d'`: `double`
- `'f'`: `float`
- `'l'`: `long`
- `'s'`: `short`
- `'z'`: `size_t`
- `'o'`: `off_t`
- `'j'`: `intmax_t`

This can be useful for VMOD authors in conjunction with workspace operations.

### SEE ALSO

- VTC
- VCL

### COPYRIGHT

    Copyright (c) 2017 Varnish Software AS
    All rights reserved.

    Author: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:
    1. Redistributions of source code must retain the above copyright
       notice, this list of conditions and the following disclaimer.
    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.

# vmod_purge

## Varnish Purge Module

Manual section: | 3

### SYNOPSIS

    import purge [from "path"] ;

    INT hard()

    INT soft(DURATION ttl, DURATION grace, DURATION keep)

### CONTENTS

- INT hard()
- INT soft(DURATION ttl, DURATION grace, DURATION keep)

### DESCRIPTION

vmod_purge contains functions that offer a finer-grained control than the `purge` transition in `vcl_recv`. The functions can only be called from `vcl_hit` or `vcl_miss` and they should in general be used in both to ensure that all variants of a same object are taken care of.

### EXAMPLE

    sub vcl_recv {
        if (req.method == "PURGE") {
            if (client.ip !~ purge_acl) {
                return (synth(405));
            }
            return (hash);
        }
    }

    sub my_purge {
        set req.http.purged = purge.hard();
        if (req.http.purged == "0") {
            return (synth(404));
        }
        else {
            return (synth(200));
        }
    }

    sub vcl_hit {
        if (req.method == "PURGE") {
            call my_purge;
        }
    }

    sub vcl_miss {
        if (req.method == "PURGE") {
            call my_purge;
        }
    }

    sub vcl_synth {
        if (req.method == "PURGE") {
            if (req.http.purged) {
                set resp.http.purged = req.http.purged;
            }
            return (deliver);
        }
    }

#### INT hard()

Description
  This is equivalent to `return(purge)` but explicitly called from `vcl_hit` and `vcl_miss`. It returns the number of purged objects.
Example
  set req.http.purged = purge.hard();

#### INT soft(DURATION ttl, DURATION grace, DURATION keep)

    INT soft(DURATION ttl=0, DURATION grace=-1, DURATION keep=-1)

Description
  Sets the TTL, grace and keep. By default, TTL is set to 0 with grace and keep periods left untouched. Setting a negative value for grace or keep periods leaves them untouched. Setting all three parameters to 0 is equivalent to a hard purge. It can only be called from `vcl_hit` or `vcl_miss`. It returns the number of soft-purged objects.

### SEE ALSO

- VCL

### COPYRIGHT

    Copyright (c) 2017 Varnish Software AS
    All rights reserved.

    Author: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:
    1. Redistributions of source code must retain the above copyright
       notice, this list of conditions and the following disclaimer.
    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.

# vmod_blob

## Utilities for the VCL blob type, encoding and decoding

Manual section: | 3

### SYNOPSIS

    import blob [from "path"] ;

    BLOB decode(ENUM decoding, INT length, STRING encoded)

    STRING encode(ENUM encoding, ENUM case, BLOB blob)

    STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length, STRING encoded)

    BOOL same(BLOB, BLOB)

    BOOL equal(BLOB, BLOB)

    INT length(BLOB)

    BLOB sub(BLOB, BYTES length, BYTES offset)

    new xblob = blob.blob(ENUM decoding, STRING encoded)

       BLOB xblob.get()

       STRING xblob.encode(ENUM encoding, ENUM case)

### CONTENTS

- new xblob = blob.blob(ENUM decoding, STRING encoded)
- STRING xblob.encode(ENUM encoding, ENUM case)
- BLOB xblob.get()
- BLOB decode(ENUM decoding, INT length, STRING encoded)
- STRING encode(ENUM encoding, ENUM case, BLOB blob)
- BOOL equal(BLOB, BLOB)
- INT length(BLOB)
- BOOL same(BLOB, BLOB)
- BLOB sub(BLOB, BYTES length, BYTES offset=0)
- transcode(…)

### DESCRIPTION

This VMOD provides utility functions and an object for the VCL data type BLOB, which may contain arbitrary data of any length.

Examples:

    sub vcl_init {
        # Create blob objects from encodings such as base64 or hex.
        new myblob   = blob.blob(BASE64, "Zm9vYmFy");
        new yourblob = blob.blob(encoded="666F6F", decoding=HEX);
    }

    sub vcl_deliver {
        # The .get() method retrieves the BLOB from an object.
        set resp.http.MyBlob-As-Hex
            = blob.encode(blob=myblob.get(), encoding=HEX);

        # The .encode() method efficiently retrieves an encoding.
        set resp.http.YourBlob-As-Base64 = yourblob.encode(BASE64);

        # decode() and encode() functions convert blobs to text and
        # vice versa at runtime.
        set resp.http.Base64-Encoded
            = blob.encode(BASE64,
                          blob=blob.decode(HEX,
                                           encoded=req.http.Hex-Encoded));
    }

    sub vcl_recv {
        # transcode() converts from one encoding to another.
        # case=UPPER specifies upper-case hex digits A-F.
        set req.http.Hex-Encoded
            = blob.transcode(decoding=BASE64, encoding=HEX,
                             case=UPPER, encoded="YmF6");

        # transcode() from URL to IDENTITY effects a URL decode.
        set req.url = blob.transcode(encoded=req.url, decoding=URL);

        # transcode() from IDENTITY to URL effects a URL encode.
        set req.http.url_urlcoded
            = blob.transcode(encoded=req.url, encoding=URL);
    }

### ENCODING SCHEMES

Binary-to-text encoding schemes are specified by ENUMs in the VMOD’s constructor, methods and functions. Decodings convert a (possibly concatenated) string into a blob, while encodings convert a blob into a string.

ENUM values for an encoding scheme can be one of:

- `IDENTITY`
- `BASE64`
- `BASE64URL`
- `BASE64URLNOPAD`
- `HEX`
- `URL`

Empty strings are decoded into a “null blob” (of length 0), and conversely a null blob is encoded as the empty string.

For encodings with `HEX` or `URL`, you may also specify a `case` ENUM with one of the values `LOWER`, `UPPER` or `DEFAULT` to produce a string with lower- or uppercase hex digits (in `[a-f]` or `[A-F]`). The default value for `case` is `DEFAULT`, which for `HEX` and `URL` means the same as `LOWER`.

The `case` ENUM is not relevant for decodings; `HEX` or `URL` strings to be decoded as BLOBs may have hex digits in either case, or in mixed case.

The `case` ENUM MUST be set to `DEFAULT` for the other encodings (BASE64* and IDENTITY). You cannot, for example, produce an uppercase string by using the IDENTITY scheme with `case=UPPER`. To change the case of a string, use the `toupper` or `tolower` functions from vmod_std.

#### IDENTITY

The simplest encoding converts between the BLOB and STRING data types, leaving the contents byte-identical.

Note that a BLOB may contain a null byte at any position before its end; if such a BLOB is decoded with IDENTITY, the resulting STRING will have a null byte at that position. Since VCL strings, like C strings, are represented with a terminating null byte, the string will be truncated, appearing to contain less data than the original blob. For example:

    # Decode from the hex encoding for "foo\0bar".
    # The header will be seen as "foo".
    set resp.http.Trunced-Foo1
        = blob.encode(IDENTITY, blob=blob.decode(HEX,
                                                 encoded="666f6f00626172"));

IDENTITY is the default encoding and decoding. So the above can also be written as:

    # Decode from the hex encoding for "foo\0bar".
    # The header will be seen as "foo".
    set resp.http.Trunced-Foo2
      = blob.encode(blob=blob.decode(HEX, encoded="666f6f00626172"));

The `case` ENUM MUST be set to `DEFAULT` for `IDENTITY` encodings.

#### BASE64*

The base64 encoding schemes use 4 characters to encode 3 bytes. There are no newlines or maximal line lengths – whitespace is not permitted.

The `BASE64` encoding uses the alphanumeric characters, `+` and `/`; and encoded strings are padded with the `=` character so that their length is always a multiple of four.

The `BASE64URL` encoding also uses the alphanumeric characters, but `-` and `_` instead of `+` and `/`, so that an encoded string can be used safely in a URL. This scheme also uses the padding character `=`.

The `BASE64URLNOPAD` encoding uses the same alphabet as `BASE6URL`, but leaves out the padding. Thus the length of an encoding with this scheme is not necessarily a multiple of four.

The `case` ENUM MUST be set to `DEFAULT` for for all of the `BASE64*` encodings.

#### HEX

The `HEX` encoding scheme converts hex strings into blobs and vice versa. For encodings, you may use the `case` ENUM to specify upper- or lowercase hex digits `A` through `f` (default `DEFAULT`, which means the same as `LOWER`). A prefix such as `0x` is not used for an encoding and is illegal for a decoding.

If a hex string to be decoded has an odd number of digits, it is decoded as if a `0` is prepended to it; that is, the first digit is interpreted as representing the least significant nibble of the first byte. For example:

    # The concatenated string is "abcdef0", and is decoded as "0abcdef0".
    set resp.http.First = "abc";
    set resp.http.Second = "def0";
    set resp.http.Hex-Decoded
        = blob.encode(HEX, blob=blob.decode(HEX,
                           encoded=resp.http.First + resp.http.Second));

#### URL

The `URL` decoding replaces any `%<2-hex-digits>` substrings with the binary value of the hexadecimal number after the % sign.

The `URL` encoding implements “percent encoding” as per RFC3986. The `case` ENUM determines the case of the hex digits, but does not affect alphabetic characters that are not percent-encoded.

#### BLOB decode(ENUM decoding, INT length, STRING encoded)

    BLOB decode(
       ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
       INT length=0,
       STRING encoded
    )

Returns the BLOB derived from the string `encoded` according to the scheme specified by `decoding`.

If `length` > 0, only decode the first `length` characters of the encoded string. If `length` <= 0 or greater than the length of the string, then decode the entire string. The default value of `length` is 0.

`decoding` defaults to IDENTITY.

Example:

    blob.decode(BASE64, encoded="Zm9vYmFyYmF6");

    # same with named parameters
    blob.decode(encoded="Zm9vYmFyYmF6", decoding=BASE64);

    # convert string to blob
    blob.decode(encoded="foo");

#### STRING encode(ENUM encoding, ENUM case, BLOB blob)

    STRING encode(
       ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
       ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
       BLOB blob
    )

Returns a string representation of the BLOB `blob` as specified by `encoding`. `case` determines the case of hex digits for the `HEX` and `URL` encodings, and is ignored for the other encodings.

`encoding` defaults to IDENTITY, and `case` defaults to DEFAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings, and is the required value for the other encodings.

Example:

    set resp.http.encode1
        = blob.encode(HEX,
                      blob=blob.decode(BASE64, encoded="Zm9vYmFyYmF6"));

    # same with named parameters
    set resp.http.encode2
        = blob.encode(blob=blob.decode(encoded="Zm9vYmFyYmF6",
                                               decoding=BASE64),
                          encoding=HEX);

    # convert blob to string
    set resp.http.encode3
        = blob.encode(blob=blob.decode(encoded="foo"));

#### transcode(…)

    STRING transcode(
       ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
       ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
       ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
       INT length=0,
       STRING encoded
    )

Translates from one encoding to another, by first decoding the string `encoded` according to the scheme `decoding`, and then returning the encoding of the resulting blob according to the scheme `encoding`. `case` determines the case of hex digits for the `HEX` and `URL` encodings, and is ignored for other encodings.

As with `decode()`: If `length` > 0, only decode the first `length` characters of the encoded string, otherwise decode the entire string. The default value of `length` is 0.

`decoding` and `encoding` default to IDENTITY, and `case` defaults to DEFAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings, and is the required value for the other encodings.

Example:

    set resp.http.Hex2Base64-1
         = blob.transcode(HEX, BASE64, encoded="666f6f");

     # same with named parameters
     set resp.http.Hex2Base64-2
        = blob.transcode(encoded="666f6f",
                              encoding=BASE64, decoding=HEX);

     # URL decode -- recall that IDENTITY is the default encoding.
     set resp.http.urldecoded
        = blob.transcode(encoded="foo%20bar", decoding=URL);

     # URL encode
     set resp.http.urlencoded
         = blob.transcode(encoded="foo bar", encoding=URL);

#### BOOL same(BLOB, BLOB)

Returns true if and only if the two BLOB arguments are the same object, i.e. they specify exactly the same region of memory, or both are empty.

If the BLOBs are both empty (length is 0 and/or the internal pointer is NULL), then `same()` returns `true`. If any non-empty BLOB is compared to an empty BLOB, then `same()` returns `false`.

#### BOOL equal(BLOB, BLOB)

Returns true if and only if the two BLOB arguments have equal contents (possibly in different memory regions).

As with `same()`: If the BLOBs are both empty, then `equal()` returns `true`. If any non-empty BLOB is compared to an empty BLOB, then `equal()` returns `false`.

#### INT length(BLOB)

Returns the length of the BLOB.

#### BLOB sub(BLOB, BYTES length, BYTES offset=0)

Returns a new BLOB formed from `length` bytes of the BLOB argument starting at `offset` bytes from the start of its memory region. The default value of `offset` is 0B.

`sub()` fails and returns NULL if the BLOB argument is empty, or if `offset + length` requires more bytes than are available in the BLOB.

#### new xblob = blob.blob(ENUM decoding, STRING encoded)

    new xblob = blob.blob(
       ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
       STRING encoded
    )

Creates an object that contains the BLOB derived from the string `encoded` according to the scheme `decoding`.

Example:

    new theblob1 = blob.blob(BASE64, encoded="YmxvYg==");

    # same with named arguments
    new theblob2 = blob.blob(encoded="YmxvYg==", decoding=BASE64);

    # string as a blob
    new stringblob = blob.blob(encoded="bazz");

#### BLOB xblob.get()

Returns the BLOB created by the constructor.

Example:

    set resp.http.The-Blob1 =
        blob.encode(blob=theblob1.get());

    set resp.http.The-Blob2 =
        blob.encode(blob=theblob2.get());

    set resp.http.The-Stringblob =
        blob.encode(blob=stringblob.get());

#### STRING xblob.encode(ENUM encoding, ENUM case)

    STRING xblob.encode(
          ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
          ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT
    )

Returns an encoding of BLOB created by the constructor, according to the scheme `encoding`. `case` determines the case of hex digits for the `HEX` and `URL` encodings, and MUST be set to `DEFAULT` for the other encodings.

Example:

    # blob as text
    set resp.http.The-Blob = theblob1.encode();

    # blob as base64
    set resp.http.The-Blob-b64 = theblob1.encode(BASE64);

For any `blob` object, encoding `ENC` and case `CASE`, encodings via the `.encode()` method and the `encode()` function are equal:

    # Always true:
    blob.encode(ENC, CASE, blob.get()) == blob.encode(ENC, CASE)

But the object method is more efficient – the encoding is computed once and cached (with allocation in heap memory), and the cached encoding is retrieved on every subsequent call. The `encode()` function computes the encoding on every call, allocating space for the string in Varnish workspaces.

So if the data in a BLOB are fixed at VCL initialization time, so that its encodings will always be the same, it is better to create a `blob` object. The VMOD’s functions should be used for data that are not known until runtime.

### ERRORS

The encoders, decoders and `sub()` may fail if there is insufficient space to create the new blob or string. Decoders may also fail if the encoded string is an illegal format for the decoding scheme. Encoders will fail for the `IDENTITY` and `BASE64*` encoding schemes if the `case` ENUM is not set to `DEFAULT`.

If any of the VMOD’s methods, functions or constructor fail, then VCL failure is invoked, just as if `return(fail)` had been called in the VCL source. This means that:

- If the `blob` object constructor fails, or if any methods or functions fail during `vcl_init`, then the VCL program will fail to load, and the VCC compiler will emit an error message.
- If a method or function fails in any other VCL subroutine besides `vcl_synth`, then control is directed to `vcl_synth`. The response status is set to 503 with the reason string `"VCL failed"`, and an error message will be written to the Varnish log using the tag `VCL_Error`.
- If the failure occurs during `vcl_synth`, then `vcl_synth` is aborted. The response line `"503 VCL failed"` is returned, and the `VCL_Error` message is written to the log.

### LIMITATIONS

The VMOD allocates memory in various ways for new blobs and strings. The `blob` object and its methods allocate memory from the heap, and hence they are only limited by available virtual memory.

The `encode()`, `decode()` and `transcode()` functions allocate Varnish workspace, as does `sub()` for the newly created BLOB. If these functions are failing, as indicated by “out of space” messages in the Varnish log (with the `VCL_Error` tag), then you will need to increase the varnishd parameters `workspace_client` and/or `workspace_backend`.

The `transcode()` function also allocates space on the stack for a temporary BLOB. If this function causes stack overflow, you may need to increase the varnishd parameter `thread_pool_stack`.

### SEE ALSO

- varnishd
- VCL
- vmod_std

### COPYRIGHT

    This document is licensed under the same conditions as Varnish itself.
    See LICENSE for details.

    Authors: Nils Goroll <nils.goroll@uplex.de>
             Geoffrey Simmons <geoffrey.simmons@uplex.de>

# vmod_unix

## Utilities for Unix domain sockets

Manual section: | 3

### SYNOPSIS

    import unix [from "path"] ;

    STRING user()

    STRING group()

    INT uid()

    INT gid()

### CONTENTS

- INT gid()
- STRING group()
- INT uid()
- STRING user()

### DESCRIPTION

This VMOD provides information about the credentials of the peer process (user and group of the process owner) that is connected to a Varnish listener via a Unix domain socket, if the platform supports it.

Examples:

    import unix;

    sub vcl_recv {
          # Return "403 Forbidden" if the connected peer is
          # not running as the user "trusteduser".
          if (unix.user() != "trusteduser") {
                  return( synth(403) );
          }

          # Require the connected peer to run in the group
          # "trustedgroup".
          if (unix.group() != "trustedgroup") {
                  return( synth(403) );
          }

          # Require the connected peer to run under a specific numeric
          # user id.
          if (unix.uid() != 4711) {
                  return( synth(403) );
          }

          # Require the connected peer to run under a numeric group id.
          if (unix.gid() != 815) {
                  return( synth(403) );
          }
    }

Obtaining the peer credentials is possible on a platform that supports one of the following:

- `getpeereid(3)` (such as FreeBSD and other BSD-derived systems)
- the socket option `SO_PEERCRED` for `getsockopt(2)` (Linux)
- `getpeerucred(3C)` (SunOS and descendants)

On SunOS and friends, the `PRIV_PROC_INFO` privilege set is added to the Varnish child process while the VMOD is loaded, see `setppriv(2)`.

On most platforms, the value returned is the effective user or group that was valid when the peer process initiated the connection.

#### STRING user()

Return the user name of the peer process owner.

#### STRING group()

Return the group name of the peer process owner.

#### INT uid()

Return the numeric user id of the peer process owner.

#### INT gid()

Return the numeric group id of the peer process owner.

### ERRORS

All functions in this VMOD are subject to the following constraints:

- None of them may be called in `vcl_init` or `vcl_fini`. If one of them is called in `vcl_init`, then the VCL program will fail to load, with an error message from the VMOD.

- If called on a platform that is not supported, then VCL failure is invoked. An error message is written to the log (with the `VCL_Error` tag), and for all VCL subroutines except for `vcl_synth`, control is directed immediately to `vcl_synth`, with the response status set to 503 and the reason string set to “VCL failed”.

  If the failure occurs during `vcl_synth`, then `vcl_synth` is aborted, and the the response line “503 VCL failed” is sent.

- If the current listener is not a Unix domain socket, or if the attempt to read credentials fails, then a `VCL_Error` message is written to the log. The STRING functions (`vmod_user` and `vmod_group`) return NULL, while the INT functions (`vmod_uid` and `vmod_gid`) return -1.

### SEE ALSO

- varnishd
- VCL
- `getpeereid(3)`
- `getsockopt(2)`
- `getpeerucred(3C)`
- `setppriv(2)`

### COPYRIGHT

    This document is licensed under the same conditions as Varnish itself.
    See LICENSE for details.

    Authors: Geoffrey Simmons <geoffrey.simmons@uplex.de>

# vmod_proxy

## Varnish Module to extract TLV attributes from PROXYv2

Manual section: | 3

### SYNOPSIS

    import proxy [from "path"] ;

    STRING alpn()

    STRING authority()

    BOOL is_ssl()

    BOOL client_has_cert_sess()

    BOOL client_has_cert_conn()

    INT ssl_verify_result()

    STRING ssl_version()

    STRING client_cert_cn()

    STRING ssl_cipher()

    STRING cert_sign()

    STRING cert_key()

### CONTENTS

- STRING alpn()
- STRING authority()
- STRING cert_key()
- STRING cert_sign()
- STRING client_cert_cn()
- BOOL client_has_cert_conn()
- BOOL client_has_cert_sess()
- BOOL is_ssl()
- STRING ssl_cipher()
- INT ssl_verify_result()
- STRING ssl_version()

### DESCRIPTION

vmod_proxy contains functions to extract proxy-protocol-v2 TLV attributes as described in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.

#### STRING alpn()

Description
  Extract ALPN attribute.
Example
  set req.http.alpn = proxy.alpn();

#### STRING authority()

Description
  Extract authority attribute. This corresponds to SNI from a TLS connection.
Example
  set req.http.authority = proxy.authority();

#### BOOL is_ssl()

Description
  Report if proxy-protocol-v2 has SSL TLV.
Example
  if (proxy.is_ssl()) {
  set req.http.ssl-version = proxy.ssl_version();
  }

#### BOOL client_has_cert_sess()

Description
  Report if the client provided a certificate at least once over the TLS session this connection belongs to.

#### BOOL client_has_cert_conn()

Description
  Report if the client provided a certificate over the current connection.

#### INT ssl_verify_result()

Description
  Report the SSL_get_verify_result from a TLS session. It only matters if client_has_cert_sess() is true. Per default, value is set to 0 (X509_V_OK).
Example
  if (proxy.client_has_cert_sess() && proxy.ssl_verify_result() == 0) {
  set req.http.ssl-verify = “ok”;
  }

#### STRING ssl_version()

Description
  Extract SSL version attribute.
Example
  set req.http.ssl-version = proxy.ssl_version();

#### STRING client_cert_cn()

Description
  Extract the common name attribute of the client certificate’s.
Example
  set req.http.cert-cn = proxy.client_cert_cn();

#### STRING ssl_cipher()

Description
  Extract the SSL cipher attribute.
Example
  set req.http.ssl-cipher = proxy.ssl_cipher();

#### STRING cert_sign()

Description
  Extract the certificate signature algorithm attribute.
Example
  set req.http.cert-sign = proxy.cert_sign();

#### STRING cert_key()

Description
  Extract the certificate key algorithm attribute.
Example
  set req.http.cert-key = proxy.cert_key();

### SEE ALSO

- varnishd
- VSL

### COPYRIGHT

    Copyright (c) 2018 GANDI SAS
    All rights reserved.

    Author: Emmanuel Hocdet <manu@gandi.net>

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions
    are met:
    1. Redistributions of source code must retain the above copyright
       notice, this list of conditions and the following disclaimer.
    2. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
    ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
    LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
    OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
    SUCH DAMAGE.