
When is What ... Deleted, Expired, Expunged or Purged?
******************************************************

The terminology sometimes plays administrators for fools -- which they
are obviously not -- but an article clarifying what it is that is
meant by either term of Deleted, Expired, Expunged or Purged goes a
long way.

**Deleted**

   A message has been flagged as "\Deleted".

   In the context of folders, *Deleted* only really applies to a
   folder having been removed (from the user's (IMAP client) view), as
   opposed to having been renamed to a hierarchy in a trash folder.

**Expired**

   The index records of a message have been expired, and usually this
   means the message file has been purged as well. However the message
   file could be purged prior to index records being expired.

**Expunged**

   The message (which has been flagged as "\Deleted") is also
   expunged, meaning that the user can in no way retrieve the message
   autonomously.

**Purged**

   The message's index record may still exist (until they are
   expired), but the message file is removed from the filesystem, or
   in the context of folders, the mail folder is removed from the
   filesystem.


Users and IMAP Clients Deleting Messages
========================================

When a **message** is deleted by a user, this means that the user's
IMAP client has in fact *flagged* the message with "\Deleted", or
alternatively, the IMAP client has moved the message to a trash folder
(and has at least flagged the original copy as "\Deleted").

What is the exact behavior depends on the IMAP client software, and if
so allowed, the user's preferences specified within the IMAP client
software, and the Cyrus IMAP server configuration.


Flagged As "\Deleted"
---------------------

When a message is merely flagged with "\Deleted", the message itself
as such remains available to the IMAP client, but the IMAP client used
may not make it possible for the user to view a list of messages
flagged with "\Deleted". As such, the user may interpret the message
as removed and unavailable -- if the removal was accidental, a support
call may be on its way.

The message in fact still resides in the Cyrus IMAP mail spool, still
resides in the same IMAP folder, and still resides on the filesystem.

Only when the user (or the user's IMAP client as is often the case)
issues an "EXPUNGE" against the folder, or a "UID EXPUNGE" against the
message [1], will the message be actually removed -- at least from the
user's perspective. It then becomes irretrievable even if the IMAP
client allows the listing of messages flagged with "\Deleted".

Issuing an "EXPUNGE" may come in the form of a button to "compact" the
folder, or an IMAP client routine that is executed periodically or at
the end of a session (e.g. as the client application is closed), such
as an "Empty Trash folder" kind of option.

It is here that the Cyrus IMAP server settings come in to play, most
prominently the "expunge_mode" setting, which has three possible
values:

**delayed** (the default *since version 2.5.0*)

   The message files as well as the index records are retained for an
   undetermined period of time -- possibly indefinitely.

   A separate job (using cyr_expire) is responsible for actually
   removing index records and message files.

**default**

   The message files are removed at the first opportunity, while the
   index records remain available to facilitate "QRESYNC".

   In this context also, when we say "message files are removed", we
   mean "purged from the filesystem".

**immediate**

   The message files as well as the index records are removed at the
   earliest opportunity.

   In this context, when we say "message files are removed", we mean
   "purged from the filesystem".

Exceptional circumstances aside, when "immediate" or "default" is the
configured "expunge_mode", message files are often purged from the
filesytem too quickly for anyone to recover.

Note: One such exceptional circumstance is a mailbox with multiple
  sessions keeping the mailbox open. Cyrus IMAP ensures no mailbox
  records dissappear from underneath an existing open mailbox session.


Moved to Trash Folder
---------------------

Should the IMAP client normally, or allow the user to specify through
preferences, that messages being deleted should be moved in to a trash
folder, then the user will usually be able to recover from accidental
deletion autonomously, for as long as a copy of the deleted message(s)
resides in such trash folder.

However, the trash folder would typically continue to grow and grow,
and usually counts towards the user's resource usage (a.k.a. Quota);
many IMAP clients therefore allow the user to specify a preference to
empty the trash folder at the end of a session, or otherwise
periodically.

If the IMAP client does not support **RFC 6851** (for "UID MOVE"), the
client may choose to "COPY" the message then flag the original with
"\Deleted", then "EXPUNGE" the folder or "UID EXPUNGE" (**RFC 4315**)
the message.

This does not fare well in situations where the user is over quota,
though, and (other) messages will need to be flagged as "\Deleted" and
expunged, and/or folders within the quota root hierarchy will need to
be deleted.


Expunged Messages
-----------------

Messages in expunged folders, or messages that have been expunged
individually, can not autonomously be restored by a user, and are gone
permanently unless "expunge_mode: delayed" is used.

Recovering expunged messages requires administrator assistance, who
can use command-line tools such as unexpunge to list and restore
messages expunged. See the documentation on unexpunge for a walk-
through on how that works.

With the use of "expunge_mode: delayed", a regular "EVENT" (see
cyrus.conf(5)) is responsible for triggering cyr_expire. This utility
takes a parameter "-X <days>" to delete from the filesystem any
messages that had been expunged (by the user or the IMAP client) more
than "<days>" days ago.

In other words, using "expunge_mode: delayed" and cyr_expire allows an
administrator to recover messages that have been deleted by the user
less than "<days>" ago.

Note: This also offers a backup program the chance to obtain all
  message files. For a monthly full cycle, for example, one could
  choose to purge message files from the filesystem only after 69
  days: two months plus the maximum margin for a first Saturday to
  Sunday night of the week.


Deleting Folders
================

When folders are deleted the IMAP client tends to either delete the
folder, or rename the folder to a hierarchy in a trash folder.

Note: Note that deleting a folder "A/B" in a hierarchy "A/B/C" also
  deletes the folder "A/B/C".

If the folder is not renamed to a hierarchy in a trash folder but
instead removed directly, then the user has no way to autonomously
recover from such event.

This is where the Cyrus IMAP server settings come in to play, most
prominently "delete_mode".

The setting holds two values:

**delayed** (the default *since version 2.5.0*)

   Mailboxes that are being deleted are not deleted from the
   filesystem, but instead renamed to a special mailbox hierarchy
   under the deleted prefix, to be removed later by cyr_expire.

**immediate**

   In immediate mode, the mailbox is removed from the filesystem
   immediately. Note that for large folders, this can be a
   comparatively expensive operation.


Where are the Messages?
=======================

This part of the documentation assumes that you have run with the
default settings of "delete_mode: delayed" and "expunge_mode:
delayed".

The result of a message having deleted in either of the former ways,
or an entire folder having been deleted, is one of the following
stages;

* The message has only been flagged as "\Deleted" and the message
  nor the folder has been expunged.

  Result: The message resides in the original folder.

* The message has only been flagged as "\Deleted" and either the
  message individually or the entire folder as a whole has been
  expunged.

  Result: The message resides in the original folder and can be
  retrieved using unexpunge.

* The message has been copied to the trash folder and at least
  flagged "\Deleted" in the source folder, and the original message or
  the entire folder in which the original message resided may or may
  not have been expunged.

  Similarly, the trash folder may or may not have been "emptied".

  Result: A copy of the message still exists in the original folder
  and can be retrieved using unexpunge.

* The message was moved in to the trash folder, implying the
  original message is expunged from the source folder -- through "UID
  MOVE" or **RFC 6851** support *since version 2.5.0*.

  The trash folder may or may not have been "emptied".

  Result: A copy of the message still exists in the original folder
  and can be retrieved using unexpunge.

* The folder was moved to a hierarchy in the trash folder, and the
  trash folder has not yet been "emptied".

  Result: A copy of the message exists in the trash folder's
  hierarchy.

* The folder was moved to a hierarchy in the trash folder, and the
  trash folder as subsequently been emptied.

  Result: The folder hierarchy has been renamed to the deleted
  namespace.

-[ Footnotes ]-

[1] Only if the IMAP client supports **RFC 4315**, the IMAP
    UIDPLUS Extension.
