## Configuration

### Permissions

To see messages from other users and the system as well as performing various log operations from a regular user add it to the group:

```sh
sudo usermod -aG systemd-journal $USER
```

### Persisting logs

If the directory `/var/log/journal` is present the systemd journal will be stored there,
thus enabling a persistent journal across reboots.

```sh
sudo mkdir -p /var/log/journal
sudo chown root:systemd-journal /var/log/journal
sudo chmod 2755 /var/log/journal
sudo killall -USR1 systemd-journald
```

Example:
```sh
tree /var/log/journal
```

Should output something like this:
```
/var/log/journal
└── fe82e4aed51e4c1db953293110eca983
    ├── system.journal
    └── user-1000.journal
```

N.B. The following command should show current storage configuration:

```sh
# By deafault it is “Storage=auto”
cat /etc/systemd/journald.conf | grep Storage
```

Another way to enable persisten logging is to set the **Storage** option to **persistent**:

```ini
[Journal]
Storage=persistent
```

## Base usage

Show all journal entries:
```sh
journalctl
```

 To display the timestamps in UTC, you can use the **--utc** flag:
 ```sh
 journalctl --utc
 ```

See boot-related messages from the current boot:
```sh
journalctl -b
```

See boot messages from _N_ boots ago use _-N_, e.g.:
```sh
journalctl -b -2
```
This requires a persistent journal to be configured. For more see “[Persisting the journal](#persisting-the-journal)”.

To see the boots that systemd journal knows about, use the **--list-boots** option:

```sh
journalctl --list-boots
```

You can also use the boot ID to call back the data from a boot:

```sh
journalctl -b 7e4e564534f5477d8eabe7b4886d42a0
```

Jump to the end of the systemd journal (**-e**), and enable “follow” mode (**-f**):
```sh
journalctl -ef
```

Show all fields stored in the systemd journal with their field name and contents:
```sh
journalctl -o verbose
```

To print most recent _M_ journal entries use _-nM_, e.g.:
```sh
journalctl -n10
```

## Examples of querying

Displaying kernel messages:

```
journalctl -k
```

Display all messages in the systemd journal with a priority in the range **emerg** up to and including **err**:
```sh
journalctl -p emerg..err
```

It is possible to use either the priority name or its corresponding numeric value.
In order of highest to lowest priority, these are:

    0: emerg
    1: alert
    2: crit
    3: err
    4: warning
    5: notice
    6: info
    7: debug

Display all messages filtered by the executable path (**/usr/bin/gnome-shell**):
```sh
journalctl /usr/bin/gnome-shell
```

Display all messages generated by the **avahi-daemon.service** systemd unit:
```sh
journalctl _SYSTEMD_UNIT=avahi-daemon.service
```

Display all messages generated by current user ID (`id -u`):
```sh
journalctl _UID=$(id -u)
```

Display all messages generated by process ID:

```sh
journalctl _PID=1115
```

Display all of the entries that involve the executable in question:
```sh
journalctl /bin/bash
```

Display all messages generated by, and about, the **avahi-daemon.service** systemd unit:
```sh
journalctl -u avahi-daemon.service
```

The next command “follows” the mysql daemon:
```sh
journalctl -u mysql.service -f
```

Specify more than one unit source:
```sh
journalctl -u openvpn.service -u sshd.service
```

Displays all messages between two dates:
```sh
journalctl --since "2019-07-05 21:30:01" --until "2019-07-05 21:30:02"
```

 Printing the last 50 messages logged within the last hour:
 ```sh
 journalctl -n50 --since "1 hour ago"
 ```
 
 To show ournal entries in reverse chronological order use **-r**.
 For example next command shows the last 15 messages from the **sshd** daemon, listed in reverse order:
 ```sh
 journalctl -u openvpn.service -r -n 15
 ```
 
 ## Output Formats
 
 The **-o** parameter specefies format of the journalctl output. 
 
 - **short-precise**: The default format with microsecond precision
 - **short**: Is the default output format. It shows messages in syslog style
 - **short-monotonic**: Is similar to **short**, but the time stamp second value is shown with precision
 - **short-iso**: The default format augmented to show ISO 8601 wallclock timestamps
 - **json**: Will show each journal entry in json format in one long line
 - **json-pretty**: Will show each log entry in easy-to-read json format
 - **json-sse**: JSON formatted output wrapped to make add server-sent event compatible
 - **export**: A binary format suitable for transferring or backing up
 - **verbose**: Will show very detailed information for each journal record with all fields listed
 - **cat**: Shows messages in very short form, without any date/time or source server names
 
 Example 1:
 ```sh
 journalctl -u avahi-daemon.service -n 1 -o json-pretty
 ```
 
 Output 1:
 ```json
 {
        "__CURSOR" : "s=f4eb8c1ac5174e75b701df381cb15c52;i=710b;b=3ee8dc71cb1b442c9ad479288f2acf42;m=16ee39e;t=58cf60b544c83;x=7ddbe5fb422a6a43",
        "__REALTIME_TIMESTAMP" : "1562363263470723",
        "__MONOTONIC_TIMESTAMP" : "24044446",
        "_BOOT_ID" : "3ee8dc71cb1b442c9ad479288f2acf42",
        "_MACHINE_ID" : "fe82e4aed51e4c1db953293110eca983",
        "_HOSTNAME" : "tower",
        "PRIORITY" : "6",
        "SYSLOG_FACILITY" : "3",
        "_SYSTEMD_SLICE" : "system.slice",
        "_TRANSPORT" : "syslog",
        "SYSLOG_IDENTIFIER" : "avahi-daemon",
        "_COMM" : "avahi-daemon",
        "_EXE" : "/usr/sbin/avahi-daemon",
        "_SYSTEMD_CGROUP" : "/system.slice/avahi-daemon.service",
        "_SYSTEMD_UNIT" : "avahi-daemon.service",
        "_UID" : "113",
        "_GID" : "119",
        "_CAP_EFFECTIVE" : "0",
        "_CMDLINE" : "avahi-daemon: running [tower.local]",
        "_PID" : "1115",
        "MESSAGE" : "Service \"FTP file sharing on tower\" (/services/ftp.service) successfully established.",
        "SYSLOG_PID" : "1115",
        "_SYSTEMD_INVOCATION_ID" : "42bac8d872d84693b33128f1efa9c034",
        "_SOURCE_REALTIME_TIMESTAMP" : "1562363263470149"
}
 ```
 
 Example 2:
 ```sh
 journalctl -u avahi-daemon.service -n 10 -o json | jq .MESSAGE
 ```
 
 Output 2:
 ```
"Network interface enumeration completed."
"Registering new address record for fe80::9e5c:8eff:fe00:e5a3 on eno1.*."
"Registering new address record for 192.168.1.241 on eno1.IPv4."
"Started Avahi mDNS/DNS-SD Stack."
"Joining mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1."
"New relevant interface docker0.IPv4 for mDNS."
"Registering new address record for 172.17.0.1 on docker0.IPv4."
"Server startup complete. Host name is tower.local. Local service cookie is 1451721814."
"Service \"tower\" (/services/ssh.service) successfully established."
"Service \"FTP file sharing on tower\" (/services/ftp.service) successfully established."
 ```
 
 Truncate output (ellipsize fields)
 ```sh
 journalctl --no-full
```
 
 Output to standard output:
 ```
 journalctl --no-pager
 ```
 
 ## Maintenance
 
 ### Disk usage
 
 Current disk usage:
 ```sh
 journalctl --disk-usage
 ```

 Output example:
 ```
 Archived and active journals take up 1.3G in the file system.
 ```
 
 Use the **--vacuum-size** option to shrink journal by indicating a size:
 ```sh
 journalctl --vacuum-size=1G
 ```
 
  Output example:
 ```
 Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@00058d3e56fc62d0-d75e1670c635efd5.journal~ (24.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@93efc46bbe3e4af2a0fc718a4564a79d-0000000000000b54-00058d33d3695c54.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@522e24407a374023974d0083e68724cf-0000000000000001-00058d3e56f917ea.journal (8.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@00058d6162251a53-903ef862cfbe33ea.journal~ (24.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@93efc46bbe3e4af2a0fc718a4564a79d-0000000000006285-00058d3e5ed217bf.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000000001-00058d6162219f66.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000010833-00058d6b1308d58a.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@93efc46bbe3e4af2a0fc718a4564a79d-000000000001083a-00058d6b131a116d.journal (8.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000013342-00058d6c7d131fc4.journal (88.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@00058de308fa8baf-15bb1c1c500dc2e1.journal~ (24.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@7639a87d761f474ab3433bedcd1e1c02-0000000000028bf9-00058de308fa47f6.journal (56.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@4f347a49f0d241ea9cb17432b605681d-0000000000036f7e-00058e19ba8e5817.journal (16.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/user-1000@7639a87d761f474ab3433bedcd1e1c02-0000000000036f87-00058e19bba7343d.journal (8.0M).
Deleted archived journal /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d/system@00058ec2300256cd-698a26e2a805618e.journal~ (32.0M).
Vacuuming done, freed 352.0M of archived journals from /var/log/journal/5811f5275de24e19ac3c1a62f2b2e27d.
 ```
 
 Keep entries from the last year, you can type:
 ```sh
 journalctl --vacuum-time=1years
 ```
 
 ### Manage journal service

To control running journal service with systemd, use the [`systemctl` utility](https://www.freedesktop.org/software/systemd/man/systemctl.html). This utility is similar to the `service` utility provided by SysVinit and Upstart. Among others:

- `systemctl status systemd-journald` indicates whether the service is running and additional information if it is
- `systemctl start systemd-journald` starts the service (systemd unit)
- `systemctl stop systemd-journald` stops the service
- `systemctl restart systemd-journald` restarts the service
- `systemctl reload systemd-journald` reloads the service's configuration if possible, but will not kill it (so no risk of a service interruption or of disrupting processing in progress, but the service may keep running with a stale configuration)
- `systemctl force-reload systemd-journald` reloads the service's configuration if possible, and if not restarts the service (so the service is guaranteed to use the current configuration, but this may interrupt something)
 
 ## References
 - `man 1 journalctl` - Query the systemd journal
 - `man 7 systemd.journal-fields` - Special journal fields
 - `man 5 journald.conf` - Journal service configuration files