Centralized Log Management
Define Logs. Its Principle and Types
Getting ready
$ vagrant up
$ vagrant ssh
How to do it
From the man pages syslog, it read and/or clear kernel message ring buffer; set console_loglevel.
System Log Architecture or System Logging.
All type of log message store
$ ls -l /var/log
total 4308
drwxrwxr-x. 2 root root 4096 May 24 14:03 anaconda
drwxr-x---. 2 root root 40 May 27 18:12 audit
-rw-r--r--. 1 root root 14214 May 30 14:13 boot.log
-rw-------. 1 root utmp 1920 May 27 14:25 btmp
drwxr-xr-x. 2 chrony chrony 6 Nov 24 2015 chrony
-rw-------. 1 root root 35866 May 30 14:30 cron
drwxr-xr-x. 2 lp sys 54 May 24 14:08 cups
-rw-r--r--. 1 root root 35584 May 30 14:12 dmesg
-rw-r--r--. 1 root root 35540 May 28 17:12 dmesg.old
-rw-r--r--. 1 root root 50 May 25 16:32 firewalld
drwx--x--x. 2 root gdm 4096 May 30 14:13 gdm
drwxr-xr-x. 2 root root 6 Mar 10 18:58 glusterfs
drwx------. 2 root root 39 May 24 15:56 httpd
-rw-r--r--. 1 root root 293460 May 30 14:14 lastlog
-rw-------. 1 root root 5955 May 30 14:13 maillog
drwxr-x---. 2 mysql mysql 24 May 27 18:08 mariadb
-rw-------. 1 root root 3372885 May 30 14:30 messages
drwxr-xr-x. 2 ntp ntp 6 Nov 20 2015 ntpstats
drwxr-xr-x. 3 root root 17 Dec 10 16:31 pluto
drwx------. 2 root root 6 Jun 10 2014 ppp
drwxr-xr-x. 2 root root 6 Nov 20 2015 qemu-ga
drwxr-xr-x. 2 root root 72 May 30 14:12 sa
drwx------. 4 root root 4096 May 25 11:31 samba
-rw-------. 1 root root 99992 May 30 14:23 secure
drwx------. 2 root root 6 Jun 10 2014 speech-dispatcher
-rw-------. 1 root root 0 Dec 10 16:31 spooler
drwxr-x---. 2 sssd sssd 6 Nov 20 2015 sssd
-rw-------. 1 root root 0 Dec 10 16:30 tallylog
drwxr-xr-x. 2 root root 22 May 24 14:08 tuned
-rw-r--r--. 1 root root 800 May 30 14:13 wpa_supplicant.log
-rw-rw-r--. 1 root utmp 72576 May 30 14:14 wtmp
-rw-r--r--. 1 root root 70973 May 30 14:25 Xorg.0.log
-rw-r--r--. 1 root root 69650 May 28 17:15 Xorg.0.log.old
-rw-r--r--. 1 root root 1017 May 27 16:52 Xorg.1.log
-rw-r--r--. 1 root root 1159 May 26 14:44 Xorg.1.log.old
-rw-------. 1 root root 6214 May 27 18:08 yum.log
Now the log message
$ sudo tail /var/log/messages
May 30 14:30:28 centos7server NetworkManager[771]: <info> (enp0s10): DHCPv4 state changed bound -> bound
May 30 14:30:28 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
May 30 14:30:28 centos7server systemd: Starting Network Manager Script Dispatcher Service...
May 30 14:30:28 centos7server dbus-daemon: dbus[641]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
May 30 14:30:28 centos7server dhclient: bound to 192.168.56.101 -- renewal in 513 seconds.
May 30 14:30:28 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 30 14:30:28 centos7server dbus-daemon: dbus[641]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 30 14:30:28 centos7server nm-dispatcher: Dispatching action 'dhcp4-change' for enp0s10
May 30 14:30:28 centos7server systemd: Started Network Manager Script Dispatcher Service.
May 30 14:33:09 centos7server firefox.desktop: 1464598989191#011addons.update-checker#011WARN#011Update manifest for {972ce4c6-7e08-4474-a285-3208198ce6fd} did not contain an updates property
Now log secure
$ sudo less secure
May 24 14:08:22 localhost polkitd[696]: Loading rules from directory /etc/polkit-1/rules.d
May 24 14:08:22 localhost polkitd[696]: Loading rules from directory /usr/share/polkit-1/rules.d
May 24 14:08:22 localhost polkitd[696]: Finished loading, compiling and executing 5 rules
May 24 14:08:22 localhost polkitd[696]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
May 24 14:09:31 localhost polkitd[696]: Registered Authentication Agent for unix-process:11118:9143 (system bus name :1.14 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
May 24 14:09:31 localhost polkitd[696]: Unregistered Authentication Agent for unix-process:11118:9143 (system bus name :1.14, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
May 24 14:09:33 localhost gdm-launch-environment]: pam_unix(gdm-launch-environment:session): session opened for user gdm by (uid=0)
May 24 14:09:38 localhost polkitd[696]: Registered Authentication Agent for unix-session:c1 (system bus name :1.25 [gnome-shell --mode=gdm], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
May 24 14:10:44 localhost gdm-password]: pam_unix(gdm-password:session): session opened for user centos7server by (unknown)(uid=0)
May 24 14:10:44 localhost gdm-launch-environment]: pam_unix(gdm-launch-environment:session): session closed for user gdm
: (press q for exit or down arrow for continue)
Now for mail log
$ sudo mail maillog
May 28 13:58:01 centos7server postfix/cleanup[6965]: 3747533C3006: message-id=<[email protected]>
May 28 13:58:01 centos7server postfix/qmgr[2756]: 3747533C3006: from=<[email protected]>, size=435, nrcpt=1 (queue active)
May 28 13:58:01 centos7server postfix/local[6979]: 3747533C3006: to=<[email protected]>, orig_to=<centos7server>, relay=local, delay=0.46, delays=0.32/0.08/0/0.06, dsn=2.0.0, status=sent (delivered to mailbox)
May 28 13:58:01 centos7server postfix/qmgr[2756]: 3747533C3006: removed
May 28 15:00:01 centos7server postfix/postfix-script[2767]: starting the Postfix mail system
May 28 15:00:01 centos7server postfix/master[2769]: daemon started -- version 2.10.1, configuration /etc/postfix
May 28 17:13:14 centos7server postfix/postfix-script[2734]: starting the Postfix mail system
May 28 17:13:15 centos7server postfix/master[2736]: daemon started -- version 2.10.1, configuration /etc/postfix
May 30 14:13:25 centos7server postfix/postfix-script[2757]: starting the Postfix mail system
May 30 14:13:25 centos7server postfix/master[2769]: daemon started -- version 2.10.1, configuration /etc/postfix
$ sudo tail cron
May 28 15:01:01 centos7server run-parts(/etc/cron.hourly)[3873]: finished 0anacron
May 28 15:01:01 centos7server run-parts(/etc/cron.hourly)[3849]: starting 0yum-hourly.cron
May 28 15:01:01 centos7server run-parts(/etc/cron.hourly)[3890]: finished 0yum-hourly.cron
May 28 17:13:07 centos7server crond[1253]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 30% if used.)
May 28 17:13:11 centos7server crond[1253]: (CRON) INFO (running with inotify support)
May 30 14:13:16 centos7server crond[1283]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 1% if used.)
May 30 14:13:23 centos7server crond[1283]: (CRON) INFO (running with inotify support)
May 30 14:20:01 centos7server CROND[4231]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 30 14:30:01 centos7server CROND[4564]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 30 14:40:01 centos7server CROND[4748]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Now for Boot log
$ sudo tail boot.log
[ OK ] Started Xinetd A Powerful Replacement For Inetd.
[ OK ] Started Permit User Sessions.
[FAILED] Failed to start DHCPv4 Server Daemon.
See 'systemctl status dhcpd.service' for details.
Starting GNOME Display Manager...
Starting Wait for Plymouth Boot Screen to Quit...
[ OK ] Started Job spooling tools.
Starting Job spooling tools...
[ OK ] Started Command Scheduler.
Starting Command Scheduler...
Reviewing Systemlog File and Its Type Of Syntax
From the man pages journalctl, it Query the systemd journal.
use helping command for journalctl
# journalctl --help
journalctl key option for basic
$ journalctl --help
journalctl [OPTIONS...] [MATCHES...]
Query the journal.
Flags:
--system Show the system journal
--user Show the user journal for the current user
-M --machine=CONTAINER Operate on local container
--since=DATE Show entries not older than the specified date
--until=DATE Show entries not newer than the specified date
-c --cursor=CURSOR Show entries starting at the specified cursor
--after-cursor=CURSOR Show entries after the specified cursor
--show-cursor Print the cursor after all the entries
-b --boot[=ID] Show current boot or the specified boot
--list-boots Show terse information about recorded boots
-k --dmesg Show kernel message log from the current boot
-u --unit=UNIT Show logs from the specified unit
-t --identifier=STRING Show entries with the specified syslog identifier
-p --priority=RANGE Show entries with the specified priority
-e --pager-end Immediately jump to the end in the pager
-f --follow Follow the journal
-n --lines[=INTEGER] Number of journal entries to show
--no-tail Show all lines, even in follow mode
-r --reverse Show the newest entries first
-o --output=STRING Change journal output mode (short, short-iso,
short-precise, short-monotonic, verbose,
export, json, json-pretty, json-sse, cat)
--utc Express time in Coordinated Universal Time (UTC)
-x --catalog Add message explanations where available
--no-full Ellipsize fields
-a --all Show all fields, including long and unprintable
-q --quiet Do not show privilege warning
--no-pager Do not pipe output into a pager
-m --merge Show entries from all available journals
-D --directory=PATH Show journal files from directory
--file=PATH Show journal file
--root=ROOT Operate on catalog files underneath the root ROOT
--interval=TIME Time interval for changing the FSS sealing key
--verify-key=KEY Specify FSS verification key
--force Override of the FSS key pair with --setup-keys
Commands:
-h --help Show this help text
--version Show package version
-F --field=FIELD List all values that a specified field takes
--new-id128 Generate a new 128-bit ID
--disk-usage Show total disk usage of all journal files
--vacuum-size=BYTES Reduce disk usage below specified size
--vacuum-time=TIME Remove journal files older than specified date
--flush Flush all journal data from /run into /var
--header Show journal header information
--list-catalog Show all message IDs in the catalog
--dump-catalog Show entries in the message catalog
--update-catalog Update the message catalog database
--setup-keys Generate a new FSS key pair
--verify Verify journal file consistency
lines 25-56/56 (END)
$ sudo journalctl
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 14:57:52 IST. --
May 30 14:12:22 localhost.localdomain systemd-journal[90]: Runtime journal is using 8.0M (max allowed 74.2M, try
May 30 14:12:22 localhost.localdomain systemd-journal[90]: Runtime journal is using 8.0M (max allowed 74.2M, try
May 30 14:12:22 localhost.localdomain kernel: Initializing cgroup subsys cpuset
May 30 14:12:22 localhost.localdomain kernel: Initializing cgroup subsys cpu
May 30 14:12:22 localhost.localdomain kernel: Initializing cgroup subsys cpuacct
May 30 14:12:22 localhost.localdomain kernel: Linux version 3.10.0-327.el7.x86_64 ([email protected]
May 30 14:12:22 localhost.localdomain kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-327.el7.x86_64 root=/dev/
May 30 14:12:22 localhost.localdomain kernel: e820: BIOS-provided physical RAM map:
May 30 14:12:22 localhost.localdomain kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
May 30 14:12:22 localhost.localdomain kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
May 30 14:12:22 localhost.localdomain kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
May 30 14:12:22 localhost.localdomain kernel: BIOS-e820: [mem 0x0000000000100000-0x000000005f2effff] usable
May 30 14:12:22 localhost.localdomain kernel: BIOS-e820: [mem 0x000000005f2f0000-0x000000005f2fffff] ACPI data
May 30 14:12:22 localhost.localdomain kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
May 30 14:12:22 localhost.localdomain kernel: NX (Execute Disable) protection: active
May 30 14:12:22 localhost.localdomain kernel: SMBIOS 2.5 present.
$ journalctl -f
-- Logs begin at Mon 2016-05-30 14:12:22 IST. --
May 30 15:01:02 centos7server systemd[1]: Started Session 7 of user root.
May 30 15:01:02 centos7server systemd[1]: Starting Session 7 of user root.
May 30 15:01:02 centos7server CROND[5190]: (root) CMD (run-parts /etc/cron.hourly)
May 30 15:01:02 centos7server run-parts(/etc/cron.hourly)[5196]: starting 0anacron
May 30 15:01:02 centos7server anacron[5208]: Anacron started on 2016-05-30
May 30 15:01:02 centos7server run-parts(/etc/cron.hourly)[5210]: finished 0anacron
May 30 15:01:02 centos7server anacron[5208]: Will run job `cron.daily' in 9 min.
May 30 15:01:02 centos7server anacron[5208]: Jobs will be executed sequentially
May 30 15:01:02 centos7server run-parts(/etc/cron.hourly)[5213]: starting 0yum-hourly.cron
May 30 15:01:02 centos7server run-parts(/etc/cron.hourly)[5219]: finished 0yum-hourly.cron
(Press CTRl + c for exit)
$ sudo journalctl -n 5
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:01:02 IST. --
May 30 15:01:02 centos7server run-parts(/etc/cron.hourly)[5210]: finished 0anacron
May 30 15:01:02 centos7server anacron[5208]: Will run job `cron.daily' in 9 min.
May 30 15:01:02 centos7server anacron[5208]: Jobs will be executed sequentially
May 30 15:01:02 centos7server run-parts(/etc/cron.hourly)[5213]: starting 0yum-hourly.cron
May 30 15:01:02 centos7server run-parts(/etc/cron.hourly)[5219]: finished 0yum-hourly.cron
$ sudo journalctl -p warning
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:34:19 IST. --
May 30 14:12:22 localhost.localdomain kernel: ACPI: RSDP 00000000000e0000 00024 (v02 VBOX )
May 30 14:12:22 localhost.localdomain kernel: ACPI: XSDT 000000005f2f0030 0003C (v01 VBOX VBOXXSDT 00000001 AS
May 30 14:12:22 localhost.localdomain kernel: ACPI: FACP 000000005f2f00f0 000F4 (v04 VBOX VBOXFACP 00000001 AS
May 30 14:12:22 localhost.localdomain kernel: ACPI: DSDT 000000005f2f0470 02106 (v01 VBOX VBOXBIOS 00000002 IN
May 30 14:12:22 localhost.localdomain kernel: ACPI: FACS 000000005f2f0200 00040
May 30 14:12:22 localhost.localdomain kernel: ACPI: APIC 000000005f2f0240 00054 (v02 VBOX VBOXAPIC 00000001 AS
May 30 14:12:22 localhost.localdomain kernel: ACPI: SSDT 000000005f2f02a0 001CC (v01 VBOX VBOXCPUT 00000002 IN
May 30 14:12:22 localhost.localdomain kernel: Zone ranges:
May 30 14:12:22 localhost.localdomain kernel: DMA [mem 0x00001000-0x00ffffff]
May 30 14:12:22 localhost.localdomain kernel: DMA32 [mem 0x01000000-0xffffffff]
May 30 14:12:22 localhost.localdomain kernel: Normal empty
May 30 14:12:22 localhost.localdomain kernel: Movable zone start for each node
May 30 14:12:22 localhost.localdomain kernel: Early memory node ranges
May 30 14:12:22 localhost.localdomain kernel: node 0: [mem 0x00001000-0x0009efff]
May 30 14:12:22 localhost.localdomain kernel: node 0: [mem 0x00100000-0x5f2effff]
May 30 14:12:22 localhost.localdomain kernel: Built 1 zonelists in Node order, mobility grouping on. Total page
May 30 14:12:22 localhost.localdomain kernel: Policy zone: DMA32
May 30 14:12:22 localhost.localdomain kernel: ACPI: All ACPI Tables successfully acquired
May 30 14:12:22 localhost.localdomain kernel: APIC calibration not consistent with PM-Timer: 132ms instead of 10
May 30 14:12:22 localhost.localdomain kernel: NMI watchdog: disabled (cpu0): hardware events not enabled
May 30 14:12:22 localhost.localdomain kernel: ACPI: Executed 1 blocks of module-level executable AML code
May 30 14:12:22 localhost.localdomain kernel: ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_]
May 30 14:12:22 localhost.localdomain kernel: ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_]
May 30 14:12:22 localhost.localdomain kernel: ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_]
May 30 14:12:22 localhost.localdomain kernel: ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_]
May 30 14:12:22 localhost.localdomain kernel: acpi PNP0A03:00: fail to add MMCONFIG information, can't access ex
May 30 14:12:22 localhost.localdomain kernel: ACPI: Enabled 2 GPEs in block 00 to 07
May 30 14:12:22 localhost.localdomain kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
May 30 14:12:22 localhost.localdomain systemd-tmpfiles[94]: Failed to parse ACL "d:group:adm:r-x,d:group:wheel:r
May 30 14:12:22 localhost.localdomain systemd-tmpfiles[94]: Failed to parse ACL "group:adm:r-x,group:wheel:r-x":
May 30 14:12:22 localhost.localdomain systemd-tmpfiles[94]: Failed to parse ACL "d:group:adm:r-x,d:group:wheel:r
lines 1-32
$ journalctl -p err
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:01:02 IST. --
May 30 14:12:38 centos7server kernel: piix4_smbus 0000:00:07.0: SMBus base address uninitialized - upgrade BIOS
May 30 14:12:39 centos7server kernel: intel_rapl: no valid rapl domains found in package 0
May 30 14:13:13 centos7server systemd[1241]: Failed at step EXEC spawning /bin/myservice1: Permission denied
May 30 14:13:14 centos7server dhcpd[1239]: Usage: dhcpd [-p <UDP port #>] [-f] [-d] [-q] [-t|-T]
[-4|-6] [-cf config-file] [-lf lease-file]
[-tf trace-output-file]
[-play trace-input-file]
[-pf pid-file] [--no-pid] [-s server]
[if0 [...ifN]]
May 30 14:13:14 centos7server dhcpd[1239]:
May 30 14:13:14 centos7server dhcpd[1239]: If you did not get this software from ftp.isc.org, please
May 30 14:13:14 centos7server dhcpd[1239]: get the latest from ftp.isc.org and install that before
May 30 14:13:14 centos7server dhcpd[1239]: requesting help.
$ sudo journalctl _SYSTEMD_UNIT=dhcpd.service
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:26:08 IST. --
May 30 14:13:14 centos7server dhcpd[1239]: Internet Systems Consortium DHCP Server 4.2.5
May 30 14:13:14 centos7server dhcpd[1239]: Copyright 2004-2013 Internet Systems Consortium.
May 30 14:13:14 centos7server dhcpd[1239]: All rights reserved.
May 30 14:13:14 centos7server dhcpd[1239]: Usage: dhcpd [-p <UDP port #>] [-f] [-d] [-q] [-t|-T]
[-4|-6] [-cf config-file] [-lf lease-file]
[-tf trace-output-file]
[-play trace-input-file]
[-pf pid-file] [--no-pid] [-s server]
[if0 [...ifN]]
May 30 14:13:14 centos7server dhcpd[1239]:
May 30 14:13:14 centos7server dhcpd[1239]: If you did not get this software from ftp.isc.org, please
May 30 14:13:14 centos7server dhcpd[1239]: get the latest from ftp.isc.org and install that before
May 30 14:13:14 centos7server dhcpd[1239]: requesting help.
May 30 14:13:14 centos7server dhcpd[1239]:
May 30 14:13:14 centos7server dhcpd[1239]: Internet Systems Consortium DHCP Server 4.2.5
May 30 14:13:14 centos7server dhcpd[1239]: Copyright 2004-2013 Internet Systems Consortium.
May 30 14:13:14 centos7server dhcpd[1239]: All rights reserved.
May 30 14:13:14 centos7server dhcpd[1239]: Usage: dhcpd [-p <UDP port #>] [-f] [-d] [-q] [-t|-T]
May 30 14:13:14 centos7server dhcpd[1239]: [-4|-6] [-cf config-file] [-lf lease-file]
May 30 14:13:14 centos7server dhcpd[1239]: [-tf trace-output-file]
May 30 14:13:14 centos7server dhcpd[1239]: [-play trace-input-file]
May 30 14:13:14 centos7server dhcpd[1239]: [-pf pid-file] [--no-pid] [-s server]
May 30 14:13:14 centos7server dhcpd[1239]: [if0 [...ifN]]
May 30 14:13:14 centos7server dhcpd[1239]: If you did not get this software from ftp.isc.org, please
May 30 14:13:14 centos7server dhcpd[1239]: get the latest from ftp.isc.org and install that before
May 30 14:13:14 centos7server dhcpd[1239]: requesting help.
May 30 14:13:14 centos7server dhcpd[1239]: If you did get this software from ftp.isc.org and have not
May 30 14:13:14 centos7server dhcpd[1239]: yet read the README, please read it before requesting help.
May 30 14:13:14 centos7server dhcpd[1239]: If you intend to request help from the [email protected]
May 30 14:13:14 centos7server dhcpd[1239]: mailing list, please read the section on the README about
May 30 14:13:14 centos7server dhcpd[1239]: submitting bug reports and requests for help.
Now Event finding
$ sudo journalctl _PID=1
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:30:01 IST. --
May 30 14:12:22 localhost.localdomain systemd[1]: Started Setup Virtual Console.
May 30 14:12:23 localhost.localdomain systemd[1]: Started dracut cmdline hook.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting dracut pre-udev hook...
May 30 14:12:23 localhost.localdomain systemd[1]: Started dracut pre-udev hook.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting udev Kernel Device Manager...
May 30 14:12:23 localhost.localdomain systemd[1]: Started udev Kernel Device Manager.
May 30 14:12:23 localhost.localdomain systemd[1]: Started dracut pre-trigger hook.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting udev Coldplug all Devices...
May 30 14:12:23 localhost.localdomain systemd[1]: Mounting Configuration File System...
May 30 14:12:23 localhost.localdomain systemd[1]: Mounted Configuration File System.
May 30 14:12:23 localhost.localdomain systemd[1]: Started udev Coldplug all Devices.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting dracut initqueue hook...
May 30 14:12:23 localhost.localdomain systemd[1]: Reached target System Initialization.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting System Initialization.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting Show Plymouth Boot Screen...
May 30 14:12:23 localhost.localdomain systemd[1]: Started Show Plymouth Boot Screen.
May 30 14:12:23 localhost.localdomain systemd[1]: Started Forward Password Requests to Plymouth Directory Watch.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting Forward Password Requests to Plymouth Directory Watch
May 30 14:12:23 localhost.localdomain systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
May 30 14:12:23 localhost.localdomain systemd[1]: Reached target Paths.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting Paths.
May 30 14:12:23 localhost.localdomain systemd[1]: Reached target Basic System.
May 30 14:12:23 localhost.localdomain systemd[1]: Starting Basic System.
May 30 14:12:27 localhost.localdomain systemd[1]: Found device /dev/mapper/cl-root.
May 30 14:12:27 localhost.localdomain systemd[1]: Starting File System Check on /dev/mapper/cl-root...
May 30 14:12:27 localhost.localdomain systemd[1]: Started File System Check on /dev/mapper/cl-root.
May 30 14:12:27 localhost.localdomain systemd[1]: Started dracut initqueue hook.
May 30 14:12:27 localhost.localdomain systemd[1]: Started dracut pre-mount hook.
May 30 14:12:27 localhost.localdomain systemd[1]: Mounting /sysroot...
May 30 14:12:27 localhost.localdomain systemd[1]: Reached target Remote File Systems (Pre).
May 30 14:12:27 localhost.localdomain systemd[1]: Starting Remote File Systems (Pre).
$ sudo journalctl _UID=1000
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:30:01 IST. --
May 30 14:13:55 centos7server gnome-session[3120]: GPG_AGENT_INFO=/run/user/1000/keyring/gpg:0:1
May 30 14:13:55 centos7server gnome-session[3120]: GPG_AGENT_INFO=/run/user/1000/keyring/gpg:0:1
May 30 14:13:55 centos7server gnome-session[3120]: SSH_AUTH_SOCK=/run/user/1000/keyring/ssh
May 30 14:13:55 centos7server gnome-session[3120]: GPG_AGENT_INFO=/run/user/1000/keyring/gpg:0:1
May 30 14:13:55 centos7server gnome-session[3120]: GPG_AGENT_INFO=/run/user/1000/keyring/gpg:0:1
May 30 14:13:55 centos7server pulseaudio[3336]: [pulseaudio] alsa-util.c: Disabling timer-based scheduling becau
May 30 14:13:55 centos7server pulseaudio[3336]: [pulseaudio] sink.c: Default and alternate sample rates are the
May 30 14:13:56 centos7server pulseaudio[3336]: [pulseaudio] alsa-util.c: Disabling timer-based scheduling becau
May 30 14:13:57 centos7server pulseaudio[3336]: [alsa-sink-Intel ICH] alsa-sink.c: ALSA woke us up to write new
May 30 14:13:57 centos7server pulseaudio[3336]: [alsa-sink-Intel ICH] alsa-sink.c: Most likely this is a bug in
May 30 14:13:57 centos7server pulseaudio[3336]: [alsa-sink-Intel ICH] alsa-sink.c: We were woken up with POLLOUT
May 30 14:13:58 centos7server gnome-session[3120]: Gjs-Message: JS WARNING: [resource:///org/gnome/shell/ui/exte
May 30 14:13:58 centos7server gnome-session[3120]: Gjs-Message: JS WARNING: [resource:///org/gnome/shell/ui/exte
May 30 14:13:58 centos7server gnome-session[3120]: Gjs-Message: JS WARNING: [resource:///org/gnome/shell/ui/exte
May 30 14:14:04 centos7server gnome-session[3120]: vmware-user: could not open /proc/fs/vmblock/dev
May 30 14:14:05 centos7server gnome-session[3120]: (uint32 1,)
May 30 14:14:05 centos7server gnome-session[3120]: Gjs-Message: JS WARNING: [/usr/share/gnome-shell/extensions/l
May 30 14:14:07 centos7server gnome-session[3120]: Entering running state
May 30 14:14:07 centos7server gnome-session[3120]: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._imsettings_2d
May 30 14:14:07 centos7server gnome-session[3120]: Failed to play sound: File or data not found
May 30 14:14:08 centos7server gnome-session[3120]: Initializing nautilus-open-terminal extension
May 30 14:14:09 centos7server gnome-session[3120]: (nautilus:3545): Gtk-WARNING **: gtk_widget_size_allocate():
May 30 14:14:10 centos7server gnome-session[3120]: (gnome-shell:3361): mutter-WARNING **: STACK_OP_ADD: window 0
May 30 14:14:10 centos7server gnome-session[3120]: (gnome-shell:3361): mutter-WARNING **: STACK_OP_ADD: window 0
May 30 14:14:10 centos7server gnome-session[3120]: (gnome-settings-daemon:3329): color-plugin-WARNING **: failed
May 30 14:14:12 centos7server gnome-shell[3361]: GNOME Shell started at Mon May 30 2016 14:14:03 GMT+0530 (IST)
May 30 14:14:14 centos7server gnome-session[3120]: (gnome-settings-daemon:3329): color-plugin-WARNING **: unable
May 30 14:14:39 centos7server gnome-session[3120]: (tracker-extract:3582): Tracker-WARNING **: Task 0, error: Un
May 30 14:14:39 centos7server gnome-session[3120]: (tracker-extract:3582): Tracker-WARNING **: Sparql update was
May 30 14:14:39 centos7server gnome-session[3120]: INSERT {
May 30 14:14:39 centos7server gnome-session[3120]: GRAPH <urn:uuid:472ed0cc-40ff-4e37-9c0c-062d78656540> {
$ sudo journalctl _UID=81
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:30:01 IST. --
May 30 14:12:50 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.systemd1'
May 30 14:12:58 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.PolicyKi
May 30 14:13:00 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'
May 30 14:13:06 centos7server dbus[641]: [system] Activating via systemd: service name='fi.w1.wpa_supplicant1' u
May 30 14:13:06 centos7server dbus[641]: [system] Successfully activated service 'fi.w1.wpa_supplicant1'
May 30 14:13:11 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.nm_dispa
May 30 14:13:11 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 30 14:13:29 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.UPower'
May 30 14:13:30 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.UPower'
May 30 14:13:31 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.ColorMan
May 30 14:13:32 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.ColorManager'
May 30 14:13:35 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.locale1'
May 30 14:13:35 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.locale1'
May 30 14:13:39 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.GeoClue2
May 30 14:13:39 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.PackageK
May 30 14:13:39 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.GeoClue2'
May 30 14:13:40 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.UDisks2'
May 30 14:13:41 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.PackageKit'
May 30 14:13:41 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.UDisks2'
May 30 14:13:43 centos7server dbus[641]: [system] Activating via systemd: service name='net.reactivated.Fprint'
May 30 14:13:43 centos7server dbus[641]: [system] Successfully activated service 'net.reactivated.Fprint'
May 30 14:13:43 centos7server dbus[641]: [system] Activating service name='org.freedesktop.realmd' (using servic
May 30 14:13:43 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.realmd'
May 30 14:13:56 centos7server dbus[641]: [system] Activating via systemd: service name='org.bluez' unit='dbus-or
May 30 14:13:56 centos7server dbus[641]: [system] Successfully activated service 'org.bluez'
May 30 14:13:56 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.hostname
May 30 14:13:57 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.hostname1'
May 30 14:14:00 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.GeoClue2
May 30 14:14:00 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.GeoClue2'
May 30 14:14:08 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.GeoClue2
May 30 14:14:08 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.GeoClue2'
$ sudo journalctl -o verbose
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:22:22 IST. --
Mon 2016-05-30 14:12:22.578972 IST [s=eea40ea0f91c4c2f93a8c67b4744373e;i=1;b=daed33253cfc4cbdae0d73099351d032;m=
PRIORITY=6
_TRANSPORT=driver
MESSAGE=Runtime journal is using 8.0M (max allowed 74.2M, trying to leave 111.3M free of 733.9M available →
MESSAGE_ID=ec387f577b844b8fa948f33cad9a75e6
_PID=90
_UID=0
_GID=0
_COMM=systemd-journal
_EXE=/usr/lib/systemd/systemd-journald
_CMDLINE=/usr/lib/systemd/systemd-journald
_CAP_EFFECTIVE=5402800cf
_SYSTEMD_CGROUP=/system.slice/systemd-journald.service
_SYSTEMD_UNIT=systemd-journald.service
_SYSTEMD_SLICE=system.slice
_BOOT_ID=daed33253cfc4cbdae0d73099351d032
_MACHINE_ID=daf3d435c34440808e30307aa76f4a81
_HOSTNAME=localhost.localdomain
Mon 2016-05-30 14:12:22.579178 IST [s=eea40ea0f91c4c2f93a8c67b4744373e;i=2;b=daed33253cfc4cbdae0d73099351d032;m=
PRIORITY=6
_TRANSPORT=driver
MESSAGE=Runtime journal is using 8.0M (max allowed 74.2M, trying to leave 111.3M free of 733.9M available →
MESSAGE_ID=ec387f577b844b8fa948f33cad9a75e6
_PID=90
_UID=0
_GID=0
_COMM=systemd-journal
_EXE=/usr/lib/systemd/systemd-journald
_CMDLINE=/usr/lib/systemd/systemd-journald
_CAP_EFFECTIVE=5402800cf
_SYSTEMD_CGROUP=/system.slice/systemd-journald.service
lines 1-32
$ sudo journalctl --since "2016-05-30 14:57:00" --until "2016-05-30 15:1:00"
-- Logs begin at Mon 2016-05-30 14:12:22 IST, end at Mon 2016-05-30 15:07:30 IST. --
May 30 14:57:52 centos7server dhclient[1195]: DHCPREQUEST on enp0s10 to 192.168.56.100 port 67
May 30 14:57:52 centos7server dhclient[1195]: DHCPACK from 192.168.56.100
May 30 14:57:52 centos7server NetworkManager[771]: <info> address 192.168.56.101
May 30 14:57:52 centos7server NetworkManager[771]: <info> plen 24 (255.255.255.0)
May 30 14:57:52 centos7server NetworkManager[771]: <info> server identifier 192.168.56.100
May 30 14:57:52 centos7server NetworkManager[771]: <info> lease time 1200
May 30 14:57:52 centos7server NetworkManager[771]: <info> (enp0s10): DHCPv4 state changed bound -> bound
May 30 14:57:52 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.nm_dispa
May 30 14:57:52 centos7server systemd[1]: Starting Network Manager Script Dispatcher Service...
May 30 14:57:52 centos7server dbus-daemon[641]: dbus[641]: [system] Activating via systemd: service name='org.fr
May 30 14:57:52 centos7server dhclient[1195]: bound to 192.168.56.101 -- renewal in 577 seconds.
May 30 14:57:52 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 30 14:57:52 centos7server dbus-daemon[641]: dbus[641]: [system] Successfully activated service 'org.freedesk
May 30 14:57:52 centos7server systemd[1]: Started Network Manager Script Dispatcher Service.
May 30 14:57:52 centos7server nm-dispatcher[5064]: Dispatching action 'dhcp4-change' for enp0s10
May 30 15:00:01 centos7server kernel: SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
May 30 15:00:01 centos7server systemd[1]: Created slice user-0.slice.
May 30 15:00:01 centos7server systemd[1]: Starting user-0.slice.
May 30 15:00:01 centos7server systemd[1]: Started Session 6 of user root.
May 30 15:00:01 centos7server systemd[1]: Starting Session 6 of user root.
May 30 15:00:01 centos7server CROND[5099]: (root) CMD (/usr/lib64/sa/sa1 1 1)
May 30 15:00:01 centos7server systemd[1]: Removed slice user-0.slice.
May 30 15:00:01 centos7server systemd[1]: Stopping user-0.slice.
Now how systemd journal store permanaently
$ sudo journalctl | tail -10
May 30 15:34:19 centos7server NetworkManager[771]: <info> lease time 1200
May 30 15:34:19 centos7server NetworkManager[771]: <info> (enp0s10): DHCPv4 state changed bound -> bound
May 30 15:34:19 centos7server dbus[641]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
May 30 15:34:19 centos7server systemd[1]: Starting Network Manager Script Dispatcher Service...
May 30 15:34:19 centos7server dbus-daemon[641]: dbus[641]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
May 30 15:34:19 centos7server dhclient[1195]: bound to 192.168.56.101 -- renewal in 484 seconds.
May 30 15:34:19 centos7server dbus[641]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 30 15:34:19 centos7server dbus-daemon[641]: dbus[641]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 30 15:34:19 centos7server systemd[1]: Started Network Manager Script Dispatcher Service.
May 30 15:34:19 centos7server nm-dispatcher[6033]: Dispatching action 'dhcp4-change' for enp0s10
$ sudo mkdir /var/log/journals
$ sudo chown root:system-journal /var/log/journals
$ sudo chmod 2755 /var/log/journals
$ sudo init 6
$ sudo killall -USR1 systemd-journald
$ ls /var/log/journal/
daf3d435c34440808e30307aa76f4a81
$ ls -l /var/log/journal/
total 4
drwxr-sr-x+ 2 root systemd-journal 75 May 30 16:01 daf3d435c34440808e30307aa76f4a81
$ lastlog
Username Port From Latest
root pts/0 Mon May 30 16:07:55 +0530 2016
bin **Never logged in**
daemon **Never logged in**
adm **Never logged in**
lp **Never logged in**
sync **Never logged in**
shutdown **Never logged in**
halt **Never logged in**
mail **Never logged in**
operator **Never logged in**
games **Never logged in**
ftp **Never logged in**
nobody **Never logged in**
avahi-autoipd **Never logged in**
systemd-bus-proxy **Never logged in**
systemd-network **Never logged in**
dbus **Never logged in**
polkitd **Never logged in**
abrt **Never logged in**
usbmuxd **Never logged in**
colord **Never logged in**
libstoragemgmt **Never logged in**
setroubleshoot **Never logged in**
rpc **Never logged in**
rtkit **Never logged in**
chrony **Never logged in**
unbound **Never logged in**
tss **Never logged in**
geoclue **Never logged in**
ntp **Never logged in**
sssd **Never logged in**
rpcuser **Never logged in**
nfsnobody **Never logged in**
pulse **Never logged in**
gdm :0 Mon May 30 16:01:23 +0530 2016
gnome-initial-setup **Never logged in**
avahi **Never logged in**
postfix **Never logged in**
sshd **Never logged in**
tcpdump **Never logged in**
centos7server :0 Mon May 30 16:01:50 +0530 2016
apache **Never logged in**
dhcpd **Never logged in**
ram **Never logged in**
shyam **Never logged in**
radha **Never logged in**
saslauth **Never logged in**
myservice **Never logged in**
mysql **Never logged in**
Now Example how we generate log from all machine
From the man pages rsyslog, it
Install and configure in server and client
$ sudo yum install rsyslog rsyslog-doc
$ sudo systemctl enable rsyslog
$ sudo systemctl start rsyslog
$ sudo systemctl status rsyslog
Now edit the configuration file
In server
$ sudo vim /etc/rsyslog.conf
#### MODULES ####
# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
$ sudo systemctl restart rsyslog
$ netstat -nltp | grep 514
tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN 21694/rsyslogd
tcp6 0 0 :::514 :::* LISTEN 21694/rsyslogd
Configure firewall
$ firewall-cmd --add-port=514/tcp --permanent
success
$ firewall-cmd --reload
success
In client
$ sudo vim /etc/rsyslog.conf
*.info;mail.none;authpriv.none;cron.none @192.168.122.158
or another option for full log
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
*.* @@192.168.122.158:514
# ### end of the forwarding rule ###
$ sudo systemctl restart rsyslog
Now check the log in the server
$ sudo tail -f /var/log/messages