logs | Electric Monk

Tag Archive for logs

Configure VMware vMA as an ESXi 4 Syslog Server

April 30, 2013 Logs No comments

Configure VMware vMA as an ESXi 4 Syslog Server

A lot of people don’t know you can set up vMA as a Syslog Server in vSphere ESXi 4. vSphere 5 has the built-in SysLog Installation but what do you do if you’re running vSphere 4?

Pre Requisites

VMware vMA

Instructions

Step 1 – Deploy a vMA Server

Deploy a vMA Server in vCenter. Follow this link for full instructions
Specify the Hostname, IP address, Subnet Mask, Default gateway and DNS information
Specify a Password for the vi-admin account

Step 2 – Configure Time

ESXi uses UTC for internal time stamping. In order to avoid timestamp issues the vMA should be set to UTC for time keeping. When vMA collects the logs from your ESXi Host, sometimes the logs have the ESXi Host timestamp and sometimes they will have the vMA Localtime timestamp

Remove the Local Time
sudo rm /etc/localtime
Create a symbolic link to the UTC timezone
sudo ln –s /usr/share/zoneinfo/UTC /etc/localtime
Edit the NTP configuration file. Find the section # Use public servers from the pool.ntp.org project and replace the current entries with your preferred NTP servers
sudo nano /etc/ntp.conf
Configure the NTP daemon to start on reboot
sudo /sbin/chkconfig ntpd on
Restart the NTP daemon:
sudo /sbin/service ntpd restart
Confirm the NTP server connections are up
sudo ntpq -np
Change the keyboard to English if required in the location below. Change KEYTABLES=”us” to use the keyboard you have, for example: KEYTABLES=”en”
sudo vi /etc/sysconfig/keyboard

Step 3 – Add additional Storage to the vMA

Shutdown the vMA VM
Attach an additional disk to the VM. Choose how big you want it. A very rough estimate of the amount of log information captured would be 500MB per host, per day.
Power on vMA
Login in using vi-admin
Format the new Disk with the following command
fdisk /dev/sdb
Use the n command to create a new partition
Use the p command to make the new partition a primary partition
Press 1 to make it partition #1
Use the default for the First Cylinder
Use the default for the Last Cylinder
Use the p command to verify the partition table
Use the w command to write the partition table to the hard disk
Press Enter a couple of times to confirm it’s finished and return to the command prompt
Now we need to format the partition using the follow command
sudo mkfs -t ext3 /dev/sdb1

Step 4 – Mount the Disk

Edit /etc/fstab using the following command
nano /etc/fstab
Enter in the following line. Use tab to line up the entries
/dev/sdb1 /var/log/syslog ext3 defaults,auto 1 2
Use Ctrl+X then Y to save the file
Next: cd /var/log/ then mkdir syslog
Change the owner of the /syslog dir
sudo chown vi-admin:root /var/log/syslog
finally mount the disk
mount /var/log/syslog

Step 5 – Edit the vilogger application file so it knows where to store the logfiles. The default location is /var.log/vmware which needs changing

Edit the vilogger config file
sudo nano /etc/vmware/vMA/vMA.conf
Change the location entries (three of them) to <location>/var/log/syslog</location>
Use Ctrl+X then Y to save the file
Start (or restart) the vilogger daemon
sudo service vmware-vilogd start
Now vilogger is set to store the logs on to your new disk

Step 6 – Configure vMA to collect your logs

Add the first host using fastpass authentication
vifp addserver –authpolicy fpauth
Add the second host, then the third host and all hosts you need before proceeding to the next step
Enter the root password when prompted
Verify the server has been added
vifp –listservers
Enable vilogging
vilogger enable –server –numrotation 20 –maxfilesize 10 –collectionperiod 300
To access the help type vilogger –help

ESXi logging collects 3 logs (vpxa.log, hostd.log, and messages.log). Rough math is number of logs X number of hosts X maxrotations X maxfilesizes = total MB for logs. You’ll need slightly more than that for vilogger’s logs (depending on where you placed those when you edited vMA.conf).

vilogger Options

Other Links

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1016621

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024122

Analyse Logs for Security Related Messages

March 4, 2013 Objective 7 Secure a vSphere environment No comments

What logs are there?

Identify Logs used to troubleshoot storage issues

February 12, 2013 Objective 6 Advanced Troubleshooting No comments

Logs

Located in var/log

The logs you will want to look at for storage issues are likely to be

/var/log/vmkeventd.log

VMkernel deamon related log

/var/log/vmkernel.log

Generic NMP messages, iSCSI and fibre channel messages, driver, device discovery, storage and networking devices

/var/log/vpxa.log

vCenter Server vpxa agent logs, including communication with vCenter Server and the Host Management hostd agent

/var/log/hostd.log

Host management service logs, including virtual machine and host Task and Events, communication with the vSphere Client and vCenter Server vpxa agent, and SDK connections

/var/log/vmkwarning.log

Generic storage messages, like disconnects. A summary of Warning and Alert log messages excerpted from the VMkernel logs.

/var/log/storagerm

If SIOC is enabled then all the logs regarding that will be here

vCenter logs

Analyse Log entries to obtain configuration information and identify and resolve issues

February 11, 2013 Objective 6 Advanced Troubleshooting No comments

When problems occur in your virtual environment, you need to know where to look for clues to the cause and what to do to resolve them. Often, just trying to figure out the exact cause is the most difficult part, because virtual servers are more complicated than physical servers and there are more potential causes of problems. When you know where to look to find the cause of a problem, the process becomes a lot easier.

Note: By default, VMware ESXi logs do not persist upon a reboot. If a VMware ESXi host experiences an abrupt reboot due to reasons other than a VMkernel error, the logs do not persist and you do not have access to the logs prior to the reboot to determine the cause

Note: Many logfiles are time stamped using UTC – if you’re host isn’t configured to use UTC this may make correlating events and logs difficult

Types of Searches

The following types of server log entries are generated:

Info: Displays basic status information. For example, status information is logged if the server is ready and waiting.
Error: Displays errors that occur but do not stop the software from functioning. For example, an error is logged if a user requests secure information that they are not allowed to access.
Error Codes: If a KB points you to an error code you can search for this in the logs
Fatal: Displays errors that stop the software from functioning. For example, a fatal error is logged if the content server cannot access the database.

How to search through logs

You can use the grep command to search for specific terms

grep is a unix command that allows you to search for a pattern in a list of files

You can use tail command tail -f /var/log/hostd.log

Many times you need to view a constantly updating file. This is a common case with logs. People usually think that tail command is only used to view the last parts of a file, but it even provides you with the ability to view growing/changing files.

Please note that, by growing I mean files to whom data is being appended constantly. Using the -f option, tail lets us view the data that is being added to the file in real time.

You can use WinSCP to open logs in Notepad and search through them

Use the built-in text editors to open a log file. You can use nano, which is a bit easier to use, or vi to open a log file. Type nano or vi log file path/name to open one (for example, nano /var/log/vmware/hostd.log).

There are many 3rd party Tools on the market which will also provide analysis and searching through to find what you need. Examples such as Splunk, XPoLog and vLogView

In addition to the methods to view individual log files, you can use the vm-support command (it’s actually a script) that you can run on the ESX Service Console that will bundle together all the log files, configuration files, and output from various commands into a single TGZ file. After the file has been created, you can copy it to your workstation and extract it using the Linux tar command or WinZip

Useful Links