Archive for August 2022

Vdbench

What is VdBench?

Vdbench is a command line utility specifically created to help engineers and customers generate disk I/O workloads to be used for validating storage performance and storage data integrity. Vdbench execution parameters may also specified via an input text file.

Vdbench is written in Java with the objective of supporting Oracle heterogeneous attachment. Vdbench has been tested on Solaris Sparc and x86, Windows NT, 2000, 2003, 2008, XP and Windows 7, HP/UX, AIX, Linux, Mac OS X, zLinux, and native VMware

Objective of Vdbench

The objective of Vdbench is to generate a wide variety of controlled storage I/O workloads, allowing control over workload parameters such as I/O rate, LUN or file sizes, transfer sizes, thread count, volume count, volume skew, read/write ratios, read and write cache hit percentages, and random or sequential workloads. This applies to both raw disks and file system files and is integrated with a detailed performance reporting mechanism eliminating the need for the Solaris command iostat or equivalent performance reporting tools. Vdbench performance reports are web accessible and are linked using HTML. Open your browser to access the summary.html file in the Vdbench output directory.
There is no requirement for Vdbench to run as root as long as the user has read/write access for the target disk(s) or file system(s) and for the output-reporting directory.

Non-performance related functionality includes data validation with Vdbench keeping track of what data is written where, allowing validation after either a controlled or uncontrolled shutdown.

How to download Vdbench

https://www.oracle.com/downloads/server-storage/vdbench-downloads.html

Vdbench comes packaged as a zip file which contains everything you need for Windows and Linux

Vdbench Terminology

Execution parameters control the overall execution of Vdbench and control things like parameter file name and target output directory name.

  • Raw I/O workload parameters describe the storage configuration to be used and the workload to be generated. The parameters include General, Host Definition (HD), Replay Group (RG), Storage Definition (SD), Workload Definition (WD) and Run Definition (RD) and must always be entered in the order in which they are listed here. A Run is the execution of one workload requested by a Run Definition. Multiple Runs can be requested within one Run Definition.
  • File system Workload parameters describe the file system configuration to be used and the workload to be generated. The parameters include General, Host Definition (HD), File System Definition (FSD), File system Workload Definition (FWD) and Run Definition (RD) and must always be entered in the order in which they are listed here. A Run is the execution of one workload requested by a Run Definition. Multiple Runs can be requested within one Run Definition.
  • Replay: This Vdbench function will replay the I/O workload traced with and processed by the Sun StorageTekTM Workload Analysis Tool (Swat).
  • Master and Slave: Vdbench runs as two or more Java Virtual Machines (JVMs). The JVM that you start is the master. The master takes care of the parsing of all the parameters, it determines which workloads should run, and then will also do all the reporting. The actual workload is executed by one or more Slaves. A Slave can run on the host where the Master was started, or it can run on any remote host as defined in the parameter file.
  • Data Validation: Though the main objective of Vdbench has always been to execute storage I/O workloads, Vdbench also is very good at identifying data corruptions on your storage.
  • Journaling: A combination of Data Validation and Journaling allows you to identify data corruption issues across executions of Vdbench.
  • LBA, or lba: For Vdbench this never means Logical Block Address, it is Logical Byte Address. 16 years ago Vdbench creators decided that they did not want to have to worry about disk sector size changes

Vdbench Quick start

You can carry out a quick test to make sure everything is working ok

  • /vdbench -t (for a raw I/O workload)

When running ‘./vdbench –t’ Vdbench will run a hard-coded sample run. A small temporary file is created and a 50/50 read/write test is executed for just five seconds.
This is a great way to test that Vdbench has been correctly installed and works for the current OS platform without the need to first create a parameter file.

  • /vdbench -tf (for a filesystem workload)

Use a browser to view the sample output report in /vdbench/output/summary.html

To start Vdbench

  • Linux: /home/vdbench/vdbench -f <parameter file>
  • Windows: c:\vdbench\vdbench.bat -f <parameter file>

There are sample parameter files in the PDF documentation in section 1.34 and in the examples folder from the zip file.

Execution Parameter Overview

The main execution parameters are

CommandExplanation
-f <workload parameter file>One parameter file is required
-o <output directory>Output directory for reporting. Default is output in current directory
-tRun a 5 second sample workload on a small disk file
-tfRun a 5 second sample filesystem workload
-eOverride elapsed parameters in Run Definitions
-IOverride interval parameters in Run Definitions
-wOverride warmup parameters in Run Definitions
-mOverride the amount of current JVM machines to run workload
-vActivate data validation
-vrActivate data validation immediately re-read after each write
-vwbut don’t read before write
-vtActivate data validation. Keep track of each write timestamp Activate data validation (memory intensive)
-jActivate data validation with journaling
-jrRecover existing journal, validate data and run workload
-jroRecover existing journal, validate data but do not run workload
-jriRecover existing journal, ignore pending writes
-jmActivate journaling but only write the journal maps
-jnActivate journaling but use asynchronous writes to journal
-sSimulate execution, Scans parameter names and displays run names
-kSolaris only: Report kstat statistics on the console
-cClean (delete existing file system structure at start of run
-coForce format=only
-cyForce format=yes
-cnForce format=no
-pOverride java socket port number
-l nnnAfter the last run, start over with the first run. Without nnn this is an endless loop
-rAllows for a restart of a parameter file containing multiple run definitions. E.g. -r rd5 if you have rd1 through rd10 in a parameter file
xxx=yyyUsed for variable substitution

There are also Vdbench utility functions – See Section 1.9 in the PDF documentation

UtilityExplanation
compareStart Vdbench workload compare
csimCompression simulator
dsimDedupe simulator
editPrimitive full screen editor, syntax ‘./vdbench edit file.name’
jstackCreate stack trace. Requires a JDK
parse(flat)Selective parsing of flatfile.html
printPrint any block on any disk or disk file
rshStart RSH daemon (For multi host testing)
sdsStart Vdbench SD Parameter Generation Tool (Solaris, Windows and Linux)
showlbaUsed to display output of the XXXX parameter

Parameter files

The parameter files get read in the following order

  • General (Optional)
  • HD (Host Definition) (Optional)
  • RG (Replay Group)
  • SD (Storage Definition)
  • WD (Workload Definition)
  • RD (Run Definition)

or for file system testing:

  • General
  • HD (Host Definition)
  • FSD (File System Definition)
  • FWD (File System Workload Definition)
  • RD (Run Definition)

General Parameters

The below parameters must be the first in the file

CommandExplanation
abort_failed_skew==nnnAbort if requested workload skew is off by more than nnnn%
Compration=nnCompression ratio
Concatenatesds=yesBy default, Vdbench will write an uncompressible random data pattern. ‘compratio=nn’ generates a data pattern that results in a nn:1 ratio. The data patterns implemented are based on the use of the ‘LZJB’ compression algorithm using a ZFS record size of 128k compression ratios 1:1 through 25:1 are the only ones implemented; any ratio larger than 25:1 will be set to 25.
create_anchors=yesCreate parent directories for FSD anchor
data_errors=nnTerminate after ‘nn’ read/write/data validation errors (default 50)
data_errors=cmdRun command or script ‘cmd’ after first read/write/data validation error, then terminate.
dedupratio=Expected ratio. Default 1 (all blocks are unique).
dedupunit=What size of data does Dedup compare?
dedupsets=How many sets of duplicates.
deduphotsets=Dedicated small sets of duplicates
dedupflipflop=Activate the Dedup flip-flop logic.
endcmd=cmdExecute command or script at the end of the last run
formatsds=Force a one-time (pre)format of all SDs
formatxfersize=Specify xfersize used when creating, expanding, or (pre)formatting an SD.
fwd_thread_adjust=noOverride the default of ‘yes’ to NOT allow FWD thread counts to be adjusted because of integer rounding/truncation.
histogram=(default,….)Override defaults for response time histogram.
include=/file/nameThere is one parameter that can be anywhere: include=/file/name When this parameter is found, the contents of the file name specified will be copied in place. Example: include=/complicated/workload/definitions.txt
ios_per_jvm=nnnnnnOverride the 100,000 default warning for ‘i/o or operations per second per slave. This means that YOU will be responsible if you are overloading your slaves.
journal=yesActivate Data Validation and Journaling:
journal=recoverRecover existing journal, validate data and run workload
journal=onlyRecover existing journal, validate data but do not run requested workload.
journal=noflushUse asynchronous I/O on journal files
journal=maponlyDo NOT write before/after journal records
journal=skip_read_allAfter journal recovery, do NO read and validate every data block.  
journal=(max=nnn)Prevent the journal file from getting larger than nnn bytes
journal=ignore_pendingIgnore pending writes during journal recovery.
loop=Repeat all Run Definitions: loop=nn repeat nn times loop=nn[s|m|h] repeat until nn seconds/minutes/hours See also ‘-l nn’ execution parameter
messagescan=noDo not scan /var/xxx/messages (Solaris or Linux)
messagescan=nodisplayScan but do not display on console, instead display on slave’s stdout.
messagescan=nnnnScan, but do not report more than nnn lines. Default 1000
monitor=/file/nameSee External control of Vdbench termination
pattern=Override the default data pattern generation.
port=nn  Override the Java socket port number.
report=host_detail report=slave_detailSpecifies which SD detail reports to generate. Default is SD total only.
report=no_sd_detail report=no_fsd_detailWill suppress the creation of SD/FSD specific reports.
report_run_totals=yesReports run totals.
startcmd=cmdExecute command or script at the beginning of the first run
showlba=yesCreate a ‘trace’ file so serve as input to ./vdbench showlba
timeout=(nn,script) 
validate=yes(-vt) Activate Data Validation. Options can be combined: validate=(x,y,z)
validate=read_after_write(-vr) Re-reads a data block immediately after it was written.
validate=no_preread(-vw) Do not read before rewrite, though this defeats the purpose of data validation!
validate=time(-vt) keep track of each write timestamp (memory intensive)
validate=reportdedupsetsReports ‘last time used’ for all duplicate blocks if a duplicate block is found to be corrupted. Also activates validate=time. Note: large SDs with few dedup sets can generate loads of output!

Host Definition Parameter Overview

These parameters are ONLY needed when running Vdbench in a multi-host environment or if you want to override the number of JVMs used in a single-host environment

CommandExplanation
hd=defaultSets defaults for all HDs that are entered later
hd=localhostSets values for the current host
hd=host_labelSpecify a host label.
System=hostnameHost IP address or network name, e.g. xyz.customer.com
vdbench=vdbench_dir_nameWhere to find Vdbench on a remote host if different from current.
jvms=nnnHow many slaves to use
shell=rsh | ssh | vdbenchHow to start a Vdbench slave on a remote system.
user=xxxxUserid on remote system Required.
clients=nnVery useful if you want to simulate numerous clients for file servers without having all the hardware. Internally is basically creates a new ‘hd=’ parameter for each requested client.
mount=”mount xxx …”This mount command is issued on the target host after the possibly needed mount directories have been created.

Replay Group (RG parameter overview

CommandExplanation
rg=nameUnique name for this Replay Group (RG).
devices=(xxx,yyy,….)The device numbers from Swat’s flatfile.bin.gz to be replayed.

Example: rg=group1,devices=(89465200,6568108,110)
Note: Swat Trace Facility (STF) will create Replay parameters for you. Select the ‘File’ ‘Create Replay parameter file’ menu option. All that’s then left to do is specify enough SDs to satisfy the amount of gigabytes needed.

Storage Definition (SD) Parameter Overview

This set of parameters identifies each physical or logical volume manager volume or file system file used in the requested workload. Of course, with a file system file, the file system takes the responsibility of all I/O: reads and writes can and will be cached (see also openflags=) and Vdbench will not have control over physical I/O. However, Vdbench can be used to test file system file performance

Example: sd=sd1,lun=/dev/rdsk/cxt0d0s0,threads=8

CommandExplanation
sd=defaultSets defaults for all SDs that are entered later.
sd=nameUnique name for this Storage Definition (SD).
count=(nn,mm)Creates a sequence of SD parameters.
align=nnnGenerate logical byte address in ‘nnn’ byte boundaries, not using default ‘xfersize’ boundaries.
dedupratio=See data deduplication:
dedupsets= 
deduphotsets= 
dedupflipflop= 
hitarea=nnSee read hit percentage for an explanation. Default 1m.
host=nameName of host where this SD can be found. Default ‘localhost’
journal=xxxDirectory name for journal file for data validation
lun=lun_nameName of raw disk or file system file.
offset=nnnAt which offset in a lun to start I/O.
openflags=(flag,..)Pass specific flags when opening a lun or file
range=(nn,mm)Use only a subset ‘range=nn’: Limit Seek Range of this SD.
replay=(group,..)Replay Group(s) using this SD.
replay=(nnn,..)Device number(s) to select for Swat Vdbench replay
resetbus=nnnIssue ioctl (USCSI_RESET_ALL) every nnn seconds. Solaris only
resetlun=nnnIssue ioctl (USCSI_RESET) every nnn seconds. Solaris only
size=nnSize of the raw disk or file to use for workload. Optional unless you want Vdbench to create a disk file for you.
threads=nnMaximum number of concurrent outstanding I/O for this SD. Default 8

Workload Definition (WD) Parameter Overview

The Workload Definition parameters describe what kind of workload must be executed using the storage definitions entered.
Example: wd=wd1,sd=(sd1,sd2),rdpct=100,xfersize=4k

CommandExplanation
wd=defaultSets defaults for all WDs that are entered later.
wd=nameUnique name for this Workload Definition (WD)
sd=xxName(s) of Storage Definition(s) to use
host=host_labelWhich host to run this workload on. Default localhost.
hotband=See hotbanding
iorate=nnRequested fixed I/O rate for this workload.
openflags=(flag,..)Pass specific flags when opening a lun or file.
priority=nnI/O priority to be used for this workload.
range=(nn,nn)Limit seek range to a defined range within an SD.
rdpct=nnRead percentage. Default 100.
rhpct=nnRead hit percentage. Default 0.
seekpct=nnPercentage of random seeks. Default seekpct=100 or seekpct=random.
skew=nnPercentage of skew that this workload receives from the total I/O rate.
streams=(nn,mm)Create independent sequential streams on the same device.
stride=(min,max)To allow for skip-sequential I/O.
threads=nnOnly available during SD concatenation.
whpct=nnWrite hit percentage. Default 0.
xfersize=nnData transfer size. Default 4k.
xfersize=(n,m,n,m,..)Specify a distribution list with percentages.
xfersize=(min,max,align)Generate xfersize as a random value between min and max.

File System Definition (FD) parameter overview

CommandExplanation
fsd=nameUnique name for this File System Definition.
fsd=defaultAll parameters used will serve as default for all the following fsd’s.
anchor=/dir/The name of the directory where the directory structure will be created.
count=(nn,mm)Creates a sequence of FSD parameters.
depth=nnHow many levels of directories to create under the anchor.
distribution=allDefault ‘bottom’, creates files only in the lowest directories. ‘all’ creates files in all directories.
files=nnHow many files to create in the lowest level of directories.
mask=(vdb_f%04d.file, vdb.%d_%d.dir)The default printf() mask used to generate file and directory names. This allows you to create your own names, though they still need to start with ‘vdb’ and end with ‘.file’ or ‘.dir’. ALL files are numbered consecutively starting with zero. The first ‘%’ mask is for directory depth, the second for directory width.
openflags=(flag,..)Pass extra flags to file system open request (See: man open)
shared=yes/noDefault ‘no’: See FSD sharing
sizes=(nn,nn,…..)Specifies the size(s) of the files that will be created.
totalsize=nnnStop after a total of ‘nnn’ bytes of files have been created.
width=nnHow many directories to create in each new directory.
workingsetsize=nn wss=nnCauses Vdbench to only use a subset of the total amount of files defined in the file structure. See workingsetsize.
journal=dirWhere to store your Data Validation journal files

Filesystem Workload Definition (FWD) parameter overview:

CommandExplanation
fwd=nameUnique name for this Filesystem Workload Definition.
fwd=defaultAll parameters used will serve as default for all the following fwd’s.
fsd=(xx,….)Name(s) of Filesystem Definitions to use
openflags=Pass extra flags to (Solaris) file system open request (See: man open)
fileio=(random.shared)Allows multiple threads to use the same file.
fileio=(seq,delete)Sequential I/O: When opening for writes, first delete the file
fileio=randomHow file I/O will be done: random or sequential
fileio=sequentialHow file I/O will be done: random or sequential
fileselect=random/seqHow to select file names or directory names for processing.
host=host_labelWhich host this workload to run on.
operation=xxxxSpecifies a single file system operation that must be done for this workload.
rdpct=nnFor operation=read and operation=write only. This allows a mix and read and writes against a single file.
skew=nn  The percentage of the total amount of work for this FWD
stopafter=nnnFor random I/O: stop and close file after ‘nnn’ reads or writes. Default ‘size=’ bytes for random I/O.
threads=nn  How many concurrent threads to run for this workload. (Make sure you have at least one file for each thread).
xfersize=(nn,…)  Specifies the data transfer size(s) to use for read and write operations.

Run Definition (RD) Parameter Overview (For raw I/O testing)

The Run Definition parameters define which of the earlier defined workloads need to be executed, what I/O rates need to be generated, and how long the workload will run. One Run Definition can result in multiple actual workloads, depending on the parameters used.


Example: rd=run1,wd=(wd1,wd2),iorate=1000,elapsed=60,interval=5


There is a separate list of RD parameters for file system testing.

CommandExplanation 
rd=defaultSets defaults for all RDs that are entered later. 
rd=nameUnique name for this Run Definition (RD). 
wd=xxWorkload Definitions to use for this run. 
sd=xxxWhich SDs to use for this run (Optional). 
curve=(nn,nn,..)Data points to generate when creating a performance curve. See also stopcurve= 
distribution=(x[,variable]I/O inter arrival time calculations: exponential, uniform, or deterministic. Default exponential. 
elapsed=nnElapsed time for this run in seconds. Default 30 seconds. 
endcmd=cmdExecute command or script at the end of the last run 
(for)compratio=nnMultiple runs for each compression ratio. 
(for)hitarea=nnMultiple runs for each hit area size. 
(for)hpct=nnMultiple runs for each read hit percentage. 
(for)rdpct=nnMultiple runs for each read percentage. 
(for)seekpct=nnMultiple runs for each seek percentage. 
(for)threads=nnMultiple runs for each thread count. 
(for)whpct=nnMultiple runs for each write hit percentage. 
(for)xfersize=nnMultiple runs for each data transfer size. 
Most forxxx parameters may be abbreviated to their regular name, e.g. xfersize=(..,..) 
interval=nnStop the run after nnn bytes have been read or written, e.g. maxdata=200g. I/O will stop at the lower of elapsed= and maxdata=. 
iorate=(nn,nn,nn,…)Reporting interval in seconds. Default ‘min(elapsed/2,60)’ 
iorate=curveOne or more I/O rates. 
iorate=maxCreate a performance curve. 
iorate=(nn,ss,…)Run an uncontrolled workload. 
nn,ss: pairs of I/O rates and seconds of duration for this I/O rate. See also ‘distribution=variable’. 
openflags=xxxxPass specific flags when opening a lun or file
pause=nnSleep ‘nn’ seconds before starting next run.
replay=(filename, split=split_dir, repeat=nn)-‘filename’: Replay file name used for Swat Vdbench replay – ‘split_dir’: directory used to do the replay file split. – ‘nn’: how often to repeat the replay.
startcmd=cmdExecute command or script at the beginning of the first run
stopcurve=n.nStop iorate=curve runs when response time > n.n ms.
warmup=nnOverride warmup period.

Run Definition (RD) parameters for file systems, overview

These parameters are file system specific parameters. More RD parameters can be found

CommandExplanation
fwd=(xx,yy,..)Name(s) of Filesystem Workload Definitions to use.
fwdrate=nnHow many file system operations per second
format=yes/no/only/ restart/clean/once/ directoriesDuring this run, if needed, create the complete file structure.
operations=xxOverrides the operation specified on all selected FWDs.
foroperations=xxMultiple runs for each specified operation.
fordepth=xxMultiple runs for each specified directory depth
forwidth=xxMultiple runs for each specified directory width
forfiles=xxMultiple runs for each specified amount of files
forsizes=xxMultiple runs for each specified file size
fortotal=xxMultiple runs for each specified total file size

Report Files

HTML files are written to the directory specified using the ‘-o’ execution parameter.
These reports are all linked together from one starting point. Use your favourite browser and point at ‘summary.html’.

Report TypeExplanation
summary.htmlContains workload results for each run and interval. Summary.html also contains a link to all other html files, and should be used as a starting point when using your browser for viewing. For file system testing see summary.html for file system testing From a command prompt in windows just enter ‘start summary.html’; on a unix system, just enter ‘firefox summary.html &’.
totals.htmlReports only run totals, allowing you to get a quick overview of run totals instead of having to scan through page after page of numbers.
totals_optional.htmlReports the cumulative amount of work done during a complete Vdbench execution. For SD/WD workloads only.
hostx.summary.htmlIdentical to summary.html, but containing results for only one specific host. This report will be identical to summary.html when not used in a multi-host environment.
hostx-n.summary.htmlSummary for one specific slave.
logfile.htmlContains a copy of most messages displayed on the console window, including several messages needed for debugging.
hostx_n.stdout.htmlContains logfile-type information for one specific slave.
parmfile.htmlContains a copy of the parameter file(s) from the ‘-f parmfile ‘ execution parameter.
parmscan.htmlContains a running trail of what parameter data is currently being parsed. If a parsing or parameter error is given this file will show you the latest parameter that was being parsed.
sdname.htmlContains performance data for each defined Storage Definition. See summary.html for a description. You can suppress this report with ‘report=no_sd_detail’
hostx.sdname.htmlIdentical to sdname.html, but containing results for only one specific host. This report will be identical to sdname.html when not used in a multi-host environment. This report is only created when the ‘report=host_detail’ parameter is used.
hostx_n.sdname.htmlSD report for one specific slave. . This report is only created when the ‘report=slave_detail’ parameter is used.
kstat.htmlContains Kstat summery performance data for Solaris
hostx.kstat.htmlKstat summary report for one specific host. This report will be identical to kstat.html when not used in a multi-host environment.
host_x.instance.htmlContains Kstat device detailed performance data for each Kstat ‘instance’.
nfs3/4.htmlSolaris only: Detailed NFS statistics per interval similar to the nfsstat command output.
flatfile.htmlA file containing detail statistics to be used for extraction and input for other reporting tools. See also Parse Vdbench flatfile
errorlog.htmlAny I/O errors or Data Validation errors will be written here.
swat_mon.txtThis file can be imported into the Swat Performance Monitor allowing you to display performance charts of a Vdbench run.
swat_mon_total.txtSimilar to swat_mon.txt, but allows Swat to display only run totals.
swat_mon.binSimilar to swat_mon.txt above, but for File System workload data.
messages.htmlFor Solaris and Linux only. At the end of a run the last 500 lines from /var/adm/messages or /var/log/messages are copied here. These messages can be useful when certain I/O errors or timeout messages have been displayed.
fwdx.htmlA detailed report for each File system Workload Definition (FWD).
wdx.htmlA separate workload report is generated for each Workload Definition (WD) when more than one workload has been specified.
histogram.htmlFor file system workloads only. A response time histogram reporting response time details of all requested FWD operations.
sdx.histogram.htmlA response time histogram for each SD.
wdx.histogramA response time histogram for each WD. Only generated when there is more than one WD.
fsdx.histogram.htmlA response time histogram for each FSD.
fwdx.histogramA response time histogram for each FWD. Only generated when there is more than one FWD.
skew.htmlA workload skew report.

Sample Parameter Files

These example parameter files can also be found in the installation directory.

  • Example 1: Single run, one raw disk
  • Example 2: Single run, two raw disk, two workloads.
  • Example 3: Two runs, two concatenated raw disks, two workloads.
  • Example 4: Complex run, including curves with different transfer sizes
  • Example 5: Multi-host.
  • Example 6: Swat trace replay.
  • Example 7: File system test. See also Sample parameter file:

There is a larger set of sample parameter files in the /examples/ directory inside your Vdbench install directory inside the filesys and raw folders

Example 1

Example 2
Example 3
Example 4
Example 5
Example 6
Example 7