• QUICK LINKS
  • answers.stanford.edu
  • apple.stanford.edu
  • computing.stanford.edu
  • courses.stanford.edu
  • datarates.stanford.edu
  • dell.stanford.edu
  • departmentphone.stanford.edu
  • email.stanford.edu
  • ess.stanford.edu
  • helpsu.stanford.edu
  • myitserviceshelp.stanford.edu
  • orderithelp.stanford.edu
  • software.stanford.edu
  • studentphone.stanford.edu
  • techtraining.stanford.edu
  • tools.stanford.edu
  • unixcomputing.stanford.edu
  • vpn.stanford.edu
  • web.stanford.edu
  • windows.stanford.edu
• SERVICES
  • IT Services Only
  • All Campus Providers
    → computing.stanford.edu
• GETTING HELP
  • Stanford Answers
  • HelpSU: IT Help Desk
  • Training
• PROJECTS
• ABOUT US
  • What We Do
  • Organization Chart
  • Strategic Plan
  • Jobs
  • Contact info
  • Service & System Metrics
• INTERNAL
• SEARCH

  • Search IT Services:
     

• Service & System Metrics
• Metrics Generation and Collection
• Internal Metrics

• IT Services
• Service & System Metrics
• Metrics Generation and Collection

Metrics Generation and Collection

IT Services constantly strives to improve the services it delivers to the Stanford community. To facilitate this effort, it captures a variety of measurements to monitor and evaluate its performance. This page provides a selection of key metrics on IT Services most visible and used services.

Each metric includes current and historical performance and, if appropriate, targets intended to reflect excellent service. The cycle for updating individual metrics varies (weekly, monthly, annually). See the notation besides each description for information about the metric; in some cases, a URL is provided for additional description of the service or system.

How Are These Metrics Generated?

IT Services collects these metrics are collected from many sources:

• Our annual campus-wide Customer Satisfaction Survey
• Our monthly/weekly spot HelpSU Customer Satisfaction requests
• Our Remedy Help Desk application (“HelpSU”)
• Our campus Automatic Call Distribution (ACD) system
• Our Nagios automated monitoring system
• Our Pinnacle IT Services billing system
• Our email server logs

Who’s responsible for collecting the metrics?

IT Services has named “Product Managers” who are responsible for collecting information related to their operational responsibility: our cross-unit Customer Satisfaction Team for the annual satisfaction survey information, our Director of Help Desk Services for the weekly/monthly spot HelpSU surveys and Help Desk measures, our Director of Networking Services for networking-related items, our Director, Order Management for our ordering throughput, our Manager, Accounting Services for our billing information and our Director, Unix Systems for email and spam statistics.

How is the information collected and presented?

High Level Overview:
We have three basic sources for this information:

• Excel files
• Log files
• XML files

Most of the charts are built from Excel data files, which are collected separately (manually) from source systems. The Administrative Firewall data is built with a script-based download generated once a month. The mail-related statistics are gathered from the mail server logs. The networking stats are built with scripts reading into files in AFS.

Each of these sources feeds a set of tab-delimited files stored in AFS. Perl scripts convert the Excel files into tab-delimited files for storage in AFS. Each tab-delimited file carries a basic few columns, with some of them structured like this:

Date	Percentage	Target
2007-01	10%	15%
2007-02	12%	15%
2007-03	16%	15%

The only column that is present in each is the Date (named Year-Month or Year-Month-Day). Percentage and Target are found in most graphs, but not all, and other columns are added depending on what information is germane to that particular metric.

The tab-delimited files are checked on an hourly basis by a Perl script, to check to see if they have been updated. If so, this data is used to build two types of charts as shown on the web page. There is a config file for each of the measures, which describes the type of graph to use. Typically, there is a [chartname].small.gif, which displays on the top-most page, and a [chartname].gif, which contains the historical record.

IT Services uses “server side includes” to take the data from each separate tab-delimited file and display it on the IT Metrics web site.

Full Details On How We Build The IT Metrics Site

Note: The following process is specific to Stanford’s technical infrastructure and toolset, but is offered here by way of general information.

Parsing The Original Data

This step can be done in a number of scripts. For data that can be gathered without any human intervention (such as information found by parsing logs in afs space), a specific script can be made for that metric. For data that has to be entered by a human being, the usual method is to create an Excel file to hold the data and then set up a config file for to tell the script that reads Excel files how to handle this new file (see below).

The tab-delimited files are saved in subdirectories of /afs/ir/service/metrics/data/. The subdirectories are created for related metrics, such as mail or financial. Scripts that grab metrics data in a way other than the Excel files can simply create a relevant sub-directory if none exists and write to a file there.

The only standards to one of these files are:

• The first line is the header describing the fields.
• The date field should be the first field and either YYYY-MM or YYYY-MM-DD in format. Its header should begin with the word Year.
• Numbers should only have digits, the minus sign, the percentage symbol, or a decimal point.

For adding a new Excel file to the script that parses their data:

Excel files are located in /afs/.ir/dist/web/services/itmetrics/data/ with the general format of generaltype_specificmetric.xls. They are usually created by the business unit, after they determine what it wants to present. The script to create data files from the metrics is located at /afs/ir/service/webapp/tools/usr/local/bin/parse-excel-metrics. It runs hourly on tools, operating for every Excel files updated in the last hour.

To add a metric, you should do the following steps:

• Go to /afs/ir/service/metrics/config/excel. This is where all config files for a metric exist, where the filename is a descriptive name for the metric.
• Copy an existing file, naming it whatever you wish the new metric to be.
• Open the new metric config file for editing. Within the file are the following values:
  • input: Set to the name (without directory) of the Excel file this configuration should read.
  • output: Set to the name of the tab-delimited file you wish to write to. This will reside within /afs/ir/services/metrics/data and you should not include that directory in the filename — only the filename with any subdirectory.
  • worksheet: The name of the Excel worksheet containing the data.
  • start_row: The starting row of the data you care for, counting the first row as 0.
  • start_column: The starting column of the data you care for, counting the first column as 0.
  • fields: Everything after this is a list of the columns in the Excel file, one per line. Each line has, separated by tabs:
    • The name of that column (you choose — it will be written as a header line in the output, but is only informative)
    • 1 or 0 based on whether we require that column to be filled in for this to be a valid row.
    • date or number, based on the format of the column so that the program will know how to clean up the data.

Once you save the file, it then exists as a metrics config. To manually create the metric for the first time, on tools run: /usr/local/bin/parse-excel-metrics --force yourmetricname

If you need specific information on the Excel file, just fill in the input and output fields, then on tools run: /usr/local/bin/parse-excel-metrics --debug yourmetricname You will get a listing of all of the information in the Excel file, which you can use to find the worksheet name, and the start_row and start_column by finding the cell where data actually begins appearing.

Creating A Graph From The Data

Graphs could be created with whatever method is liked, but the standard way is to create via GD::Graph and the graph-metrics script. This script is installed on tools, running every hour between the excel parsing script and the includes file creation script. The creation of graphs is a bit more intensive … for the most part it’s not that hard, but it takes a little time to learn GD::Graph enough to get things looking the best.

The configuration files for metrics are available in /afs/ir/services/metrics/config/graphs/, where each configuration file defines a graph and its thumbnail version. To create a new graph, the best method is to find a graph on the ITMetrics webpage that is close to the result you want. Find the configuration file that has created the graph you are copying from (if the name doesn’t make it obvious, you can grep for the filename of the graph) and copy it to a new file named for the metric graph you are creating.

For specifics in the configuration, it helps to be familiar with GD::Graph. Most everything available in a set() command there is available by simply typing the setting and its value into the config file. For example:

• $graph->set(y_max_value => 8) is equivilant to, in a config file,
• y_max_value 8

There are the following special cases:

• input_file defines the tab-delimited file (created in the first section above) that will be read for data.
• lines that begin with 'column' define each column of said file read for values.
  They are read in the following format: column <legend> <column> <datatype> <graphtype> <color> <showlabel>
  • Legend is the legend for this data to appear on the chart.
  • Datatype is the type of data — valid types are percentage and number. and linepoints.
  • Color is the hexidecimal color for this data on the graph.
  • Showlabel determinds if the legend is shown or not. If not shown, the legend only exists to help you understand the config file.
• The lines [Full] and [Mini] designate that anything below that line will apply only to full-sized graphs or the mini, thumbnail graphs.
• file designates the output file (coming after [Full] or [Mini]). Directory is already set in the script.
• periods is the number of periods this graph should be for. Each period is a row of the input file, and might be a week, a month, or a year, depending on how often it is updated. This number of rows (the most recent) will be displayed on the graph.
• title is the title of the graph itself. This should only be set for the full-sized graphs. Putting a | will designate a linebreak, and wrapping in '' will help preserve whitespace if needed.
• legends is not normally needed, but is an override for the legends shown in the column settings above. These only are required when the legends asked for need to be reduced in size for the mini-graph. If, for instance, the legend for one piece of data should be "Wireless Availability from 2005-2006 by Residence", that might not all fit in a thumbnail graph. Thus, legends lets you specify an override legend for only one size of graph. It should be placed after a [Full] or [Mini] statement, and legends are separated by a |. In the above example, you might place after [Mini] the following (if it had two columns):
• legends Wireless Avail. from 05-06|Target Availability

Once you are done, run '/usr/local/bin/graph-metrics --force --all' on tools to generate your graph. It will be viewable on the web at http://www-preview.stanford.edu/services/itmetrics/images/. Tweak as needed until you like it.

This is, yes, a somewhat complicated setup. Most of that is due to the variety of graphs available.

Creation Of Files For The itmetrics Web Page To Use Via SSI

This is the last step of metrics, creating a number of files for each metric. Each file contains one piece of data, which the itmetrics web page then uses server side includes to display. The files are created by the script /ir/service/webapp/tools/usr/local/bin/output-metric-includes. This script runs hourly on tools, and for any tab-delimited file updated in the past hour, it reads specific row/columns and writes the data in each to a specified file. After that is done, if any file was updated then the script releases the web.dist.itmetrics volume.

To add a new metric, our web documentation group must specify what filenames they want data in, and what data they want in each. No data should be reformatted or calculated — we only pull data straight from one file to another in this step.

Once you have the information you need, do the following:

• Look at the tab-delimited file for the metric that you are adding include files for. Find the column numbers for whatever data they want in an include file, starting the count with 0.
• Go to /afs/ir/service/metrics/config/includes. This is where the configuration files for the include files reside.
• Copy one of these files, where the new file's name should be the name you wish to give the metric.
• Open the file for editing, changing the following information:
  • fname: the tab-delimited metric file you wish to read for the data. Do not include /afs/ir/service/metrics/data/ in the path, but only whatever subdirectory and filename come after that.
  • A number of rows that each describe the row/column of the data to pull and the filename to pull it to. This is in the format of "$i_$j filename", where $i and $j are numbers that show the position of the data we want in the metric array (see below).

The metric array is a two-dimensional array of the data in the metrics file with the last line first. The $i_$j is row_column, and save for one or two exceptions, which puts the latest data for column 2 in 0_2, for example. The latest date is always 0_0, since dates are always in the first column. The date previous to our last would be in 1_0, if you needed that for some reason.

Once you save the file, you have a new includes setup ready. To test it, on tools run: /usr/local/bin/output-metric-includes --force yourmetricname

Credits

Credit for the overall structure and business analysis of this effort goes to:
Lynne Myers (then Director, Business Analysis and Metrics) and the IT Services Customer Satisfaction team. The team can be reached at itss-survey@lists.stanford.edu

Credit for the technical implementation goes to:
Jon C. Robertson
jonrober@stanford.edu
System Software Developer
IT Services, Stanford University