A Note From Bill...
Last week, a contractor damaged a power line feeding Forsythe Hall. This
started an unfortunate chain of events that led to loss of power in parts
of the Data Center and to service outages affecting much of the Stanford
community. The following summary reinforces the importance of redundancy
and fallback strategies in avoiding such incidents. It also highlights
that in rare events when the unfortunate does happen, it is incumbent on
us to provide a calm, coordinated, and well-communicated response as the
incident at hand is resolved. I ask that service areas revisit existing
recovery and business continuity plans to incorporate the lessons learned
from this recent incident.
Following is a brief summary of what happened.
Late Wednesday morning, a contractor performing pre-work for a new power
feed to Forsythe Hall hit an existing power line with his jackhammer.
Fortunately, he was not hurt. The accident damaged a connected transformer,
tripped electrical breakers, and shut down power to Forsythe through the
affected line. The building's generator—tested just six days earlier
and setup to automatically start under such circumstances—failed
to auto-start. This resulted in about half of Forsythe Hall operating from
backup battery power.
Due to uncertainty about when electricians might be able to restore stable/continuous
power to the building, many service owners with affected equipment chose
to manually power down their machines to avoid a potential hard crash.
Backup battery power was exhausted at about 12:15 p.m. Affected machines
not yet manually shut down experienced a hard crash. We were
able to start the generator at about 12:40 p.m. Once generator power was
established, systems began to come back online quickly. However, some systems
were not back online until 6:30 p.m. Although the damaged power line was
repaired earlier in the evening, we chose to have the generator continue
to provide power to the building until about 8:30 a.m. on Thursday, when
we determined it was safe to return to standard power.
The IT Services Satellite Operations Center (SOC) convened at 12:15 p.m.
on Wednesday and ran until about 7:00 p.m. that evening. It reconvened
at 8:00 a.m. on Thursday and was closed at about 12:30 p.m. when it was
determined that operations were once again stable. Participants also conducted
a full debriefing of the incident on Friday. The response and support from
groups across the campus (e.g., Public Safety, LBRE, Facilities, EH&S,
Hospitals, and countless others) could not have been better. The SOC is
made up of staff from IT Services and other campus departments and organizations
who come together during significant events to manage the response and
coordinate communication.
I believe that such outages are rare because we recognize that
keeping our systems running safely and securely is a fundamental purpose
of our organization. We all work hard toward that objective. Still,
regardless of how rare, we must carefully look at these events to see where
we can make improvements to our processes and procedures that will further
minimize the impact of such occurrences to our clients. That effort has
already begun. If you would like more detail about the event, or the lessons
learned, I encourage you to contact your manager or director.
I want to thank everyone in IT Services, and our incredible partners across
the campus, for their hard work and truly collaborative effort. I also
want to thank our clients for their patience and willingness to work with
us through the event.
- Bill Clebsch
IT Services
Account Management
For those of you who work directly with clients, and others who are
just curious, this is an update on the current Account Management program
and team.
The brief definitions: the Account Managers (AMs) build and maintain
partnerships with clients who use the services we provide.
They provide a single point of contact for those clients,
help them navigate our services, and negotiate service
level agreements (SLAs). The Account Management function
also supports a liaison program in an ongoing effort
to keep the communication open with those groups for
whom we don't have Account Managers in place. This
enables us to understand their business needs and
to share information with them. The liaison program
is staffed by the Client Services Executive Director
and Directors.
Over the past months, there have been some changes in responsibility
for the AMs and a new staff member has joined the group.
Brian Leetham started in May as the primary
AM for Administrative Systems (AS) and the Department
of Athletics, Physical Education, and Recreation (DAPER).
We don't know what we'd do without him.
Liz Goesseringer has transitioned the AS support to Brian, and now
has an additional role as Demand Manager for the Data
Center in addition to continuing as the AM for the
School of H&S.
She is keeping track of Data Center hosting requests
and working with the Technical Facilities group to
accommodate as many requests as possible (see the article
in the September
5 itsinbits for more information on the
demand management effort).
Phil Reese, in addition to his client responsibilities, is active
in the area of Research Computing and continues to serve as the support
for C-ACIS—the faculty committee on computing.
Meighan McWilliam, after transitioning the DAPER support to Brian,
has taken on a new role as the internal
AM for IT Services in addition to her other clients.
She's handling all of the orders for our internal
projects and operations, which turns out to be a very
time-consuming task.
Last, but not least, Jane Marcus continues to manage her client relationships
in addition to staffing the TIPS group—a University-wide
group of administrative experts. She is also helping
to organize the IT Open House and chairing the Support
Strategies and Planning (SSP) group.
The Account
Management site includes a matrix that
maps Account Managers to their clients, and also shows
which clients have a liaison assigned.
- Nan McKenna
Client Support; Client Relations
Billing Process Improvements
The Order Management Redesign (OMR) efforts improved the quality of our
clients’ ordering and billing experience. OrderIT is now the single
point of entry to request most of IT Services' billable services,
and those orders feed directly into the Pinnacle billing system. Clients
receive email notifications when the order is reviewed, scheduled,
or completed. They can also check order status on the OrderIT
web site.
One of the key changes of OMR is the creation of the new Service Desk
as the client point-of-contact for all ordering and billing questions.
The Service Desk is responsible for the initial review of all orders
entered into Pinnacle, including confirming that an order is well-formed
and that it includes a chargeable Project Task Award (PTA). Once an
order is fulfilled, Order Processing runs a quality assurance (QA)
task, which includes new triggers to assure accurate accounting, and
then sends the order to Billing Services in a Ready-to-Bill status.
Billing Services performs
a final QA check on each billable work order and then closes over
100 orders each day. The initial order review process, custom triggers,
new QA checks, and an updated interface between Oracle Financials
and Pinnacle PTAs have eliminated many billing errors and rejected
billing charges due to closed or missing PTAs.
Monthly billing rejects are down significantly and the total billing
reject amount has decreased from $150,000 at this time last year to
less than $8,000 for all of FY07, thanks to the efforts of Judy Pincus,
who corrected PTAs on inaccurate monthly recurring charges (MRCs).
Before OMR, clients had difficulty reconciling the financial expenditure
statements from Oracle to the billing reports available on the OrderIT
web site. The billing reports were long, difficult to decipher, and
only available in PDF format. In addition, most services were grouped
together under one expenditure code: "58210 Voice Services."
As part of OMR, 58210 expanded into seven expenditure codes. Clients
can easily distinguish among services such as Voice, Network Access,
Campus Card, or Construction Projects and then reconcile the Oracle
financial statements to the new IT Services billing reports available
on OrderIT. Clients rave about the new summary and detailed billing
reports available in PDF or downloadable Excel format and they appreciate
that the reports are also available by departmental organization (ORG).
In addition, OrderIT users can now capture and schedule billing reports.
For each account or organization, clients can set up a report to run
each month and have that report emailed to them automatically without
having to visit OrderIT to set up the report each time.
The training
guide is updated with instructions and a video
helplet was created to guide users through the capture and schedule
process. Individual billing questions have dropped dramatically since
the new client reports became available. Since OMR go-live, 90% of
the recorded billing questions have been handled by the Service Desk
and 10% were handled by the Billing Services group.
The OMR project goals included greater consistency in orders and order
fulfillment of billable services, efficient order tracking, improved
accuracy in billing, a single location for clients to order services
and, most importantly, providing a better overall client experience
from the time the order is placed through order tracking and fulfillment.
Our clients are experiencing positive results since OMR has fulfilled
these goals.
- Christine Soldahl
Finance and Administration
WebEx Upgrade
The Stanford version
of WebEx is scheduled to be upgraded to the latest
version of the online web collaboration tools in
October. There have been enhancements and changes made
to Meeting Center, Support Center, and Training Center—the three
most widely-used tools at Stanford—plus changes to Sales Center
and Event Center.
Some of the enhancements include redesigned screens for starting,
joining, and ending meetings, as well as expanded controls
and options for network-based recording. Meeting Center
now supports up to six cameras simultaneously. In Support
Center, used by our Help Desk and CRC staff, there
is a new Customer Support Representative dashboard
that includes methods to connect customers to the support
session without requiring them to download the entire
WebEx client.
We have arranged an online briefing to cover the changes and enhancements,
which we think all current and prospective users would
benefit from seeing. The "webinar" is scheduled for
Thursday, September 20, from 1:30–2:30 p.m. via WebEx.
If you are interested, please send an email to Lori
Wisneski and she will email you the WebEx
invitation to this online update meeting.
- Chris Lundin
Client Support; Help Desk Services
New(er) Bike Racks in Jordan Quad
For all of you biking enthusiasts who are helping us to reduce our
General Use Plan travel numbers, or who are using bikes
around campus, I am pleased to announce that we will
be installing upgraded bike racks across Jordan Quad.
These will be a style that has a wheel-well and permit
easier securing of your bike than the various models
deployed around Jordan Quad today. Check out the racks
by the MLK buildings if you'd like to view the model
first-hand. The first four locations will be:
- Old rack between Spruce and Cypress
- Spine racks in two locations at
Pine rack between Polya and Forsythe
- Racks in front
of Forsythe
These should be done in the next couple of weeks. Following this we'll
replace the busier racks at Polya.
If you are pleased to hear this news, please thank Christopher Kittle
and Chris Lundin, without whom the upgrades wouldn't be going forward.
If you have concerns to express or questions about the replacements,
please feel free to contact me.
- Jay Kohn
Shared Communication Services
Employee Referral Drawing Winner
Digant Kasundra is the winner of the Employee
Referral $50 Visa card drawing for the month of September.
- Nilda Bonet
Human Resources