its in bits

A Note From Bill...

I have often spoken of two central elements in our efforts to improve our organization: accountability and focus on Stanford's mission. By being accountable to our clients and ourselves, we create an organization that delivers precisely what we have promised, an organization with present and future value, an organization whose whole is greater than the sum of its parts. By focusing on Stanford's key mission of research, teaching, and creating the leaders of tomorrow, we work to understand what the faculty, students, and staff are trying to accomplish so we can provide technology that assists them and contributes to elevating Stanford beyond where it is today. These elements are especially important in the current uncertain financial environment.

An important tool is our annual client survey. Each year, we ask key questions about client satisfaction with our current services and questions about the central technology gaps clients see that we may be able to fill. Naturally, the answers don't always exactly line up. Sometimes the faculty are more concerned about one thing and the students another. But by being rigorous about the content of our questions and the validity of our methodology, we are able to more precisely define the technology needs of the university community.

The client survey is especially important this year because all of our clients must continue to improve with fewer resources. This is an invigorating challenge for IT Services. Continuing to deliver the same services at the same cost will not help Stanford University in these times. We need to think in new ways about how to deliver our services more efficiently and effectively, which new services might really contribute to our clients' ability to solve their problems, and which services are no longer needed and should be retired.

In the coming weeks, we will be talking more about how we can significantly reduce the cost of existing services—while helping more clients use them—and which new services we need to develop to make a bigger difference for our clients. Already, many clients have asked us to centrally provide services they are currently maintaining themselves and to help them find ways to reduce their technology spending.

Never before have we had a more immediate need to focus on the cost of our services. Times of change are also times of transformation. As we talk with our clients, and as we see the results of our client survey in late March, we must find ways to transform what we do. By continuing to be accountable and focusing further on Stanford's core mission, we can be an essential element to the University, turning these challenging times into triumph.

- Bill Clebsch
IT Services

Stanford's Emergency Response Exercise

Stanford just concluded its biennial university-wide exercise to test its emergency preparedness, and IT Services played a key role in this effort. Exercises like this are very important toward improving our readiness in the event of an actual emergency. They help us to test our current processes and to inform us of what needs to be improved.

The scenario for the exercise was that a 6.9 magnitude earthquake had occurred on the Hayward Fault that morning. Damage was widespread, and the Stanford community would be on its own for the next 24–36 hours. Supplies of electricity, natural gas, and water were all affected. Forsythe was still up and running, as well as the voice and data networks—but the only diesel fuel we’d have available was that which was already on campus, so unless electricity was restored, the generators would soon run out of diesel fuel, and Forsythe would have to be shut down.

On February 5, the EOC (Emergency Operations Center), including the President and Provost, convened in the Faculty Club to exercise Stanford’s response to this scenario. Within the EOC, there were five subteams: Command, Operations, Public Information, Intelligence & Planning, and Logistics & Finance. Once the exercise started, the Command team informed the EOC that the priorities were to house and feed those on campus, and to carry out building assessments. It was also a priority to ensure that the hospitals were up and running, as well as the data and voice networks. For the exercise, all 25 SOCs (Satellite Operations Centers) were activated, along with the hospital. Twenty-two simulation team members inserted 675 “injects” during the exercise, and there were observers within the EOC from local governments and other universities.

As an input to the university-wide exercise, various schools and departments carried out their own field exercises on January 30. Within IT Services, 33 individuals (IT Services Emergency Response Team members and volunteers) convened for the afternoon to conduct building assessments and medical triage, review the contents of the “ark,” and set up a portion of the “tent city” that would be deployed for medical operations, communications, and shelter. IT Services buildings around Jordan Quad were also evacuated as a part of the drill.

In case you missed them, there were a couple of good articles about the exercise, in the Stanford Report and in the Stanford Daily.

Overall, the exercise was deemed to be successful, but there is still room for improvement. We will be taking notes from the EOC, IT SOC, and field exercise to develop action plans toward making improvements to our processes. We will also be charting out quarterly exercises to test our readiness—from building evacuation drills, to convening the IT SOC, to recovery exercises once the Livermore Server Room is fully provisioned.

Thanks to everyone for your participation and cooperation in the January 30 field exercise and building evacuations, and in the February 5 university-wide exercise. Special thanks to Tom Prussing (Computing Services) for his leadership of the field exercise and to Chris Lundin (Client Support) for his leadership of the IT SOC.

- Matthew Ricks
Computing Services

Remedy Upgrade Scheduled for March 7–8

Back in November, we deferred a Remedy upgrade because of problems with the server software we encountered during testing. The vendor, BMC Software, has since released a highly-tested production update which we are confident (while continuing to test!) will perform without issues. So we're back on track to schedule a Saturday–Sunday outage to get our server software updated. We've tentatively identified the weekend of March 7–8.

HelpSU will be unavailable from 7:30 a.m. Saturday, March 7, until noon on Sunday, March 8. Customers will be able to submit HelpSU requests but they will be queued and added to the application once it is up and running again. Beginning Saturday morning, March 7, customers will see this web page.

The release delivers numerous back-end changes. The most notable benefit to Stanford is an upgrade of CMDB v2.0 to CMDB v2.1, which promises substantial performance increases in processing and reconciling CMDB data. With the CMDB Project under way, we believe that now is the best time strategically to do the upgrade.

Some significant benefits come in the form of increased browser compatibility:

Windows Vista support for Firefox 2.0 and higher
Mac OS X 10.5 support for Safari 3.0 and higher

Continued thanks and appreciation to the Computing Services teams (Application Support, Data and Systems Integration, and Systems Administration) who are working together on this major update.

- Chris Lundin
Client Support

Pinnacle Application Hardware and OS Upgrade

The house is being lifted and a new foundation is being poured! On the weekend of March 7–8, the Application Support Team will be busy implementing Pinnacle on new hardware and platforms. The legacy Solaris environment has been churning away on eight-year-old Sun Fire servers running Solaris 8. The decision was made to migrate the Pinnacle Application to Dell servers running Red Hat AS 4.

The benefits of this project include:

Performance: Users will notice a tremendous increase in performance. Initial benchmarking has shown 15-minute imports are completing in two minutes. Billing processes that took five hours on the legacy system are completing in under an hour!
Alignment with Computing Services Initiatives: UNIX Team: end of support for Solaris OS. Storage Team: End of support for EMC DMX.

The Pinnacle System will be unavailable from 5 p.m. Thursday, March 5 through 7:59 a.m. Monday, March 9 as we upgrade the hardware. The department Order IT web site and the student My IT Services web site will be not be available during the upgrade. All Pinnacle users must be logged out of the Pinnacle PRD system by 5 p.m. on Thursday, March 5.

The Pinnacle Backup PRD version will be available to view service profile information during the upgrade. Please keep in mind that any data that is updated in the Backup version will need to be re-entered into Production after go-live on Monday, March 9. Technicians can do work, but changes will need to be given to Order Processing for update to Production on Monday.

Reports in Reportmart1 that point to Pinnacle PRD will not be available. Some reports point to Backup PRD and you can continue to run those reports during the upgrade. On Monday, March 9 Pinnacle users will access Pinnacle and run reports as they normally would.

Very special thanks go out to Greg Janicki, who has managed this project and assumed the role of both Project Manager and Technical Lead. There are many teams and many individuals who have had their hand in this embedded project, and we would like to thank everyone who has been involved.

- Anne Pinkowski
Application Support

Don't Ask Us To Do Something We'll Get In Trouble For!

The IT Operations Center (ITOC) monitors most of our critical IT infrastructure and services, alerting Subject Matter Experts when certain alerts appear in our SMARTS monitoring system. ITOC staff have access to the AMCOM paging system and various on-call lists, and are available to contact the proper staff 24x7. All of the groups we support have provided us with on-call lists and documentation about specific troubleshooting steps, including who should be contacted under certain circumstances. Our staff work hard to follow these procedures.

From time to time, IT Services colleagues call in to the IT Operations Center and ask us to short-circuit or circumvent our documented procedures. That places ITOC staff in the difficult position of 1) trying to be responsive to the request while 2) honoring the written procedures that we have been given. From time to time we bend the rules only to be reminded "that's not what we agreed to."

So, I've asked ITOC staff to follow our written procedures. If our procedures say to page the On-Call Networking person, we are going to do that rather than page the networking staff member you just know is the right person. If our written procedures indicate a device or service is supported only during business hours, we're going to demur if you ask us in the middle of the night to wake someone up. (Of course, we'll work quickly to correct any inaccurate documentation.)

If you are a Subject Matter Expert calling in to the IT Operations Center, we ask you to bear with us as we work through our documented procedures and contact protocols. Please let me know if you have any questions.

- Chris Lundin
Client Support

Just-in-Time Subtickets Process Improvement

I am happy to announce that the "Just-in-Time Subtickets" process became reality yesterday, Tuesday, February 17. Just-in-Time is a series of process improvements that further refines the current data center order fulfillment process.

Expected benefits include:

Reduction in overall fulfillment processing time
Subtickets can be processed immediately—groups no longer need to keep checking the superticket to determine if it is their turn in the process
Networking can assign ports before TFAC subticket, eliminating costly rewiring
Parallel processing
More client focused. For those who choose, we will interview the client and fill out form on their behalf
Metrics can now be attained on the individual fulfillment steps, so further process refinement can continue
The breakdown of the process and sequencing of the tasks can be used as the specification to automate workflow in the future

Proposed Next Steps:

Client-friendly questionnaire with conditional logic
Collaborative internal form to replace fulfillment form
Automated workflow

We have scheduled three Thursday drop-in sessions on 2/19, 2/26, and 3/5 from 2–3 p.m. in Polya 152 to address any questions or issues with the new process.

Many thanks to all who contributed and endured the thrilling process maps! Order Processing: Jared Sanz-Freilich; TFAC: Allen Penny and Pat Luma; Networking: Dave Macia and Alvin Chew; Windows: Sean Riordan; UNIX: Jon Pilat; Storage: Gerald Villabroza; Database: Cindy Lou Chapman; Business Services: Steve Loving, Meighan McWilliam, Ramon Herrera, Judy Pincus, and Caren Kammeyer.

- Katherine Pappas-Kassaras
Project & Process Management

Stanford Email & Calendar Update

After comprehensive regression testing, the IEC project team upgraded Stanford Email & Calendar to Zimbra version 5.0.12 on Friday, February 13. The new version provides many improvements, including:

elimination of double booking of conference rooms when times overlap but don’t match exactly
faster performance when using the Find Attendees search
faster performance when initially loading Zimbra
a new version of the Outlook Connector that synchronizes much more quickly
improvements to the iPhone sync

For more details, visit our Known Issues web page. It includes the resolution on all bugs that were fixed as well as outstanding issues.

Stanford is still waiting for one additional release prior to migrating Sundial calendar data to the Stanford Calendar for the rest of campus.

Over the next few weeks, we ask that you please submit any new issues or observations via HelpSU. It’s important to track and document any new, unexpected behavior after the upgrade.

- Ammy Hill
Campus Readiness

Vacation Accrual Policy Changes

By now you should have heard that Stanford has adopted a new, reduced vacation accrual maximum beginning in 2010. For more details about this, please see the related article in the Stanford Report.

With adequate notice, departments are permitted to require use of vacation to bring balances down. The following dates have been identified as suggested vacation days, which occur before or after a remaining 2009 University-paid holiday. As always, use of vacation should be requested, scheduled, and approved by your manager, with as much notice as possible.

For more information on vacations, please see Administrative Guide Memo 22.5, [PDF].

Please direct inquiries to your supervisor or IT Services HR. Thanks!

- Nilda Bonet and Cheryl Miller
Human Resources