Search Stanford:

Strategic Vision

• Overview
• Technology Principles

Documents by Service Area

• Authentication
• Authorization
• Backups
• Calendar
• Desktop Management
• Directory
• Email
• Identity Management
• Integration
• Networking
• Storage
• Voice
• WWW Services
• Windows Infrastructure

Related Pages

• Planning Documents
• IT Services Projects
• Project Management Office

• IT Services
• Strategic Vision
• Storage

Storage Strategic Vision

Written by Scotty Logan, Russ Allbery and Ross Wilper

This document attempts to outline a strategic vision for storage at Stanford University, focusing specifically on central offerings provided by and managed by IT Services. Note that WebDAV is mentioned here but covered in more depth in the the WWW vision.

Principles

Storage is an essential infrastructure component for most applications and systems, so a storage service must balance widely divergent requirements: from high throughput, highly available storage for production databases, to low cost disk for backups and infrequently used files. Since no single storage device can satisfy the full set of requirements, some amount of administrative simplicity must be sacrified in order to provide appropriate storage at a reasonable cost. In the next few years, the emergence of standards-based devices and management software should reduce the risk and complexity of multi-vendor storage environments.

A storage area network provides server access to a shared pool of block-level storage. While it is possible to share the same volumes between different servers on a SAN, volumes are more commonly dedicated to a single server. A SAN allows efficient storage utilization and easy allocation and de-allocation of storage to and from servers.

A distributed file system provides universal access to the same file space, allowing users to freely move between computers and locations while working on the same files, allowing easier collaboration among users, and providing an integration point with other central services that read data out of the distributed file system. A distributed file system is vital to the support of computer clusters and shared work spaces, and in growing use by clients who want to access the same data from multiple places.

More than many other technologies, distributed file systems do not work well piecemeal. Particularly in the context of growing cross-disciplinary programs and cooperation between departments, workgroup-level file servers are sharply limited in their ability to meet client needs. If the same file system is not available universally to the entire campus, many of the advantages of having a distributed file system are lost. IT Services is uniquely positioned to provide these services to campus; if we do not do a good job in doing so, departments cannot easily fill in for our shortcomings.

A backup service enables reliable backup copies to be made of all data, including work-related data, administrative data, personal data and application and system configuration information. The service should also balance fast restores, which require local backups, with disaster protection, which requires off-site backups.

Some principles apply to storage services in general:

• Security. Storage systems must support authenticated access, network encryption, and authorization and access control. It must be flexible in its authentication requirements to meet the needs of private data, shared data confidential to a particular group, and distribution of data to the Stanford campus as a whole.
  
  
• Cross-platform support. Our clients use Windows, Mac OS X, and multiple versions of Unix and Linux on their desktops, laptops and servers, and want to be able to share storage between all platforms; in the case of file services, clients want to be able to share files between all platforms. In the future, we will likely also need to provide access from mobile devices.
  
  
• Scalability. A central file service must handle simultaneous connections from thousands of separate clients; a central backup service must handle hundreds of simultaneous client connections and reliably manage backups for thousands of clients; a central SAN must handle multiple storage devices and hundreds of servers, with varied performance and capacity requirements.
  
  

ITS's storage services should provide a centrally-managed data storage environment for use by both IT Services and client systems, applications and services.

Specific areas of focus are upgrading the main SAN to make it more scalable and reliable, deploying improved AFS clients for Windows and Mac, deploying CIFS and DFS services and investigating desktop backup tools.

Technologies

Stable core technologies:

• Switched Fibre Channel (aka Fibre Channel SAN) as the primary storage mechanism, using 1Gb/s, 2Gb/s and eventually 4Gb/s interconnects.
• AFS, specifically the OpenAFS implementation, including the Windows, Mac OS X, and Unix (including Linux) clients and the OpenAFS file servers.
• CIFS, currently primarily for Windows systems.
• PC Leland and MacLeland as the distribution platforms for the Windows and Mac OS X AFS clients.
• EMC Symmetrix DMX for applications that require high availability and high throughput, albeit at a high cost.
• EMC CLARiiON, with Fibre Channel disks, for applications that do not need DMX-class throughput and reliability.
• EMC CLARiiON, with ATA disks, for on-site backup copies and general purpose file serving.
• Business Continuance Volumes (BCVs) on EMC Symmetrix (including the new DMX) arrays for creating database clones for backups and reporting builds.
• IBM's Tivoli Storage Manager for server backup and archiving.
• LTO and 3590 tape technology for backups.

Technologies new to IT Services:

• Microsoft DFS for linking CIFS servers into a single, federated namespace.
• Basic CIFS for Linux and Mac OS X using the Samba-based clients.
• CIFS offers Offline Files and Folders, which enables automated replication of files between Windows client computers and CIFS file servers.
• Kerberos v5 as the authentication technology for AFS and possibly NFSv4 (it's already in use for CIFS).
• WebDAV (covered in the WWW vision).
• Desktop backup. While some users and departments still use the centrally managed TSM server (aka BaRS), the service was sunsetted in Fall 2004; a new project was recently started to investigate desktop backup options.
• NFSv4 is still developing, and while it is available both from a few vendors (Sun and Network Appliance) and in the Linux 2.6 kernel, none of those are complete implementations of the specification. IT Services should continue to track NFSv4 progress. NFSv4 improves on NFSv3 in many ways; the most obvious is the required Kerberos 5 support in all clients and servers.

Emerging technologies: (see Projects and Research)

• VSAN (Virtual SAN) and IVR (Inter-VSAN Routing). VSANs are very similar to IP VLANs, and are available on the new Cisco Fibre Channel directors and switches. VSANs can be used to isolate groups of hosts and devices, minimize the impact of reconfigurations, and interface with non-Cisco SAN fabrics. IVR enables communication between hosts or devices in different SAN fabrics without merging those fabrics.
• MetroSAN. As IT Services moves servers out of the Forsythe Data Center, the SAN must be extended to Sweet Hall, the ECHs and any other location where IT Services houses servers. Basic long wave Fibre Channel over single mode fiber should suffice for Sweet Hall and the ECHs; other technologies (such as extended long wave Fibre Channel, iSCSI, FCIP, or iFCP) might be needed for other locations.
• iSCSI (SCSI over TCP/IP) is emerging as a useful technology for low-throughput hosts, "stranded" hosts (those which are not easily connected to the FC SAN fabric) and for some long distance SAN solutions. Windows and Linux are well supported as iSCSI clients, and various iSCSI arrays are available, as are gateways to FC SANs. Debian systems, which are rarely supported by FC vendors, should be classified as "stranded" and connected to central storage via open source iSCSI software initiators.
• Inline encryption devices are now available for both SAN (Fibre Channel) and NAS (NFS, CIFS), enabling data to be protected while it is stored on disk. Investigation is required to determine whether there is sufficient need for this level of data protection to justify the cost.
• Various replication products are available for SAN and NAS. As with encryption, investigation is required to determine if there is a need; any such investigation should be tied to a Business Continuance / Disaster Recovery effort.
• Provisioning storage is currently a multi-step manual process. The EMC management tools owned by IT Services have the ability to automate many of those steps. Similar tools are also available from other vendors, as are tools to automate change management.

Deprecated technologies: (see Projects)

• Kerberos v4 as authentication technology for AFS. See the authentication vision for more information on the project to phase out Kerberos v4 entirely.
• Direct Attached Storage (DAS) for application data should be phased out. DAS is hard to share between servers, which results in most servers having too much or too little storage; it also complicates failover since it requires recabling to another server (or that every server have a dedicated spare which is twin-tailed to the same DAS device).
• Non-RAID and RAID0 offer no protection and should also be phased out; they are only acceptable in situations where the service or application is built on a pool of identical servers.
• CIFS file servers that are not integrated with the central Active Directory forest should either be integrated, or be replaced with ones that are.
• NFSv3 is insecure and should not be used on open, public networks. Kerberized NFSv3 is more secure, but not widely available. AFS, CIFS or NFSv4 should be used instead. NFSv3 is suitable for limited use on private networks, e.g. as the shared storage for a clustered Oracle database.

Other technologies in use:

These technologies are curently deployed and useful in specific circumstances, but either are not attractive for broader user or have a limited scope of applicability; we have no recommendations on their use at this time.

• Dantz Retrospect for Windows and Mac backup. Retrospect is used by many departments who either have their own IT staff, or who contract with IT Services for support.
• Connected.com for Windows desktop and laptop backup. IT Services has an existing contract with Connected, but so far less than 150 systems are being backed up.
• Internal disk (another form of DAS) is still used for many boot drives and for servers where the application data is widely replicated.
• NFSv3 is used for sharing some IT Services home directories between various Administrative Systems servers, and is is used read-only as part of the system build process for Linux and Solaris systems. It is also used for software distribution to administrative systems and database systems, but this use may be reduced or phased out in favor of another distributed file system.

Projects

First:

• Complete current SAN upgrade. This includes the replacement of the EMC CLARiiON CX600 FC array with a CX700, and migration from the old environment (four McData directors, one EMC Symmetrix 8830, one EMC Symmetrix 8530) to the new environment (two core-edge fabrics with one Cisco MDS directors and four Cisco MDS switches each, one EMC Symmetrix DMX2000 and one EMC CLARiiON CX700).
  
  
• Integrate PTS groups with directory and registry workgroups so that they can be managed through the same interface and users don't have to duplicate their group maintenance work in multiple places. Part of this project will be working through the implications for namespace, group nesting, and group ownership.
  
  
• Move the AFS VLDB servers to Debian (which includes the Kerberos v4 kaserver) and redo the template and documentation to match our current Linux server practices.
  
  
• Upgrade all of the AFS servers to 1.2.13 and put better monitoring and file server parameters in place based on consultation from Sine Nomine.
  
  
• Develop DFS & CIFS service. The hardware, an EMC NS704G NAS gateway, is on-site, but there are currently no resources available to work with it.
  
  
• Deploy a TSM server for SUL/AIR. Most likely, this will be on a Solaris system for use as part of the Stanford Digital Repository.
  
  
• Desktop backup service. The production selection phase of the project is about to kick off, and should finish in early August. The duration of the implementation will depend on the selected product.
  
  
• Expand the use of ATA disk for onsite backup copies; this will likely be a combination of TSM and the product chosen for desktop backups.
  
  
• Upgrade TSM servers to version 5.3. Several of the TSM servers are running older, unsupported versions of TSM; upgrading them will require help from the DBAs, who will need to upgrade BMC SQL-Backtrack (DataTools).
  
  
• Determine requirements for encryption and replication.
  
  

Next:

• Select and deploy tools to automate common storage provisioning tasks and assist with change management.
  
  
• Switch AFS from Kerberos v4 to Kerberos v5 authentication. This is also covered in the authentication vision.
  
  
• Migrate the AFS servers to Linux. This involves testing use of the SAN on Linux, including both Red Hat and Debian, deciding whether to use Red Hat or Debian for the AFS servers, building a new template based on standard AFS server packages, and migrating the data to the new systems. The new configuration should also use the dual-redundant SAN connection paths recommended by the block storage strategy.
  
  
• Deploy OpenAFS 1.4 clients, giving us Linux 2.6 support and multiple other client improvements.
  
  
• Acquire or develop an abstraction layer for the management of volume snapshots and clones. The current set of scripts used by the DBAs are tied to EMC's SymmAPI interface for BCVs on Symmetrix.
  
  
• Use IVR to link the various SANs in the data center; this is primarily to allow NAS heads direct access to tape drives on the TSM SAN for backups.
  
  

Later:

• Enable nested PTS group membership on the AFS VLDB servers, with related changes to our documentation and processes around PTS groups.
  
  
• Deploy OpenAFS 1.4 servers, most notably giving us large file support and a threaded volserver among other improvements.
  
  
• The current "new" SAN maintenance contract will expire at the end of 2007, so planning for a maintenance extension or another upgrade should start no later than Fall 2006.
  
  
• Replacement of all AIX TSM servers with Linux TSM servers. Solaris TSM servers should also be migrated to Linux if there are no large Solaris database servers which require backups direct from cloned SAN volumes to tape. All the AIX and Solaris TSM servers will need to be upgraded to a recent version of TSM before migrating to Linux.
  
  
• Deprecation of internal storage. Low cost, reliable mechanisms are needed to replace internal disks. iSCSI might be an option - NIC chipset manufacturers such as Broadcom already have iSCSI support in their newest products, but server vendors will need another 12 to 18 months for integration and testing.
  
  

Research

The following areas should be explored with an eye to their long-term inclusion in our storage strategy. Without more information, it is premature to specifically identify any of these areas for projects, but if the research pans out, they may move from this section into the project section for a full production implementation. Each research project is associated with an initial application that we can use to test the results of the research.

• Evaluate distance solutions such as long wave Fibre Channel, iSCSI, FCIP and iFCP. These technologies will be needed to provide storage to the ECHs and off-site data centers.
  
  
• NFSv4 has a great deal of industry momentum around it and is becoming more and more widely deployed. It supports strong Kerberos v5 authentication, rich authorization, and location-independence and is platform-neutral, making it the most likely possibility to be able to contribute to the space where currently AFS stands alone. Since NFSv4 will likely be the native network file system technology shipped with future Unix-based operating systems (basic implementations are currently available in AIX5.3, Solaris 10 and the 2.6 Linux kernel), we need to continue to track and pilot NFSv4 and watch for applications that would benefit from using it instead of other file systems. We don't yet know what role NFSv4 will be able to play in our infrastructure, and need to actively experiment to see where it fits.
  
  
• Support for CIFS and DFS on platforms other than Windows is becoming more stable and more widespread, and we should explore the viability of deploying CIFS on Unix and Mac OS X as well as Windows.
  
  
• Evaluate NAS and SAN encryption products.
  
  
• Evaluate iSCSI for low-throughput Windows and Linux servers. iSCSI could provide cheap and easy block-level storage access for clients across campus, although it remains to be seen how much demand exists, and how much of it could be satisified with file-level networked storage (CIFS, etc.)
  
  
• Significant new development is being done on AFS, particularly in the area of the native Windows client, disconnected operation, and better integration with Kerberos v5. We should be alert for and take advantage of opportunities to pilot and beta-test this work and help contribute to and steer its development, possibly by funding development through our relationship with Sine Nomine.
  
  
• Investigate low cost SAN equipment. Currently, our low end SAN storage uses large EMC CLARiiON CX600 arrays which are near the top of the CLARiiON family. For tasks, such as backup storage, that require absolutely no "intelligence" in the array other than basic hardware RAID (ie. no snapshots, replication, etc.), or for situations where the full capacity and performance of a CX600 will never be used, IT Services might be better served with a simpler, lower cost solution.