Doña Ana County



Recent media attention may leave business leaders wondering about Doña Ana County’s computer system problems. We are hopeful that the following facts will help clarify matters.

* The current county management and Information Systems director inherited a woefully inadequate Information Systems operation.

* Last summer, the county recognized that its computer systems were vulnerable and hired Cervantes Consulting Center, Inc. (C3i) to conduct an assessment and implement an initial phase of improvements.

* C3i identified: ineffective and improperly configured security systems; outdated and poorly configured network equipment; outdated and overloaded hardware; an insufficient and irregular back-up system; and programming without documentation operating inside a closed system.

* Under contract, C3i beefed up the system’s security, installed new equipment, reconfigured and added speed to the network, spread applications over several host servers, prepared most systems for automatic backup, and developed an open-data-access concept that makes training easier and data migration more efficient.

* On Oct. 11, 2001, the county hired Raymond Long as director of the Information Systems Department. Long holds a bachelor’s degree in computer science from the University of Texas at El Paso, and he was formerly a systems and support specialist for the Texas Department of Transportation’s El Paso office. He also has experience with the City of El Paso as a senior network administrator for the Information Services Department and as a network administrator for the city’s library system. He is a certified systems engineer for Microsoft, a certified network associate for Cisco and a certified administrator for Novell Systems.

* On Nov. 2, 2001, the county’s outdated server – a Data General machine – suffered a serious and fatal hardware failure, and it was discovered that no hardware maintenance contract had been in effect for several years. Also, no back-up tapes were in place. The antiquity of the system made it cost-prohibitive to attempt a rehabilitation.

* The replacement Hewlett-Packard (HP) server was already partially running and in testing mode when the old server crashed, necessitating an emergency migration of data from the old systems to the new.

* Input of data into the new system began the week of Nov. 12, but on Nov. 16, the new system also suffered a hardware failure and all the inputted data was lost. Hewlett-Packard technicians were brought in immediately to assist in the weekend’s accelerated conversion process.

* By Sunday, Nov. 18, the system was partially operable again with new, fault-tolerant features, such as mirroring. On Nov. 21, the Hewlett-Packard technicians identified other hardware issues, which were resolved by Friday, Nov. 23. Working through the Thanksgiving holiday weekend, county staff and the Hewlett-Packard team rebuilt the operating systems and reloaded applications and data.

* With the system operational on Nov. 26, staff discovered software problems that were corrupting data and turned their attention to resolving those issues. By Nov. 29, the offices of the County Assessor, County Treasurer and County Clerk were testing the system from their internal computers. By Friday, Nov. 30, the Clerk’s Office was able to allow outside agencies to come to the Courthouse to perform routine research.

* On Monday, Dec. 3 – one month after the initial system crash – the county had restored Tel-Net access and was moving quickly toward having fully functional operations for internal county departments.

* By Friday, Dec. 7, all internal systems were fully functional, and by Dec. 13, fault-tolerant, hot swappable, bootable tape drives were functioning, along with automatic back-up procedures in place.

* IS Director Raymond Long has renegotiated hardware and software contracts to ensure better coverage and quicker response to future problems, and Hewlett-Packard is now able to dial into both of the county’s HP systems nightly to check for problems.

To summarize, the recent system failures are rooted in critical hardware failures complicated by historically shoddy back-up procedures and a lack of maintenance contracts. C3i brought these issues to the attention of management, which acted to correct them immediately. The migration of the system from the Data General server to the new Hewlett-Packard equipment did not cause the recent problems, but upon completion of the migration and the implementation of standard back-up procedures, the likelihood of future failures of this magnitude is virtually eliminated. Although there remains more work to be done, upon completion of the present conversion, Doña Ana County’s system will enjoy, for the first time in years, a significantly more solid and secure status.