|
|
|
||
|
|
A Case Study: |
||
|
Research
and |
Connecticut Department of Transportation
|
|
|
DOT-VNTSC-FHWA-03-06 Final Report
John A. Volpe National Cambridge,
Massachusetts
Transportation Systems Center November
2003
Planning
and Policy Analysis Division
Final Report DOT-VNTSC-FHWA-03-06
A Case Study:
Connecticut Department of Transportation
An Innovative Procurement and Implementation Process
PDF Version 692KB
Allan J. DeBlasio
David W. Jackson
November 2003
Prepared by
U.S. Department of Transportation
Research and Special Programs Administration
John A. Volpe National Transportation Systems Center
Planning and Policy Analysis Division
Cambridge, Massachusetts
Prepared for
U.S.
Department of Transportation
Federal Highway Administration
Office of Operations
and
Intelligent
Transportation Systems Joint Program Office
Washington, D.C.
Foreword
This report was prepared by staff of the U.S. Department of Transportation (U.S. DOT), John A. Volpe National Transportation Systems Center (Volpe Center), Planning and Policy Analysis Division for the Federal Highway Administration (FHWA) Office of Operations and the U.S. DOT Intelligent Transportation Systems (ITS) Joint Program Office (JPO). Volpe Center staff was requested to conduct a program review to examine what delays continue to affect ITS projects and to identify actions taken by staffs at state DOTs to address these issues. From October 2002 to September 2003, the Volpe Center team gathered information from visits to 12 states and interviewed FHWA division office ITS specialists and field engineers and state DOT ITS managers and engineering staffs. During their investigation, the team learned of an innovative procurement and implementation process used by the Connecticut Department of Transportation (ConnDOT). This report documents the case study conducted to explore that approach.
The Volpe Center study team consisted of Allan J. DeBlasio, the project manager, David W. Jackson, and Dana M. Larkin of the Volpe Center Planning and Policy Analysis Division; Margaret E. Zirker of Cambridge Systematics, Inc.; and Terrance J. Regan of Planners Collaborative, Inc. Joseph I. Peters of the ITS JPO and Larry Swartzlander of FHWA Office of Transportation Management were the task managers. Messrs. DeBlasio and Jackson authored this paper.
The authors thank Mr. Robert Ramirez from the FHWA Connecticut Division Office; Mr. Harold Decker, Mr. John Korte, and Ms. Mary Matuszak from the ConnDOT; and Ms. Celeste Cashman from the Connecticut Department of Administrative Services for their cooperation and for the time they spent supporting this case study. The authors also thank Mr. Carl-Henry Piel from the IBI Group, Inc. and Mr. Saul Katz from Traffic Control Systems for their input during this case study.
Connecticut Department of Transportation
An Innovative
Procurement
and Implementation Process
In a recent study conducted by the John A. Volpe National Transportation Systems Center (Volpe Center), researchers found that intelligent transportation systems (ITS) projects that included the development of software, the integration of systems, or both continue to experience a significant number of delays. These delays may be caused by incorrectly estimating the level of effort needed to successfully complete such projects, improperly defining system requirements, inappropriately managing changing requirements, or employing a contractor not skilled in the required areas. The study also revealed that staffs at state departments of transportation (DOTs) are addressing these issues using innovative processes. This case study describes one such unique procurement process used by the Connecticut Department of Transportation (ConnDOT).
Project Background
The ConnDOT is implementing a large-scale freeway traffic management system in the greater Hartford area. The goal of the project is to alleviate congestion due to incidents and other activities. The project was the outgrowth of an ITS early deployment plan completed in 1997 by ConnDOT, the Capital Region Council of Governments, and the Federal Highway Administration (FHWA).
This project covers 60 centerline miles of Interstate Highways 84 and 91 and State Route 2 and includes the installation of a fiber-optic cable network, 104 video camera sites, 50 traffic flow detector stations, and 39 variable message signs (VMS). A video and data transport system will transmit video and traffic data to and from the ConnDOT Newington Highway Operations Center. The system will also be used to share video images and data between the Newington Highway Operations Center, the ConnDOT Bridgeport Traffic Operations Center, the State Police Communications Center located within the Connecticut Department of Public Safety Headquarters in Middletown, the State Police Troop H Barracks in Hartford, and the City of Hartford Traffic Operations Center. Figure1 illustrates the project location and Figure 2 shows the locations of the operations centers.
Due to the size and complexity of the project, ConnDOT management decided to divide it into five phases. Each project phase was designed to stand alone, which allowed work to commence on early deployment phases while subsequent phases were still in design. This was extremely important, as the aggressive deployment schedule for the project required that the overall project be completed less than three years after design commenced.
The first phase called for renovation of the ConnDOT Newington headquarters operations center. The second phase covered the installation of the mainline fiber-optic conduit. The third phase included the installation of 39 VMS, which are connected to the operations center by dial-up telephone lines (and in the future by fiber-optic cable). The fourth phase focused on central-control software development. Work on these first four phases occurred simultaneously.
Figure
1. Hartford Area
Freeway Traffic Management System
The fifth and final phase involved the installation of the electronics and communication infrastructure, including closed circuit television (CCTV) cameras and controllers, microwave vehicle detectors, a high speed synchronous optical network< (SONET) fiber-optic backbone system and multiplexing/de-multiplexing equipment, as well as interagency Ethernet and teleconferencing subsystems. It also included the integration of the system components.
ConnDOT staff executed two contracts to complete the final phase. The first contract was a traditional low-bid construction contract for the installation of a fiber-optic network and field equipment, such as cameras and detectors. The second contract was for the purchase and system integration of the video and data transport system, which supports the transmission of video and data from the CCTV cameras, pan-tilt-zoom controllers, VMS, and microwave vehicle detectors and includes the video switch, the video demultiplexers, and the SONET. The second contract also includes installation of control equipment in the two ConnDOT operations centers and the two State Police locations. This case study focuses on the second contract, which is called the Purchase of a Video and Traffic Data Transport System for the Department of Transportation Hartford Area Incident Management System or the Procurement Project.
The Procurement Project totaled $5,380,856.63 and was funded using federal Congestion Mitigation and Air Quality (CMAQ) funds and state matching funds. The contract specified a 12-month implementation period followed by a 12-month warranty period.
Figure 2. Operations Centers
Project Design
Transmitting video and data through the communications system and between the centers that had to be linked required a new and complex design. ConnDOT wanted to ensure that all the system equipment deployed by the contractor would work according to specifications. They wanted a methodical step-by-step installation and a system of checks and balances.
The ConnDOT ITS staff with support from a consultant designed the project in-house. ConnDOT staff developed the operational needs, functional requirements, and quantity of products; the consultant prepared the detailed specifications and technical plan. The design was unique to the ConnDOT staff because the level of communications and control system detail was extensive. Typical design details, as used in many construction projects, could not be used for this project. As one ConnDOT staff member stated, “Everything had to be written out.” There was no example to follow. ConnDOT staff sought to provide bidders with complete information including equipment requirements and block diagrams. The contractor noted that the design for this project contained one of the better set of specifications with which he has worked.
ConnDOT staff did not want to use propriety protocols but instead used existing and emerging standards whenever they could. Their design included national interface standards, fiber-optics standards, and communications protocols. They also used National Transportation Communications for ITS Protocol (NTCIP) standards for the VMS and will use center-to-center standards. They are currently using the center-to-center data dictionary and the traffic incident management data dictionary.
Procurement Process
The procurement and installation of the video and data transport system was perceived to be complicated. ConnDOT staff wanted flexibility and appropriate control of the procurement. ConnDOT staff wanted to ensure that bidders were qualified to perform the work and that the selected equipment was proven and reliable. Although ConnDOT staff still identified price as a key criterion, they also placed a heavy emphasis on quality.
In the State of Connecticut, most procurements greater than $10,000 are administered by the Connecticut Department of Administrative Services (DAS). Therefore, ConnDOT purchasing and ITS design staffs worked closely with DAS staff to prepare the procurement package. All parties agreed that the use of a request for proposal (RFP) would produce the best outcome for this highly technical project. Figure 3 shows the key organizations in the procurement process.
Figure 3. Key Organizations
Their decision was facilitated by state statute. In 1999, the Connecticut State Legislature passed an act that granted the Commissioner of Administrative Services authority to use competitive negotiation as a permissible method for purchase and contracts.[1] It also required the commissioner to establish standards and procedures for using competitive negotiation for purchases and contracts, including but not limited to, criteria which shall be considered in making purchases by competitive negotiation and the weight which shall be assigned to each such criterion.[2]
Previous to this act, the commissioner was permitted to use competitive negotiations just for purchasing advertising space or time, and the state’s chief information officer was allowed to use competitive negotiations for the purchase of information system and telecommunication system facilities, equipment, and services.[3] The procurement package was also reviewed and accepted by staff at the state’s attorney general office and DAS legal staff.
The DAS advertised the project in two Connecticut newspapers and posted the RFP on their Web site. DAS also sent notices by e-mail to firms registered with the agency. Several trade magazines also notified their members of the RFP. Over 20 RFPs were requested and three proposals were submitted.
In the RFP, ConnDOT requested bidders to submit a detailed physical system design and equipment specifications or “catalog cuts” with their bid package. ConnDOT staff and their consultant evaluated the design and technical submittals against project specifications. The selected bidder noted that his firm typically has to design a system in order to bid on it, but not in as much detail as was required by the ConnDOT RFP.
Bidders were also given the opportunity to suggest improvements to the ConnDOT design. For example, the selected bidder suggested a change in the way video images would be sent to the two ConnDOT operations centers. The bidder’s project manager suggested the video camera signal be equalized locally at the camera site in the field rather than equalizing the signal at the central operations center.
In addition, ConnDOT required the bidders to submit a list of projects that they completed dealing with fiber optics and data communications. Moreover, the bidders had to provide references for each of the projects they listed. The selected bidder stated that supplying references is common. He had previously provided references to other agencies when bidding on jobs.
Using the protocol presented in Appendix A, ConnDOT staff and their consultant contacted the bidders’ clients, other agencies, subcontractors, and vendors of the bidders. In order to distribute the workload, ConnDOT staff called public agency references, while their consultant spoke to private sector references. ConnDOT and consultant staff felt that valuable input was gained during the reference-checking process. It enabled team members to learn more about the firms and vendors identified in the bidders’ proposals.
There was a pre-bid meeting, and prospective bidders were given an opportunity to submit questions. Nine questions were received, all on technical aspects of the RFP. ConnDOT staff supplied answers to DAS staff, who released an addendum to the RFP noting the questions and the responses.
Contractor Selection and Contract Award
ConnDOT and DAS staffs developed a technical proposal assessment matrix to evaluate the bids, and a team of three ConnDOT staff members and one DAS staff member reviewed the proposals. The evaluation matrix, shown in Table 1, gave bidders an overall (weighted) score that included pricing, quality of design, and opinions of references. Although bidders were aware that they would be evaluated on qualifications as well as price, the selection criteria was not made public until after the contract was awarded. The selected firm actually had the higher price of the two qualified bidders.
Once a bidder was selected, DAS and ConnDOT staffs entered negotiations with the firm. The negotiations included discussions on price as well as implementation procedures and other technical aspects of the project. The two staffs wanted to ensure that the selected bidder understood all aspects of the project including maintenance responsibilities. ConnDOT staff adjusted the schedule proposed by the bidder in order to coordinate this project with other related ITS projects currently underway. After one round of negotiations, DAS signed a fixed-price contract with the selected bidder, and the ConnDOT purchasing staff issued a purchase order.
Implementation Process
As previously mentioned, ConnDOT staff recognized the complexity of the design and wanted to ensure compatibility among the different types of field equipment and the infrastructure. This system was further complicated because the contractor was supplying equipment from more than 20 vendors and sub-vendors.
The first step to ensure the design would work was the requirement that the contractor develop a “proof of concept.” This required the contractor to develop and test a prototype within the first “two months or so” of the contract. Pricing for the prototype was specifically called out in the bid request.
The prototype was a subset of the entire system; one or more of each type of component had to be installed and tested. A prototype test plan document was developed by the contractor and approved by ConnDOT staff prior to commencement of the test procedure. Although it was more detailed than a normal prototype test, the test only lasted one full day due to the preparation of the contractor. It was conducted at the contractor’s facility. During the test, some noise and static appeared on video images. An issue with a loose fiber-optics connector assembly was easily located and corrected during the prototype test.
Table 1. Technical Proposal Assessment Matrix
|
Category |
Weight within category |
Weight among categories |
|
|
|
|
|
1. Pricing |
|
35% |
|
100% |
|
|
|
|
|
|
2. Technical requirements |
|
30% |
|
70% |
|
|
10% |
|
|
10% |
|
|
10% |
|
|
|
|
|
|
3. Bidder qualifications |
|
20% |
|
20% |
|
|
20% |
|
|
25% |
|
|
25% |
|
|
10% |
|
|
|
|
|
|
4. Bid requirements |
|
10% |
|
15% |
|
|
15% |
|
|
15% |
|
|
20% |
|
|
15% |
|
|
20% |
|
|
|
|
|
|
5. Warranty and maintenance |
|
5% |
|
25% |
|
|
25% |
|
|
25% |
|
|
25% |
|
|
|
|
|
The second step was the requirement that the contractor develop a “proof of performance.” This required the contractor to conduct a system demonstration test at his facility. The system demonstration included all of the field equipment that the contractor would be installing in the field. This test would ensure the system would be integrated properly. It differed from normal system testing that usually occurred after the equipment was installed in the field and done using software loaded on a laptop computer. A system demonstration test plan document was developed by the contractor to detail the test procedure process and the equipment required for the test. ConnDOT staff and the contractor used this document to log the performance of the equipment and note any issues that needed to be addressed prior to installing the equipment in the field. Figures 4 and 5 show the contractor’s facility during the system demonstration test.
Figure 4. System Demonstration
The contractor noted that this was an expensive and time-consuming activity because of the amount of equipment that needed to be included. There was more manual labor required than he estimated. Furthermore, the contractor had to bring additional electrical service to his facility to power all the equipment being tested. He also had to purchase and install approximately 5 miles of cable, all of which was used for the equipment installation. Because the CCTV cameras were being installed under the infrastructure contract, the contractor also had to purchase 12 cameras to integrate into the system test. The system demonstration test took two days even though the contract allotted three days.
During the process of implementing the system design, the contractor found a problem with the approach used by the video matrix switch vendor in interconnecting the video switches between the Newington Highway Operations Center and the main fiber hub. After discussion with the video switch manufacturer, the contractor devised a solution that required the design and building of special circuitry to enable the connection of the two switches. The circuitry was incorporated into the switch design and is functioning without any problems. ConnDOT staff was very pleased with how the contractor handled this issue and felt that they actually received a better product from the “extra effort” of the contractor. Other minor fabrication problems were also resolved during the system demonstration.
Figure 5. System Demonstration
ConnDOT also included requirements for an integrated system test and a 30-day continuous operational test. As shown in Table 2, ConnDOT allocated contract payments to coincide with the completion of the various tests. This was an incentive to conduct testing as soon as it was appropriate for the project.
Table 2. Payment Schedule
|
Item |
Percent of Contract Award |
|
1. Acceptance of prototype demonstration |
5% |
|
2. and 3.
Equipment purchased by contractor prior to system |
45% |
|
4. Acceptance of system demonstration test |
10% |
|
5. Acceptance of integrated system test |
20% |
|
6. Acceptance of 30-day operational testing |
10% |
|
7. Final acceptance of system |
8% |
|
8. Completion of warranty/maintenance period |
2% |
Project Management
To ensure the Procurement Project moves smoothly and is integrated with the project installing the field equipment, ConnDOT staff holds periodic coordination meetings for all the contracts for traffic management systems in the Hartford area. Most often the meetings are held weekly and include not only the ConnDOT ITS staff, consultants, and contractors, but also representatives from the design and construction areas of the ConnDOT. ConnDOT staff dedicates time in their calendars weeks in advance to attend these meetings and require the consultants and contractors to do the same. These weekly meetings have well-structured agendas and can last from 30 minutes to two hours depending on the issues needed to be discussed. ConnDOT staff use the meetings to resolve issues, note action items, and adjust schedules to stay on target. All parties agreed that there were good communications among them and a good rapport was established. Very often the contractor’s representative did not wait for weekly meetings or to be contacted by ConnDOT staff. He would be in contact with ConnDOT staff almost on a daily basis by telephone or e-mail to discuss work status and resolve any issues.
Furthermore as part of their management process, ConnDOT staff holds consultants and contractors to contracted schedule dates. They also hold themselves accountable to complete their reviews of documents and other material as required to maintain funding commitments and project schedules. The ITS staff realizes that if they cannot provide appropriate and timely guidance, then projects will get delayed and implementation schedules will not be maintained. Moreover, the ConnDOT ITS staff realizes they need input from their construction staff and work closely with the construction field personnel assigned to the project.
ConnDOT, their consultant, and the contractor all encountered staff turnover during the project, which presented a challenge to the team. They learned, however, that if a primary point of contact for each organization is designated for the project or specific stages of the project and if good project documentation is maintained, the challenge can be addressed.
Change Orders
Another reason that ConnDOT staff wanted to ensure the adequacy of the design early in the project was to limit the number of change orders that might be required. There were only three change orders. Two change orders adjusted the payment milestones -- one milestone was separated into two and another one was added because of a delay of one vendor in supplying necessary equipment. The other change order was a $14,000 increase in price to purchase additional equipment required as part of the final system implementation.
Outcomes of the Process
ConnDOT staff achieved many of their desired outcomes and some benefits that were not expected. For example, using an RFP rather than an invitation for bid (IFB) actually reduced the amount of time and cost required to put a bid package together because fewer offices were involved and the RFP development time was shorter than for a traditional IFB.
The review of the design and equipment catalog cuts during the proposal submittal process and the subsequent ConnDOT request that the bidders supply further information or modifications did not lengthen the procurement process. This review ensured that any concerns or issues on the capability of the equipment to comply with specifications were addressed before a contractor was chosen. Also, once the contractor was selected, only a single round of contract negotiations was required. The bid package was issued July 24, 2001, the bid opening was September 6, 2001, and the contract was awarded January 11, 2002.
The prototype test and the system demonstration test gave ConnDOT staff and the contractor “a leg up” on reducing the amount of problems that could occur once the system was installed. Prototype testing ensured that the equipment met functional requirements. The system demonstration test allowed integration and communications issues of all procured equipment to be worked out in a laboratory setting. As one interviewee noted, “It is much easier to troubleshoot when all of the equipment is in close proximity.”
The full benefits of the tests were more visible as the project moved forward from testing to installation. Because the prototype test and the system demonstration identified problem areas sooner, they made field testing easier. The tests did not necessarily find more problems than would have been uncovered in the field, but they did force an early and efficient resolution of the problems. The tests also reduced the potential for a large number of change orders that could occur in complex ITS projects.
Even though the prototype test and the system demonstration test were performed, they naturally did not eliminate all the problems that might occur in the field. The interviewees agreed that there is just no way to simulate every situation that might present itself in the field. For example, some fiber-optic transceivers produced some static when installed in the field. Although the transceivers worked well using short fiber-optic connections in the system demonstration, they did not perform adequately when connected by long runs of cable in the field. ConnDOT staff gained knowledge from this situation and may modify the testing process for future projects by changing test requirements such as evaluating equipment performance over a full reel of fiber-optic cable during the prototype test or system demonstration.
Summary
ConnDOT staff achieved their goal of procuring a qualified bidder and reliable equipment. Through their procurement process, they were able to ascertain the bidders’ qualifications as well as the bidders’ understanding of this complex project.
Furthermore, the prototype test and the system demonstration test gave ConnDOT staff a level of comfort and assurance they desired by using intermediate steps to install and test the integrated system. “Backend” testing was reduced. As previously noted, equipment was tested in the prototype and some changes were made before all equipment was purchased, reducing the potential for delay. In addition, the system demonstration allowed the system to be built and bugs worked out in one location. It streamlined the testing process and corrections were made on site. More testing early in the project definitely benefited ConnDOT as it eased field installation and minimized the number of change orders. The testing did, however, place more responsibility on the prime contractor.
The team from the ConnDOT, the DAS, the FHWA, and the consultant along with the contractor proved that their innovative process was the best method of gaining the assurances that they desired. To build on their success, ConnDOT staff will be using a qualifications-based procurement for a maintenance contract covering the equipment used in the freeway traffic management system in the Hartford area, when the warranty period expires, and for other intelligent transportation systems resident throughout the state.
ConnDOT Newington
Highway Operations Center
ConnDOT
Bridgeport
Highway
Operations Center
Appendix A
Contractor Reference Questionnaire
Hartford Area Incident Management System
Video and Data Transport System Project
Contractor: (If subcontractor note parent firm)
Client:
Project:
Project Cost:
Client Contact:
Phone Number:
Address:
Date Work Performed:
Opening for DOT
This is ________ from the Connecticut Department of Transportation. We have recently received technical submittals from _____________ in response to a RFP. _____________ have given you/your agency as a reference as part of their submittal and we would like to ask you a few questions about your experience with these firms. The project the firms have submitted proposals for entails the installation of communications equipment for the transmission of video and data from field devices along the Interstate to the Department's Operations Center. The project will require the contractor to provide and install transceivers, multiplexers, channel banks, and SONET equipment in compliance with the project requirements. We appreciate your assistance in gathering information about _______________.
Opening for consultant
This is ________ from _______________. We are consultants to the Connecticut Department of Transportation. The Department has recently received technical submittals from _____________ in response to a RFP. _____________ have given you/your agency as a reference as part of their submittal and we would like to ask you a few questions about your experience with these firms. The project the firms have submitted proposals for entails the installation of communications equipment for the transmission of video and data from field devices along the Interstate to the Department's Operations Center. The project will require the contractor to provide and install transceivers, multiplexers, channel banks, and SONET equipment in compliance with the project requirements. We appreciate your assistance in gathering information about _________________________.
1.) Did the contractor perform the work described in the reference provided (read and note whether they actually did each item provided in their description)?
2.) Did the work performed meet your expectations?
2.a.) Was there anything specifically that exceeded your expectations?
2.b.) Was there anything specifically that did not meet your expectations?
3.) What factors were most important in selecting this contractor over others?
4.) Was the work performed on budget? If not, why?
5.) Did the contractor provide the initiative to contain project costs by suggesting alternative methods? If yes, were these methods efficient?
6.) Did the contractor meet the milestones deadlines? Y/N If not,
6.a.) How much of a deviation occurred?
6.b.) Was there a justification for missing the deadlines?
6.c.) Was the project completed on time?
7.) On a scale of 1 to 10, with 10 being high and 1 being low, how would you rate the contractor in the following areas:
7.a.) Overall quality of the work?
7.b.) Technical competence of the staff?
7.c.) Responsiveness of the
Contractor to your needs?
(Was the contractor distracted
by other commitments?)
7.d.) Management of responsibilities?
7.e.) Overall Communication?
7.e.1.) Meetings organization and effectiveness?
7.e.2.) Coordination between involved parties?
7.f.) Cooperativeness?
7.g.) Overall availability of resources?
8.) Again on a scale of 1 to 10, how would you describe your satisfaction with the quality of the system installed?
8.a.) Equipment Selected?
8.b.) Software?
8.c.) System Reliability?
8.d.) Maintenance?
8.e.) Fulfillment of service contract or warranty?
8.f.) Quality of Documentation?
9.) Who was your main point of contact in the contractor’s company?
9.a.) Who were the personnel with whom you worked closest?
10.) Did the contractor have any subcontractors working on the job? Y/N Who?
10.a.) Did they work as a sub to another prime? Y/N Who?
10.b.) How well did the subcontractors perform?
10.c.) Did the contractor and subcontractors work well as a team?
11.) Did any major claims arise in the course of this contract? Y/N
11.a.) What was the nature of the claim?
11.b.) How was the claim resolved?
12.) What was your overall impression of the experience of working with the contractor?
13.) What do you think of the contractor in general?
14.) Has this contractor worked for your organization previously?
15.) Would you hire this contractor again to perform similar work?
16.) Do you know of any other projects this contractor has worked on that we could contact?
17.) Any additional comments?
Date Contacted:
Interviewer:
[1] Connecticut Public Act No. 99-161, An Act
Concerning Revision of State Purchasing Statutes and Procedures.
[2] General Statutes of Connecticut, Title 4a,
Section 4a-57.
[3] Connecticut Public Act No. 97-9, An Act
Concerning the Management of State Agency Information and
Telecommunication Systems.