RESULTS

Results as of May 2004

This section contains the results of our independent assessment of nominated TB PM software applications. This section is divided into three parts, one for each of the tiers for the software assessment:

Tier 1 - Technical Assessment
Tier 2 - Functional Assessment
Tier 3 - Hands-on Evaluation

Tier 1 Results. The first-tier evaluation was considered a back-of-the-box evaluation. The first-tier selection matrix was sent out to vendors for each software package nominated. The matrix focused on reliability, sustainability, and technology. The following table shows the vendors that were nominated and the results of the first-tier evaluation.

Commercial-off-the-shelf (COTS) applications

Application Vendor Deployment Sites as of May 2004[1] Evaluation Outcome Comments
Universal Client Management System Advanced Business Software Imperial Co., CA Evaluate in 2nd tier Meets the technical requirements for first tier.
CDCIS (Alabama CIMS) Casto Information Management System AL, San Diego, CA Evaluate in 2nd tier Meets the technical requirements for first tier.
Equicare Cogent Integrated Solutions North Side HIV and Health Coalition - Chicago, IL;
Access Community Health - Chicago, IL; Toronto Rehabilitation Institute - Toronto, Ontario;
Oak Bay Lodge - Victoria, BC
Evaluate in 2nd tier Meets the technical requirements for first tier.
HealthDoc Denver Dept. of Health, El Paso Health Dept. Denver Health and El Paso County Health Department Evaluate in 2nd tier Meets the technical requirements for first tier.
Integrated Client Health Management System Persimmony Software Nowhere Evaluate in 2nd tier Meets the technical requirements for first tier.
Siemens system COTS Not applicable. Is being integrated as part of HealthDoc (see above) Not evaluated Does not seem to stand on its own as an application; however, it might work well integrated as part of another application. For example, it could be used as part of the HealthDoc application (which itself will be evaluated in the second tier)

back to Top

Government-off-the-shelf (GOTS) Applications

Application State/Jurisdiction Deployed Where? Evaluation Outcome Comments
Stop TB Arizona Registry (STAR) Arizona AZ Does not meet first tier requirements Not web-enabled. based on Microsoft Access database (not scalable); limited documentation; no vocabulary control; no administrative utilities; limited scheduling and calendaring functions.
TB Database System (TBDB) Colorado CO Evaluate in 2nd tier Meets the technical requirements for first tier.
TB Information System (TBIS) Massachusetts MA Evaluate in 2nd tier Meets the technical requirements for first tier.
TB Surveillance Illinois IL Does not meet first tier requirements Not web-enabled; based on Microsoft Access97database (not scalable); no vocabulary control; n o technical documentation (other than comments in code) and no user documentation (user training is internal); no export capabilities; no scheduling and calendaring functionality.
Tuberculosis Contact Investigation System (TBCIS) South Carolina Currently installed and in use at TB Division headquarters in Columbia, SC and on laptops in 9 project area sites. Statewide deployment scheduled for 2004 Does not meet first tier requirements This is a new application being built. It should be available in the spring 2004. Not planned to be web-enabled; no plans for import or export capabilities; no plans for vocabulary control; no plans for scheduling or calendaring functionality.
Denver Health staffTRAK Denver, Colorado Denver only Does not meet first tier requirements Not web-enabled; based on Microsoft Access 97 database (not scalable); no vocabulary control; no current scheduling and calendaring functionality (under construction).
Minnesota System for Tuberculosis Information (MSTI) Minnesota MN Evaluate in 2nd tier Meets the technical requirements for first tier.
HAWK Kansas Dept. of Health KA Evaluate in 2nd tier Meets the technical requirements for first tier.
San Francisco TB Control Ward (Oaxaca) San Francisco, California San Francisco, CA Does not meet first tier requirements Not yet released; no technical or user documentation; based on Microsoft Access database (not scalable); not web-enabled; no vocabulary control; local access only via LAN/WAN.
Open eMED New Mexico / Oklahoma Nowhere Evaluate in 2nd tier Meets the technical requirements for first tier.
Community Health Activities and Referral System (CHARTS) California Alameda Co., CA Does not meet first tier requirements Not web-enabled, standalone application; no vocabulary control; no import or export capabilities; no scheduling and calendaring functionality. This application is being upgraded in the future to support newer technologies (e.g., Microsoft .NET, BizTalk) This evaluation addressed current technologies.
Contact Investigation Improvement Project (CIIP) California Department of Health CI Does not meet first tier requirements Not web-enabled; based on Microsoft Access database (not scalable); no vocabulary control; no technical documentation and no user documentation; no import capabilities; no scheduling and calendaring functionality.
Oklahoma TB Information System (OTIS) Oklahoma OK Does not meet first tier requirements Not web-enabled (stand-alone Windows application written in VB6); no reporting capabilities; no security; no vocabulary control; no scheduling or calendaring functionality.
Public Health Management System (PHMS) Florida Planned for 2004 Evaluate in 2nd tier Meets the technical requirements for first tier.
PA NEDSS Pennsylvania Philadelphia, PA Evaluate in 2nd tier Meets the technical requirements for first tier.
staffTRAK-TB Centers for Disease Control and Prevention (CDC) CDC/DTBE/SEOIB Does not meet first tier requirements Not web-enabled; based on Microsoft Access 97 database (not scalable); no vocabulary control; no current scheduling and calendaring functionality (under construction).
HealthDoc COTS Colorado State Health Department Evaluate in 2nd tier Meets the technical requirements for first tier.

back to Top

Tier 2 Results . The second tier focused more on the core business functionality of the software applications. An assessment was developed and sent out containing 76 questions regarding functional capabilities. Weights were assigned to each question (Must have = 3, Should have = 2, and Nice to have = 1) and scores were assigned to each vendor based on written/verbal responses (Full credit = 2, Partial credit = 1, and None = 0). Systems whose average was above 3.5 continue to the next tier (except for Persimmony, who opted out of the evaluation process). The following tables show the results of the second tier evaluation.

Second Tier Composite Scores

System/Max Weight 3 2 1
ABS 5.96 3.00 2.00 0.96
CIMS 3.80 2.32 1.09 0.39
CO TBDB 2.96 1.84 0.82 0.30
Equicare 4.67 2.61 1.36 0.70
FL PHMS 3.04 1.16 1.18 0.70
HAWK 2.88 1.94 0.64 0.30
HealthDoc 1.57 0.68 0.64 0.26
MATBIS 3.90 2.32 1.27 0.30
MSTI (MN) 2.54 1.55 0.82 0.17
Open Emed 3.76 2.32 1.00 0.43
PA NEDSS 3.81 2.42 1.09 0.30
Persimmony 5.65 3.00 1.91 0.74

back to Top

Tier 3 Results. The third tier consisted of extensive hands-on evaluation of the applications. Demonstration sessions were set up for each of the applications and access to the applications and appropriate training were provided for project evaluators. De-identified data was gathered from the field (Alabama, Boston, California, Denver, Florida, Louisiana, New Mexico, Oklahoma, and Oklahoma City) and used to create diverse test scenarios for the third tier evaluation. Two basic scenarios which were used for this purpose are an ideal scenario called the Strawman and a multi drug resistant scenario called MDR . These test scenarios were used by evaluators to input data into each of these systems and provide a basis for evaluating their functionality. The applications were re-scored after the third tier in-depth evaluation and testing. During the third tier evaluation, information for other factors besides functionality were also gathered and evaluated. Usability, Maintenance Program, Versions, Number of Installations, and Documentation were other categories which were reviewed by evaluators. These additional factors are all part of information needed when choosing a software product. Successful software installations need these additional factors as well as functional capabilities. The following table shows the results of the third tier evaluation.

Product Functionality Usability Maintenance Program Other
Version No. of Installations Documentation
ABS Best Average Poor 1.0 1 User Guide
CIMS Good Average Poor 3.5.1 3 User/Technical
Equicare Average Good Poor 2.2 0 User/Some Technical
MA TBIS Poor Good Worst Beta 0 User/Technical
Open Emed Worst Worst Worst 0.99.1 0 User/Technical
PA-NEDSS Good Best Poor 5.0 1 User/Technical

Scale for Tier 3 Results

Results from the third tier are also presented in the Decision Model Tool spreadsheet (Microsoft Excel) format for viewing .

For a copy of the TB-PM Application Assessment Report (PDF file) please click here.


[1]As of May 2004

Skip Navigation | Assessment Guide | How to Get Started | Requirements Definition | Business Models | Support | Data Migration
Local Enhancements and Customizations | Costs | Funding | PHIN Compatibility | Decision Model Tool | Glossary | Results