This section contains the results of our independent assessment of nominated TB PM software applications. This section is divided into three parts, one for each of the tiers for the software assessment:
Tier 1 - Technical Assessment
Tier 2 - Functional Assessment
Tier 3 - Hands-on Evaluation
Tier 1 Results. The first-tier evaluation was considered a back-of-the-box evaluation. The first-tier selection matrix was sent out to vendors for each software package nominated. The matrix focused on reliability, sustainability, and technology. The following table shows the vendors that were nominated and the results of the first-tier evaluation.
Commercial-off-the-shelf (COTS) applications
| Application | Vendor | Deployment Sites as of May 2004[1] | Evaluation Outcome | Comments |
|---|---|---|---|---|
| Universal Client Management System | Advanced Business Software | Imperial Co., CA | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| CDCIS (Alabama CIMS) | Casto Information Management System | AL, San Diego, CA | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| Equicare | Cogent Integrated Solutions |
North Side HIV and Health Coalition - Chicago, IL; Access Community Health - Chicago, IL; Toronto Rehabilitation Institute - Toronto, Ontario; Oak Bay Lodge - Victoria, BC |
Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| HealthDoc | Denver Dept. of Health, El Paso Health Dept. | Denver Health and El Paso County Health Department | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| Integrated Client Health Management System | Persimmony Software | Nowhere | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| Siemens system | COTS | Not applicable. Is being integrated as part of HealthDoc (see above) | Not evaluated | Does not seem to stand on its own as an application; however, it might work well integrated as part of another application. For example, it could be used as part of the HealthDoc application (which itself will be evaluated in the second tier) |
Government-off-the-shelf (GOTS) Applications
| Application | State/Jurisdiction | Deployed Where? | Evaluation Outcome | Comments |
|---|---|---|---|---|
| Stop TB Arizona Registry (STAR) | Arizona | AZ | Does not meet first tier requirements | Not web-enabled. based on Microsoft Access database (not scalable); limited documentation; no vocabulary control; no administrative utilities; limited scheduling and calendaring functions. |
| TB Database System (TBDB) | Colorado | CO | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| TB Information System (TBIS) | Massachusetts | MA | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| TB Surveillance | Illinois | IL | Does not meet first tier requirements | Not web-enabled; based on Microsoft Access97database (not scalable); no vocabulary control; n o technical documentation (other than comments in code) and no user documentation (user training is internal); no export capabilities; no scheduling and calendaring functionality. |
| Tuberculosis Contact Investigation System (TBCIS) | South Carolina | Currently installed and in use at TB Division headquarters in Columbia, SC and on laptops in 9 project area sites. Statewide deployment scheduled for 2004 | Does not meet first tier requirements | This is a new application being built. It should be available in the spring 2004. Not planned to be web-enabled; no plans for import or export capabilities; no plans for vocabulary control; no plans for scheduling or calendaring functionality. |
| Denver Health staffTRAK | Denver, Colorado | Denver only | Does not meet first tier requirements | Not web-enabled; based on Microsoft Access 97 database (not scalable); no vocabulary control; no current scheduling and calendaring functionality (under construction). |
| Minnesota System for Tuberculosis Information (MSTI) | Minnesota | MN | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| HAWK | Kansas Dept. of Health | KA | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| San Francisco TB Control Ward (Oaxaca) | San Francisco, California | San Francisco, CA | Does not meet first tier requirements | Not yet released; no technical or user documentation; based on Microsoft Access database (not scalable); not web-enabled; no vocabulary control; local access only via LAN/WAN. |
| Open eMED | New Mexico / Oklahoma | Nowhere | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| Community Health Activities and Referral System (CHARTS) | California | Alameda Co., CA | Does not meet first tier requirements | Not web-enabled, standalone application; no vocabulary control; no import or export capabilities; no scheduling and calendaring functionality. This application is being upgraded in the future to support newer technologies (e.g., Microsoft .NET, BizTalk) This evaluation addressed current technologies. |
| Contact Investigation Improvement Project (CIIP) | California Department of Health | CI | Does not meet first tier requirements | Not web-enabled; based on Microsoft Access database (not scalable); no vocabulary control; no technical documentation and no user documentation; no import capabilities; no scheduling and calendaring functionality. |
| Oklahoma TB Information System (OTIS) | Oklahoma | OK | Does not meet first tier requirements | Not web-enabled (stand-alone Windows application written in VB6); no reporting capabilities; no security; no vocabulary control; no scheduling or calendaring functionality. |
| Public Health Management System (PHMS) | Florida | Planned for 2004 | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| PA NEDSS | Pennsylvania | Philadelphia, PA | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
| staffTRAK-TB | Centers for Disease Control and Prevention (CDC) | CDC/DTBE/SEOIB | Does not meet first tier requirements | Not web-enabled; based on Microsoft Access 97 database (not scalable); no vocabulary control; no current scheduling and calendaring functionality (under construction). |
| HealthDoc | COTS | Colorado State Health Department | Evaluate in 2nd tier | Meets the technical requirements for first tier. |
Tier 2 Results . The second tier focused more on the core business functionality of the software applications. An assessment was developed and sent out containing 76 questions regarding functional capabilities. Weights were assigned to each question (Must have = 3, Should have = 2, and Nice to have = 1) and scores were assigned to each vendor based on written/verbal responses (Full credit = 2, Partial credit = 1, and None = 0). Systems whose average was above 3.5 continue to the next tier (except for Persimmony, who opted out of the evaluation process). The following tables show the results of the second tier evaluation.
Second Tier Composite Scores
| System/Max Weight | 3 | 2 | 1 | |
|---|---|---|---|---|
| ABS | 5.96 | 3.00 | 2.00 | 0.96 |
| CIMS | 3.80 | 2.32 | 1.09 | 0.39 |
| CO TBDB | 2.96 | 1.84 | 0.82 | 0.30 |
| Equicare | 4.67 | 2.61 | 1.36 | 0.70 |
| FL PHMS | 3.04 | 1.16 | 1.18 | 0.70 |
| HAWK | 2.88 | 1.94 | 0.64 | 0.30 |
| HealthDoc | 1.57 | 0.68 | 0.64 | 0.26 |
| MATBIS | 3.90 | 2.32 | 1.27 | 0.30 |
| MSTI (MN) | 2.54 | 1.55 | 0.82 | 0.17 |
| Open Emed | 3.76 | 2.32 | 1.00 | 0.43 |
| PA NEDSS | 3.81 | 2.42 | 1.09 | 0.30 |
| Persimmony | 5.65 | 3.00 | 1.91 | 0.74 |
Tier 3 Results. The third tier consisted of extensive hands-on evaluation of the applications. Demonstration sessions were set up for each of the applications and access to the applications and appropriate training were provided for project evaluators. De-identified data was gathered from the field (Alabama, Boston, California, Denver, Florida, Louisiana, New Mexico, Oklahoma, and Oklahoma City) and used to create diverse test scenarios for the third tier evaluation. Two basic scenarios which were used for this purpose are an ideal scenario called the Strawman and a multi drug resistant scenario called MDR . These test scenarios were used by evaluators to input data into each of these systems and provide a basis for evaluating their functionality. The applications were re-scored after the third tier in-depth evaluation and testing. During the third tier evaluation, information for other factors besides functionality were also gathered and evaluated. Usability, Maintenance Program, Versions, Number of Installations, and Documentation were other categories which were reviewed by evaluators. These additional factors are all part of information needed when choosing a software product. Successful software installations need these additional factors as well as functional capabilities. The following table shows the results of the third tier evaluation.
| Product | Functionality | Usability | Maintenance Program | Other | ||
|---|---|---|---|---|---|---|
| Version | No. of Installations | Documentation | ||||
| ABS |  |
 |
 |
1.0 | 1 | User Guide |
| CIMS |  |
|
|
3.5.1 | 3 | User/Technical |
| Equicare | |
|
|
2.2 | 0 | User/Some Technical |
| MA TBIS | |
|
 |
Beta | 0 | User/Technical |
| Open Emed | |
|
|
0.99.1 | 0 | User/Technical |
| PA-NEDSS | |
|
|
5.0 | 1 | User/Technical |
Results from the third tier are also presented in the Decision Model Tool spreadsheet (Microsoft Excel) format for viewing .
For a copy of the TB-PM Application Assessment Report (PDF file) please click here.
[1]As of May 2004
Skip Navigation | Assessment Guide | How to Get Started | Requirements Definition | Business Models | Support | Data Migration
Local Enhancements and Customizations | Costs | Funding | PHIN Compatibility | Decision Model Tool | Glossary | Results