Message-Id: <9109051623.AA01967@cert.sei.cmu.edu>
Date: Thu, 5 Sep 91 10:16:41 MDT
From: Chris McDonald  ASQNC-TWS-R-SO <cmcdonal@wsmr-emh03.army.mil>
Subject: Product Test - - SEER, version 3.32
To: /usr/cmcdonal/maillist:@wsmr-emh03.army.mil
Cc: /usr/cmcdonal/virrevlist:@wsmr-emh03.army.mil


*******************************************************************************
                                                                          PT-43
							         September 1991
*******************************************************************************


1.  Product Description:  SEER is a commercial program advertised to provide
information management and anti-virus software protection.  This test report
addresses version 3.32 for a single system.  The network version as of this
date is version 3.34.

2.  Product Acquisition:  The program is available from KWARE Inc., 2952
Timberwood Way, Herndon, VA 22071.  A flyer included with a program manual
listed REB Management Consultants Inc., 8518 Spartan Road, Fairfax, VA 22031
as an authorized distributor.  REB Inc. has a telephone number of 703-560-2076.

3.  Product Tester:  Chris Mc Donald, Computer Systems Analyst, Information
Systems Command, White Sands Missile Range, NM 88002-5506, DSN:  258-4176, DDN:
cmcdonal@wsmr-emh03.army.mil or cmcdonald@wsmr-simtel20.army.mil.

4.  Product Test:

    a.  I acquired an evaluation copy in August 1991 directly from REB Inc.
Although I received both a stand-alone and a network version of the program, I
chose to test only the former.

    b.  Product tests occurred on a Unisys 286 PC, MS-DOS 3.3, 640K during the
entire month of August 1991.  Program documentation states that SEER will run
"properly on IBM compatible PCs, XTs, ATs (286 & 386) and on PS/2s".  IBM DOS 
or MS-DOS 3.1 or greater is necessary.  

    c.  The SEER documentation proposes that "executives are responsible for
the computers which they manage".  If knowledge provides the tools to manage
computers, then SEER provides three major components:  (1)  prevention; (2)
detection; and (3)  restoration.  The prevention tool resides in SEER's ability
to analyze software before use and to inform an individual of potentially
dangerous code.  It is this capability which potentially can provide malicious
program detection.

    d.  SEER has an automatic installation component which performed as
documented.  Once installed on a hard drive a user executes the program by
entering the command SEER which is case insensitive.  The program displays a
master menu with various options.  The menu selection to provide a user "with
the capability to determine the danger potential of software programs" is
SUSPECT.  

    e.  SUSPECT has options to analyze a single file, an entire sub-directory,
or an entire drive.  The analysis is fairly quick and results in the creation
of a summary report on any file which SUSPECT considers dangerous.  I ran the
analysis against the hard drive and against a dozen floppy disks to understand
the operation of the analysis and to create summary reports for review.  I then
ran the analysis against floppy disks containing 80+ files infected either with
known viruses or trojan horses, or against disks with infected boot sectors.
The results of some of those tests were as follows:

    (1)  SUSPECT identified IO.SYS as a program that "may alter internal
identification on your floppy or hard disk! . . . Unless your (sic) really
trust this program, don't use it". 

    (2)  SUSPECT identified FORMAT.COM as a program that "may FORMAT your Hard
DISK!!! . . . Unless your (sic) really trust this program, don't use it!"

    (3)  SUSPECT did not identify as "dangerous" floppy disks infected with 
boot sector viruses, such as Stoned or MusicBug.

    (4)  SUSPECT did not identify as "dangerous" files infected with these
common viruses/trojan horse:  Cascade, Cascade-B, Twelve-Tricks.

    (5)  SUSPECT created summary reports for files infected with these common
viruses:  Dark Avenger, Jerusalem, Jerusalem-B, Sunday, Vacsina, Yankee Doodle.
In each case SUSPECT identified interrupt functions in the infected files which
suggested each program "alters DOS in memory only".  But the last line in all
summary reports stated:  "This program appears safe to use."

    (6)  SUSPECT created summary reports and positively identified all files
infected with the DataCrime or Columbus Day Virus.  The report identified the
virus by name, instructed the user to "please turn off your computer, then call
your Security people".

    f.  The program documentation describes five descriptions which can appear
in the SUSPECT summary reports.

    (1)  "This program will format either a floppy or a hard disk".

    (2)  "This program by-passes DOS's built-in disk protection".

    (3)  "This program alters DOS in memory only".

    (4)  "This program is designed to remain in memory".

    (5)  "This program appears safe to use".

I was able to activate all five descriptions during my tests, but clearly not
with the results described in the documentation.  With one exception SUSPECT
either left as "undecidable" or failed to detect whether a file or floppy disk
was potentially dangerous.  Although SUSPECT issued a "mild warning" that
certain programs alter DOS in memory, or that certain programs remain in
memory, a user might be deterred from further investigation when the summary
report concludes that the "program appears safe to use".  

    g.  The results of the SUSPECT testing suggested that SEER in its present
form does not reliably locate known malicious programs.  While SEER has adopted
a generic approach to malicious program detection, the success of that approach
is indeed suspect.  

    h.  While "prevention" is only one component of the overall SEER strategy,
I chose to suspend testing after reviewing the summary reports a second and in


				       2

some cases a third time.  I did test other master menu items which functioned
as documented.  However, the amount of testing was not sufficient to describe
the effectiveness of the additional components.

5.  Product Advantages:

    SEER appears to incorporate some reasonable controls to manage computer
resources, which is "one" of its stated objectives.  The information management
options discussed in the program documentation, however, were not fully
examined during the test period.

6.  Product Disadvantages:

    a.  The program's "prevention" component did not perform with the
effectiveness described in the documentation.  Since information management and
anti-virus protection are the two major objectives identified, it seems
reasonable to have expected better results.

    b.  Even if the SUSPECT component had performed well, the program has no
disinfection or removal capabilities.  Therefore, a user would have to utilize
other programs.  

7.  Comments:

    Fred Cohen's original paper on his first computer virus experiments
concluded that detection of viruses by their appearance or behavior was
"undecidable".  Yet seven years after the publication of his work, detection of
viruses by their appearance and behavior remains the most common form of viral
defense for the MS-DOS environment.

    SEER in its present state reinforces the "undecidable" aspect.  I
consciously chose to suspend testing after evaluating the "prevention"
component.  Therefore, readers should recognize that this report does not
address the total program.  


[The opinions expressed in this evaluation are those of the author, and should
not be taken as representing official Department of Army positions or a
commercial endorsement.]


				       3