         fffff       lll
        fff  ff       lll
 iii    fff fff iii   lll
 iii    fff     iii   lll
      fffffff         lll   eeeeee           000    4   4    55555
iiii    fff    iiii   lll  eee   ee         0   0   4   4    5
 iii    fff     iii   lll  eeeeeeee     v v 0   0   44444    5555
 iii    fff     iii   lll  eee          v v 0   0       4        5
iiiii  fffff   iiiii lllll  eeeeee       v   000  *     4  * 5555

Sat Sep 13 00:32:35 EDT 1997

If you try to use ifile at all, I would definitely like to hear from you.
I would greatly appreciate hearing about your installation and/or the actual
operation of the system.
Please send me e-mail at "jr6b+@andrew.cmu.edu".

NOTE TO ALL USERS: Please read the CHANGES file.

Official ifile web site --> "http://www.cs.cmu.edu/~jr6b/ifile/" <--

There is now an ifile mailing list! (two, in fact)  The first,
ifile-announce, is meant for all users of ifile.  Posts to this list will
be limited to new version and major developmental announcements.
The second, ifile-discuss, is meant for all those who wish to help in the
developmental process of ifile by making suggestions, submitting bug
reports and discussing various aspects of ifile.  All those who are
subscribed to ifile-discuss will automatically be subscribed to ifile-announce.

To subscribe to ifile-discuss, send mail to jr6b+@andrew.cmu.edu with the
subject header "subscribe ifile-discuss".  To subscribe to ifile-announce, send
mail to jr6b+@andrew.cmu.edu with the subject header
"subscribe ifile-announce".

What is ifile?

ifile is an intelligent mail filtering program which uses framework provided by
the MH mailing system and the slocal filtering system.  It uses the text-based
learning algorithm, Naive Bayesian, to determine what type of mail you like
to have filtered into your different folders.  It learns by observing any
actions you take to move (refile) messages to different folders than they were
originally filtered to.  The great benefit of ifile is that once ifile is in
place, ifile will update its knowlege about your preferences with every
message that you refile.  Normally, with other filters, you must write
down a strict set of rules and then change those rules as your preferences
change.  ifile does not deal with keywords, but rather with the word frequency
of the messages you receive, thus putting a much stronger emphasis on message
content and the actual meaning of the message.  As it is well integrated in
the EXMH package, it requires little maintainance after the initial
installation.

Does it actually work?

Yes.  I've had a number of users tell me that ifile does a great job of
filtering mail.  In my own experiences, it has also done a nice job.
Since I've added code to keep track of the number of filters vs. refiles,
ifile has correctly filtered over 3/4 of my mail.  What is impressive about
that number is that I have over 30 folders into which I filter mail.  Getting
such a high accuracy for such a difficult task isn't easy.

What exactly does it do?

The ifile mail filtering system adds two pieces to the system commonly
found with MH.  The first of those is a wrapper for the MH refile program.
When a refile request is made, it is first sent to irefile, where information
about the message(s) is gathered.  Once the necessary information is
acquired, it is stored in the data file, "~/.idata", and control is passed
to the MH refile to perform the actual moving of the message(s).

ifile also adds an extension to the slocal filtering program.  A line is
added to your ~/.maildelivery file so that whenever slocal is executed to
filter your mail, control is immediately passed to ifilter.  ifilter reads
the message to be filterd, determines which mailbox the message should be
filed into and then calls rcvstore to do the actual filing.  One benefit of
this system is that lines may be added to your .maildelivery file which will
catch mail before it gets to ifilter.  Mail which you wish to filter simply
based on a From: header might be better filtered via slocal by adding a line
to your ~/.maildelivery file.

How do I know if I can actually use this system?

To use ifile without any modifications, you must...

- use MH (or an MH-based program, such as EXMH) to handle your mail.
- have slocal installed on your system
- have "root" (or system administration) priviledges on your system
- have access to a C compiler (gcc is the compiler used for testing)

As of v0.4.0, MH command line options are almost completely supported.  There
is still no support for using 'cur' as a message name.

So, how do I install this system?

Quick install (for a full explaination, read the INSTALL file):

- Unzip this package to a directory of your choice.
- If you do not have perl in /usr/bin/, edit the first line of
  install.perl so that it reflects the location of perl on your system.
- Execute install.perl and answer any prompts given by the program
- Set up your mail program to use slocal for mail filtering

If you receive any error messages while following the above instructions,
read the full installation instructions included in the INSTALL file.

Is that it?

Yup.  Once you have completed those tasks, ifile will be installed.  If you
choose to run the knowledge_base.perl script, ifile should have a good idea of
how you like your mail organized and you should be able to see the beauty
of ifile after the first few messages you receive.

In general, mailboxes with a specific theme, or those which only contain
messages from a certain person should be easy for ifile to learn.  Those with
varying topics or which contain mail from a wide spread of people may be more
difficult for ifile to learn.

What do the version numbers indicate?

This program is still under fairly heavy development.  The first two numbers
indicate the major version number.  The last number indicates the minor
version number.  Each new major version is likely to contain major
developments and lots of new code unseen in previous new versions.  Each new
minor version within a major version will comprise primarily of bug fixes and
other less major changes.

Why are there ifile.jpg and Ifile.jpg files in the distribution?

I felt like ifile could use pictures of some sort, so I decided to create
some ifile images.  ifile.jpg is more of a logo, where Ifile.jpg might serve
as a nice web page image link, indiating that you filter your mail with ifile.
They are hardly works of art, but I think they are kind of neat :)  Btw,
for those of you interested, they were created with the GIMP v0.99.3.

What's up with the error.c and ifile.c files?

I'm starting to put some serious work into modularizing ifile.  error.c is
the library which is being used for message handling and error reporting,
while ifile.c is the library being used for general ifile functions.  I would
like to strip ifilter.c and irefile.c to nothing more than a few function
calls from the ifile.c library.

v0.3.3 brings in a new system of message/error reporting.  I do all message
reporting through a new function, ifile_verbosify().  It allows each message
to be prioritized so that the output of ifile programs can be regulated
with the change of a single variable.  The variable which regulates the
output (ifile_verbosity_level) can be found in error.c and the definitions
for its possible values (ifile_verbosity_levels) can be found in ifile.h
The normal distribution setting will be ifile_quiet, which will only report
to standard error warnings and errors, yet will keep a log of progress
information in the /tmp/*.info file.

As of v0.4.0, the whole modularization idea is starting to take shape.  The
main ifilter function has been stripped down to just a few function calls and
the main irefile function is starting to look a little nicer.  A number of
variable names have also been changed and reorganized so that the code is more
readable and easier to understand.

v0.4.2 - database writing modularized!

Can I use ifile with other mail filters?

Most likely yes.  ifile is set up to install on top of slocal, but could
easily be used to work with other mail filters.  If you will notice, ifilter
and irefile both work as command line programs.  There is no absolute
need for them to be called from MH or slocal.  You can give either program
the command line argument '--help' to receive a listing of usage options.
ifilter should be easy to use with another filter as all it requires is that
a message be passed to it via standard input.

If you currently use procmail to filter your mail, try adding the following
line to your .procmailrc file, commenting out all other mail recipies you
use to filter mail:

:0
| /usr/local/lib/ifile-v0.4.2/ifilter

Please note that the above line HAS NOT BEEN TESTED and very well may not
work!  Also note that you will need to replace the path with the correct
location of the ifilter program on your system.

What is the news2mail.perl script for?

This script was written by Chris Browne <cbbrowne@hex.net> under the
assumption that you have a newsfeed on your personal machine.  The idea
is that news isn't always posted to the group/groups to which it best
pertains.  This script will allow you to have your news filtered through
ifile in similar fashion to the way mail is filtered.  Thus, for each of
the newsgroup mailboxes you have set up in your mail program, with any luck,
only the relevant articles from a particular newsgroup will end up in that
mailbox.

Please read news2mail.perl for details.  news2mail.perl MUST be edited before
it will be usable with your system!

What is all this talk about an irefile queue?

Included with the v0.4.2 distribution is primitive code which allows for
less frequent writing of the entire .idata file, thus speeding up the
average call to irefile.  The basic idea is to have each call to irefile
add entries to a queue instead of having it read and rewrite the entire
.idata file (which is the current procedure).  After the queue entries are
added, a data collector program would be called.  If the queue has
accumulated enough entries, the data collector would clean out the queue
and update the .idata file (reading and rewriting it) accordingly.  As
the data collection program could be run separately from the irefile
program, it would never be necessary for the irefile program to read from
or write to the .idata file, thus speeding up execution time immensly.

That's the theory anyway.  Included in this distribution are two new files:
irefile.queue.c and data_collector.  "irefile.queue.c" acts as a replacement
for irefile.c.  Currently it will NOT periodically run the data_collector
script, so for any information to be added to the .idata file, you must
run the data_collector script every so often.  irefile.queue.c will add an
entry to the file ~/.idata.queue for every message it is given to refile
(creating the file if necessary).  When the data_collector script is run,
it will read in ~/.idata.queue, add the entries to .idata and then delete
~/.idata.queue.

To use irefile.queue.c instead of the regular irefile.c program, you must
execute "make irefile.queue".  This will compile the executable
"irefile.queue".  To make MH use this as its refile program you will need
to change the link from your MH refile binary location so that it points
to irefile.queue instead of irefile.  The path of the MH_REFILE #define in
ifile.h will tell you the path of your MH refile binary if you are not
sure where to find it.  You would do something like the following:

rm /usr/mh/bin/refile
ln -s /usr/local/ifile/irefile.queue /usr/mh/bin/refile

The queue system still needs quite a bit of work, but it seems to be
functional enough at this point to serve well as a nice play toy :)

Is there any way to nicely integrate ifile into EXMH?

I'm glad you asked, because as of v0.4.3, there is.  I will point you to
three files included in the ifile distribution which describe how you might
better integrate ifile into EXMH.

The first is "filter_button".  This describes how to add a button to EXMH
which will filter the currently highlighted message using ifilter.  This
would probably be most useful if you like to use your inbox as a holding
box for messages you are planning to take some action.  This button would
allow you to incorporate mail to your inbox and then use ifilter (via
the button) when you decide the mail can be stored away.

The second is "exmh_integration".  This describes how to set up ifile
as one of the incorporate options under EXMH.  Normally when you go
to Preferneces->Incorporate Mail, you see a list of ways to incoporate
your mail.  This text file describes how to add ifile as one of those
options so that incoming mail will be filtered without needing it to
first be routed through slocal (although there are benefits to having
it first go through slocal).

The third is "exmh_integration2".  This describes how one goes about
incorporating irefile into EXMH so that it is not necessary to set a link
from the MH refile binary location to irefile.

Note that the above three files are very "fresh off the presses" and probably
do not explain everything well enough for all ifile users to get things
working without some hacking.

Who wrote this?

Why, I did!  'I' am Jason Rennie, a junior majoring in Computer Science at
Carnegie Mellon University.  I've been programming for most of my life,
starting with BASIC as a youth, moving through Pascal during high school and
then widening my horizons in college with such languages as C/C++, perl, Java
and ML.  Since installing it in December of '95, I have become a big-time linux
fan, running it exclusively.
I've been doing quite a bit of work with learning algorithms, especially
text-based learning,  and I decided it would be cool to apply
one of the text learning algorithms to mail filtering.  So, I did.

Other code contributors include Diego Zamboni and Chris Browne.

How do I contact this computer geek?

By e-mail, of course!  jr6b+@andrew.cmu.edu  You can also check out my web
page at "http://www.andrew.cmu.edu/~jr6b/"


Credits
-------
Tom Mitchell - Professor who was willing to hire me for a summer during
	       which I learned vast amounts of information about machine
               and text learning.  He is also a very nice guy to work with :)
Dave Robinson - Alpha tester, pointed out some major bugs in version 0.1.x.
                Good source of encouragement :)
Diego Zamboni - Wrote first port of ifilter to C; also wrote the first
                server/client version of ifilter (in the search of efficiency).
Chris Browne - Proposed numerous suggestions and ideas for increasing the
               efficiency of the ifile C code.  Wrote a perl/DBM version of
               ifile.

Thanks also go to such people as Colin McCormack, Alfred Werner,
Mikael Nilsson, Jim Grinnell and Ken Wallich who have made their own little
contributions to the success of ifile.

Neat web sites related to ifile/text learning:

http://www.wisewire.com/ - WiseWire - WWW document filtering - sort of like Yahoo, only better
http://www.cs.cmu.edu/afs/cs/project/theo-11/www/naive-bayes.html - General code for text classification
http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/student-projects.html - Tom Mitchell's home page

If you do find this program helpful and would like to support it and see
it grow into a powerful, full-featured, robust mail filtering system (gotta
make sure to include all the buzzwords) please fill out the following short
survey and mail it to jr6b+@andrew.cmu.edu:

------------------------->8--------------------------

What version of ifile do you have? :

What operating system do you run? :

What is your default C compiler? :

What versions of exmh and mh do you run? :

What version of perl is on your system? :

Did you have any problems installing ifile?  If so, please describe them here:








Did you encounter any problems while running ifile (lost mail - poor job of
filtering)?  If so, please describe them here (copies of /tmp/refile.info and
/tmp/ifilter.info would be really helpful!) :








Do you have any suggestions on how to improve ifile?  (this would be most
helpful if you have specific code changes)








----------------------8<---------------------------

Thank you for your interest!!!! :)
