1.1   What is b8?

b8 is a spam filter implemented in PHP. (formerly called "bayes-php"). It is intended to keep your weblog or guestbook spam-free. The filter can be used anywhere in your PHP code and tells you if a text is spam or not, using statistical text analysis. See How does it work? for details about this.
To be able to do this, b8 first has to learn some spam and some ham example texts to decide what's good and what's not. If it makes mistakes classifying unknown texts, they can be corrected and b8 learns from what was going wrong, getting better with each example text.

At the moment of this writing, b8 has classified 14411 guestbook entries and weblog comments on my homepage since december 2006. 131 were ham. 39 spam texts (0.27 %) have been rated as ham (false negatives), with not even one false positive (ham message classified as spam).
This results in a sensitivity of 99.73 % and a specifity of 100 % for me. I hope, you'll get the same good results :-)

Basically, b8 is a Bayesian spam filter like Bogofilter or SpamBayes, but it is not intended to classify emails. On the other hand, when I begun writing b8, I didn't know a good spam filter (or any spam filter that isn't just example code how one could implement a Bayesian spam filter in PHP) that was intended to filter weblog or guestbook entries. That's why I had to write my own ;-)
Caused by it's purpose, the way b8 works is slightly different from most of the Bayesian email spam filters out there. See What's different? if you're interested in the details.

1.2   How does it work?

b8 is a naive Bayesian Spam filter, basically using the technique described in Paul Graham's article "A Plan For Spam" [1]. The improvements proposed in Graham's article "Better Bayesian Filtering" [2] and Gary Robinson's article "Spam Detection" [3] have also been considered. See also the article "A Statistical Approach to the Spam Problem" [4].

b8 cuts the text to classify to pieces, extracting stuff like email addresses, links and HTML tags. For each such token, it calculates a single probability for it being spam, based on what the filter has learned so far. When the token was not seen before, b8 trys to find similar ones using the "degeneration" described in [2] and uses the most relevant value found. If really nothing is found, a default rating is set.
Then, b8 takes the most relevant values (which have a rating far from 0.5, which would mean we don't know what it is) and calculates the probability that the whole text is spam by the inverse chi-square function described in [3]. There are some parameters that can be set which influence the filter's behaviour (see below).

In short words: you give b8 a text and get back a value between 0 and 1, saying it's ham when it's near 0 and saying it's spam when it's near 1.

1.3   What do I need for it?

Not much! You just need PHP 5 on the server where b8 will be used (b8 version 0.5 finally dropped PHP 4 compatibility – thankfully ;-) and a proper storage possibility for the wordlists. I strongly recommend using Berkeley DB. See below how you can check if you can use it and why you should use it. If the server's PHP wasn't compiled with Berkeley DB support, a MySQL table can be used alternatively.

1.4   What's different?

b8 is designed to classify weblog or guestbook entries, not emails. So, it uses a slightly different technique than the others use.

My experience was that such spam entries were quite short, sometimes just "123abc" as text and a link to a suspect homepage. Some spam bots don't even make a difference between e. g. the "name" and "text" fields and post their text as email address, for example. So, b8 just takes one string to classify, making no difference between "headers" and "text".
The other thing is, that most Bayesian filters count one token one time, no matter how often it appears in the text (as Graham describes it in [1]). b8 does count how often a token was seen and learns or considers this. Additionally, the number of learned ham and spam texts are saved and used as the calculation base for the single possibilities. Why this? Because a text containing one link (no matter where it points to, just indicated by a "http://" or a "www.") might not be spam, but a text containing 20 links might be.

This means that b8 might be good for classifying weblog or guestbook entries – but very likely, it will work quite poor when being used for something else (like classifying emails). But as said above, for this task, there are a lot of very good filters out there to choose from.

1.5   And why is it called b8?

The initial name for the filter was (damn creative!) "bayes-php". There were two main reasons for searching another name: 1. "bayes-php" sucks. 2. the PHP License does not want that the name of a script written in PHP does contain "PHP". Read the License FAQ for a reasonable argumentation about this.

Luckily, Tobias Lang proposed the new name "b8". And these are the reasons why I chose this name:

• "bayes-php" is a "b" followed by 8 letters.
• "b8" is short and handy. Additionally, there was no program with the name "b8" or "bate"
• The English verb "to bate" means "to decrease" – and that's what b8 does: it decreases the number of spam entries in your weblog or guestbook!
• "b8" just sounds way cooler than "bayes-php" ;-)

So … have a lot of fun using b8 :-)

2.1   Update from bayes-php version 0.2.1 or earlier

Please first follow the database update instructions of the bayes-php-0.3 release if you update from a version prior to bayes-php-0.3 and then read the following paragraph about updating from a version <0.3.3.

2.2   Update from bayes-php version 0.3 or later

You use Berkeley DB?

Everything's fine, you can simply continue using your database.

You use MySQL?

The CREATE statement of b8's wordlist has changed. The best is probably to create a dump via your favorite administration tool or script, create the new table and re-insert all data. The layout is still the same: there's one "token" column and one "data" column. Having done that, you can keep using your data.

You use SQLite?

Sorry, at the moment, there's no SQLite backend for b8. But we're working on it :-)

The configuration model of b8 has changed. Please read through the Configuration section and update your configuration accordingly.

b8's lexer has been partially re-written. It should be now able to handle all kind of non-latin-1 input, like cyrillic, chinese or japanese texts. Caused by this fact, much more tokens will be recognized when classifying such texts. Therefore, you could get different results in b8's ratings, even if the same database is used and although the math is still the same.

4.1   b8's base configuration

All these values can be set in the "config_b8" array (the first parameter) passed to b8. The name of the array doesn't matter (of course), it just has to be the first argument.

These are some basic settings telling b8 which backend classes to use:

storage

This defines which storage backend to use to save b8's wordlist. Currently, two database backends are available: Berkeley DB (dba) and MySQL (mysql). At the moment, b8 0.5 does not support SQLite, but it will be (hopefully) re-added in one of the next releases. The default is dba (string).

Berkeley DB

This is the preferred storage backend. It was the original backend for the filter and remains the most performant. b8's storage model is optimized for this database, as it is really fast and fits perfectly to what the filter needs to do the job. All content is saved in a single file, you don't need special rights or a database server.
If you don't know whether your server's PHP can use a Berkeley DB, simply run the script install/setup_berkeleydb.php. If it shows a Berkeley DB handler, please use this backend.

MySQL

As some webspace hosters don't allow using a Berkeley DB (but please be sure to check if you could use it!), but most do provide a MySQL server, using a MySQL table for the wordlist is provided as an alternative storage method. As said above, b8 was always intended to use a Berkeley DB. It doesn't use or need SQL to query the database. So, very likely, this will work less performant, produce a lot of unnecessary overhead and waste computing power. But it will do fine anyway!

See Configuration of the storage backend for the settings of the chosen backend.

degenerator

The degenerator class to be used. See How does it work? and [2] if you're interested in what "degeneration" is. Defaults to default (string). At the moment, only one degenerator exists, so you probably don't want to change this unless you have written your own degenerator.

lexer

The lexer class to be used. Defaults to default (string). At the moment, only one lexer exists, so you probably don't want to change this unless you have written your own lexer.

The behaviour of the lexer can be additionally configured with the following variables:

min_size

The minimal length for a token to be considered when calculating the rating of a text. Defaults to 3 (integer).

max_size

The maximal length for a token to be considered when calculating the rating of a text. Defaults to 30 (integer).

allow_numbers

Should pure numbers also be considered? Defaults to FALSE (boolean).

The following settings influence the mathematical internals of b8. If you want to experiment, feel free to play around with them; but be warned: wrong settings of these values will result in poor performance or could even "short-circuit" the filter.
Leave these values as they are unless you are sure that your changes will result in a better performance!

The "Statistical discussion about b8" [5] shows why the default values are the default ones.

use_relevant

This tells b8 how many tokens should be used when calculating the spamminess of a text. The default setting is 15 (integer). This seems to be a quite reasonable value. When using to many tokens, the filter will fail on texts filled with useless stuff or with passages from a newspaper, etc. not being very spammish.
The tokens counted multiple times (see above) are added in addition to this value. They don't replace other ratings.

min_dev

This defines a minumun deviation from 0.5 that a token's rating must have to be considered when calculating the spamminess. Tokens with a rating closer to 0.5 than this value will simply be skipped.
If you don't want to use this feature, set this to 0. Defaults to 0.2 (float). Read [5] before increasing this.

rob_x

This is Gary Robinson's x constant (cf. [3]). A completely unknown token will be rated with the value of rob_x. The default 0.5 (float) seems to be quite reasonable, as we can't say if a token that also can't be rated by degeneration is good or bad.
If you receive much more spam as ham or vice versa, you could change this setting accordingly.

rob_s

This is Gary Robinson's s constant. This is essentially the probability that the rob_x value is correct for a completely unknown token. It will also shift the probability of rarely seen tokens towards this value. The default is 0.3 (float)
See [3] for a closer description of the s constant and read [5] for specific information about this constant in b8's algorithms.

4.2   Configuration of the storage backend

All these values can be set in the "config_database" array (the second parameter) passed to b8. The name of the array doesn't matter (of course), it just has to be the second argument.

5.1   Setting up a new database

bayes*dbversion  => 2
bayes*texts.ham  => 0
bayes*texts.spam => 0

5.2   Using b8 in your scripts

Just have a look at the example script located in example/index.php to see how you can include b8 in your scripts. Essentially, this strips down to:

# Include the b8 code
require "{$_SERVER['DOCUMENT_ROOT']}/b8/b8.php";

# Do some configuration

$config_b8 = array(
        'some_key' => 'some_value',
        'foo' => 'bar'
);

$config_database = array(
        'some_key' => 'some_value',
        'foo' => 'bar'
);

# Create a new b8 instance
$b8 = new b8($config_b8, $config_database);

b8 provides three functions in an object oriented way (called e. g. via $b8->classify($text)):

learn($text, $category)

This saves the reference text $text (string) in the category $category (string, either ham or spam, case-sensitive!).

unlearn($text, $category)

This function just exists to delete a text from a category in which is has been stored accidentally before. It deletes the reference text $text (string) from the category $category (string, either ham or spam, case-sensitive!).
Don't delete a spam text from ham after saving it in spam or vice versa, as long you don't have stored it accidentally in the wrong category before! This will not improve performance, quite the opposite: it will actually break the filter after a time, as the counter for saved ham texts will reach 0, although you have ham tokens stored (because the filter will try to remove texts from the ham data which have never been stored there and just skip the non-existing words).

classify($text)

This function takes the text $text (string), calculates it's probability for being spam it and returns a value between 0 and 1 (float).
A value close to 0 is more likely ham and a value close to 1 is more likely spam. What to do with this value is your business ;-) See also Tips on operation below.