1.1 What is b8?

b8 is a spam filter implemented in PHP. It is intended to keep your weblog or guestbook spam-free. The filter can be used anywhere in your PHP code and tells you whether a text is spam or not, using statistical text analysis. What it does is: you give b8 a text and it returns a value between 0 and 1, saying it's ham when it's near 0 and saying it's spam when it's near 1. See How does it work? for details about this.
To be able to do this, b8 first has to learn some spam and some ham (non-spam) texts. If it makes mistakes when classifying unknown texts or the result is not distinct enough, b8 can be told what the text actually is, getting better with each learned text.

b8 is a statistical spam filter. I'm not a mathematician, but as far as I can grasp it, the math used in b8 has not much to do with Bayes' theorem itself. So I call it a statistical spam filter, not a Bayesian one. Principally, It's a program like Bogofilter or SpamBayes, but it is not intended to classify emails. Therefore, the way b8 works is slightly different from email spam filters. See What's different? if you're interested in the details.

An example of what we're talking about here:

At the moment of this writing (November 2012), b8 has, since December 2006, classified 26,869 guestbook entries and weblog comments on my homepage. 145 were ham. 76 spam texts (0.28 %) have been falsely rated as ham (false negatives) and I had to remove them manually. Only one single ham message has been falsely classified as spam (false positive) back in June 2010, but – in defense of b8 – this was the very first English ham text I got. Previously, each and every of the 15,024 English texts posted have been spam. Texts with Chinese, Japanese or Cyrillic content (all spam either) did not appear until 2011.
This results in a sensitivity of 99.7 % (the probability that a spam text will actually be rated as spam) and a specifity of 99.3 % (the probability that a ham text will actually be rated as ham) for my homepage. Before the one false positive, of course, the specifity has been 100 % ;-)

1.2 How does it work?

In principle, b8 uses the math and technique described in Gary Robinson's articles "A Statistical Approach to the Spam Problem" 4 and "Spam Detection" 3. The "degeneration" method Paul Graham proposed in "Better Bayesian Filtering" 2 has also been implemented.

b8 cuts the text to classify to pieces, extracting stuff like email addresses, links and HTML tags and of course normal words. For each such token, it calculates a single probability for a text containing it being spam, based on what the filter has learned so far. When the token has not been seen before, b8 tries to find similar ones using "degeneration" and uses the most relevant value found. If really nothing is found, b8 assumes a default rating for this token for the further calculations.
Then, b8 takes the most relevant values (which have a rating far from 0.5, which would mean we don't know what it is) and calculates the combined probability that the whole text is spam.

1.3 What do I need for it?

Not much! You just need PHP (at least 5, 4 is not supported anymore) and a database to store the wordlist.

The probably most efficient way to store the wordlist is using a Berkeley DB. This also has been the original approach, and b8 comes with an appropriate storage backend. Of course, other databases can be used as well, as an example, there's also a backend using a mysqli object-style MySQL connection.

It should be quite trivial to create a storage backend for whatever database you want to use. Simply write a class extending \b8\storage\storage_base which implements the abstract functions listed there.

1.4 What's different?

b8 has been designed to classify forum posts, weblog comments or guestbook entries, not emails. For this reason, it uses a slightly different technique than most of the other statistical spam filters out there use.

My experience was that spam entries on my weblog or guestbook were often quite short, sometimes just something like "123abc" as text and a link to a suspect homepage. Some spam bots don't even made a difference between e. g. the "name" and "text" fields and posted their text as email address, for example. Considering this, b8 just takes one string to classify, making no difference between "headers" and "text".
The other thing is that most statistical spam filters count one token one time, no matter how often it appears in the text (as Paul Graham describes it in 1). b8 does count how often a token has been seen and learns resp. considers this. Why this? Because a text containing one link (no matter where it points to, just indicated by a "http://" or a "www.") might not be spam, but a text containing 20 links might be.

This means that b8 might be good for classifying weblog comments, guestbook entries or forum posts (I really think it is ;-) – but very likely, it will work quite poor when being used for something else like classifying emails. At least with the default lexer. But as said above, for this task, there are lots of very good filters out there to choose from.

2.1 Update from bayes-php version 0.2.1 or earlier

Please first follow the database update instructions of the bayes-php-0.3 release if you update from a version prior to bayes-php-0.3, then read on.

2.2 Update from bayes-php version 0.3 to any pre-0.5 version of b8

Version 0.5 introduced some changes. Here they are. Please read Update from b8 0.5.* for how to update your database.

If you use SQLite: Sorry, at the moment, there's no SQLite backend for b8. You can probably create a dump of your database and insert it into a MySQL table.

b8's lexer has been partially re-written. It should now be able to handle all kind of non-latin-1 input, like Cyrillic, Chinese or Japanese texts. Caused by this fact, much more tokens will be recognized when classifying such texts. Therefore, you could get different results in b8's ratings, even if the same database is used and although the math is still the same.

b8 0.5 introduced two constants that can be used in the learn() and unlearn() functions: b8::HAM and b8::SPAM. The literal values "ham" and "spam" can still be used anyway.

2.3 Update from b8 0.5.*

The lexer and the degenerator can now be configured via an additional config array. If you want to use the new lexer and/or degenerator config, read through the Configuration section.

The database format has changed. There's an update script for DBA and MySQL databases which can be found in the directory update/. Simply edit the configuration array inside the respective script and run it. A new database with the current structure (v3) will be created. When the update was okay, simply replace your current database with the new one or change your configuration in a way that the new database will be used by b8.

The validate() functions have been removed in favor of throwing exceptions when something goes wrong instantiating b8. So if you set up b8 like this

$b8 = new b8($config_b8, $config_storage);

$started_up = $b8->validate();

if($started_up !== true) {
    echo "Error: ", $started_up;
    do_something();
}

you will have to change your code to something like this:

try {
    $b8 = new b8($config_b8, $config_storage, $config_lexer, $config_degenerator);
}
catch(Exception $e) {
    echo "Error: ", $e->getMessage();
    do_something();
}

When an error occurs while instantiating b8, the object will simply not be created.

2.4 Update from b8 0.6.*

$b8 = new b8\b8(...);

$db = dba_open('wordlist.db', 'w', 'db4');
$config_dba = [ 'resource' => $db ];

4.1 b8's base configuration

All these values can be set in the "config_b8" array (the first parameter) passed to b8.

These are some basic settings telling b8 which backend classes to use:

storage

This defines which storage backend will be used to save b8's wordlist. It's the name of the class in the b8\storage namespace. b8 comes with two example backends, the default setting is dba (string). It should be quite easy to add a custom backend. Simply create a class fitting your needs that extends b8\storage\storage_base and implements it's abstract functions.

dba

This one uses a Berkeley DB, the original storing approach of the filter. All content is saved in a single file, you don't need special user rights or a database server. Most probably a good choice, as this is very performant and fits exactly to b8's needs.
If you don't know whether your server's PHP installation supports Berkeley DB, simply run the script install/setup_berkeleydb.php. If it shows a Berkeley DB handler, you can use this backend.

mysql

An example implementation using a MySQL table to store the wordlist.

lexer

The lexer class to be used. Defaults to standard (string).
At the moment, only one lexer exists, so you probably don't want to change this unless you have written your own lexer.

degenerator

The degenerator class to be used. See How does it work? and 2 if you're interested in what "degeneration" is. Defaults to standard (string).
At the moment, only one degenerator exists, so you probably don't want to change this unless you have written your own degenerator.

The following settings influence the mathematical internals of b8. If you want to experiment, feel free to play around with them; but be warned: wrong settings of these values will result in poor performance or could even "short-circuit" the filter. Leave these values as they are unless you know what you are doing.

The "Statistical discussion about b8" 5 shows why the default values are the default ones.

use_relevant

This tells b8 how many tokens should be used to calculate the spamminess of a text. The default setting is 15 (integer). This seems to be a quite reasonable value. When using too many tokens, the filter will fail on texts filled with useless stuff or with passages from a newspaper, etc. not being very spammish.
The tokens counted multiple times (see above) are added in addition to this value. They don't replace other interesting tokens.

min_dev

This defines a minimum deviation from 0.5 that a token's rating must have to be considered when calculating the spamminess. Tokens with a rating closer to 0.5 than this value will simply be skipped.
If you don't want to use this feature, set this to 0. Defaults to 0.2 (float). Read 5 before increasing this.

rob_x

This is Gary Robinson's x constant (cf. 3). A completely unknown token will be rated with the value of rob_x. The default 0.5 (float) seems to be quite reasonable, as we can't say if a token that also can't be rated by degeneration is good or bad.
If you receive much more spam than ham or vice versa, you could change this setting accordingly.

rob_s

This is Gary Robinson's s constant. This is essentially the probability that the rob_x value is correct for a completely unknown token. It will also shift the probability of rarely seen tokens towards this value. The default is 0.3 (float)
See 3 for a closer description of the s constant and read 5 for specific information about this constant in b8's algorithms.

4.2 Configuration of the storage backend

The used storage backend itself defines what it wants to have passed in it's configuration array. The two example backends have this configuration:

4.3 Configuration of the lexer

The lexer disassembles the text we want to analyze to single words ("tokens"). The way it does this can be customized.

All the following values can be set in the "config_lexer" array (the third parameter) passed to b8. The name of the array doesn't matter (of course), it just has to be the third argument.

min_size

The minimal length for a token to be considered when calculating the rating of a text. Defaults to 3 (integer).

max_size

The maximal length for a token to be considered when calculating the rating of a text. Defaults to 30 (integer).

allow_numbers

Should pure numbers also be considered? Defaults to false (boolean).

get_uris

Look for URIs. Defaults to true (boolean).

old_get_html

Extracts HTML. This is the old search function used up to b8 0.5.2. If you have an existing b8 installation and want the exactly same behaviour as before, use this. This function will probably removed in a future release. Defaults to true (boolean).

get_html

Extracts HTML. This has been added in b8 0.6 and should work better then the "old_get_html" procedure. Defaults to false (boolean).

get_bbcode

Extracts BBCode, which is often used in forums. Defaults to false (boolean).

4.4 Configuration of the degenerator

When a token is not found in the database, b8 tries to find similar versions of that token. The degenerator provides these similar versions (cf. 2). The way it generates these "degenerates" can be customized.

All the following values can be set in the "config_degenerator" array (the fourth parameter) passed to b8. The name of the array doesn't matter (of course), it just has to be the fourth argument.

multibyte

Use multibyte operations when searching for degenerated versions of an unknown token. When activating this, b8 needs PHP's mbstring module to work. Defaults to false (boolean).

encoding

The internal encoding to use when doing multibyte operations. This will only be used when multibyte is set to true. Defaults to UTF-8 (string).

The difference of using or not using multibyte operations will only show up when non-latin-1 text is processed by b8. For example, if we have an unknown token HeLlO!, the degenerator will provide the degenerated versions hello!, HELLO!, Hello!, hello, HELLO, Hello and HeLlO, no matter if multibyte operations are used or not.

When we have a non-latin-1 word, we may get a different result. For example, if we have the unknow token ПрИвЕт!, the degenerator will only provide one degenerated version of it: ПрИвЕт. Using multibyte operations, we get the same variants as with the latin-1 word: привет!, ПРИВЕТ!, Привет!, привет, ПРИВЕТ, Привет and ПрИвЕт.

Using multibyte operations will simply make the degenerator more effective.

5.1 Setting up a new database

"b8*dbversion" => "3"
"b8*texts"     => "0 0"

5.2 Using b8 in your scripts

Just have a look at the example script example/index.php to see how you can include b8 in your scripts. Essentially, this strips down to:

# Include b8's code
require_once($path_to . 'b8.php');

# Do some configuration
$config_b8          = array('some_key' => 'some_value', ...);
$config_storage     = array('some_key' => 'some_value', ...);
$config_lexer       = array('some_key' => 'some_value', ...);
$config_degenerator = array('some_key' => 'some_value', ...);

# Create a new b8 instance
try {
    $b8 = new b8\b8($config_b8, $config_storage, $config_lexer, $config_degenerator);
}
catch(Exception $e) {
    do_something();
}

b8 provides three functions in an object oriented way (called e. g. via $b8->classify($text)):

classify(string $text)

This function takes the text $text (string), calculates it's probability for being spam and returns it in the form of a value between 0 and 1 (float).
A value close to 0 says the text is more likely ham and a value close to 1 says the text is more likely spam. What to do with this value is your business ;-) See also Tips on operation below.

learn(string $text, string $category)

This saves the text $text (string) in the category $category (b8 constant, either b8\b8::HAM or b8\b8::SPAM).

unlearn(string $text, string $category)

You don't need this function in normal operation. It just exists to delete a text from a category in which is has been stored accidentally before. It deletes the text $text (string) from the category $category (b8 constant, either b8::HAM or b8::SPAM).
Don't delete a spam text from ham after saving it in spam or vice versa, as long you don't have stored it accidentally in the wrong category before! This will not improve performance, quite the opposite! The filter will always try to remove texts from the ham or spam data, even if they have never been stored there. The counters for tokens which are found will be decreased or the word will be deleted and the non-existing words will simply be ignored. But always, the text counter for the respective category will be decreased by 1 and will eventually reach 0. Consequently, the ham-spam texts proportion will become distorted, deteriorating the performance of b8's algorithms.

9.1 FAQ

9.2 The database layout

The database layout is quite simple. It's essentially just a key-value pair for everything stored. There are two "internal" variables stored as normal tokens. A lexer must not provide a token starting with b8*, otherwise, we will probably get collisions. The internal tokens are:

b8*dbversion

This indicates the database's version.

b8*texts

The number of ham and spam texts learned.

Each "normal" token is stored with it's literal name as the key and it's data as the value. The backends store the token's data in a different way. The DBA backend simply stores a string containing both values separated by a space character. The SQL backends store the counters in different columns.

A database query is always done by searching for a token's name, never for a count value.