my million dollar idea of the week

I use spamassassin a lot (see the counter near the upper right of my main blog page), 'cause it's amazingly effective. A lot of its effectiveness is due to its use of a couple of distributed "hive mind" kind of information sharing services, namely Razor2 and DCC.

What these two services (free, internet-community provided services) do, in a simplified explanation is take a given message (that your mailer submits for spam testing) and runs it through a "hash", which (much like it sounds) is kind of a computational meatgrinder. If you tossed the contents of your fridge and pantry into a big blender, and kept a careful tally of what the mix tasted like after every couple of new ingredients, you could call that record your "kitchen hash".

Now, at the end, you certainly couldn't tell just from tasting the mix, or looking at that progressive record (salty, saltier, now spicier, now with a mango zing), what exactly was in there, but what you could tell is whether someone else, following the same instructions, had tossed roughly the same things into the blender in the same order. Or, more appropriately, you could tell if they did not use the same ingredients.

Computer hashes work in sort of the same way. You can have an email, and "hash" it, and end up with a couple lines of what looks like computer garbage, but if you ran the same email through ten times, you'd always get the same exact garbage, and if someone else did the same, they'd get that same "garbage".

This is useful, 'cause it lets us compare our emails with thousands of others over the 'net fairly efficiently (the hash is small) and privately (you don't see my love letters, just the hash). What Razor2 and DCC do (in slightly different ways) is take the hashes you submit, compare their database of already received message-hashes, and tell you whether they've seen it before, how many times, and whether people said "oh, and by the way, i opened it, and believe you me, you don't wanna visualize grandma wearing THAT."

(DCC, I believe, just measures message counts, and therefore is a good measure of how "bulky" a mail is, while Razor2 does something similar and also features "trusted submitters" who can say "this is confirmed spam". )

This means that I can use the fact that spammers spew their stuff far and wide against them, because if i've received a message that i can confirm that 1700 other people have also received, it's probably spam. Also, if i've got a handful of "spamtrap" addresses (sprinkled on webpages and in usenet postings, or submitted to that stupid "somebody's got a crush on you" site), you can know that any mail at ALL to those is crap, and report that as known spam. If I'm told "this matches known spam" by Razor2, then out the email goes. Nifty, huh?

Ok, now for the new, physical-world application of a "share your input with the hive" idea like that:

I see on gizmodo.com that they've got a radar detector that contains a GPS receiver.

Consider adding wireless data or an SMS-messaging carphone to that mix (I live in Manhattan, so I don't drive, but you can believe that the next car I own will be POSIX compliant, and you'll be able to NFS mount it from the garage's wireless network)

With inter-car communication, your radar detector with GPS can tell the mothership "I see a speedtrap at 41°38'N/91°33'W" and then other cars can query for that info, which shows up on their dashboard display maps as little smokey-bear icons.

Furthermore, since the GPS system knows your speed and direction (rate of change), it can report back to the hive, and requestors can get a feel for how congested roads are, given enough input, and map those too. ("i've got 100 subscribers in one mile approaching the Brooklyn Bridge, going 2mph, but i've got 40 subscribers approaching the 59th street bridge, and they're all going 45mph... mark the good bridge bright green, and the bad one red.")

You'd probably want to anonymize your data submissions, rather than admit to a central authority that you've been doing 95mph between lights on broadway. On the other hand, given the sheer number of taxis in NYC, and that they're licensed strictly by the city, maybe mandating each cab be a roving traffic reporter might be good for everyone, especially those taxis.

A cheaper, lower-tech solution, might be to read EZ-pass tags every mile or two from overhead signs, (without charging them) so they have an idea of how fast traffic is moving and where.

Cars that can say "hey, everybody, I'm at these coordinates, and i just sensed this" becomes pretty powerful for more reasons than I can probably think of right now.

Imagine somebody's car "knows" it's been parked away from its driveway for awhile, and now it's pulling out of a parallel parking spot (reverse with a turn, straighten out, forward)... now it can say "hey, a parking spot possibly just freed up here."

Coupling with your Mobil Speed Pass's info, you can report to the hive "just filled up with gas for this price over here", and the map can share gas prices with all the other members.

Note that I'm not talking about some systems that exist already that monitor fleets of trucks to see if their drivers are speeding, or gps systems that make sure the kids aren't taking the minivan up to Lover's Lane... those are interesting applications, but what really interests me is sensor data from your car (or your cellphone, for people walking around, or heck, from commercial aircraft maybe reporting data back to the NOAA) that can be shared, anonymously, for the benefit of the group.

I can imagine economies rising out of this, crediting users a few pennies for giving a traffic or weather update, charging users a bit to find cheaper gas, or get the quickest route from A to B, maybe giving rewards to sensors that report a traffic accident. Imagine being able to reward drivers who avoid a congested route with a buck off their next EZ-pass fare.

The privacy concerns for stuff like this will really need to be examined closely before tossing the system out there, but I predict that this is where we're headed. I'd love to hear any else's thoughts about stuff like this.

Posted by ryan at July 01, 2003 12:38 AM