My-CL   Email Alert System For Craig's List

Home Projects My-CL

My-CL

I was working at CNET.com when the internet boom happened. Right about that time, Craig's List was launched, and it was instantly the go-to site in the Bay Area. It took many years for it to gain wider acceptance around the U.S., but it was a trustworthy lifeline for housing, furniture, bikes, jobs, electronics, and even relationships in the Bay Area right away.

As so many of my projects do, this one started out as something I wanted. Even that I needed. I couldn't possibly be on the Craig's List site right when a listing that mattered to me was posted, so I wrote an email alert system. It was written in perl and used a highly configurable crawler to only fetch results for queries that counted. The interface I build allowed myself, and eventually other users, to choose by category, keyword(s), and price range. Later, I added the ability to search all the Craig's List cities in the entire U.S.

Word quickly spread amongst my friends about the interface I had written. I bought the domain name my-cl.com, though I sure didn't like the hyphen! I wanted a short, memorable domain name that was to the point, and that did the trick. I often attended industry events in San Francisco, and people asked me about it. Soon, I had 300 users signed up, searching for all manner of items on Craig's List. My system was generating about 1,200 alerts per day. I got emails from people thanking me for the system, telling me they would never have found such-and-such job, item, apartment, etc. were it not for my system.

I added the ability to search "deep" instead of just search by headlines. Also, I started to parse results pages for images, taking the first one in most cases. All in all, my Craig's List alert system was humming along doing just fine, and I let it run for a years without much maintenance.

At some point, Craig's List buckled down on crawlers and spiders, and as they often do (as recently as July 2013), they changed up their HTML layout. Whenever that happens, it means I need to review and revise my HTML parser. Eventually, my code could no longer do a "deep" crawl, and various other sites sprouted up for saving and executing Craig's List searches. Since this was just a fun project to begin with, I eventually saved off all my code and shut down the site.

I now run my own personal Craig's List script to alert me to listings. I can certainly help any business that needs to make use of Craig's List data, either by analyzing existings listings across the U.S. or by being alerted to new listings or trends in listings.

My-CL

Newsletter Subscription
Follow me on:

Latest news

  • Twitter Bootstrap
    Twitter Bootstrap 3.0.0 RC1
    It's here - and now just one CSS file
    Time to bootstrap more websites!
  • QBike Store News
    QBike Store 1,000th Sale
    QBike just surpassed 1,000 sales
    That's 1,000 satisfied customers and counting!

© 2012-2013, All Rights Reserved