How to Build an SMS Service

by Brian Retford, Jordan Schwartz

Publisher: O'Reilly

Pub Date: June 15, 2007

Print ISBN-10: 0-596-51513-8

Print ISBN-13: 978-0-59-651513-3

Pages: 72

Copyright

Introduction

Chapter 1. Introducing SMS

Section 1.1. Message Format

Section 1.2. Definitions

Section 1.3. International Issues

Chapter 2. Types of SMS Service

Section 2.1. Notification

Section 2.2. Lookup

Section 2.3. Messaging/Social Networking

Section 2.4. Action

Section 2.5. Art

Chapter 3. SMS Interaction Design Considerations

Section 3.1. Short message length

Section 3.2. Intrusiveness

Section 3.3. High Barrier to Input

Section 3.4. Slow Interactivity

Section 3.5. Minimal Affordances

Section 3.6. Limited Character Set

Section 3.7. Always On Everywhere

Section 3.8. Store and Forward

Section 3.9. Multitasking and Stealth

Chapter 4. SMS Short Codes

Section 4.1. More Is Less

Chapter 5. SMS Fundamentals

Section 5.1. Pricing: Paying for SMS Messages

Section 5.2. Premium SMS: Getting Paid for Your Service

Chapter 6. Six Ways to Implement SMS

Chapter 7. Using an SMS Aggregator

Section 7.1. The Basics

Section 7.2. Advantages and Disadvantages

Section 7.3. Example Service: Conference Messaging

Chapter 8. Using a Smartphone as an SMS Modem

Section 8.1. The Basics

Section 8.2. Advantages and Disadvantages

Section 8.3. Example Service: Echo and Hello World Using a Smartphone

Section 8.4. Server Architecture

Section 8.5. Tips and Tricks

Chapter 9. Using Email to SMS Gateways

Section 9.1. The Basics

Section 9.2. Example Service: Weather Updates

Chapter 10. Using a Mashup: 411Sync

Section 10.1. 

Chapter 11. Using Twitter to Build a SMS Service

Section 11.1. Twitter Overview

Section 11.2. The Twitter API

Section 11.3. Advantages and Disadvantages

Section 11.4. Example Service: Echo

Chapter 12. Using Windows Live Alerts

Chapter 13. Summary

Copyright

Copyright © 2007, O'Reilly Media, Inc.. All rights reserved.

Published by O'Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O'Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com.

Nutshell Handbook, the Nutshell Handbook logo, and the O'Reilly logo are registered trademarks of O'Reilly Media, Inc. The vulpine phalanger and related trade dress are trademarks of O'Reilly Media, Inc.

Many of the designations uses by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

Introduction

SMS, or Short Message Service, is rapidly emerging as a multibillion dollar a year industry. A leading research firm reports that consumer spending on mobile services represents a $140 billion opportunity worldwide, with person-to-person SMS making up half of that. Portio Research reported that there were 946 billion SMSs sent in 2006. Clearly, the medium has hit the mainstream.

Reach and mobility are SMSs most powerful draws. There are 2.41 billion SMS-ready handsets in the world today, enough for a third of the world's population. In fact, some 30 countries have more than one cell phone per person.

As for mobility, anywhere a cell phone can go, a text message can go. This gives you, the service or application developer, the opportunity to deeply integrate your features into daily life. Coupled with the fact that it is a true push technology, this means that all types of information can be delivered to people wherever they are, exactly when it's most meaningful or useful.

Creating your own SMS service can be quite simple, as there are a number of different implementation options suitable to a variety of needs and circumstances. This guide will describe several of them to help you pick the option most appropriate to your needs.

However, there is more to building a successful SMS service than implementing code that sends and receives text messages. SMS presents a different set of opportunities and constraints relative to a traditional browser-based web service or desktop client application. The 160 character limit is the most obvious limit, and the "Always On Everywhere" nature of the mobile phone the most obvious opportunity, but there are a host of other less apparent factors to consider when designing your service.

So, beyond describing the mechanics of creating a SMS service, this guide seeks to describe the art of creating a good SMS service.

Introducing SMS > Message Format

Chapter 1. Introducing SMS

An SMS is a message delivered to or from a mobile phone. In the industry, messages originating from a phone are referred to as MO (mobile originated) or MT (mobile terminated). As author or operator of a service that employs SMS, you don't need to know a lot about the specific protocols and pathways the carriers use to send these messages, but there are aspects of the format that may impact your use.

1.1. Message Format

A text message is limited to 140 bytes, or 160 7-bit characters. If you are sending Arabic, Korean, or other languages that require 16-bit encoding, you will be limited to 70 characters.

It's possible to send 8-bit characters, but they are not typically viewable on phones (they are used instead for data such as ringtones and over the air provisioning of WAP settings). Additional information, such as the sender's number and the time stamp, are included as well, but are not counted against the 160 character limit.

NOTE

MMS, or Multimedia Message Service, is an evolution of the SMS system that allows users to send photos, videos, and audio in addition to text. Unlike an SMS, there is no set limit on text length, either. Instead of being sent on the GSM signaling channel, MMSs are sent over an IP connection from an MMS-C to the phone. The IP connection to an MMS-C must always be initiated from the phone, however, a carrier will typically send an SMS to the phone that contains instructions for the phone to connect back to an MMS-C via IP to collect the MMS message.

The system for sending and receiving MMS messages is quite different from SMS, and varies somewhat by carrier, so it won't be covered in this guide.

Introducing SMS > Message Format

Chapter 1. Introducing SMS

An SMS is a message delivered to or from a mobile phone. In the industry, messages originating from a phone are referred to as MO (mobile originated) or MT (mobile terminated). As author or operator of a service that employs SMS, you don't need to know a lot about the specific protocols and pathways the carriers use to send these messages, but there are aspects of the format that may impact your use.

1.1. Message Format

A text message is limited to 140 bytes, or 160 7-bit characters. If you are sending Arabic, Korean, or other languages that require 16-bit encoding, you will be limited to 70 characters.

It's possible to send 8-bit characters, but they are not typically viewable on phones (they are used instead for data such as ringtones and over the air provisioning of WAP settings). Additional information, such as the sender's number and the time stamp, are included as well, but are not counted against the 160 character limit.

NOTE

MMS, or Multimedia Message Service, is an evolution of the SMS system that allows users to send photos, videos, and audio in addition to text. Unlike an SMS, there is no set limit on text length, either. Instead of being sent on the GSM signaling channel, MMSs are sent over an IP connection from an MMS-C to the phone. The IP connection to an MMS-C must always be initiated from the phone, however, a carrier will typically send an SMS to the phone that contains instructions for the phone to connect back to an MMS-C via IP to collect the MMS message.

The system for sending and receiving MMS messages is quite different from SMS, and varies somewhat by carrier, so it won't be covered in this guide.

Introducing SMS > Definitions

1.2. Definitions

Before we get much further, it will be useful to establish some common terms that will come up throughout the guide. These include:

Short code

A 4- to 12-digit (in the U.S.) number to which mobile phones can send and receive text messages. While they are technically unique to each carrier, the Common Short Code Administration (http://www.usshortcodes.com) acts as the common registrar in the U.S., ensuring that the same short code is not issued by different carriers to different application providers.

GSM

Digital cellular technology used by virtually all carriers worldwide (but see below).

CDMA

Digital cellular technology used in the U.S. by Sprint and Verizon.

SIM card

Removable smart card found in GSM phones. Contains configuration data (e.g., phone number, SMS-C connectivity info) as well as user data (e.g., portable phone book). Contains a small CPU that processes SMS messages.

WAP

Wireless Application Protocol. Open international standard for encoding Internet data for mobile devices. WAP sites are generally written in XHTML MP.

SMS aggregator

A service that provides an interface to application providers (that's you) to send and receive text messages. An SMS Aggregator typically exposes web interfaces to the application providers and connects directly to the SMS gateways (SMS-C) of multiple carriers to send and receive the messages.

Application Provider

A service that provides the content and logic behind an SMS service. This could be you!

Carrier (aka Operator)

A company that sells or rents telecommunication transmission services such as telephony and data communication. For example, T-Mobile, Cingular, and Sprint in the U.S., and Orange and British Telecom in Europe.

Introducing SMS > International Issues

1.3. International Issues

This short cut was written in the United States, and, as much of the writing coming out of that country, tends to be U.S.-centric. You will see dollars quoted, U.S. short code formats addressed, and U.S. pricing schemes assumed. Apologies are offered to readers worldwide, and assurances given that, even if some of the details may differ, the underlying concepts and technologies are the same everywhere.

Types of SMS Service > Notification

Chapter 2. Types of SMS Service

In some ways, SMS is just an input/output protocol for an application. A quick survey of existing applications can help frame what types of requirements your service may generate. While this list is far from exhaustive, it is useful to recognize five basic categories, which we'll explore in the following sections: notification, lookup, messaging and social networking, action and what we call, for want of a better term, art.

2.1. Notification

Notification is the simplest type of SMS service: you sign-up to receive a message on your phone when some event occurs (i.e., not in response to a proximal user action). For example:

Microsoft's Windows Live Alerts sends stock quotes, traffic updates, horoscopes, and more via SMS.

Google's Calendar service sends reminders for events via SMS.

University of Amsterdam sends students their exam results via SMS.

While these types of services tend to be "set and forget," they can also be interactive. While there is no response required when receiving a weather report via SMS, but a stock quote received via SMS could engender a "sell" reply and a resulting action on your behalf.

Notification services tend to be the easiest to create. As we will show in later sections, any of the implementations of SMS service will lend itself to a notification service.

Types of SMS Service > Notification

Chapter 2. Types of SMS Service

In some ways, SMS is just an input/output protocol for an application. A quick survey of existing applications can help frame what types of requirements your service may generate. While this list is far from exhaustive, it is useful to recognize five basic categories, which we'll explore in the following sections: notification, lookup, messaging and social networking, action and what we call, for want of a better term, art.

2.1. Notification

Notification is the simplest type of SMS service: you sign-up to receive a message on your phone when some event occurs (i.e., not in response to a proximal user action). For example:

Microsoft's Windows Live Alerts sends stock quotes, traffic updates, horoscopes, and more via SMS.

Google's Calendar service sends reminders for events via SMS.

University of Amsterdam sends students their exam results via SMS.

While these types of services tend to be "set and forget," they can also be interactive. While there is no response required when receiving a weather report via SMS, but a stock quote received via SMS could engender a "sell" reply and a resulting action on your behalf.

Notification services tend to be the easiest to create. As we will show in later sections, any of the implementations of SMS service will lend itself to a notification service.

Types of SMS Service > Lookup

2.2. Lookup

In a lookup service, the user initiates the SMS conversation. For example:

Send "Define <word>" to 46645 (GOOGL) or 95482 (WLIVE) and you will receive an SMS response with the word's definition.

Send a book's ISBN number to TicTap at 763-807-3927 to find out how much it costs on http://Amazon.com.

Lookup services have an added requirement beyond those of a notification service: you have to be able to send them a text message, not just receive one (making the email to SMS gateway implementation described later inappropriate). They are otherwise as straightforward to create as a notification service: take text input, parse, process, and respond via text.

Types of SMS Service > Messaging/Social Networking

2.3. Messaging/Social Networking

Messaging or social networking services like Dodgeball, Twitter, 3Jam, UPOC, and Microsoft's Slam, act as an intermediary between users, receiving a message from one user, then sending on some form of the message to other users based on some logic.

Dodgeball, for example, enables location-aware social networking. The Dodgeball system has a database of known locations. Users establish a network of friends, then "check-in" by sending an SMS to the service with their location. For example, let's say I wanted to let people know I was at the club "Hive Mind" on 4th Avenue in Seattle. I could send:

@Hive Mind

And all my friends would receive:

dodgeball.com says: Jordan S. @ Hive Mind
(4065 4th Ave NE at 42nd St)
at 10:28 PM. Reply w/"@venue name" to check in!"

Twitter offers a similar system, but more focused on social blogging and less on location-specific check-ins. Twitter allows users to be "friends" or "followers" of one another. Messages sent to Twitter are redistributed to all the friends and followers of the sender. Person to person messages can also be sent directly through the Twitter system, using the d <username> <message> syntax.

3Jam, UPOC and Slam all offer group-centric text messaging. Users establish shared groups (rather than social networks) and message with them. Any message sent to a group is redistributed to everyone in that group via SMS, and replies are sent to the same set of people who received the original message. This allows for conversations and threads in a way that social network-based services do not.

Messaging and social networking SMS services generally require a relatively complex syntax for interaction, so they are often supplemented with a web site for some of the "heavy lifting" user interface (e.g., group management or preferences).

Types of SMS Service > Action

2.4. Action

Services in this category allow the user to initiate an event. For example:

The American Idol television show allows viewers to send in a text message to vote for their favorite contestant.

CIMB Investment Bank allows users to buy and sell stock via text message.

DPS-Promatic allows users to control their home thermostat and lights via text message.

Types of SMS Service > Art

2.5. Art

We're just such big fans of these services that we couldn't help mentioning them. There are some great examples out there of SMS services that don't fit cleanly into any utilitarian category, but are valuable in their own way. We'll restrain ourselves to a couple examples:

Troika's SMS Guerilla Art Projector projects SMS messages from a distance onto any surface (http://www.troika.uk.com/sms-guerrilla-projector.htm)

Three Maryland Institute of the Arts students created a museum piece in which museum-goers send their mood into the piece via SMS and the piece adapts throughout the evening (read about it in The Washington Post at http://www.washingtonpost.com/wp-dyn/content/article/2007/03/16/AR2007031602193.html)

SMS Interaction Design Considerations > Short message length

Chapter 3. SMS Interaction Design Considerations

Many of the same principles that apply to designing a software application apply to designing an SMS service: keep it simple, interface text should be meaningful and understandable to the user, always give the user a means to solve a problem that arises, etc. However, the unique qualities of SMS introduce some unique challenges and pitfalls, as well.

3.1. Short message length

The most obvious constraint on designing an SMS service is the maximum size of the message: in English, the best case, you get 160 characters to get your message across to your user. To put that into perspective, this sentence is 84 characters (including the period).

There are a several strategies you can consider as a service author:

Aggressive editing: After you think you've pared down your message as short as it can go, take off another 10 characters. Try dropping nouns, verbs, and prepositions and see if the message still makes sense. Ask yourself if you need all the information and context you are providing in the message. Brevity is the soul of wit and the heart of SMS.

Text speak:It may compromise the professional air of your service, but, if used properly, it could win you street cred. U can m8k yr msg shrtr! (Used inappropriately, this will win you ridicule.)

Multiple messages: This should be your last resort, but you can send a single "message" as multiple SMSs. If you attempt to send a message longer than 160 characters, the carrier will typically break the message up into multiple messages for you, but the breaks may not be where and when you want. Avoid orphaned words or forcing users to click between messages to complete a thought by breaking a message at logical stop points (e.g., the end of a sentence).

When you send multiple messages, it is standard practice to number messages and include the total number of messages in the communication, as they may not be delivered in the order sent (or may not be delivered at all). For example "1/2: Lorem Ipsum" would indicate the first message of 2. An ellipsis (...) at the end of the first message is another cue to your user that another message is on its way.

NOTE

Many phones and carriers support segmented or concatenated SMS. A segmented SMS message is a long message broken into multiple standard-size messages with a header containing segmentation information. The receiving phone uses the header information to re-assemble the multiple messages into the single message.

An advantage of segmented SMS is that it is backward compatible. If the receiving phone does not support segmented SMS, it will simply display each segment as an individual message.

SMS Interaction Design Considerations > Short message length

Chapter 3. SMS Interaction Design Considerations

Many of the same principles that apply to designing a software application apply to designing an SMS service: keep it simple, interface text should be meaningful and understandable to the user, always give the user a means to solve a problem that arises, etc. However, the unique qualities of SMS introduce some unique challenges and pitfalls, as well.

3.1. Short message length

The most obvious constraint on designing an SMS service is the maximum size of the message: in English, the best case, you get 160 characters to get your message across to your user. To put that into perspective, this sentence is 84 characters (including the period).

There are a several strategies you can consider as a service author:

Aggressive editing: After you think you've pared down your message as short as it can go, take off another 10 characters. Try dropping nouns, verbs, and prepositions and see if the message still makes sense. Ask yourself if you need all the information and context you are providing in the message. Brevity is the soul of wit and the heart of SMS.

Text speak:It may compromise the professional air of your service, but, if used properly, it could win you street cred. U can m8k yr msg shrtr! (Used inappropriately, this will win you ridicule.)

Multiple messages: This should be your last resort, but you can send a single "message" as multiple SMSs. If you attempt to send a message longer than 160 characters, the carrier will typically break the message up into multiple messages for you, but the breaks may not be where and when you want. Avoid orphaned words or forcing users to click between messages to complete a thought by breaking a message at logical stop points (e.g., the end of a sentence).

When you send multiple messages, it is standard practice to number messages and include the total number of messages in the communication, as they may not be delivered in the order sent (or may not be delivered at all). For example "1/2: Lorem Ipsum" would indicate the first message of 2. An ellipsis (...) at the end of the first message is another cue to your user that another message is on its way.

NOTE

Many phones and carriers support segmented or concatenated SMS. A segmented SMS message is a long message broken into multiple standard-size messages with a header containing segmentation information. The receiving phone uses the header information to re-assemble the multiple messages into the single message.

An advantage of segmented SMS is that it is backward compatible. If the receiving phone does not support segmented SMS, it will simply display each segment as an individual message.

SMS Interaction Design Considerations > Intrusiveness

3.2. Intrusiveness

SMS is intrusive. Part of the attractiveness of SMS lies in its direct access to user attention. Email is filtered, snail mail glutted, but who doesn't reach immediately for their phone when the familiar chirp of an incoming message sounds?

However, attention is a limited resource, and if you squander it, you will lose the access to the very users you are trying to reach.

Many users of Dodgeball, for example, loved the service on first signing up. Establish links to a few friends, set up your phone and the next thing you know, you're receiving an SMS whenever a friend is out somewhere.

Anywhere. All the time.

What began as a fun way to meet up with friends resulted in a torrent of SMS messages, 10, 20, or 30 per night, at all hours. You're trying to sleep, and your phone is chirping and buzzing, letting you know that somewhere, people you know, or people who know somebody you know, are somewhere you're not!

It would be easy to blame your friends for prattling, but, of course, that is just blaming the user. They are only using the service as it was designed. A well designed service will channel its users into appropriate behaviors.

Dodgeball provided some mechanisms for controlling this flow (e.g., send "off" to turn off all messages for the night, "stop" to staunch the flow indefinitely, and a blocked list of friends you don't want to receive messages from temporarily), but these course controls could still leave the user frustrated. How do I get just the important messages when I want?

Twitter has begun to address this problem by creating a variety of controls for the receiver:

Users can set "sleep" times during which they don't wish to receive any text messages.

Users can choose to receive only direct messages (or no messages at all) as SMS (the others are read on the PC).

Some guidelines:

When building a social networking or messaging service, establish behavioral norms through your user interface and through your syntax to coax your users to send messages to their contacts appropriately.

Give your users a means of filtering and controlling the messages they receive.

Users should opt-in to your service, not just out. Foisting a service on someone who doesn't want it erodes your brand and earns you nothing but ill will.

NOTE

In the U.S., SMS spam is covered by the CAN-SPAM Act of 2003 and the Telephone Consumer Protection Act of 1991.

Worldwide, laws vary. Ethics don't:

Do not send unsolicited SMS messages to people.

Make sure your users have an obvious and easy means of quitting your service if they so choose.

Customers should be wooed, not tricked or shackled. Let your users opt-in and opt-out.

Let social networks build organically. Mining address books is just nasty.

SMS Interaction Design Considerations > High Barrier to Input

3.3. High Barrier to Input

Texting is hard. Whether you're using multitap (e.g., tap the 2 on your keypad once for "a," twice for "b," and thrice for "c") or T9 (e.g., allow your phone to guess whether 4663 means "good," "home," "gone," etc.), it's still tough to enter a long message on a cell phone.

The most obvious mitigation strategy is to require little or no input. For example, Dodgeball keeps a database of street addresses of known locations, allowing users to simply name the location, or some unique substring of the location. Twitter allows users to send messages directly to other users simply by preceding the message with the letter "d."

However, an oft-overlooked fact is that many phones support T9 input, meaning words recognizable by a phones internal dictionary may be easier for a user to type than a shorter set of characters that do not constitute a real word. For example, entering the character "@" on a Windows Mobile 5 phone requires seven key taps (the "1" key followed by the down arrow six times in T9 mode or the "1" key seven times in multitap mode). Entering the word "at" requires only two key taps. Choose your syntax accordingly.

NOTE

One December 19, 2006, 18-year-old Ben Cook or Portland, OR reclaimed the title of Guinness Book of World Records Fastest Text Messager from Singapore student Ang Ghuang, 16.

Cook took the championship by texting the standard phrase "The razor-toothed piranhas of the genera Serrasalmus and Pygocentrus are the most ferocious freshwater fish in the world. In reality they seldom attack a human." in 41.00 seconds. Ghuang's record, set little more than a month earlier, was 41.52 seconds.

How fast are you?

SMS Interaction Design Considerations > Slow Interactivity

3.4. Slow Interactivity

SMS can be slow. Imagine the heaviest web page you can. Now imagine viewing that web page over a 2,400 baud modem. That, essentially, is the delay you face when waiting for a reply from a text message service. Messages may come back quickly, in a matter of seconds, or you could be waiting for minutes.

Fortunately, user expectations are usually set appropriately by the medium, so you get some latitude, but you still need to minimize the round trips required for an interaction. A user should never have to send a response to a response. If that much interactivity is required, they are likely to put down the phone and seek out a computer (or another service).

Here are some tricks for minimizing round trips:

Use intelligent defaults: Whatever you think the user will want to do 80% of the time, just do it. Don't ask "are you sure" type questions if you can avoid it.

Combine steps: Allow the user to include multiple pieces of information or decisions in a single message.

SMS Interaction Design Considerations > Minimal Affordances

3.5. Minimal Affordances

In user interface design, an affordance is a clue to a user that a given action or behavior is possible. However, as Grady Leno of http://Phonesherpa.com put it, text messaging is a lot like playing Zork:

>You are in an open field west of a big white house with a boarded front door.

>There is a small mailbox here.

What are you supposed to do? Look around?

>I don't understand that

When you're staring at your phone, there is very little interface available to tell you what to do. No blue underlined hyperlinks, beveled Submit buttons, or labeled fields, just a blank text message input form.

Consequently, the need for your interface to be as intuitive as possible is very high; it's critical that your users' first guess regarding how to get something done (or what can be done) be right.

The first strategy for increasing the intuitiveness of your interface is to leverage existing syntaxes. The Microformats wiki has a good list of standards being used for SMS and command line interfaces at http://microformats.org/wiki/picoformats. For example, many services give the user the option of silencing incoming messages, but not all in the same way (see Table 3-1).

Table 3-1. How to silence incoming messages

| Dodgeball | Twitter | Slam | Facebook | Mixd
Stop messaging tonight option | Off | N/A | N/A | N/A | N/A
Stop messages indefinitely option | stop | off | .off | off | stop all

As you can see, there is some convergence, but no standardization to syntax. Accepting multiple inputs to mean the same thing may be a necessary mitigation.

NOTE

Whether you decide to use "stop" or "off" as your "official" means of turning off messages, it's important to make it very easy for users to ask not to be disturbed anymore, not just because it's the respectful thing to do, but because carriers may not allow you to operate a service on their network if you don't.

Don't be surprised if you're required to support "stop," "quit," "leave," "unsubscribe," and "unsub" as means of hushing up.

A second strategy involves providing affordances in all or selective outgoing messages. Dodgeball appends the instruction reply with "@venue name" to check-in when users check-in and Reply to xxx-xxx-xxxx to say hi when users send a non-location "shout." A more generic Send ? for help can let a curious user know what her options are...in only 15 characters.

NOTE

Dennis Crowley, a Dodgeball co-founder, reports:

We saw a big spike in usage when we added [reply with "@venue name" to check-in]. Receiving a check-in became a "me too" call to action rather than just a reminder.

A great reminder of how suggestible users are. They've never used your service before. Don't be afraid to tell them what to do.

SMS Interaction Design Considerations > Limited Character Set

3.6. Limited Character Set

While it's tough to forget that phones don't have a full keyboard, it's still easy to overlook some of the implications of that fact. It's not just that text input is hard, it means that some characters are simply impossible to enter.

?, @, ., and ! are usually easily accessible on phones

#, $, %, and : may require a special symbols menu, at which point you may lose some of your less dedicated users.

Even with the "safe" characters, some may be easier to access then others. Count the number of taps it takes on your phone to access a special character and ask yourself whether it's OK to force users to go through that every time they want to access the functionality it unlocks.

NOTE

According to Wikipedia:

In Mandarin Chinese, numbers that sound similar to words are used in place of those words. For example, the numbers 520 in Chinese (wu er ling) sound like the words for "I love you" (wo ai ni). The sequence 748 (qi si ba) sounds like the curse for "drop dead."

Was that useful 4 U?

SMS Interaction Design Considerations > Always On Everywhere

3.7. Always On Everywhere

It's not all grief and tears, though. SMS offers a number of advantages over other modes of communication, and if you design your service to take advantage of these, you can distinguish yourself smartly from your competition.

The most obvious of these is always on everywhere. Unlike a PC, people tend to carry their mobile phones with them everywhere they go, and they never turn them off. Because this is the primary distinguishing characteristic of an SMS service, you need to ask yourself "what do people need access to everywhere they go, at any moment?"

If what you're delivering doesn't fit this description, you ought to ask yourself whether you need to be delivering your information via SMS. Is it really critical that I know what my horoscope is while I'm on the bus? Well, perhaps if the influence of Mercury on my moon sign makes travel dangerous, but that ought to be rare.

On the other hand, if you are offering a service that is relevant to the untethered user, providing some information critical in the field, you ought to have a winner. TsunamiSMS (http://www.tsunamisms.com) is an excellent example of this. Do you really want to wait until you check your email to find out there is a tsunami warning for your area?

SMS Interaction Design Considerations > Store and Forward

3.8. Store and Forward

Like email, SMS is a store and forward system. In other words, the server will receive sent messages and hold on to them until they can be delivered (although, like email, the period during which delivery is attempted will be limited, typically to 24—48 hours).

This makes SMS messaging an ideal way to deliver information or to communicate with someone who has infrequent or unreliable connectivity.

For example, after Hurricane Katrina ravaged the Mississippi and Louisiana coast, I spent a week working with a relief organization, World Shelters, building temporary shelters for the displaced. The work was organized on the fly, with volunteers identifying those in need at relief centers or through word of mouth and then trying to direct the building teams to destinations without the benefit of reliable maps or local guides.

With the infrastructure decimated, cellular coverage was spotty, and thus calling teams to get status was impractical: the odds of both caller and callee having service at the same time was low. Email was out of the question.

Fortunately, we were able to use the Slam prototype developed in collaboration with Shelly Farnham and Scott Counts in Microsoft Research. Slam is a group-centric text messaging service that allows people to send and receive text messages to a group. Messages could be sent with the comfortable knowledge that, even if they weren't delivered immediately, they would be received as soon as the other people in the group moved through an area of cellular coverage.

While disaster situations like that are (thankfully) rare, spotty cellular coverage is not. Thick-walled buildings or theater rules are enough to prevent someone from receiving a phone call. Whether it's a message from a friend letting you know where people are meeting up or a reminder from your calendar, store and forward means the message can still get through as soon as possible.

SMS Interaction Design Considerations > Multitasking and Stealth

3.9. Multitasking and Stealth

How many times have you done it? You're in a meeting, a call comes in, but you can't answer it. So, you push it to voicemail and send a quick text message to the caller asking, "What's up?"

Why is it OK to send a text message in the middle of a meeting but not OK to answer the phone? Text messaging isn't merely less intrusive, it can be invisible. Schoolchildren all over the world have already figured this one out, tapping out messages to each other with hands held under desks, studiously nodding at an math lesson played out on a blackboard.

And if you've got nothing to hide, just enjoy the ease of multitasking: it is possible to read or send a text message while doing something else entirely (although not, of course, driving). Ever tried to eat a burrito with one hand and send email with the other? No go. But eat a burrito and tap out an SMS? No problemo.

Use SMS in situations where other, more conventional modes of communication are inconvenient or inappropriate and you'll provide value and relevance where others do not.

SMS Short Codes > More Is Less

Chapter 4. SMS Short Codes

A short code is a five or six digit (in the U.S.) phone number that is used to address SMS and MMS messages from a mobile phone. They are typically used because they are easier to remember than 10-digit or longer phone numbers.

When initially introduced, short codes were unique to each carrier (so the same short code might be owned by different application providers on T-Mobile and on Cingular, for example). However, in 2003, cross-carrier short codes were introduced in the U.S., and the ability to offer a single number accessible to all users, regardless of carrier, became practical.

In the United States, short codes are registered by a Common Short Code Administration, the Cellular Telecommunications & Internet Association. The application process for short codes is straightforward and can be completed quickly and easily at the CTIA web site, http://www.usshortcodes.com.

As of April 2007, pricing for short codes in the U.S. was as follows:

Randomly assigned 5- or 6-digit code: $500/month

Custom chosen 5- or 6-digit code: $1,000/month

Note, though, that short codes must be leased in three-month blocks of time and paid for upfront, so your initial payment for turning on a short code will be $1,500 to $3,000.

The fee, however, merely gives you the right to use the short code, but does not provide connectivity to the carrier. For that, you must go through an SMS aggregator service (see the "Chapter 7" section later in this tutorial), where you'll end up paying for its service and for the messages sent and received. Your aggregator may charge an additional short code fee even if you bring your own.

NOTE

The following words correspond to a sampling of unregistered, available short codes as of April 2007:

ADVICE | CHURCH | HONEY | PICNIC
BEAUTY | DINNER | LOVERS | SCUBA
BROKER | DESIGN | MEETME | SLEEP
BUYNOW | GARDEN | NURSE | SKIING
CAMERA | HELPME | PIANO | YOUTH

4.1. More Is Less

Also note that short codes are not a necessary piece of an SMS service. SMS aggregators will rent you "normal" phone numbers (e.g., 10 digit in the United States, up to 16 digits elsewhere) for your use. The primary drawback of these longer numbers is that they are difficult to remember and you may only have temporary use of them. If users only need to reply, or you can count on users saving your number in their address book, you can reduce your investment considerably by settling for a longer number.

SMS Short Codes > More Is Less

Chapter 4. SMS Short Codes

A short code is a five or six digit (in the U.S.) phone number that is used to address SMS and MMS messages from a mobile phone. They are typically used because they are easier to remember than 10-digit or longer phone numbers.

When initially introduced, short codes were unique to each carrier (so the same short code might be owned by different application providers on T-Mobile and on Cingular, for example). However, in 2003, cross-carrier short codes were introduced in the U.S., and the ability to offer a single number accessible to all users, regardless of carrier, became practical.

In the United States, short codes are registered by a Common Short Code Administration, the Cellular Telecommunications & Internet Association. The application process for short codes is straightforward and can be completed quickly and easily at the CTIA web site, http://www.usshortcodes.com.

As of April 2007, pricing for short codes in the U.S. was as follows:

Randomly assigned 5- or 6-digit code: $500/month

Custom chosen 5- or 6-digit code: $1,000/month

Note, though, that short codes must be leased in three-month blocks of time and paid for upfront, so your initial payment for turning on a short code will be $1,500 to $3,000.

The fee, however, merely gives you the right to use the short code, but does not provide connectivity to the carrier. For that, you must go through an SMS aggregator service (see the "Chapter 7" section later in this tutorial), where you'll end up paying for its service and for the messages sent and received. Your aggregator may charge an additional short code fee even if you bring your own.

NOTE

The following words correspond to a sampling of unregistered, available short codes as of April 2007:

ADVICE | CHURCH | HONEY | PICNIC
BEAUTY | DINNER | LOVERS | SCUBA
BROKER | DESIGN | MEETME | SLEEP
BUYNOW | GARDEN | NURSE | SKIING
CAMERA | HELPME | PIANO | YOUTH

4.1. More Is Less

Also note that short codes are not a necessary piece of an SMS service. SMS aggregators will rent you "normal" phone numbers (e.g., 10 digit in the United States, up to 16 digits elsewhere) for your use. The primary drawback of these longer numbers is that they are difficult to remember and you may only have temporary use of them. If users only need to reply, or you can count on users saving your number in their address book, you can reduce your investment considerably by settling for a longer number.

SMS Fundamentals > Pricing: Paying for SMS Messages

Chapter 5. SMS Fundamentals

When SMS was originally designed, it was not intended for person-to-person messaging. Instead, it was conceived as a means of delivering a service-based message to the user, such as a voicemail notification. As a result, the signaling channel of the GSM system was used, and the architecture not optimized for the kinds of messaging we do today.

A text message is limited to 140 bytes, or 160 7-bit characters, though languages that require 16-bit encoding will be limited to 70 characters.

When an SMS message is sent, the mobile phone uses the configuration data found in its SIM card to locate and contact the Short Message Service Center (SMS-C) operated by the carrier. The sender's carrier will forward the message on to the recipient's carrier using SS7 signaling protocols, and the recipient's carrier holds the message until it can be delivered on to the recipient's phone, as shown in Figure 5-1.

Figure 5-1. SMS network architecture simplified

The fact that the message must cross so many networks can lead to low reliability in the system: delivering SMS messages across carriers is more likely to fail than delivering messages within a carrier's network.

It is worth noting that "true" SMS exists only on GSM networks. GSM is the system used by carriers across most of the world, but in the United States, several carriers (including Sprint and Verizon) use an alternative system, CDMA.

This is unlikely to impact you directly, as CDMA-based carriers have created their own SMS-like systems and interface directly to GSM carrier's SMS system. You may notice, though, some idiosyncrasies when sending and receiving messages from these carriers. For example, Sprint phones will typically allow users to compose 220 character messages, rather than 160.

5.1. Pricing: Paying for SMS Messages

In the United States, an SMS message is charged to both the sender and the receiver. Pricing varies by carrier and plan, but typically a message will cost a consumer $0.10 to send and to receive (although some carriers offer "all you can eat" unlimited flat rate plans).

In Europe, Asia and elsewhere in the world, only the sender of the message pays, it is free to receive unlimited text messages. Messages can cost anywhere from €0.03 to €0.14.

SMS Fundamentals > Pricing: Paying for SMS Messages

Chapter 5. SMS Fundamentals

When SMS was originally designed, it was not intended for person-to-person messaging. Instead, it was conceived as a means of delivering a service-based message to the user, such as a voicemail notification. As a result, the signaling channel of the GSM system was used, and the architecture not optimized for the kinds of messaging we do today.

A text message is limited to 140 bytes, or 160 7-bit characters, though languages that require 16-bit encoding will be limited to 70 characters.

When an SMS message is sent, the mobile phone uses the configuration data found in its SIM card to locate and contact the Short Message Service Center (SMS-C) operated by the carrier. The sender's carrier will forward the message on to the recipient's carrier using SS7 signaling protocols, and the recipient's carrier holds the message until it can be delivered on to the recipient's phone, as shown in Figure 5-1.

Figure 5-1. SMS network architecture simplified

The fact that the message must cross so many networks can lead to low reliability in the system: delivering SMS messages across carriers is more likely to fail than delivering messages within a carrier's network.

It is worth noting that "true" SMS exists only on GSM networks. GSM is the system used by carriers across most of the world, but in the United States, several carriers (including Sprint and Verizon) use an alternative system, CDMA.

This is unlikely to impact you directly, as CDMA-based carriers have created their own SMS-like systems and interface directly to GSM carrier's SMS system. You may notice, though, some idiosyncrasies when sending and receiving messages from these carriers. For example, Sprint phones will typically allow users to compose 220 character messages, rather than 160.

5.1. Pricing: Paying for SMS Messages

In the United States, an SMS message is charged to both the sender and the receiver. Pricing varies by carrier and plan, but typically a message will cost a consumer $0.10 to send and to receive (although some carriers offer "all you can eat" unlimited flat rate plans).

In Europe, Asia and elsewhere in the world, only the sender of the message pays, it is free to receive unlimited text messages. Messages can cost anywhere from €0.03 to €0.14.

SMS Fundamentals > Premium SMS: Getting Paid for Your Service

5.2. Premium SMS: Getting Paid for Your Service

Premium SMS, sometimes called reverse billed SMS or MT billed SMS is an SMS that incurs an extra charge on the receiver beyond that normally levied for receiving an SMS. For example, after voting via SMS on the American Idol TV show, voters receive an SMS reply that charges their phone bill. Phone Sherpa (http://www.phonesherpa.com) uses premium SMS instead of MMS to deliver multimedia content by sending a premium SMS that contains a link the user can open with the phone's browser to download ringtones and wallpaper.

From a technical perspective, premium SMS is simply a normal MT (mobile terminated) SMS with a special tariff-level header indicating to the carrier how much to charge the recipient. However, it's not simply a matter of tagging a header onto your messages and waiting for the dollars to start rolling in.

For one, for your tariff-level header to be respected by the receiving carrier, the short code that you send it on must be approved by that carrier for that tariff level. The process is not, unfortunately, simple.

Due to some poorly behaved players early in the history of premium SMS, the carriers now keep a tight rein on who is allowed to send premium SMS messages across their networks. Each application must be individually reviewed and tested by each carrier before that carrier will approve the service, and each carrier's rules and requirements are slightly different. Some carriers will require specific wording in the sign-up messages (e.g., opt-out) while others require the presence of a well-advertised toll-free number on the web site.

Fortunately, the SMS aggregator services not only provide unified access to the carrier's SMS gateways, they will also aggregate the process of applying to the carriers for premium SMS approval. Typically, an application provider will fill out a single, long application provided by the SMS aggregator that contains the rules and requirements of all participating carriers. The aggregator will then farm out the application to the individual carriers and help shepherd the approval process.

However, the approval and testing process can be quite lengthy, upwards of four months, and you should expect to have your application fully functional and testable throughout. Carriers will want to review all the messages your service sends for rules compliance (and they will continue to audit your service on an ongoing basis after approval). Of course, you must pay for your short code and services during this waiting period, whether or not any customers can pay.

Once approval is obtained and your service is underway, the slice of the pie that you, as the application provider, will see can be surprisingly small. Between the carriers and the aggregators, a typical application provider may only end up collecting 40 to 50% of the amount charged to the end users (see Figure 5-2).

Figure 5-2. Premium SMS architecture simplified

And don't count on an immediate cash flow. The period between the time the premium SMS is sent and the consumer charged and when a check arrives from a carrier can be over three months, a potentially damaging wait for a small business.

Finally, high spoilage has been reported by those in the industry. If a consumer complains, or if a message is sent but not logged correctly by the carrier, no money is collected from the end-user and no money is delivered to the application provider.

Don't be too disheartened, though. It's not easy, but it's not impossible. A leading research firm has estimated that almost €1 billion was spent on premium SMS in Europe in 2006.

Chapter 6. Six Ways to Implement SMS

There are many ways you can easily implement an SMS service today, each with its own pros, cons, and idiosyncrasies. Some implementations are more suited to certain types of services than others.

In this final section we'll consider six ways you might implement an SMS service: aggregator, smartphone, email, Twitter, 411Synch, and Windows Live. There are, of course, many other options. Table 6-1 lists the six implementations we discuss, the type of service for which each is suited, and their relative cost, complexity, and scalability.

Table 6-1. Six implementations and types of service

| Cost | Service types | Complexity | Scalability
Aggregator | High | All | Moderate | High
Smartphone | Moderate | All | Moderate | Low
Windows Live | Free | Notification | Simple | High
Email to SMS | Free | Notification messaging | Moderate | Low
411Sync | Free | Lookup | Simple | High
Twitter | Free | All w/quirks | Moderate | Moderate

Let's look more closely at the four factors that determine the best use of each implementation type: cost, service type, complexity, and scalability.

Cost

There are several free options (or at least no incremental cost beyond hosting a web service). The two options that involve money are smartphone, which can cost as little as $200 or $300 at start-up and a $40 or $50 monthly fee for the phone service, and an SMS aggregator, where you will end up paying anywhere from 1.5 cents to 8 cents per message, inbound and outbound.

If you are experimenting and prototyping, you may wish to design your service so that you can take advantage of a free service in the early stages, then swap it out for a more robust but costly service at the appropriate time.

Service type

Some implementations come with inherent constraints that may make them unsuitable for certain services. For example, it is difficult to initiate an email service from a mobile phone (although replies are generally simple). This makes it a poor choice for a look-up service. The Windows Live service only supports MT messaging, making it inappropriate for anything but notification services.

Complexity

Don't discount the value of simplicity. None of the implementations we describe here are highly complex, but some are certainly simpler to implement than others. However, what's easy to one person may be complex to another, depending on your existing knowledge and skills. For example, one of the authors of this guide had a background in C# and Windows applications, while the other was more familiar with Python and Linux/Apache-based solutions, meaning what was simple to one was complex to the other.

Scalability

There are two low rankings in the scalability column in Table 6-1: smartphone and email. The bottleneck for a smartphone implementation is that it can only handle approximately 30 messages per minute. For email, the bottleneck will be set by the carriers, which will attempt to detect services running against their gateways and may block messaging temporarily in an attempt to limit spam.

Using an SMS Aggregator > The Basics

Chapter 7. Using an SMS Aggregator

Using an SMS aggregator is the most traditional way to implement an SMS service SMS aggregators, or SMS brokers, have negotiated deals with all the major carriers to deliver SMS messages through a thick pipe directly to the SMS-Cs. In turn, the aggregators expose web interfaces to third parties that wish to send and receive SMS messages programmatically.

7.1. The Basics

Figure 7-1. SMS aggregator network architecture simplified

Figure 7-1 illustrates the basic set up. Your web service communicates with an SMS aggregator via SOAP, FTP, SMPP, or any number of other protocols. The aggregator, in turn, communicates with each individual carrier's SMS-C via SMPP to send and receive messages on your behalf.

There are many aggregators in business today and all offer slightly different sets of services, coverage, and pricing. We can't endorse or recommend any one service in particular, but we can describe two basic tiers of SMS aggregator on the market.

The first is intended for smaller companies that are either experimenting in SMS or operating at low volume (e.g., less than 100,000 messages). These services offer you the ability to pay by the message, usually by purchasing batches of credits. Clickatell is a well known example of a service in this category.

In the second category are companies such as mBlox, OpenMarket (which acquired SimpleWire), and Sybase 365. These companies generally charge a monthly service fee of a few thousand dollars regardless whether you send any messages at all. However, the per-message fee they charge can be considerably lower (as low as a penny and a half per message, if you buy in batches that approach a million). If you reach high volumes of messages, this can end up making financial sense.

Using an SMS Aggregator > The Basics

Chapter 7. Using an SMS Aggregator

Using an SMS aggregator is the most traditional way to implement an SMS service SMS aggregators, or SMS brokers, have negotiated deals with all the major carriers to deliver SMS messages through a thick pipe directly to the SMS-Cs. In turn, the aggregators expose web interfaces to third parties that wish to send and receive SMS messages programmatically.

7.1. The Basics

Figure 7-1. SMS aggregator network architecture simplified

Figure 7-1 illustrates the basic set up. Your web service communicates with an SMS aggregator via SOAP, FTP, SMPP, or any number of other protocols. The aggregator, in turn, communicates with each individual carrier's SMS-C via SMPP to send and receive messages on your behalf.

There are many aggregators in business today and all offer slightly different sets of services, coverage, and pricing. We can't endorse or recommend any one service in particular, but we can describe two basic tiers of SMS aggregator on the market.

The first is intended for smaller companies that are either experimenting in SMS or operating at low volume (e.g., less than 100,000 messages). These services offer you the ability to pay by the message, usually by purchasing batches of credits. Clickatell is a well known example of a service in this category.

In the second category are companies such as mBlox, OpenMarket (which acquired SimpleWire), and Sybase 365. These companies generally charge a monthly service fee of a few thousand dollars regardless whether you send any messages at all. However, the per-message fee they charge can be considerably lower (as low as a penny and a half per message, if you buy in batches that approach a million). If you reach high volumes of messages, this can end up making financial sense.

Using an SMS Aggregator > Advantages and Disadvantages

7.2. Advantages and Disadvantages

The biggest advantages to using an SMS aggregator are scale and reliability. Sending millions of messages directly to the carrier's infrastructure is exactly what these companies specialize in, and they've been doing it for years. If you're going to run a large scale operation, working with an aggregator is really your only long-term option (short of working directly and individually with the carriers, an option that won't be explored in this guide).

Another advantage is the rich feature set that the carriers make available. You have access to their APIs, including batch sending, via a wide variety of protocols, and you can even send premium SMS messages.

The biggest downside to using an SMS aggregator is cost. As described in the "Chapter 5" section, you end up paying for all the messages sent and received by your service. Success can be costly. It's also worth reading the Premium SMS section of this guide carefully, as much of what we discuss there applies to aggregators in general (e.g., the 3 to 4 month approval process).

Using an SMS Aggregator > Example Service: Conference Messaging

7.3. Example Service: Conference Messaging

Let's imagine we're organizing a large conference and wish to allow participants and organizers to communicate with one another via SMS. In the hypothetical service, participants will register with their:

Phone number

Nickname

Aggregators such as Clickatell allow you to lease a phone number with exclusive access to send and receive SMSs from that number. They also allow batch messages to be created that allow large amounts of messages to be quickly sent. So, for the purposes of our example, our service supports the following functionality:

Conference organizers can broadcast messages via a web interface.

Other participants broadcast messages ("shout") to conference participants by prefixing their messages with '!'.

Any message not prefixed by '!' will be considered feedback for the organizers and simply stored for later review.

For example, the conference organizer wants to let participants know that the party that was scheduled for the opening evening has been moved from 6:00 to 8:00 p.m. She navigates to an internal web page and types:

Tonight's party has been moved from 6:00pm to 8:00pm

Every conference participant receives a message on their phone:

TestCon says: 'Tonight's party has been moved from 6:00pm to 8:00pm'

After learning that the time of the party has changed, one of the participants decides to organize a dinner event. He replies to the text message sent by the admin:

!I'm going to Luigi's for dinner @6:30, anyone want to join? Msg me

Our service looks for messages pre-pended with '!' and broadcasts it to every conference participant's mobile phone. Each participant will see:

Bob(2063333333) says: I'm going to Luigi's for dinner @ 6:30, anyone want to join? Msg me

7.3.1. How to: Conference messaging using Clickatell

To illustrate how we would build our conference messaging service using an SMS aggregator, we will use the APIs provided by Clickatell. Clickatell is one of the most accessible SMS aggregators: You pay only for the messages sent and received (i.e., there is no set-up fee) and they give you 10 free credits on sign-up for testing. You can sign-up for a free developer account, its APIs are well documented, it connects to more than 600 different carrier networks, and it offer connectivity via a wide variety of protocols (e.g., HTTP/S, SMPP, SMTP, FTP, XML, and COM).

7.3.1.1. What you'll need

To implement this service, you'll need a few things:

A Clickatell developer account (http://www.clickatell.com/products/sms_gateway.php)

Python 2.4 or higher (http://www.python.org/download)

A web server that can execute Python CGI scripts (most web hosting providers support this). The following files described in this section will need to be placed in the web server's root path:

config.py: Static configuration data that is unique for each project.

broadcast.py: Functions that support sending SMSs to all registered participants.

admin.html: A web form that allows the conference admins to send broadcasts to all the participants.

admin_send.py : The script that processes the form that admin.html submits. It actually sends the SMSs.

responder.py : A script that is called by Clickatell when participants send SMSs to our leased number.

NOTE

Testing the incoming SMS portion of this example requires leasing a phone line from Clickatell, which can be expensive if you're just getting your feet wet. However, you can test just the web-based broadcast (outbound only) portion of this example with a free account.

7.3.1.2. Clickatell basics

We'll be using the Clickatell HTTP/S API for implementing our service. HTTP/S is one the simplest ways to access Clickatell, but it's also very feature rich. The API Guide is available at (http://www.clickatell.com/downloads/http/Clickatell_HTTP.pdf).

7.3.1.3. Static configuration

Our project relies on certain configuration elements that will be unique to every person running the example. You'll need to edit this file to match your particular situation.

config.py:

#change these to match your registration info
api_id = '12345678'
user = 'test'
passwd = 'testpasswd'
reply_num = '13334445555'

conference = 'TestCon'
nick_number_dict = {
       '14445556666' : 'bob',
       '15556667777' : 'ted',
       '16667778888' : 'teresa'
}

Our Clickatell set up require four pieces of configuration:

api_id: A unique ID given to you when you sign up for Clickatell's service.

user: The user name you created when you signed up for Clickatell.

passwd: The password you set up for Clickatell.

reply_num: The phone number replies will be sent to. You need to lease this number from Clickatell (explained in more detail below). If you don't want to lease a number, just leave this alone.

This example uses two other pieces of configuration data:

conference: The name of the conference we're setting up.

nick_number_dict: keeps track of all our conference participants. More advanced applications may want to use a SQL database or other more flexible data format for this.

7.3.1.4. Simple Clickatell script

Let's take a look at a short Python script that will use the Clickatell HTTP API to send one message. This script is just an example, and isn't one of the files you'll be installing on your web server to implement the conference messaging service. It still relies on our configuration data though.

simple_send.py:

Code View: Scroll / Show All

import urllib
import config
def send_message(to, text):
       paramdict = {
              'api_id': config.api_id,
              'user' : config.user,
              'password' : config.passwd,
              'to' : to,
              'from' : config.reply_num,
              'text' : text
       }
       params = urllib.urlencode(paramdict)
       result = urllib.urlopen("https://api.clickatell.com/http/sendmsg", params)
       print result.readlines()

if __name__ == "__main__":
       #change the phone number to your own
       send_message('13334445555', 'test message')

The first thing this script does is import urllib and config. Python comes with urllib, it allows communication over most common Internet protocols including HTTPS, which is what we'll be using it for. We created config ourselves in the last section.

We then define a function called send_message that takes two parameters:

text: The text of the message we want to send.

to: The phone number of the person we want to send the message to.

The send message function creates a dictionary that contains all the parameters that we'll need to send to Clickatell's sendmsg function.

The script then calls urlencode (from the urllib module), which converts the dictionary into a string that is URL-encoded.

All we need to do now is send a POST request to the sendmsg command at Clickatell. We use the urlopen command to do so, passing the URL as the first parameter and the POST variables as the second.

Lastly, we print out the response so you can see what it looks like.

The simple script above can be run on any computer with python installed. The send_message function takes a phone number and a message. Change the number to your own, and the message to whatever you'd like, then invoke the script (from the command line):

python simple_send.py

or on Windows:

python.exe simple_send.py

You could use this method to implement the conference broadcast functionality of our example service, but Clickatell provides a better mechanism.

7.3.1.5. Batching

The example above sends an individual message, but most aggregators provide a batching mechanism that will allow you to a message to a large number of people using only one API invocation. This will speed up the sending of the messages and reduce the likelihood of messages getting dropped.

Batching using Clickatell's API consists of two steps (three if you use the auth command to obtain a session ID—we just pass our credentials each time):

Set up the text of the message, obtain a batch_id

Send the set up message to any number of phones by referencing the batch_id and passing a comma delimited list of phone numbers to one of the send commands (we use quicksend)

Broadcast.py implements two methods: batch_setup and broadcast.

In order to send a message to all the conference participants, broadcast is called. It calls batch_setup to set a new batch up, and then sends a message to everyone in the configuration file.

This file can be executed at the command prompt as with simple_send.py. It will send two messages to every number in your configuration file (so make sure it's accurate—it should probably only have your number in it at this point).

broadcast.py:

Code View: Scroll / Show All

import urllib
import re
import config

def batch_setup(message):
       batch_setup_params = {
              'api_id': config.api_id,
              'user' : config.user,
              'password': config.passwd,
              'from' : config.reply_num, # the number replies go to
              'mo' : '1', # allow responses
              'template' : message
       }
       batch_id_re = re.compile("ID:[ ]*([0-9a-f]+)")
       params = urllib.urlencode(batch_setup_params)
       res = urllib.urlopen(
              "https://api.clickatell.com/http_batch/startbatch", params
       )
       bid = 0
       # extract the batch id from the response
       for line in res.readlines():
              print line
              m = batch_id_re.search(line)
              if m:
                     bid = m.group(1)
                     print "BID: %s" % bid
                     return bid
       return None

def broadcast(msg, shout_from = None):
       if shout_from:
              nick = config.nick_number_dict[shout_from]
              message = '%s(%s) says: %s' %(nick, shout_from, msg)
       else:
              message = '%s Announcement: %s' %(config.conference, msg)
       bid = batch_setup(message)
       if not bid:
              raise Exception("Error Getting Batch ID")
       # get a comma-delimited list of numbers from our dictionary
       to_nums = ','.join(config.nick_number_dict.keys())
       batch_send_params = {
              'api_id': config.api_id,
              'user' : config.user,
              'password' : config.passwd,
              'batch_id' : bid,
              'to' : to_nums
       }
       params = urllib.urlencode(batch_send_params)
       res=urllib.urlopen(
              "https://api.clickatell.com/http_batch/quicksend", params
       )
       print res.readlines()

# simple test if we run the file directly
# it will send a message to all the numbers
# in conf.py
if __name__ == "__main__":
       broadcast('test message', '12063497060')
       broadcast('test message')

Our function broadcast takes two parameters:

msg: The text of the message we wish to broadcast.

shout_from: The phone number of the person shouting. The default (None) will send an official announcement.

The first thing broadcast does is create the text of the message based on the contents of the shout_from parameter. It then calls the batch_setup function, passing it the message we wish to send. The command is set up by submitting a POST with the following parameters to (https://api.clickatell.com/http_batch/startbatch).

api_id, user, password: Clickatell configuration parameters described previously.

from: The phone number replies will be sent to. Important note: all phone numbers should be specified in international format (i.e., beginning with country code).

mo: Indicates whether the receiver can send replies to the message (remember, "mo" means "mobile originated"). Set to one for true, zero for false.

template: The message you wish to send. This will have to be URL encoded to be passed as a parameter.

The code calls this URL from batch_setup in the way that it called sendmsg in our first example. Clickatell replies with a batch_id that will be used to actually send our message. Our code uses a regular expression to parse out the batch_id and returns it to the user.

Once broadcast has retrieved the batch_id, it takes all of the phone numbers listed in our configuration file and joins them together into one long string, with each number separated by commas.

We then use urlopen to POST with the following parameters to (https://api.clickatell.com/http_batch/quicksend).

api_id, user, password: Clickatell configuration parameters described previously.

batch_id: The batch_id you obtained as a response to your startbatch command above.

to: A comma-delimited list of numbers to send the message you configured as template in the startbatch command.

And with that, your broadcast message is on its way.

The sample code prints out the responses to each message so you can see what they look like. Feel free to remove or disable the print statements as you wish.

Our sample doesn't track the messages once they are sent, but check whether your messages were successful by examining the response to your quicksend POST.

The response will contain one line for each number you send the message to. Each line will contain an apimsgid and the phone number of the recipient:

       ['ID: cc959204830b71ed27a014c3622356ee0 To: 12223334444\n']

You can then use the querymsg command to get the status of the particular recipient/message combination that the api_id refers to:

Code View: Scroll / Show All

http://api.clickatell.com/http/querymsg?user=xxxx&password=xxxx&api_id=xxxx&
apimsgid=xxxxx

NOTE

Clickatell's quicksend command supports a maximum of 100 numbers on the To: line if submitting the request via HTTP GET, or 300 if submitting the request via HTTP POST. Break your request into multiple calls if you are sending to more numbers than these.

7.3.1.6. Web frontend

Our web frontend consists of three pieces. The first is a static HTML page that presents a simple form to the conference administrator. The second is a CGI script that processes the form submission. The third is a CGI script that will be used as a callback by Clickatell when they receive inbound SMSs for your leased phone number.

The interface for the conference administrator is presented by one simple HTML form.

admin.html:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
       <head>
              <title>
                     Conference Announcement System
              </title>
       </head>
       <body>
              <h3> Send an Announcement </h3>
              <form action="admin_send.py" method="post">
                     <p> Message: <input type="text" name="message"/> </p>
                     <p> <input type="submit" value="Announce"/> </p>
              </form>
       </body>
</html>

It has one field message that should be filled in with the text of the message that should be sent. The form is submitted via POST to the CGI script admin_send.py.

The second part of our web frontend is a python CGI script that processes the form described in admin.html. It imports the broadcast functionality we wrote earlier and uses it to send a message to everyone.

admin_send.py:

import sys
import cgi
import cgitb; cgitb.enable()

from broadcast import *

def result(text):
 print "Content-type: text/html\n\n"
 print """
  <html>
  <head><title>Announcement Made</title></head>
  <body>%s</body>
  </html> """ % text
  sys.exit(0)

form = cgi.FieldStorage()
message = form.getfirst("message")
broadcast(message)
result("%s sent!" % message)

First the script imports several needed libraries and the broadcast functionality we wrote earlier. The cgitb.enable() command enables tracebacks on the web form, which makes it much easier to debug any potential problems.

We then define a function result() that will return a status page to the user.

The next line after the function definition uses the cgi module to fetch the parameters that were passed to our script.

We grab the message parameter (you'll recall that's the name of the field from admin.html) and then we pass it to broadcast.

Finally, we call our result function which returns a page to the user that confirms that he's sent the message.

NOTE

This short cut assumes you have access to a web hosting service that is running Apache or other server that will let you run python CGI scripts.

The easiest way to get everything working is to put all the files in the same directory. Everything with a .py extension will need to have correct permissions. This is usually 711 (rwx..x..x) on Unix systems, though the correct permissions can vary from host to host. For more advanced applications, you'll want to separate your static pages from your scripts or follow other practices designed to prevent source code compromise.

There are many other Python web frameworks that can be used, but they are not necessarily as straightforward or widely supported as Apache and pure CGI. Read about them at http://wiki.python.org/moin/WebFrameworks.

7.3.1.7. Receiving messages

To receive messages using Clickatell, you will need to set up two-way messaging, which involves leasing a phone number and setting up a callback URL. When an SMS is received to our leased number, Clickatell will invoke our callback URL with parameters containing the message details.

The Clickatell guide for setting up two-way messaging is at https://www.clickatell.com/downloads/Clickatell_Two_Way_Technical_Guide.pdf.

NOTE

As a quick reference, here is Clickatell's pricing for two-way messaging as of May 2007:

Number type | Length | Once-off set up fee (in euros) | Monthly rental (in euros)
Standard number | Up to 12 digits | €100 | €25; €75
Tagged number | 16 digits | €50 | €15
Tagged number range | 1 digit (10 tagged numbers) | €100 | €30
| 2 digits (100 tagged numbers) | €100 | €60
| 3 digits (1,000 tagged numbers) | €100 | €80
| 4 digits (10,000 tagged numbers) | €100 | €90

The URL for this script needs to be accessible to Clickatell (so you'll need a static IP address or domain). Place it in the same directory as the rest of the scripts we've created. Our callback URL will receive a post from Clickatell with two parameters: text (the body of the message) and from (the number the message was received from).

responder.py:

import sys
import cgi
import cgitb; cgitb.enable()
import re
from broadcast import *

form = cgi.FieldStorage()

text = form.getfirst("text") #message is in the 'text' variable
from_pn = form.getfirst("from")
response_file = file("responses.txt", "a")

#Check if message is of the form "!message body"
m = re.match("!(.*)", text)
if m:
       #m.group(1) is the text of the shout
       broadcast(m.group(1), from_pn)
else: #if we didn't get a 'shout', write the response into a file
       response_file.write("%s : %s\n" % (from_pn, text))
       response_file.close()

First, we import the standard raft of libraries we'll be using, including our own.

Next, we grab the parameters via the cgi.FieldStorage object. We pick out the text and from parameters

We open a file responses.txt in append mode. Append mode ensures that we will add new lines to the file every time we write to it, instead of overwriting it. This file path should be changed to be outside the web path to prevent unauthorized users from viewing it. This file will be used to store feedback messages (i.e., those not prepended with a '!').

The re module is used to create a regular expression that will match text that has a '!' as the first character and will group everything else for later retrieval. We use the regex to try to match the message we were sent.

For instance, this will match the regex:

!What's Up?

This won't:

This talk is boring

If there was a match, we call broadcast and pass it the text (not including the '!') and the number we received the request from.

If there isn't a match, we write the text out to the file we opened earlier.

NOTE

If you change the script you may need to debug it. An easy way to debug the script is to call it from a web browser and supply your own text and from parameters.

Another way to test and debug the script from the command line, without a web server, is to invoke python with a URL-encoded string of form parameters as the second parameter:

  python responder.py text='!what%20is%20up'\&from=13334445555

The escape character \ before the & is necessary on many Unix shells to prevent the shell from interpreting the ampersand. The same goes for the ticks around the text parameter—'!' is a special character in most shells. Windows and some Unix shells will be different.

7.3.1.8. Running the service

Install the files onto your hosting account as described earlier, open the admin page, and send a message.

Make sure you register your callback URL and try sending a shout. Or, try using the debugging methods above to test the shouting functionality if you don't want to lease a phone number.

Good work, you've just built a send and receive broadcast message service using an SMS aggregator! Now let's look at some other methods for implementing an SMS service.

Using a Smartphone as an SMS Modem > The Basics

Chapter 8. Using a Smartphone as an SMS Modem

The term "smartphone" refers to any mobile phone that is sufficiently feature rich that it starts to resemble a little PC. In this section, we'll be taking advantage of smartphones that run the Windows Mobile 5 or Windows Mobile 6 operating system, because they expose APIs that you can use to send text messages programmatically.

Using a smartphone as your SMS server can be a cheap and easy way to get true SMS connectivity: all you need is a Windows Mobile smartphone, a PC to connect it to and (ideally) an unlimited text messaging plan from your carrier.

8.1. The Basics

Think of your phone as an SMS modem attached to your computer. All the logic for sending and receiving the messages will execute on your Windows PC, you'll just use the phone to actually send out messages and receive messages on your application's behalf.

Using a Smartphone as an SMS Modem > The Basics

Chapter 8. Using a Smartphone as an SMS Modem

The term "smartphone" refers to any mobile phone that is sufficiently feature rich that it starts to resemble a little PC. In this section, we'll be taking advantage of smartphones that run the Windows Mobile 5 or Windows Mobile 6 operating system, because they expose APIs that you can use to send text messages programmatically.

Using a smartphone as your SMS server can be a cheap and easy way to get true SMS connectivity: all you need is a Windows Mobile smartphone, a PC to connect it to and (ideally) an unlimited text messaging plan from your carrier.

8.1. The Basics

Think of your phone as an SMS modem attached to your computer. All the logic for sending and receiving the messages will execute on your Windows PC, you'll just use the phone to actually send out messages and receive messages on your application's behalf.

Using a Smartphone as an SMS Modem > Advantages and Disadvantages

8.2. Advantages and Disadvantages

Using a smartphone gives you the addressability of an SMS Aggregator (i.e., you get a real phone number that you can give out for your users to send messages to) without the high set-up and per message costs. Pick up a used smartphone on E-Bay for a couple hundred dollars, and shell out $20/month on an unlimited SMS plan, and you're in business.

In business for a while, anyway. Scale will likely be your biggest limiting factor when using a smartphone-based SMS solution. In our experience, a single phone can process only 30 messages per minute. If your service succeeds, you may wish to support orders of magnitude more messages than this at peak use.

Using a Smartphone as an SMS Modem > Example Service: Echo and Hello World Using a Smartphone

8.3. Example Service: Echo and Hello World Using a Smartphone

To implement an SMS service using a smartphone, you don't need a web server at all. Our example smartphone service will do two things:

When an incoming message is received (e.g., "Testing"), it will echo back the message it received (e.g., "You said Testing").

It will display a form with text boxes for a phone number and a message body. When you click a button on the form, it will send your message to that number.

8.3.1. How to: Using a Smartphone to send and receive SMS

Microsoft Research India has released a free toolkit that abstracts the SMS functionality on the phone and remotes them through ActiveSync to the PC. This allows you to write and execute your application on your Windows PC. The smartphone functions as an SMS modem.

It includes the Windows PC side and Windows Mobile smartphone software, as well as a C# template for creating a service. Quite helpfully, it also includes a phone emulator, so you can develop and test your application without actually using a smartphone.

8.3.1.1. Installing the Microsoft Research SMS Toolkit

You can download the toolkit from http://www.codeplex.com/smstoolkit. The instructions on the web site are relatively self-explanatory. However, it is worth highlighting a few gotchas to avoid frustration:

On Windows Vista, the installer for toolkit must be run as administrator even if you are logged in with an account that has administrator privileges. To do so:

Save the installer to your local hard drive

Click Start->More Programs->Accessories

Right-click "Command Prompt"

Choose "Run as administrator"

Use the command line to execute the installer

The PC application requires SQL Server 2005 Compact Edition and ActiveSync 4.0. These are both free downloads available from Microsoft's web site, but the toolkit installer will not install them for you, neither will it check to make sure they are installed. You will simply hit errors when you try to run your application.

Most Windows Mobile smartphones come "application locked," meaning they are limited in terms of what types of applications can be installed and what access the install process has to the system. It is a straightforward process to unlock your phone, and instructions on doing so are provided on the SMS Toolkit web site, but at the bottom of the page. Be sure to complete this step before you try to install the Toolkit software.

Windows Mobile 5 or Windows Mobile 6 is required. Smartphones with earlier versions of the Windows Mobile operating system do not have the necessary APIs to support the Toolkit.

8.3.1.2. Getting started

Once you have installed the toolkit according to the instructions on the Codeplex web site and above, simply create a new project in Visual Studio using the SMS Service template. It will create several files for you, but Form1.cs is where the core of the example lies.

When the component is initialized, the following code executes:

Code View: Scroll / Show All

SmsService = new SmsServiceClass("SMS_Service");
SmsService.OnDeliveryReportReceived +=
    new OnDeliveryReportDelegate(SmsService_OnDeliveryReportReceived);
SmsService.OnSmsReceived += new OnSmsReceivedDelegate(SmsService_OnSmsReceived);

The first line creates the SMS service. You'll select this service when running the emulator, and invoke it when sending an SMS.

The second and third lines create event handlers for the delivery report and for incoming SMS messages. SMSService_OnSMSReceived is what will be invoked when you receive an SMS.

Taking a quick look at SMSService_OnSMSReceived, we see exactly how easy it is to write an SMS server using Microsoft Research India's SMS Toolkit:

SmsResponseClass SmsService_OnSmsReceived(SmsClass SmsMessage)
{
SmsResponseClass Response = new SmsResponseClass();
SmsClass SmsItem = new SmsClass();
SmsItem.Body = "Your Response";
SmsItem.Number = SmsMessage.Number;
Response.AddResponseSms(SmsItem);
return Response;
}

If you make no changes to this code at all and simply connect the phone and run the application as it comes out of the template, the above function will be invoked and reply to whatever message was received with "Your Response."

To implement our echo response, simply modify the line:

SmsItem.Body = "Your Response";

to:

SmsItem.Body = "You said" + SmsMessage.Body;

This will place the body of the incoming message into the body of the outgoing message. When you return the Response class object that contains the SmsItem you just created, the application will send out your message to the same number it received a message from.

NOTE

If you have used your smartphone as a phone, and have entries in the address book, SmsClass.Number may have more information than you expect. For example, when receiving a text message from my friend Peter Brown, SmsMessage.Number resolved to Brown, Peter" <+12065551234>. If you simply take SmsMessage.Number, stuff it into SmsItem.Number and add it to Response, it will work...sometimes.

I found that this format worked fine when replying to messages sent from the T-Mobile and Cingular networks, but Sprint simply failed to deliver my replies.

To work around the problem, clean it up:

using System.Text.RegularExpressions;
...
Regex regex = new Regex("[^0-9]");
SmsMessage.Number = regex.Replace(SmsMessage.Number, "");

This will strip out everything but actual numbers and ensure that it is respected by all carriers.

8.3.1.3. Sending a message

To send a message programmatically, we will use the SmsService.SendSms(SmsClass Sms)function.

In designer mode of Form1.cs, add two text boxes and a button. Name the text boxes txtPhoneNumber and txtMessageBody and name the button btnSendSMS. You'll probably want to make txtMessageBody multiline and add a couple of labels to keep track of what's going on as well. When you're done, it should look something Figure 8-1.

Figure 8-1. Sending a message

Now, double-click the btnSendSMS button to create a new function for click event and fill it in as follows:

private void btnSendSMS_Click(object sender, EventArgs e)
    {
      SmsClass SmsItem = new SmsClass();
      SmsItem.Number = txtPhoneNumber.Text;
      SmsItem.Body = txtMesssageBody.Text;
      SmsService.SendSms(SmsItem);
    }

You're ready to go. Run your application, fill in a number and message in the body, click your button and your message is on its way.

The first line of this function creates a new SmsClass instance. The next two lines fill in the number and body of the instance and the fourth line sends the message we just created. Easy, huh?

NOTE

You cannot invoke the SmsService.SendSMS method from within the SmsService.OnSmsReceived function. Because of threading issues, your application will hang if you attempt this.

Instead, if you wish to send more than one message from within the OnSmsReceived function, just create a new SmsClass instance and add it to the Response class instance of the SMSResponseClass array:

SmsClass secondResponse = new SmsClass();
secondResponse.Body = "This is the second response";
secondResponse.Number = "2065555555"
Response.AddResonse(secondResponse);

Using a Smartphone as an SMS Modem > Server Architecture

8.4. Server Architecture

Setting up the SMS server with a smartphone is a bit of an unusual task in that you can't simply install some software on a web host's servers and be done with. You need physical access to the PC and the phone, so this will typically be run out of your home or office directly.

In addition, ActiveSync supports only one phone connection at a time. If you want to operate multiple numbers, you will have to run the software on multiple PCs.

When we built Slam, our group-centric SMS service, it was important to have multiple addressable numbers, which would allow users to have a separate number for each group they were a member of. We also provided a rich smartphone and PC web UI interface. This resulted in the architecture in Figure 8-2.

Figure 8-2. Microsoft Slam server set up

By using four different phones with four different phone numbers, we could allow each user to be a member of up to four groups. Which phone number corresponded to which group would vary from person to person (even if they were a member of the same group), we simply kept a mapping of phone servers and groups for each user.

Using a Smartphone as an SMS Modem > Tips and Tricks

8.5. Tips and Tricks

Having built several SMS services based on this architecture and operated one of them for several years, we learned a few things along the way.

Phones Leak:Microsoft is constantly improving the Windows Mobile platform, but as of Windows Mobile 5, there were still some significant memory leaks present in the OS. As a result, we typically ran into out of memory errors after several weeks of operation. Plan on rebooting the phones every week or so to ensure your service doesn't go down unexpectedly.

Scale: Using a smartphone SMS server implementation is intrinsically limited in scale. In my experience, the smartphone was able to send and receive approximately 30 messages per minute. Because SMS is a store and forward system, if the incoming messages get backed up, you needn't worry, they will flow down to your phone at the rate your phone can handle it. This should help you ride out any activity spikes as you scale your service up.

International: Sending SMS messages across national boundaries can be expensive and may not be covered by your standard SMS plan (even if it is unlimited). Save yourself an unexpected phone bill and decide whether you want to filter out international messages.

Using Email to SMS Gateways > The Basics

Chapter 9. Using Email to SMS Gateways

Because most U.S. carriers and many international carriers will convert emails to SMS messages and forward them on to their customers, email to SMS gateways present a highly accessible means of rapidly prototyping SMS services. However, in most cases they are not scalable solutions for a full-fledged service, because the carriers may arbitrarily block messages being sent from a particular service when volume reaches a certain point (perhaps as few as 50 messages in a day).

9.1. The Basics

Every major U.S. carrier and most international carriers provide a gateway to the Internet so that their customers can receive SMS messages sent via email. The sender addresses the message to an email address of the form ###########@carrier-sms-domain.com, it is converted by the carrier into an SMS and delivered. There are several lists online with the carrier to domain name mappings, including one at http://www.notepage.net/smtp.htm.

NOTE

One thing you'll learn quickly is that there is very little that is "standard" whe using email to SMS gateways. Each carrier has a few idiosyncrasies in how they send and receive messages. For example, while most carriers will fill in the From address with the phone number plus their domain (e.g., 2065551212@tmomail.net), a few don't. Verizon PCS, for example, allows its users to substitute a handle instead of the number, so a message sent to a Verizon PCS user with the address 2065551212@vtext.com" might be replied to with a From address of johnsmith@vtext.com. This makes it difficult to keep track of who you're receiving messages from.

You'll want to accommodate this in your service. For example, when you user registers, have them reply to the registration mail with a unique response code that will allow you to keep track of who is sending them message, then match that up with the address it's coming from.

Figure 9-1. Email to SMS architecture

To allow the receiver to reply via SMS and have the email user receive the reply, the carriers will assign a temporary response code to the sender's email address, and then forward any SMS messages sent by that user to that response code back to the appropriate email address (see Figure 9-1).

For example, let's say Bob is a T-Mobile subscriber with the phone number (333) 444-5555. Suppose you were to send an email to Bob as follows:

From: yourname@yourdomain.com

To: 3334445555@tmomail.net

Subject: This is the subject

This is the message

Bob would receive the following SMS message:

To: 3334445555

From: 501

yourname@yourdomain.com/This is the subject/This is the message

If he responded to that SMS (i.e., sent an SMS to 501), you would receive the following email:

To: yourname@yourdomain.com

From: 3334445555@tmomail.net

Subject: Re: This is the subject

Thanks!

------------

yourname@yourdomain.com /This is the subject /This is the body

Note that T-Mobile keeps track of the subject and copies back the text of the original message when forming a reply to a response code.This is not universal across carriers.

NOTE

T-Mobile @tmomail.net

Verizon @vtext.com

Cingular @cingularme.com

Sprint @messaging.sprintpcs.com

AT&T @mmode.com

Find lots more at http://www.notepage.net/smtp.htm!

9.1.1. Advantages and disadvantages

The primary advantage of using an email to SMS gateway is that it's free (to you). There are some disadvantages, however.

First, you will need to know the carrier for each of your users in advance. If one of your users should change carriers, you will not be notified automatically.

Second, it may be a violation of some carrier's terms of use to operate a service against their email gateways. If they detect multiple messages coming from a particular SMTP server, they may block that server from addressing their customers for some period of time.

Carriers have at least two reasons to throttle incoming messages: 1) they do not want their customers to receive spam and 2) the email gateway is free, whereas they monetize their SMPP gateways (the gateways used by SMS aggregators).

Each carrier uses a different algorithm to detect abuse, so it is difficult to predict what will cause an interruption in your service and how to avoid it. Anecdotal evidence indicates that as few as 50 messages per day may trip the blacklist for some carriers. You may also find that sending from multiple addresses (even if it from the same mail server) is enough to mitigate this issue for some carriers.

Third, while most carriers allow their users to respond via SMS to email, many do not provide a means for a user to a conversation via email (i.e., they can only respond). This makes certain types of interactivity difficult or impossible. A look-up service, such as those described earlier, for example, are difficult to implement using an email to SMS gateway.

It is worth noting that the Dodgeball service began with a pure SMS—email implementation. Dennis Crowley, the co-founder, has said that he believes that the service did suffer from throttling issues, but never received any user complaints. The lesson here is that certain services may be more resilient to temporary interruption than others, based on user expectation and need.

NOTE

During the 2004 Republican National Convention, protestors tried to use TxtMOB, a group-centric text messaging service, to coordinate their actions. On the critical day, though, they found their messages blocked. Evidence of political dirty tricks? Perhaps not. On September 7, 2004, a reader posted this message to the popular web site Boing Boing:

"I'm a network data analyst for T-Mobile. I've actually tested the network to see why those messages were blocked, and from the response our email-to-sms gateway is giving, apparently our immensely retarded spam filter thinks that txtmob's SMTP server is spamming us. Basically, if the network sees more than about a hundred messages coming from the same SMTP server within an hour, it just blacklists it. Stupid but true."

Using Email to SMS Gateways > The Basics

Chapter 9. Using Email to SMS Gateways

Because most U.S. carriers and many international carriers will convert emails to SMS messages and forward them on to their customers, email to SMS gateways present a highly accessible means of rapidly prototyping SMS services. However, in most cases they are not scalable solutions for a full-fledged service, because the carriers may arbitrarily block messages being sent from a particular service when volume reaches a certain point (perhaps as few as 50 messages in a day).

9.1. The Basics

Every major U.S. carrier and most international carriers provide a gateway to the Internet so that their customers can receive SMS messages sent via email. The sender addresses the message to an email address of the form ###########@carrier-sms-domain.com, it is converted by the carrier into an SMS and delivered. There are several lists online with the carrier to domain name mappings, including one at http://www.notepage.net/smtp.htm.

NOTE

One thing you'll learn quickly is that there is very little that is "standard" whe using email to SMS gateways. Each carrier has a few idiosyncrasies in how they send and receive messages. For example, while most carriers will fill in the From address with the phone number plus their domain (e.g., 2065551212@tmomail.net), a few don't. Verizon PCS, for example, allows its users to substitute a handle instead of the number, so a message sent to a Verizon PCS user with the address 2065551212@vtext.com" might be replied to with a From address of johnsmith@vtext.com. This makes it difficult to keep track of who you're receiving messages from.

You'll want to accommodate this in your service. For example, when you user registers, have them reply to the registration mail with a unique response code that will allow you to keep track of who is sending them message, then match that up with the address it's coming from.

Figure 9-1. Email to SMS architecture

To allow the receiver to reply via SMS and have the email user receive the reply, the carriers will assign a temporary response code to the sender's email address, and then forward any SMS messages sent by that user to that response code back to the appropriate email address (see Figure 9-1).

For example, let's say Bob is a T-Mobile subscriber with the phone number (333) 444-5555. Suppose you were to send an email to Bob as follows:

From: yourname@yourdomain.com

To: 3334445555@tmomail.net

Subject: This is the subject

This is the message

Bob would receive the following SMS message:

To: 3334445555

From: 501

yourname@yourdomain.com/This is the subject/This is the message

If he responded to that SMS (i.e., sent an SMS to 501), you would receive the following email:

To: yourname@yourdomain.com

From: 3334445555@tmomail.net

Subject: Re: This is the subject

Thanks!

------------

yourname@yourdomain.com /This is the subject /This is the body

Note that T-Mobile keeps track of the subject and copies back the text of the original message when forming a reply to a response code.This is not universal across carriers.

NOTE

T-Mobile @tmomail.net

Verizon @vtext.com

Cingular @cingularme.com

Sprint @messaging.sprintpcs.com

AT&T @mmode.com

Find lots more at http://www.notepage.net/smtp.htm!

9.1.1. Advantages and disadvantages

The primary advantage of using an email to SMS gateway is that it's free (to you). There are some disadvantages, however.

First, you will need to know the carrier for each of your users in advance. If one of your users should change carriers, you will not be notified automatically.

Second, it may be a violation of some carrier's terms of use to operate a service against their email gateways. If they detect multiple messages coming from a particular SMTP server, they may block that server from addressing their customers for some period of time.

Carriers have at least two reasons to throttle incoming messages: 1) they do not want their customers to receive spam and 2) the email gateway is free, whereas they monetize their SMPP gateways (the gateways used by SMS aggregators).

Each carrier uses a different algorithm to detect abuse, so it is difficult to predict what will cause an interruption in your service and how to avoid it. Anecdotal evidence indicates that as few as 50 messages per day may trip the blacklist for some carriers. You may also find that sending from multiple addresses (even if it from the same mail server) is enough to mitigate this issue for some carriers.

Third, while most carriers allow their users to respond via SMS to email, many do not provide a means for a user to a conversation via email (i.e., they can only respond). This makes certain types of interactivity difficult or impossible. A look-up service, such as those described earlier, for example, are difficult to implement using an email to SMS gateway.

It is worth noting that the Dodgeball service began with a pure SMS—email implementation. Dennis Crowley, the co-founder, has said that he believes that the service did suffer from throttling issues, but never received any user complaints. The lesson here is that certain services may be more resilient to temporary interruption than others, based on user expectation and need.

NOTE

During the 2004 Republican National Convention, protestors tried to use TxtMOB, a group-centric text messaging service, to coordinate their actions. On the critical day, though, they found their messages blocked. Evidence of political dirty tricks? Perhaps not. On September 7, 2004, a reader posted this message to the popular web site Boing Boing:

"I'm a network data analyst for T-Mobile. I've actually tested the network to see why those messages were blocked, and from the response our email-to-sms gateway is giving, apparently our immensely retarded spam filter thinks that txtmob's SMTP server is spamming us. Basically, if the network sees more than about a hundred messages coming from the same SMTP server within an hour, it just blacklists it. Stupid but true."

Using Email to SMS Gateways > Example Service: Weather Updates

9.2. Example Service: Weather Updates

To show how we might use an email to SMS gateway, let's build an application that will automatically send users the weather forecast for their zip code once per day.

To register for our service, users will browse to a web page where they will provide their phone number, zip code, and the name of the carrier.

Shortly thereafter, they'll receive a text message on their mobile phone from our service that looks something like this:

info@myweathertxt.com /Your code is: 7f6bc210

We provide this code to ensure that the user isn't signing up someone else for our service. They will then enter that code onto our web page to confirm their registration.

Thereafter, once a day, at a time of our choosing (let's say 7:00 p.m. PST), the user will receive a weather forecast on her phone for the next day that will look like:

info@myweathertxt.com /Weather for Seattle, WA on 31 May 2007 will be Partly Cloudy with High/Low of 78/53

We'll also add code to allow users to stop receiving forecasts. All they have to do is reply to a forecast with the word stop.

9.2.1. How to: Weather updates using email to SMS

Here's what you'll need to get started:

SMTP and IMAP access: To implement our example, you'll need a mail account with an SMTP server and an IMAP server that you can access. Many free services such as Gmail and Hotmail do not typically provide both SMTP and IMAP access. Most hosting providers do. Some free email services are starting to add IMAP support, so it may be worth checking to see if your favorite one does.Note, however, that a similar solution with POP and SMTP is of course also possible.

Web server: A web server that can execute Python CGI scripts (exactly the same as in the aggregator example). This example has quite a few files, which may make it seem a bit intimidating. Rest assured they are all quite small. Place these in the root path of your web server:

config.py

createDb.py

index.html

challenge.py

smsEmailSender.py

register_new_phone.py

verify_phone.py

get_forecast.py

do_forecasts.py

smsEmailServer.py

responseServer.py

SQLite v2 or greater: If you already have Python installed, you most likely also have SQLite. SQLite is a fast and easy to use single user SQL database. You can learn more about it here: http://www.sqlite.org/.

Scheduled tasks: Your web server also needs to have the ability to run a process at scheduled intervals. This is accomplished using cron on Unix and the Windows Task Scheduler on Windows.

9.2.1.1. Static configuration data

The config.py file will hold configuration information for our service, such as the user name and password we use to access our SMTP server. It will also store the name of the database file with our user list in it.

Create a file called config.py with the following content and place it in your web server's root directory.

Code View: Scroll / Show All

config.py:
#CHANGE THIS
db_file = "database.db" # the name of the file user data is stored in
server = "mail.myhost.com" # your SMTP server's URL
account = "myaccount" # the user name for accessing your SMTP server
password = "mypasswd" # the password for your SMTP server
email_address = "myemail@mydomain.net" # the return address for outgoing email

9.2.1.2. Setting up the database

We're going to use SQLite to store our user's data in a SQL database. Before we can use the database we'll need to set up the tables for our application. CreateDb.py will take care of this for us.

createDb.py:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import config
from pysqlite2 import dbapi2 as sqlite
tables = ["users"]
table_descs = [
  """
       CREATE TABLE users(
              phonenumber VARCHAR(10),
              provider VARCHAR(30),
              zipcode VARCHAR(5),
              challenge VARCHAR(8),
              verified BOOLEAN DEFAULT FALSE,
              PRIMARY KEY(phonenumber)
       );
       """,
]
def main():
       con = sqlite.connect(config.db_file)
       con = con.cursor()
       #sqlite2 doesn't support IF EXISTS
       for t in tables:
              try:
                     con.execute("DROP TABLE %s" % t)
              except:
                     pass
       for t in table_descs:
              print t
              con.execute(t)

if __name__ == '__main__':
       main()

A complete explanation of SQL and how it works is outside the scope of this short cut. Our example should be pretty easy to understand, though. This script is set up in such a way as to allow many tables to be created, though for this example we'll only be making one.

We store the SQL command to create the users table in the table_descs list. It's a simple table with a few strings, including the user's phone number, provider, and zip code. The verified field lets us know whether or not the user has validated that they actually have the phone with the phone number they registered (recall the validation step from the example). We set phonenumber to be the primary key because we know it will be unique, and lookups will often be done by phone number.

Once you upload this script to your server, you'll want to run it (just like you run other python scripts). The rest of the code connects to a SQLite database with the name we set up in config.py and then executes our CREATE TABLE command.

9.2.1.3. The registration service

It's important that any service that pushes SMSs to users allows its users to sign-up and remove themselves from that service. Our service will let the user sign-up by providing his phone number, provider, and zip code.

Code View: Scroll / Show All

index.html:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
<head>
       <title>Weather Updates</title>
</head>
<body>
<p>First, register your phone</p>
<p>Then confirm it by typing the confirmation code into the form</p>
<p>You'll receive a weather forecast for the next day, every day</p>

<hr/>
       <h3> Register </h3>
       <form action="register_new_phone.py" method="get">
              <p>Phone Number (10 Digit US only): <input type="text"
name="phonenumber" size="10"/></p>
              <p>Zip: <input type="text" name="zip" size="5"/></p>
              <p>Provider:
              <select name="provider">
                     <option value="tmobile">T-Mobile</option>
                     <option value="cingular">Cingular/AT&T</option>
                     <option value="verizon">Verizon</option>
                     <option value="sprint">Sprint</option>
                     <option value="nextel">Nextel</option>
              </select>
              </p>
              <p><input type="submit" value="Register"/></p>
       </form>
<hr/>
       <h3> Verify </h3>
       <form action="verify_phone.py" method="get">
              <p>Phone Number: <input type="text" name="phonenumber" size="10"/></p>
              <p>Code: <input type="text" name="code" size="8"/></p>
              <p><input type="submit" value="Verify"/></p>
       </form>
<hr/>
</body>
</html>

There are two forms on this page. The first one allows initial registration, which will send a code to the entered phone number. The second one allows the user to verify that he owns the phone he entered by entering the secret code and his phone number.

You'll notice that the form action for the first form is register_new_phone.py and the action for the second is verify_phone.py. Registration requires that we perform two additional tasks that we haven't covered yet:

Generate a unique code to send to the user

Send an SMS to the user via email

We'll cover these tasks first, and then tie them all together to register a phone.

9.2.1.4. Generating unique codes

There are a lot of mechanisms that could be used to generate unique codes. In order to be useful for SMS, codes need to be:

Composed of a short sequence of alphanumeric characters

Long enough that they can't be guessed

For our application, we're SMS'ing a code to the user's phone and having him enter it on a web page. You might think it should be easier to display a code to the user on the web page itself and have the user SMS it to you. However, this approach can't be used when using email to implement an SMS service because users can't typically can't initiate communication with an email service from SMS.

Challenge.py will create our verification code.

challenge.py:

import sys
import random
import sha
CHALLENGE_LENGTH = 8
def generate_challenge():
       nonce = str(random.random())
       challenge = sha.sha(nonce).hexdigest()[0:CHALLENGE_LENGTH]
       return challenge

First, we import random and sha. The challenge length is set to be 8 characters.

One function, generate_challenge is defined. The function takes the user's phone number as an input. First, it generates a random number and converts it to a string. Then it uses the sha algorithm to generate a cryptographic hash of the random number, gets the result in hex, and truncates the hash to the first 8 characters.

There are a many mechanisms for generating user readable characters from the random number (including just passing the random number to the user). We use a cryptographic hash to help protect against the compromise of a weak random number generator.

The challenge is returned to the caller after its generated.

9.2.1.5. Sending an SMS via email

This is the part most of you have probably been waiting for. We're going to connect to our SMTP server and send SMSs using the carriers' SMTP gateways.

Code View: Scroll / Show All

smsEmailSender.py:

import sys
import smtplib

emails = {
       "tmobile" : "tmomail.net",
       "cingular" : "mobile.mycingular.net",
       "verizon" : "vtext.com",
       "sprint" : "messaging.sprintpcs.com",
       "nextel" : 'messaging.nextel.com',
}

class SmsEmailSender(object):
       def __init__(self, server, account, password, from_addr):
              self.__smtp = smtplib.SMTP()
              self.__account = account
              self.__password = password
              self.__from_address = from_addr
              self.__server = server

       def sendsms(self, phone_number, provider, body, from_addr = None):
              if not from_addr:
                     from_addr = self.__from_address

              self.__smtp = smtplib.SMTP()
              self.__smtp.connect(self.__server)
              self.__smtp.starttls()
              self.__smtp.login(self.__account, self.__password)
              self.__smtp.sendmail(
                     from_addr,
                     "%s@%s" % (phone_number, emails[provider]),
                     "%s" % (body)
              )

First, we define a dictionary that maps user-friendly provider names to the mail address suffixes for those providers. A nearly complete database of these mappings is available through the CoreSMS project (http://www.corelevel.com/sms/).

SmsEmailSender is defined as a class to allow easier reuse. The SMTP connection could also be reused, but in our class we recreate it each time we call sendmail, as the error handling for the other case is more complicated.

SmsEmailSender takes four parameters in its constructor:

server: The SMTP server to connect to

account: The account name on the SMTP server

password: The password for the account

from_addr: The FROM address to use on outgoing mail

We could use these values from our configuration and not require them to be passed in. SmsEmailSender is designed to be independent of configuration mechanisms and parameters so that it can be reused in other projects.

Our class has only one function, sendsms. The function takes five parameters:

phone_number: The phone number to send to.

provider: The provider to use, which must be in the emails dictionary defined at the top or an exception will be thrown.

body: The text of the email.

from_addr [optional]: An optional FROM address that will override the one used when constructing the object.

Next we connect to the server and log in. Calling starttls before sending your password ensures that it will be transmitted encrypted over the wire, instead of in plaintext (the default for SMTP).

The actual SMTP sendmail function simply requires you to pass it the address that the email will be from, the address you are sending the mail to, and the entire body of the message.

9.2.2. Processing a registration

Our form submits a GET request to a CGI script register_new_phone.py, passing parameters describing the user. The script generates a unique code.

register_new_phone.py:

Code View: Scroll / Show All

#!/usr/bin/python

import sys
import re
import cgi
import cgitb; cgitb.enable()
import smsEmailSender
import challenge
import config
from pysqlite2 import dbapi2 as sqlite

def result(text):
       print "Content-type: text/html\n\n"
       print """
       <html>
              <head><title>Registration Code Sent</title></head>
              <meta http-equiv="refresh" content="5; URL=/">
              <body>
                     %s
              </body>
       </html>
       """ % text
       sys.exit(0)

form = cgi.FieldStorage()
phonenumber = form.getfirst("phonenumber")
zipcode = form.getfirst("zip")
provider = form.getfirst("provider")

if not phonenumber or len(phonenumber) > 10:
       result("Please enter a valid 10 digit phone number")
phonenumber = phonenumber[0:10]
if not re.match("[0-9]{10}", phonenumber):
       result("Please enter a valid 10 digit phone number")
if not zipcode or len(zipcode) > 5 or not re.match("[0-9]{5}", zipcode):
       result("Please enter a valid 5 digit zipcode")
if not provider in smsEmailSender.emails:
       result("Your cell phone provider is not yet supported.")

con = sqlite.connect(config.db_file)
cur = con.cursor()

serv = smsEmailSender.SmsEmailSender(
       config.server,
       config.account,
       config.password,
       config.email_address
)

cur.execute("SELECT * FROM users WHERE phonenumber = ?", [phonenumber])
res = cur.fetchone()
if res:
       if res[4]:
              result("This phone has already been registered and verified")
       serv.sendsms(res[0], res[1], "Your code is: %s" % res[3])
       result("That number has already been registered, resending code")
else:
       challenge = challenge.generate_challenge()
       cur.execute("INSERT INTO users VALUES(?,?,?,?,?)", [phonenumber, provider,
zipcode, challenge, False])
       cur.close()
       con.commit()
       serv.sendsms(phonenumber, provider, "Your code is: %s" % challenge)
       result(
       """
       <p> The registration code has been sent to your phone. </p>
       <p> Please return to the home page and enter it to verify your
phone number. </p>
       <p> Redirecting in 5 seconds...</p>
       """)

There's quite a bit of code here. The first part of this script is identical to the one described in the aggregator chapter. We import needed modules, define our function to display results to the user, and we grab our three parameters from the query (phonennumber, zip,provider).

The next three blocks of conditionals validates that all the input parameters are like we expect them to be:

phonenumber: should be exactly 10 digits

zipcode: should be exactly 5 digits

provider: must be in the list of smsEmailSender.py's supported list of providers

If any of these conditions aren't met, we call result and return an informative error message to the user.

If everything checks out, we go ahead and open a connection to our database and get a cursor object. Cursor objects are transactions in Python's database API.

We instantiate a new SmsEmailSender object because we'll be using it in a few lines.

If the user is already registered, we want to let them know that and not reregister him, so we use our database connection to look for a user with the phone number that was passed onto us.

If we get a result, we do one of two things:

If the user isn't verified yet, we resend his confirmation code (he may have lost it).

If the user is already verified, we inform him that he is in fact already registered and good to go.

If the user isn't in the database, we generate a new challenge for him, and insert a new record into the database. It's important to call commit on cursor objects if they change a database's state, otherwise the changes won't have any effect.

Now we send the user his registration code via the sendsms method we wrote earlier. We call result and inform him that the code has been sent to his phone. As a courtesy to the user, our result HTML has a meta-refresh directive in it that will send him back to the home page in five seconds.

9.2.2.1. Verifying users

Once the user receives his registration code, he'll enter it into the form on the front page.

verify_phone.py:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import re
import cgi
import cgitb; cgitb.enable()
import config
from pysqlite2 import dbapi2 as sqlite

def result(text):
       print "Content-type: text/html\n\n"
       print """
       <html>
              <head><title>Verified</title></head>
              <meta http-equiv="refresh" content="5; URL=/">
              <body>
                     %s
              </body>
       </html>
       """ % text
       sys.exit(0)

form = cgi.FieldStorage()

phonenumber = form.getfirst("phonenumber")
code = form.getfirst("code")

if not phonenumber or len(phonenumber) > 10:
       result("Please enter a valid 10 digit phone number")
phonenumber = phonenumber[0:10]
if not re.match("[0-9]{10}", phonenumber):
       result("Please enter a valid 10 digit phone number")
if not code or len(code) != 8:
       result("Invalid Code")

con = sqlite.connect(config.db_file)
cur = con.cursor()
cur.execute("SELECT * FROM users WHERE phonenumber = ?", [phonenumber])
res = cur.fetchone()
if res and code == res[3]:
       cur.execute("UPDATE users SET verified = 'True' WHERE phonenumber
= ?", [phonenumber])
       con.commit()
       result("<p> Success - Phone Registered </p>")
else:
       result("Invalid Code")

Just as in the last CGI script, we import needed libraries, define our result function, and get our two parameters from the form: phonenumber and code. We then validate our parameters.

If our parameters are valid we connect to the database and look for a record with the phone number of the one that was passed in. If there is no record or the record doesn't have the same challenge code stored in it as the user passed to us, we error out.

If the code matches, we update our database by setting the verified field to True. A message indicating success is returned to the user.

9.2.2.2. Getting the Weather

Now that we have registered and verified users, we need to get forecasts for them. Luckily for us, Yahoo has a great RSS API for retrieving weather forecasts at (http://developer.yahoo.com/weather/).

Our code talks to the web service defined there and grabs tomorrow's forecast:

get_forecast.py:

Code View: Scroll / Show All

import re
import urllib
from xml.dom.minidom import parse

yweather_ns = "http://xml.weather.yahoo.com/ns/rss/1.0"
def get_forecast(zipcode):
       res = urllib.urlopen("http://weather.yahooapis.com/
forecastrss?p=%s&u=f"%zipcode)
       dom = parse(res)
       fdict = {}
       #get the location
       loc = dom.getElementsByTagNameNS(yweather_ns, "location")[0]
       fdict["city"] = loc.getAttribute("city")
       fdict["state"] = loc.getAttribute("region")
       #get tomorrow's forecast
       forecast = dom.getElementsByTagNameNS(yweather_ns,"forecast")[1]
       fdict["date"] = forecast.getAttribute("date")
       fdict["high"] = forecast.getAttribute("high")
       fdict["low"] = forecast.getAttribute("low")
       fdict["text"] = forecast.getAttribute("text")
       return fdict

if __name__ == "__main__":
       get_forecast("98112")

We use the python module minidom to handle XML processing. We define one function, get_forecast that takes the zip code for which to obtain forecast data.

The function first fetches the forecast (using urllib) and then uses the parse command of minidom to create a DOM.

Next, we look up the location element and grab the city and state of our forecast and put them into a dictionary.

There are two forecasts included with each request—today's and tomorrow's. Our service gives people the next day's weather in the evening the day before, so we want to grab tomorrow's forecast.

We retrieve a list of all the forecast nodes and grab the second one (index '0' is the first forecast). That element is used to get the date of the forecast, the high and low temperatures, and a textual description of the expected conditions. All of these elements are added into a dictionary that is then returned to the caller.

9.2.2.3. Sending our forecasts

Now that we can get a forecast, and we know who wants forecasts for which locations, we need to write a script that will loop through our users, get their forecasts, and send them.

do_forecasts.py:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import re
import smsEmailSender
import config
from get_forecast import *
from pysqlite2 import dbapi2 as sqlite

con = sqlite.connect(config.db_file)
serv = smsEmailSender.SmsEmailSender(
       config.server,
       config.account,
       config.password,
       config.email_address
)

cur = con.cursor()
users = cur.execute("SELECT phonenumber, provider, zipcode FROM users
WHERE verified='True'")
for u in users.fetchall():
       fd = get_forecast(u[2])
       forecast = "Weather for %(city)s, %(state)s on %(date)s will be %(text)s
with High/Low of %(high)s/%(low)s" % fd
       serv.sendsms(u[0], u[1], "Daily Forecast", forecast)

The first thing we do here is import the modules we need, including our own smsEmailSender, config, and get_forecast modules.

Next we connect to our database and instantiate an SmsEmailSender. A cursor object is created and used to find all the users whose verified flag is set to True.

The fetchall command is used to retrieve a list of users that we will iterate over. For each user, we get a forecast for his zip code. The zip code is stored in the third element (index 2) of the returned row, because that's how we asked for it to be returned in our SELECT statement.

Next we use Python's string formatting to create a text string that looks like the one in the example at the start. The slightly different form of string formatting used here allows you to explicitly grab elements of a dictionary instead of relying on positioning alone.

After creating the message, we use our SmsEmailSenderinstance, calling its sendsms member with the phone number (element 0 of the returned row), the provider (element 1), a subject of "Daily Forecast," and the text of our forecast.

This script will loop through all of our users and SMS them tomorrow's forecast.

9.2.2.4. Scheduling

Now we need to get do_forecasts.py to run once a day. On Unix you'll type:

       crontab -e

This will open your crontab file in an editor (probably vi, but you can change your editor by setting the EDITOR shell variable) and make it look like this:

SHELL=/bin/bash
PATH=/bin:/usr/bin:/usr/local/bin
00 19 * * * cd WEBDIR; python do_forecasts.py

Replace WEBDIR with the absolute path to the directory where all the files from this project are placed. This will cause your script to be run once a day at 7:00 p.m. (19:00). If you want it to run at a different time in order to test it, just change the "00 19" to be the 24-hour time, minute first, of when you want it to run.

If you're implementing this service on Windows, you can use the Windows Task Scheduler to implement something similar.

Wow. That was a decent bit of code, but we've implemented our first SMTP-based SMS service.

9.2.2.5. Receiving SMSs via email

While most carriers don't enable initiating a conversation with an email service from SMS, most do allow users to respond. This can be useful for your service. For example, you may wish to allow users to quit via SMS, or to ask for the forecast at some time other than that scheduled.

To demonstrate how we would go about receiving responses from SMS via email, we'll create code to allow the user to stop receiving forecasts by replying "stop." This is a little more complicated than the previous bits of code we've gone over, as it uses threading. It relies on our config.py and requires that the same account you use to send mail can be used to access an IMAP server.

First, we'll create a class that will poll for new messages and dispatch them to a callback when they are received.

smsEmailServer.py:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import time
import imaplib
import threading
import email
import traceback

class SmsEmailServer(object):
       def __init__(self, server, account, password, on_email_func,
polling_interval = 10):
              self.__account = account
              self.__password = password
              self.__on_email = on_email_func
              self.__polling_interval = polling_interval
              self.__running = False
              self.__server = server
              self.__imap = imaplib.IMAP4_SSL(server)
              self.__running = True
              self.__poll_thread = threading.Thread(target = self.__poll_mail_server)
              self.__poll_thread.start()

       def shutdown(self):
              if self.__running:
                     self.__running = False
                     self.__poll_thread.join()

       def __poll_mail_server(self):
              while (self.__running):
                     try:
                           self.__imap = imaplib.IMAP4_SSL(self.__server)
                           self.__imap.login(self.__account, self.__password)
                           self.__imap.select()
                           typ, data = self.__imap.search(None, 'ALL')
                           for num in data[0].split():
                                  typ, data = self.__imap.fetch(num, '(RFC822)')
                                  m = email.message_from_string(data[0][1])
                                  self.__on_email(m)
                                  self.__imap.store(num, "+FLAGS", "(\\Deleted)")
                           self.__imap.expunge()
                           time.sleep(self.__polling_interval)
                     except:
                           traceback.print_exc()
              self.__imap.logout()

We start by importing modules that we need, most notably: imaplib,threading, and email.

We then define a class called SmsEmailServer. It takes the standard server, account, and password parameters that SmsEmailSender used.

It takes a function as a parameter for on_email_func.This is a function that will be called whenever an email message is received (which is what SMS replies look like—an email).

It also takes an optional polling_interval parameter, which determines how often it looks for new mail. The default poll interval is 10 seconds.

SmsEmailServer sets a bunch of variables that it will use inside its polling thread, launches the polling thread, and returns.

The polling thread runs the function __poll_mail_server. This function has a loop that will run forever unless shutdown is called from another thread (ResponseServer.py, in this example).

Inside the loop we connect to the IMAP server over SSL and login. The IMAP connection can be reused, but it makes the code more complicated, so it is not presented here.

The rest of the loop is esoteric IMAP specific code that you needn't worry about, except for the part that creates a new email object and calls on_email_func with results.

After that function is called our code flags the message for deletion to make sure we don't process it again.

Being able to receive emails alone doesn't help us interact with our users. We'll need to process them. This script creates a new SmsEmailServer to listen for incoming SMSs. When it gets one it checks to see if the first word is "stop." If it is, we'll see if the sender is one of our users and remove them from the system

response_server.py:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import time
import config
from smsEmailServer import *
from smsEmailSender import *
from pysqlite2 import dbapi2 as sqlite

def on_email(email):
       contents = None
       #some phones send mms' as responses
       phonenumber = email["From"].split("@")[0].strip()
       if email.is_multipart():
              for p in email.get_payload():
                     #we only support plain text...
                     if p.get_content_type() == "text/plain":
                           contents = p.get_payload()
                           break
       else:
              contents = email.get_payload()

       if contents[0:4].lower() == "stop":
              print "Got stop from: %s" % phonenumber
              con = sqlite.connect(config.db_file)
              cur = con.cursor()
              cur.execute("SELECT * FROM users WHERE phonenumber = ?", [phonenumber])
              u = cur.fetchone()
              if u:
                     cur.execute("DELETE FROM users WHERE phonenumber = ?",
[phonenumber])
                     cur.close()
                     con.commit()
                     serv = SmsEmailSender(
                           config.server,
                           config.account,
                           config.password,
                           config.email_address
                     )
                     serv.sendsms(u[0], u[1], "You'll no longer receive forecasts
from us.")
                     print "User Removed"
              else:
                     print "No registered user with this phonenumber"
       else:
              print "Received Unknown Command: %s" % contents

def main():
       se = SmsEmailServer(config.server, config.account, config.password, on_email)
       try:
              while True:
                     time.sleep(.5)
       except:
              print "\n Shutting Down..."
              se.shutdown()

if __name__ == '__main__':
       main()

The main function just creates a new SmsEmailServer and sits in a loop—hitting Ctrl-C (or Ctrl-Break) will stop the process. When it creates the server it registers the on_email function as the handler for received email. SmsEmailServer will call our on_email function every time it gets a message.

The on_email function looks kind of complicated. The code at the start of our function grabs the From address out of the message. This is typically the phone number of the originating user. However, some phones, such as the T-Mobile Sidekick, don't behave this way. Instead, they send mail from an email address account associated with the phone. See the 'Response addresses' blurb at the bottom of this chapter for a method to deal with this.

To further complicate things, some phones send MMSs as responses to SMSs sent via email. This is a bit easier for us to deal with, however, because we still know who the message is from, it just happens to be in an unexpected format. MMS's are multipart MIME messages. So, after our code gets the From address, it checks to see if the message is multipart. If it is, it looks for the plain text portion and considers that the message content.

Next it looks to see if "stop" is the first word. If it isn't, we bail out and do nothing (add your own custom handling here; for example, you can allow users to look up the forecast for their city).

If it is "stop," we have a bit of work to do. First we connect to the database and look for a user with a phone number equal to the From address. If there's a match, we remove the entry from the database. Right after we remove the entry, we send a confirmation SMS to the user who asked to be removed.

If we can't find a user with a phone number equal to the From address, we simply do nothing.

In order to receive SMSs, you'll need to leave this script running on a server. The machine needs to be the same one that the web frontend is running on, because the database is stored in a file. Contact your hosting provider for information on how to set up a background process, as it varies from provider to provider.

NOTE

The address each SMS appears to be from is an important consideration when designing your service. The user will always see the From address at the start of every message, and any responses will be delivered to that address.

For example, if you send mail to 2065551212@tmomail.net from myservice@mydomain.com, your user will receive an SMS that looks something like:

myservice@mydomain.com / body of your message

Conversely, when your service receives a reply from an SMS user, you'll want to know who sent you the message.

The typical way to do this is to simply look at the address of the sender when checking mail for the service. Most emails will appear to be from something similar to 1112223333@carrier.net. You could strip the number of the front portion, verify that the phone number is a registered user, and respond as appropriate given the content of the request.

This approach works for most services, but it's less than ideal for a couple of reasons:

It allows hackers to impersonate other users by forging From addresses on emails.

If you have sent multiple messages to the user, there may be no way to know which message the user is responding to.

As an alternative, you may pick a unique From address for every outgoing message or conversation and store it in a table. In this sense, the unique address is exactly like a session key on a web site. This approach has several advantages:

Impersonating a user is more difficult, providing you check that the From address for a message matches the expected From address.

It reduces the likelihood you will be throttled based on From addresses. There are many other heuristics that could be used by carriers to throttle, so this is not a complete solution (some carriers throttle by the IP address of the SMTP server, for example).

It enables threading, as each From address can correspond to a separate conversation with the user.

Using a Mashup: 411Sync >

Chapter 10. Using a Mashup: 411Sync

411Sync (http://www.411sync.com) is a service that provides a web-based API to let you create your own application on top of their SMS functionality. It's a free, easy to use method to implement a lookup-type SMS service. The usage paradigm is straightforward:

Register a keyword with 411Sync, such as smsbook.

Provide an URL for 411Sync to call when someone sends them an SMS with that keyword.

Your URL should return an RSS or Atom feed to 411Sync, which they use to construct a reply SMS.

For example, we registered the keyword smsbook in the process of writing this guide. Its purpose was to allow people to easily add themselves to a mailing list that would inform them when the guide was published.

Here's an example of sending using the service (you can try it yourself).

If you send the following text message to (415) 676 8397(411Sync's SMS number),

smsbook youremail@yourdomain.com

You will receive an SMS back from (415) 676 8397 that reads:

More Info at http://tinyurl.com/2phuke -- youremail@yourdomain.com added to mailing list -- - Reply back ?? to get alerts

Now, that probably looks a bit confusing. 411Sync adds the header ("More info at http://tinyurl.com/xxxxxx") and footer ("- Reply back ??to get alerts") to your message (and, yes, that comes out of your 160 character allotment).

10.1.

10.1.1. Advantages and disadvantages

This biggest advantage of 411sync is the cost: free. With no money and very little code, you can run a service that sends and receives text messages. Great, right?

What you get in cost, you give up in flexibility. The chief disadvantage of using 411Sync is the loss of control over the message format. Remember that header and footer that you saw in the example above? Do you want that as part of your service? Even if you don't, you can't change or remove it.

10.1.2. Example service: Email registration

As we mentioned above, we created a simple sign-up service when we were writing this short cut that let people who ask about it sign up to receive a notification when the short cut was ready. Anyone who sent the message:

smsbook youremail@yourdomain.com

to 411Sync would get a message back telling them we'd notify them when the short cut was ready (and we added their email address to our database).

Let's take a look at how we did it.

10.1.3. How to: Email registration service with 411Sync

To run our service, you'll need:

A web server capable of executing Python CGI scripts

SQLite on the same server

See the "Chapter 7" section for information on how to set these up if you haven't already.

10.1.3.1. Register with 411Sync

First step to creating your service will be to register with 411Sync. Just fill out the simple form at http://www.411sync.com/cgi-bin/Register, provide the verification number he'll send to you as a text message and sign back in.

NOTE

Did you notice that 411Sync asks for your service provider (carrier) when you register, in order to text message you your verification code? Can you guess why it needs this information?

Flip back to the "Chapter 9" section in this guide for a hint!

10.1.3.2. Create the service

The code running our smsbook service is quite simple:

smsbook.cgi:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import cgi
import cgitb; cgitb.enable()

form = cgi.FieldStorage()
request = form.getfirst("request")
if not request:
       response = "--Include your email after smsbook--"
elif len(request.split("@")) != 2:
       response = "--Include a valid email after smsbook--"
else:
       email_file = file("emails.txt", "a")
       email_file.write(request + "\n")
       response = "--%s added to mailing list--" % request
       email_file.close()

print """
<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
       <title>Email Reg</title>
       <link></link>
       <description>test</description>
       <language>en-us</language>
       <copyright></copyright>
       <item>
              <title> %s </title>
              <description></description>
       </item>
</channel>
</rss>
""" % response

When 411Sync invokes this script, it will pass in any parameters (text included in a message after our keyword) as a request, so the first thing our script does is grab that for processing. In our example, we check it to ensure that it isn't empty and to make sure it looks at least a little bit like an email address.

Provided that everything checks out, we open a file called emails.txt in append mode, and write the email address that was stored in request into the file.

The last thing we do is return a response to 411Sync. 411Sync is pretty peculiar about the format of the response, and it isn't incredibly well documented on its web site.

The text that actually gets displayed to the user is tucked inside the title tag of the item element. The content of that tag is %s in our example, which is being substituted with the value of response.

10.1.3.3. Register your keyword

In order to register your keyword with 411Sync, you'll need to actually have the XML response ready for it to test, so make sure you've completed the previous step before you start this step.

From 411Sync's home page, click the "My Mobile Search Keywords" link and then click "Create Mobile Search Keyword." The form looks a bit more complicated than it is. Most of the questions it asks have to do with how your service will be advertised on its web site. The fields you need to worry about are the ones where you specify your keyword and the HTTP location of your XML data (i.e., the callback URL 411Sync will invoke when someone sends them a text message with your keyword).

10.1.3.4. Try it

You're ready to go. Try sending a text message with your keyword and an email address to 411Sync. I'd suggest sending a message with the smsbook keyword, too, but I guess if you're reading this, you're all set.

If you're interested in doing more with 411Sync than we describe here, you should check out its full API description at http://www.411sync.com/cgi-bin/Developer.

Using a Mashup: 411Sync >

Chapter 10. Using a Mashup: 411Sync

411Sync (http://www.411sync.com) is a service that provides a web-based API to let you create your own application on top of their SMS functionality. It's a free, easy to use method to implement a lookup-type SMS service. The usage paradigm is straightforward:

Register a keyword with 411Sync, such as smsbook.

Provide an URL for 411Sync to call when someone sends them an SMS with that keyword.

Your URL should return an RSS or Atom feed to 411Sync, which they use to construct a reply SMS.

For example, we registered the keyword smsbook in the process of writing this guide. Its purpose was to allow people to easily add themselves to a mailing list that would inform them when the guide was published.

Here's an example of sending using the service (you can try it yourself).

If you send the following text message to (415) 676 8397(411Sync's SMS number),

smsbook youremail@yourdomain.com

You will receive an SMS back from (415) 676 8397 that reads:

More Info at http://tinyurl.com/2phuke -- youremail@yourdomain.com added to mailing list -- - Reply back ?? to get alerts

Now, that probably looks a bit confusing. 411Sync adds the header ("More info at http://tinyurl.com/xxxxxx") and footer ("- Reply back ??to get alerts") to your message (and, yes, that comes out of your 160 character allotment).

10.1.

10.1.1. Advantages and disadvantages

This biggest advantage of 411sync is the cost: free. With no money and very little code, you can run a service that sends and receives text messages. Great, right?

What you get in cost, you give up in flexibility. The chief disadvantage of using 411Sync is the loss of control over the message format. Remember that header and footer that you saw in the example above? Do you want that as part of your service? Even if you don't, you can't change or remove it.

10.1.2. Example service: Email registration

As we mentioned above, we created a simple sign-up service when we were writing this short cut that let people who ask about it sign up to receive a notification when the short cut was ready. Anyone who sent the message:

smsbook youremail@yourdomain.com

to 411Sync would get a message back telling them we'd notify them when the short cut was ready (and we added their email address to our database).

Let's take a look at how we did it.

10.1.3. How to: Email registration service with 411Sync

To run our service, you'll need:

A web server capable of executing Python CGI scripts

SQLite on the same server

See the "Chapter 7" section for information on how to set these up if you haven't already.

10.1.3.1. Register with 411Sync

First step to creating your service will be to register with 411Sync. Just fill out the simple form at http://www.411sync.com/cgi-bin/Register, provide the verification number he'll send to you as a text message and sign back in.

NOTE

Did you notice that 411Sync asks for your service provider (carrier) when you register, in order to text message you your verification code? Can you guess why it needs this information?

Flip back to the "Chapter 9" section in this guide for a hint!

10.1.3.2. Create the service

The code running our smsbook service is quite simple:

smsbook.cgi:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import cgi
import cgitb; cgitb.enable()

form = cgi.FieldStorage()
request = form.getfirst("request")
if not request:
       response = "--Include your email after smsbook--"
elif len(request.split("@")) != 2:
       response = "--Include a valid email after smsbook--"
else:
       email_file = file("emails.txt", "a")
       email_file.write(request + "\n")
       response = "--%s added to mailing list--" % request
       email_file.close()

print """
<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
       <title>Email Reg</title>
       <link></link>
       <description>test</description>
       <language>en-us</language>
       <copyright></copyright>
       <item>
              <title> %s </title>
              <description></description>
       </item>
</channel>
</rss>
""" % response

When 411Sync invokes this script, it will pass in any parameters (text included in a message after our keyword) as a request, so the first thing our script does is grab that for processing. In our example, we check it to ensure that it isn't empty and to make sure it looks at least a little bit like an email address.

Provided that everything checks out, we open a file called emails.txt in append mode, and write the email address that was stored in request into the file.

The last thing we do is return a response to 411Sync. 411Sync is pretty peculiar about the format of the response, and it isn't incredibly well documented on its web site.

The text that actually gets displayed to the user is tucked inside the title tag of the item element. The content of that tag is %s in our example, which is being substituted with the value of response.

10.1.3.3. Register your keyword

In order to register your keyword with 411Sync, you'll need to actually have the XML response ready for it to test, so make sure you've completed the previous step before you start this step.

From 411Sync's home page, click the "My Mobile Search Keywords" link and then click "Create Mobile Search Keyword." The form looks a bit more complicated than it is. Most of the questions it asks have to do with how your service will be advertised on its web site. The fields you need to worry about are the ones where you specify your keyword and the HTTP location of your XML data (i.e., the callback URL 411Sync will invoke when someone sends them a text message with your keyword).

10.1.3.4. Try it

You're ready to go. Try sending a text message with your keyword and an email address to 411Sync. I'd suggest sending a message with the smsbook keyword, too, but I guess if you're reading this, you're all set.

If you're interested in doing more with 411Sync than we describe here, you should check out its full API description at http://www.411sync.com/cgi-bin/Developer.

Using Twitter to Build a SMS Service > Twitter Overview

Chapter 11. Using Twitter to Build a SMS Service

Twitter (http://twitter.com) is a social networking service that allows users to broadcast their goings-on to one another via SMS (as well as email and instant messenger). Its API is described at: http://twitter.com/help/api and a healthy developer community has grown around its API at http://groups.google.com/group/twitter-development-talk/web/api-documentation.

11.1. Twitter Overview

Fundamentally, Twitter is about updating your status—you tell Twitter what you're doing, and it tells all your friends.

For example, let's say you and I have registered with Twitter and we've marked each other as friends. If I were to send this SMS to Twitter's short code (40404):

I'm working on my SMS guide!

You and all my other friends on Twitter would receive an SMS from 40404 that reads:

Jordan : I'm working on my SMS guide!

Twitter also lets you send direct messages to specific users by prefacing a message with the letter "d" followed by the recipient's user name, or handle. So, if instead of sending a general "tweat" (that's what they call status update messages like the one above), I sent this SMS to Twitter:

d BrianR How's the sample code coming along?

my co-author Brian (and only Brian) would receive this message:

direct from JojoSchwa: How's the sample code coming along? (reply? Send: d JojoSchwa hi.)

Using Twitter to Build a SMS Service > Twitter Overview

Chapter 11. Using Twitter to Build a SMS Service

Twitter (http://twitter.com) is a social networking service that allows users to broadcast their goings-on to one another via SMS (as well as email and instant messenger). Its API is described at: http://twitter.com/help/api and a healthy developer community has grown around its API at http://groups.google.com/group/twitter-development-talk/web/api-documentation.

11.1. Twitter Overview

Fundamentally, Twitter is about updating your status—you tell Twitter what you're doing, and it tells all your friends.

For example, let's say you and I have registered with Twitter and we've marked each other as friends. If I were to send this SMS to Twitter's short code (40404):

I'm working on my SMS guide!

You and all my other friends on Twitter would receive an SMS from 40404 that reads:

Jordan : I'm working on my SMS guide!

Twitter also lets you send direct messages to specific users by prefacing a message with the letter "d" followed by the recipient's user name, or handle. So, if instead of sending a general "tweat" (that's what they call status update messages like the one above), I sent this SMS to Twitter:

d BrianR How's the sample code coming along?

my co-author Brian (and only Brian) would receive this message:

direct from JojoSchwa: How's the sample code coming along? (reply? Send: d JojoSchwa hi.)

Using Twitter to Build a SMS Service > The Twitter API

11.2. The Twitter API

Because Twitter has already done the hard work of implementing an SMS service (not to mention picked up the tab), you can use its API to send and receive messages on your behalf.

The API conforms to the principles of REST, as does the architecture of the site in general. This is important because the official API does not yet support everything needed to make a seamless SMS service. This guide will fill in some of the missing pieces. The API is under ongoing development, so the functions we talk about here may have changed or may be part of the official API by the time you read this.

Using Twitter to Build a SMS Service > Advantages and Disadvantages

11.3. Advantages and Disadvantages

Twitter is an attractive option for implementing an SMS service because it's free and comes with a built in user base and social network. You don't need a web server to create a Twitter mashup, either, just a machine with an Internet connection.

On the other hand, anyone who wants to use your service must first become a member of Twitter. This isn't really that big of a deal, though, when you consider that some kind of registration system is required for most services.

The other big downside to using Twitter is that your service must be addressed by name (e.g., "d yourservicename message") on every message, making interactions more complex and arduous. Twitter also takes up a bit of your character limit with reply instructions (but it's nothing compared to the extras 411Sync adds in).

Using Twitter to Build a SMS Service > Example Service: Echo

11.4. Example Service: Echo

As an example, we'll create a service in Python that will echo back at you whatever you send to it.

For example, if you send the following message to Twitter's 40404 short code:

    d echo "What's up?"

it will respond with:

    ECHO: What's up? (reply? send: d echo hi)

The text in parentheses is added automatically by Twitter—you can't get rid of it.

11.4.1. How To: Creating a Simple Twitter Service

Start by creating a new Twitter user for your service. We chose echo for this example, but you will have to choose your own. For the purpose of simplicity, we will set its password to 'echo' as well.

Users can add the service as their friend and then send it direct messages, to which the service will respond.

The Echo service will use a Twitter interface class that we'll be creating. We'll go over that class first and then over the details of the Echo service itself.

To create your Twitter service, you'll need:

Python 2.4 or higher

The json-py module (called python-json in pypi) for JSON support (http://cheeseshop.python.org/pypi/python-json)

A Twitter account for your service (http://www.twitter.com/)

A code listing will be provided after we go over the basic functionality provided by the Twitter class.

11.4.2. Functionality

Our Twitter interface provides most of the functions that you'll want to use to implement a Twitter service. Some of them are documented in the public API, but some of them are not. Here are the methods we'll be using:

get_followers (public): Gets a list of users that are following you. People who are following you will hear your messages. http://twitter.com/statuses/followers.json

get_friends (public): Gets a list of users that you consider your friends—you'll receive their updates and their direct messages. http://twitter.com/statuses/friends.json

get_direct_messages (public): Returns every (non-deleted) direct message you've ever received. (Ideally there would be a function just to return unread messages, but it doesn't exist as of this writing.) http://twitter.com/direct_messages.json

befriend_all (undocumented): This function makes everyone who is following you your friend. This is important because in Twitter, you can only receive message from people who are your friends. http://twitter.com/followers/befriend_all

send_direct_message (public): Given a user name or id and a message to send, this method will send the message to the user - provided that user has accepted you as a friend. As noted in the API documentation, this method must be called using POST. http://twitter.com/direct_messages/new.json

delete_direct_message (undocumented): Lets you delete a message, given its message id. A service needs to delete messages every time it reads them to avoid acting twice on the same message. http://twitter.com/direct_messages/destroy/[MessageID]

delete_one_page_of_sent_messages (undocumented): Gets the Message IDs of all the sent messages that will show up on the user's sent direct message panel. It uses the IDs and repeatedly calls delete_direct_message. There might be a better way of doing this, but as of this writing it is the best mechanism we could find. http://twitter.com/direct_messages/sent(This is normal user-visible page, not an API call.)

twitter.py:

Code View: Scroll / Show All

#!/usr/bin/python
import sys
import os
import urllib2
import urllib
import json
import time
import re
import random

TWITTER_URL = "http://twitter.com"

class Twitter(object):
       def __init__(self, useremail, password):
              authstring = "%s:%s"%(useremail, password)
              self.__twitter = urllib2.build_opener()
              self.__twitter.addheaders = [
               ('Authorization', 'Basic %s'% (authstring.encode("base64").strip()))
              ]
              self.__sent_destroy_re = re.compile("/direct_messages/destroy/([0-9]+)")

       def get_followers(self):
              page = self.__twitter.open(TWITTER_URL + "/statuses/followers.json")
              return json.read(page.read())

       def get_friends(self):
              page = self.__twitter.open(TWITTER_URL + "/statuses/friends.json")
              return json.read(page.read())

       def befriend_all(self):
              page = self.__twitter.open(TWITTER_URL + "/followers/befriend_all")

       def get_direct_messages(self):
              page = self.__twitter.open(TWITTER_URL + "/direct_messages.json")
              return json.read(page.read())

       def send_direct_message(self, friend_id, text):
              params = {
                     'text' : text,
                     'user' : friend_id,
              }
              page = self.__twitter.open(TWITTER_URL + "/direct_messages/new.json",
data=urllib.urlencode(params))
              return json.read(page.read())

       def delete_direct_message(self, direct_message_id):
              page = self.__twitter.open(TWITTER_URL + "/direct_messages/destroy/%s"
% direct_message_id)

       def delete_one_page_of_sent_messages(self):
              page = self.__twitter.open(TWITTER_URL + "/direct_messages/sent")
              for line in page.readlines():
                     m = self.__sent_destroy_re.search(line)
                     if m:
                           self.delete_direct_message(m.group(1))
def main():
       #simple usage example
       twit = Twitter("username", "password")
       print twit.get_friends()

if __name__ == '__main__':
       main()

Twitter requires that most commands be authenticated using Basic Authentication. The constructor builds an object that will attach the appropriate authorization header every time we make a request. The code inside the constructor for Twitter base64 encodes the text "username:password" and then puts it in an "Authorization" header prefixed with the word "Basic." You can read more about Basic Authentication at http://www.ietf.org/rfc/rfc2617.txt.

Almost every method simply uses the opener we created in the constructor to access one URL. The URL returns a JSON dictionary that we convert into a python dictionary.

The .json appended to the URL tells Twitter to return the results as JSON dictionaries. XML and RSS are supported for many methods as well. JSON was chosen here because it is lightweight and translates easily into Python dictionaries. All we have to do is call the read method of the JSON module on the results, and we get a Python dictionary back with all the results

A JSON dictionary for a get_friends call looks something like this (this user has one friend named testinos:

[{"name":"testinos","description":null,"location":null,
"profile_image_url":"http:\/\/assets0.twitter.com\/images\/
default_image.gif?1181015407","url":null,"id":5719462,
"screen_name":"testinos","protected":false}]

After calling json.read() on the text of the JSON, we get a Python dictionary that looks like this:

[{'name': 'testinos', 'url': None, 'profile_image_url':
'http://assets0.twitter.com/images/default_image.gif?1181015407',
'screen_name': 'testinos', 'location': None, 'protected': False,
'id': 5719462, 'description': None}]

There are a few methods that warrant some additional explanation:

The Twitter API requires that POST be used when sending direct messages. The opener knows to use POST instead of get when it's passed a urlencoded string as the second parameter to open. You'll notice in the send_direct_message function that we call urlencode on a parameter dictionary for just that purpose

The delete_one_page_of_sent_messages function has to load a normally user-visible web page in order to do its duty. This is because there is no API call to retrieve sent messages. A regular expression extracts the Message ID for each sent message out of the page and passes it to our delete_direct_message.

11.4.3. The Echo application

We'll use the commands described above to build our service. The Twitter API only supports polling right now, so we'll be doing everything in a loop.

echo.py:

#!/usr/bin/python
import twitter
import time
twit = twitter.Twitter('servicename', 'servicepass')
while True:
       twit.befriend_all()
       print "checking..."
       dms = twit.get_direct_messages()
       for dm in dms:
              twit.delete_direct_message(dm['id'])
              print "Echoing! %s" % dm['text']
              twit.send_direct_message(dm['sender_screen_name'],
"ECHO: " + dm['text'])
       time.sleep(120)
       twit.delete_one_page_of_sent_messages()

Once you have the Twitter interface written, the service itself is trivial. Every two minutes we befriend anyone who has added us as their friend, and check for new messages. If we have a new message, we get the contents of it, delete it, and then send it back to the originator prefixed with ECHO:.

To try this out for yourself:

Edit echo.py to contain the username and password for your service.

Create a test Twitter account or a personal one.

Follow your service (Follow is a Twitter concept similar to making someone your friend, except it is a one-way relationship, instead of two-way)

Copy twitter.py and echo.py into the same directory on any machine that meets the requirements discussed at the start of the chapter, then type:

python echo.py

Now have a friend send a direct message to your service (e.g., "d echo this is a test").

Have fun tweating!

Chapter 12. Using Windows Live Alerts

Microsoft's Windows Live Alerts provides a quick, easy, and free means of integrating SMS into your service, depending on your needs. Windows Live Alerts will poll any RSS feed and generate SMS messages (as well as IM and email) for all subscribers whenever new content is generated.

To build a Windows Live Alert SMS feed, simply visit http://signup.alerts.live.com/ and follow the instructions there. You will fill out a short form, including your contact information, the URL of your RSS feed, and some configuration information (e.g., whether you want headlines delivered or just notification of new content). The Alerts service will produce a widget for you to embed in your web page that looks something like this:

When one of your users clicks on the link, Windows Live will guide them through a process of validating and configuring their phone to receive your alerts. A web-based Message Manager application allows you to monitor subscriptions, view the outgoing queue, and produce usage reports.

The service was designed for blog updates, but any information that you wish to publish as an RSS feed will be digested and sent to your users, making this an ideal means for producing a pure broadcast service.

Some important notes on implementing a Windows Live Alerts mashup:

As of May 2007, Alerts supports RSS 0.9x, 1.0 and 2.0 as well as Atom formats.

Alerts is a free, ad-supported service. As of today, that means that alerts delivered via email and IM may be accompanied by an advertisement. In the future, this may extend to SMS, as well. An ad-free version of the service is also available, however. Contact AlertsSu@microsoft.com for more information.

A richer, SOAP-based API is also available for driving the Alerts service. More information on this service is available through MSDN.

Chapter 13. Summary

We hope this has been helpful to you in planning and implementing your own SMS service. We've covered both aspects of SMS service interface design as well as the backend implementation. To those of you with web service or client application development experience, many of the concepts should have been familiar, just arranged in a new way.

Ultimately, there is nothing truly special about SMS: it is a simple text-based input/output protocol to your connected application. What is special is what you do with it. Go do something special.