Multimodal Multiple Choice

Enabling a simultaneous voice and data session via a wireless device may not change the world, but the capability, referred to as multimodal, is getting a lot of attention.

“Just as browser technology drove the worldwide explosion of Internet-based applications and businesses, multimodality will drive the adoption of new, easy-to-navigate wireless applications worldwide,” said Arvind Rao, OnMobile CEO.

For example, a user could locate a restaurant by asking a wireless device for directions from a current location. A map then could be displayed on the screen as directions are read using text-to-speech technology. Or the user could receive an SMS message with text directions along with a visual.

That's a far cry from today's model where users have to turn off a voice session to enter data mode. Analysts say multimodal technology truly will enable subscribers to control their wireless experiences. The ability to enter and retrieve data via speech, touch and graphics on a device allows unprecedented ease of data access. For carriers, the capability may be key finally to getting people to use wireless data.

Compelling Combination

Multimodal applications combine speech, touch and graphics into a single interface, combining input (touchscreen, keypad and voice commands) and output (images and sounds) modes in a device to allow users to access wireless content in almost any way they'd like.

SpeechWorks is creating enhanced user interfaces that provide multimodal access to wireless content through input and output mechanisms that use speech recognition and text-to-speech technology. Lobby7 and SpeechWorks' solution features Lobby7's multimodal application server, x-mode, SpeechWorks' text-to-speech and voice recognition technologies, and integrated speech, touch and graphics. For example, the user can tell the device where he is going and simultaneously tap the screen and say, “show me how to get here.” He also can view a map on the device's screen while receiving audio directions.

Multimodal combines the best of both worlds — the speed and convenience with the high utility and relevance of high-value applications that make people want to use wireless data.

“Wireless (carriers) love this idea because it should bring about more consumption,” said Steve Chambers, SpeechWorks vice president of marketing. “There's more information being conveyed so the applications should be stickier. All of them have thought about this as a revenue-enhancing line item.”

Two Asian carriers are testing OnMobile Systems' multimodal wireless platform, MMP2500, and soon will launch the service. The technology combines voice access with SMS or WAP to receive and respond to voice or text messages. The platform is installed in the carrier's network and plugged into the SMSC, MSC, location server, billing systems and WAP gateway.

The MMP2500 relies on VoiceXML and XHTML standards and supports applications that integrate voice, text and graphics. Users can choose whether they want driving directions read to them, delivered in a text-based form such as e-mail or downloaded as a map.

Greg Wolfond, 724 Solutions chairman, said the company will integrate interactive voice response (IVR) and voice-recognition technologies with its current wireless transaction platform and applications to allow consumers to move from voice to text seamlessly when making wireless purchases. With ASP InterVoice-Brite, 724 will offer an alerts-based multimodal solution later this year.

“Multimodal is about bringing together all channels to create a compelling user experience on all wireless devices,” he said. “You might be talking to a speech interface and say, ‘What were the last 10 transactions?’ It takes a long time to read 10 transactions. But while you're talking, an SMS message can send that information to your phone. All of that is workable over today's networks and creates a better customer experience.”

Simultaneous Sessions

Andy Willett, AT&T Wireless vice president, business development, said customers are frustrated when they enter data mode on the carrier's CDPD network and get disconnected from inbound voice calls, which go directly to voice mail.

“There's been this challenge for them in terms of how to toggle back and forth between the two, and calls going to voice mail more than they would like them to,” he said.

Willett said AT&T already has an alternate alert function available in markets where it has overlayed a 2.5G GPRS data network on top of its GSM voice network.

“You can't do simultaneous (sessions),” he said. “But what you can do, which starts to address this problem, is use SMS messaging.”

For example, users in voice mode can receive an SMS alert that says “you have an e-mail,” or “you have an order you need to check.” In that message, there's an embedded URL that the user can click on to enter data mode immediately, get information and then click back out into voice mode. This capability is a first step toward simultaneous sessions, which will be automatic in the 3G world.

“When you get both voice and data going over the same network, all of it over IP, then you can get into scenarios when you're actually sending data and talking on the phone at the same time,” Willett said. “That means that you really and truly need to be on a 3G network that is utilizing IP for voice as well as for data.”

Handsets must be able to multiplex both voice and data and multimedia onto an IP stream in the 3G environment, according to Terry Boland, Nortel Networks director of strategic marketing, Intelligent Internet.

“As the IP stream or session is sent across into the carrier's network, devices inside the core network can then look inside that packet and be able to qualify whether that's a voice call … or whether it's data,” he said. “That has to be put in place prior to those devices being able to multiplex both types of sessions across one IP or core infrastructure.”

Muzib Kahn, Samsung Telecommunications product engineer, said that future handsets will be multimodal.

“(Multimodal capability) is done through the call-processing software that resides in the phones,” he said.

Still, the entire implementation system — handsets, network, testing — has yet to be worked out.

“There are some synchronicity issues because you're mixing streams of visual and audio, but there are some trials going on over current networks, so that's not the bottleneck,” Chambers said. “That's exactly why you need new devices, so you can process multiple streams and sync them up, and why the protocols need to be established to sync up these different data streams as well.”

The first wave of multimodal applications will use today's networks to deliver thin content, but Willett said multimodal capability won't make or break the wireless Internet even when 3G arrives.

“There are definitely applications where that kind of capability would prove useful,” he said. “So from that perspective, would it have a positive impact on data usage? Absolutely. Do I think it's one of the key drivers that will make data useful for our customers and make us have a very viable business in that area? No, I don't think it's that critical.”

---

A Dash of SALT 231

Many companies are writing multimodal protocols, but that's not good enough, according to Steve Chambers, SpeechWorks vice president of marketing.

“If we waited for these formal boards, it would be too long,” he said. “Protocols are needed that will allow the client to talk to the server, that will allow the data streams to sync up, that will allow audio and visual to mix, (and to establish) markup languages so content can be used through these multimodal applications.”

Six companies have established an initiative to promote and accelerate standards for multimodal applications. Cisco, Comverse, Intel, Microsoft, Philips and SpeechWorks recently formed the Speech Application Language Tags (SALT) Forum to create a formal specification.

Multimodal access will enable users to input data using speech and/or a keyboard, keypad, mouse or stylus, and produce data as synthesized speech, audio, plain text, motion video and/or graphics. Modes could be used independently or concurrently.

SALT will create a device-, network- and platform-independent standard by extending existing markup languages such as HTML, XHTML and XML, Chambers said.

One of its goals is to create a development environment where Web programmers can code multimodal applications.

SALT will make its specification publicly available in 1Q02 and submit it to a standards body by mid-year.

For more information, visit www.saltforum.org.