Mobile Application Prototyping with Python

1

• Mobile Application Prototyping with Python
• A 3-Day Crash Course turned into a 1-Hour Talk

Nathan Eagle, PhD Research Scientist MIT Design
Laboratory Massachusetts Institute of Technology
August 4, 2006 GSTIT
2
Why Program Phones?
Stats for bottom graph made up, but probably
pretty accurate

• Mobile phones are computers that are carried by
  over 1,000,000,000 people around the world.
• They are no longer single use devices but rather
  can be harnessed to provide a variety of
  functionalities.
• You'll be part of the first mobile phone
  application developers in the world.

3
Programmable Phones Hardware
Data from Python for Series 60, Jukka Laurila,
EuroPython 2005.

• 100-220 MHz ARM processor (and a separate
  processor for telephony functions)
• Typically 4 8 MB of free RAM
• FAT formatted Flash as mass storage
• GSM, GPRS, UMTS, Bluetooth, IrDA (although
  getting phased out), WLAN
• Display 176x208, LCD panel or sometimes
  touch-screen
• Integrated camera
• 20 million shipped by Nokia alone
• In other words A pretty capable computer
• ...with an always-on Internet connection
• ... carried with millions of people, all the time
• ...that you can write own software for
• ...IF you spend enough effort
• And the numbers are just going up faster
  processors, more memory, and lots more available
  devices.

4
Programmable Phones Operating Systems

• Symbian
• Many platform dependent flavors (UIQ, Series 40,
  60, 80, )
• Independent company but partially owned by
  Nokia
• Microsoft PocketPC / Smartphone
• Aggressive Marketing, Vendor Connections, and
• Linux
• rare and invisible
• Motorola, maybe Nokia in the future?

5
Programmable Phones Languages

• Java (MIDP)
• Major Sandboxing
• Functionality growing with MIDP2, but still not
  quite there
• ie bluetooth (!), but no phone control, no cell
  tower information, etc.
• C (Symbian)
• Very steep learning curve
• Frustrating features
• Designed for serious developers
• and now
• PYTHON!

6
So Whats So Special About Python?
Image from Michele Marchetti, Python brings
application ideas to life NRC 2004

• Cross Platform
• Open Source
• Successful (Google, NASA, etc)
• Scripting Language
• Extending and embedding abilities
• Good standard library
• Reasonable memory footprint
• Access to full phone functionality

import appuifw appuifw.note(uHello World!)
7
Snake in 93 lines!
Images from Kari Pulli, Rapid Development on
PyS60 2006
8
The Goal for this Talk

• Become a Mobile Phone Hacker!
• Write applications that send SMS messages
• Turn the phone's microphone on and record audio
  files to the memory card.
• Develop an Address Book application that has more
  functionality than the built-in Contacts app.
• Write a program that has a GUI that automatically
  changes depending on the user's location
• Be able to write a mobile phone virus (but don't).

9
Day 1 Exercises User Interfaces

• Hello World
• Making Sure Everyone Can Program the Phone
• Hail!
• Introduction to Queries
• Class Survey
• Advanced UIs
• file i/o

10
Day 2 Application Development

• Building Applications
• Title, Screen Size, Tabs,
• GUI Design
• Customizing Your Own Graphical User Interfaces
• Graphics and Drawing
• Keyboard Keys
• XML
• Contacts and Calendar Databases

11
Day 3 Using the Phone as a Sensor

• OS Read and Writes
• File IO
• System Information
• Sound Recording and Playing
• Call Logs
• .wav processing?
• Location
• Logging the Cell Towers
• BT GPS Interface
• Imaging
• Image Capture using the Camera
• Image Handling
• Bluetooth Sensing
• Identifying Who is Around

12
Extra Exercises

• Networking
• Creating a Logging Script?
• Mobile Phone Virus?
• Standalone Apps py2sis

13
A Taste of Python STRINGS
From Guido van Rossums tutorial
http//www.python.org/doc/essays/ppt/lwnyc2002/int
ro22.ppt

• Guido van Rossums Intro to Python, 2002
• "hello""world" "helloworld" concatenation
• "hello"3 "hellohellohello" repetition
• "hello"0 "h" indexing
• "hello"-1 "o" (from end)
• "hello"14 "ell" slicing
• len("hello") 5 size
• "hello" lt "jello" 1 comparison
• "e" in "hello" 1 search

14
A Taste of Python LISTS
From Guido van Rossums tutorial
http//www.python.org/doc/essays/ppt/lwnyc2002/int
ro22.ppt

• Flexible arrays, not Lisp-like linked lists
• a 99, "bottles of beer", "on", "the",
  "wall"
• Same operators as for strings
• ab, a3, a0, a-1, a1, len(a)
• Item and slice assignment
• a0 98
• a12 "bottles", "of", "beer"
• 98, "bottles", "of", "beer", "on", "the",
  "wall"
• del a-1 -gt 98, "bottles", "of", "beer"

15
A Taste of Python LIST METHODS
From Guido van Rossums tutorial
http//www.python.org/doc/essays/ppt/lwnyc2002/int
ro22.ppt

• gtgtgt a range(5) 0,1,2,3,4
• gtgtgt a.append(5) 0,1,2,3,4,5
• gtgtgt a.pop() 0,1,2,3,4
• 5
• gtgtgt a.insert(0, 42) 42,0,1,2,3,4
• gtgtgt a.pop(0) 0,1,2,3,4
• 42
• gtgtgt a.reverse() 4,3,2,1,0
• gtgtgt a.sort() 0,1,2,3,4

16
A Taste of Python Dictionaries

• Dictionaries
• Associative Arrays / Hash Tables of Keys and
  Items
• d "Brand" "Nokia", "Model" "6600"
• Methods keys(), values(), items(), has_key()
• Look Up
• d"Model"
• 6600
• Delete
• del d"Brand"

17
A Taste of Python Indenting
From Guido van Rossums tutorial
http//www.python.org/doc/essays/ppt/lwnyc2002/int
ro22.ppt

• Control Structures
• Indenting instead of braces
• Easy, but Dangerous

0 Bingo! --- --- --- 3 --- --- --- 6 --- --- --- 9
--- --- --- 12 --- --- --- 15 Bingo! --- --- ---
18 --- ---
/ in C / for (i 0 i lt 20 i) if (i3
0) printf("d\n", i) if
(i5 0) printf("Bingo!\n")
printf("---\n")
in Python for i in range(20) if i3
0 print i if i5 0
print "Bingo!" print "---"
18
A Taste of Python Functions and Modules

• Functions
• Example greetings.py
• Modules
• Example import appuifw
• Check out what you get with dir
• ie dir(appuifw)

def greetings() "this is an example
function" name appuifw.querry(u'What is
your name?','text') appuifw.note(u'Hail 'name,
'conf') greetings()
19
The appuifw Module

• Queries

• Forms

20
The Better Way to Learn HACK!

• Download the API Reference here
• http//reality.media.mit.edu/pdfs/pys60_api.pdf
• Get a computer set up with the phone emulator and
  Python
• S60 SDK for 2nd Edition, FP 2 http//www.forum.nok
  ia.com/main/0,,034-483,00.html
• Python for S60 SDK http//www.forum.nokia.com/ma
  in/0,,034-821,00.html

21
Running Your Own Scripts

• Transferring Scripts from PC to Phone
• Bluetooth Send / Sync / Console
• Infrared Transfer
• Memory Card Transfer
• GPRS Transfer
• Using the Symbian Emulator
• Path for .py scripts (or something similar)
• C\Symbian\8.0a\S60_2nd_FP2\epoc32\release\wins\ud
  eb\z\system\APPS\python
• Run Emulator in debug mode
• Open Python -gt Options -gtRun Scripts

22

• Part 2 Application Development

23

• DOWNLOAD
• http//reality.media.mit.edu/code/nairobi_code.zip
  
• Path to put .py files
• C\Symbian\8.0a\S60_2nd_FP2\epoc32\release\wins\ud
  eb\z\system\APPS\python

24
Day 2 Application Development

• Creating Our Own Modules
• Building Applications
• Title, Screen Size, Tabs, Threads, Body, Menus
• GUI Design
• Customizing Your Own Graphical User Interfaces
• Graphics and Drawing
• Keyboard Keys
• XML
• Contacts and Calendar Databases

25
Get More User Input
import appuifw planets uMars, uEarth,
uVenus prompt uEnter your home planet
index appuifw.menu(planets, prompt)
appuifw.note(uHello planetsindex , uinfo)

• The menu method pops up the list of items in
  first param
• It returns with an index into the list of menu
  items
• Note that the prompt param must also be a unicode
  string

26
Our Own Interface

• NAIROBI.py
• There are a bunch of annoyances in the current UI
• Lets put wrappers around basic calls
• We should go back and do this for location

27
Hiding the Unicode
Example from Larry Rudolph's Intro to Python
slides
this is file nairobi.py wrappers to
appuifw def note( str , type
info) appuifw.note( unicode(str), type )
def query( str , type text ) return
appuifw.query( unicode(str), type ) def menu(
list, prompt select one ) ulist
unicode(u) for u in list return
appuifw.menu( ulist , unicode( prompt ) )
28
Using nairobi.py
import nairobi.py planets Mars, Earth,
Venus prompt Enter your home planet
index nairobi.menu(planets, prompt)
nairobi.note(Hello planetsindex )
29
appuifw.app

• Appuifw contains an instance of the class
  application, called app

30
The 9-Steps to Application Development

• 1. import all modules needed
• 2. set the screen size (normal, large, full)
• 3. create your application logic ...
• 4. create an application menu (if necessary)
• 5. set an exit key handler
• 6. set the application title
• 7. deal with active objects if necessary
• 8. set the application body (text or canvas or
  listbox or none)
• 9. create a main loop (e.g. while loop) if
  suitable

31
1. Importing Modules

• import appuifw
• import e32

32
2. Setting Screen Size

• screen has 3 different values
• (a normal screen with title pane and softkeys
• appuifw.app.screen'normal'
• (only softkeys visible)
• appuifw.app.screen'large'
• (a full screen)
• appuifw.app.screen'full'
• Example script app_screen.py

33
3. Application Logic

• This is where the heart of your application lies.

34
4. Application Menus

• An application menu uses the left softkey and can
  always be accessed while your application is
  running. An application menu can contain also a
  submenu
• create the callback functions that shall be
  executed when when selecting an item in the
  menu def item1()    print "item one"
• def subitem1()    print "subitem one"
• def subitem2()    print "subitem two"
• create the menu using appuifw.app.menu(title,
  callback1), (title, (subtitle, callback2))
• appuifw.app.menu (u"item 1", item1),
  (u"Submenu 1", ((u"sub item 1", subitem1), (u"sub
  item 2", subitem2)))
• Example script app_menu.py

35
5. The Exit Key Handler

• The exitkey handler gets activated when you press
  the right (exit) softkey. By assigning an extra
  function to the .exit_key_handler you can define
  what shall happen when it is pressed.
• def quit()    appuifw.app.set_exit()
• app.exit_key_handlerquit

36
6. The Application Title

• appuifw.app.title u"SMS sending"

37
7.0 UI Threads

• places objects on screen
• registers callbacks procedures associated with
  screen keyboard events
• when event occurs, want to pass control to the
  callback procedure.
• what if thread is executing something else?
• Callbacks should execute quickly
• UI thread should spend most of the time idle

38
7.1 e32 module Coordination
Example from Larry Rudolph's Intro to Python
slides

• Dont use normal thread locks
• import thread
• lock thread.allocate_lock()
• Whole application gets blocked, since no UI
  actions would be handled
• Use e32.Ao_lock instead

39
7.2 Active Objects

• If Symbian written today, AOs would be called
  listeners
• Get called by a thread scheduler (have a little
  bit of state)
• Run to completion then return to scheduler
• They preserve the responsiveness of the UI and
  sockets
• You need to import the e32 module import e32
• create an instance of the active object
  app_lock e32.Ao_lock()
• starts a scheduler -gt the script processes
  events (e.g. from the UI) until lock.signal() is
  callled.app_lock.wait()
• stops the scheduler app_lock.signal()
• For more detail see the python_api.pdf.

40
8. Application Body

• text or canvas or listbox or none
• body as Listboxappuifw.app.body
  appuifw.Listbox(entries,shout) Example script
  app_body_listbox.py
• body as Textappuifw.app.body
  appuifw.Text(u'hello') Example script
  app_body_text.py
• body as Canvasappuifw.app.bodyappuifw.Canvas(
  event_callbackNone, redraw_callbackhandle_redraw
  ) Example script app_body_canvas.py

41
9. The Main Loop

• put in the main loop the things that need to be
  run through again and again in your script
• running 1
• while running     e.g. redraw the
  screen    handle_redraw(())

42
Application Skeletons

• 1. no main loop because the application logic
  works without
• Example script app_skeleton.py
•  
• 2. with mainloop (if suitable)
• Example script app_skeleton_with_mainloop.py

43
Day 2.5-3 Phone As Sensor

• OS Read and Writes
• File IO
• System Information
• Sound Recording and Playing
• Call Logs
• .wav processing?
• Location
• Logging the Cell Towers
• BT GPS Interface
• Imaging
• Image Capture using the Camera
• Image Handling
• Bluetooth Sensing
• Identifying Who is Around

44
OS Reads and Writes

• Get the Contents of a Directory
• import e32
• import os
• define the directory to read
• imagediru'c\\nokia\images'
• read the directory
• filesmap(unicode,os.listdir(imagedir))
•  
• Example script os_dir_read.py

45
Reading

• Read an image into a variable
• import e32import os
• read the directoryfopen('c\\nokia\\images\\
  testimg.jpg'','rb')test f.read()f.close()

46
Writing Files

• define the directory and file name to write the
  file intoimagediru'c\\writetest.txt'
• epoc32\wins\c
• create the filefile open(imagedir,'w')
• write some text into itfile.write('hello, this
  works')
• close the filefile.close()
• Example script os_dir_write.py

47
Reading and Writing OS Settings

• Write and read settings to/from the OS
• Write several variables with attached values as
  settings into a file on the OS, and also read the
  them.
• SYSINFO
• import sysinfo
• Sysinfo.battery(), imei, free_space,total_ram,
  signal()
• SEE PYTHON API!

48
Sound Recording and Playback

• Program an application that can record and play a
  sound, controlled from the application menu
• import audio
• audio.Sound.play() for playing the sound
• audio.Sound.record() to record
• 2. You need to open and close sound files by
  using
• audio.Sound.open(filename)
• audio.Sound.close()
• Example ex_soundrecorder_descr.py

49
Bluetooth Scans Who is Around?

• import socket
• socket.bt_discover()
• See the python_api for more details

50
The Camera

• Program an application that lets you take a
  picture. The picture shall be stored on the c
  drive.
• 1. from graphics import and import camera
• 2. Use the SELECT key to take the picture.
• 3. Use the LeftSoftKey to to activate the camera
  mode again.
• Example ex_camera_descr.py
• Check out the python_api.pdf for more
  parameters! (exposure, white balance, .)

51
Image Handling

• Play with Nokia's Example Scripts
• Image Rotation
• Example script image_rotation.py
• Image Viewing
• Example script imageviewer.py

52
Location Messaging

• Location
• import location
• Mobile Country Code, Mobile Network Code,
• Location Area Code, and Cell ID
• (mcc, mnc, lac, cellid) location.gsm_location()
• SMS Messaging
• import messaging
• sms_send(recipient, message)

53
Exercise

• Create an Application that Sends Your Phone's
  Location by SMS
• Use Example ex_sms_sending_descr.py
• Email me if you are interested in a summer
  position
• nathan_at_media.mit.edu

54

• Mobile Application Prototyping with Python
• A 3-Day Crash Course for the University of
  Nairobi
• Day 3 Bluetooth, GPRS, Keyboard

Nathan Eagle, PhD Research Scientist MIT Design
Laboratory Massachusetts Institute of Technology
July 3-6 2006 School of Computing and
Informatics University of Nairobi
55
Day 3 Wrap-Up

• Programming with the Help of Bluetooth
• The BT Console
• BT Sync
• Networking with GPRS
• Download
• Upload
• TCP/IP
• Keyboard Commands

56
Logging into your Phone Bluetooth Console

• BT Console let's you log into your phone!
• Terminal Client Needed
• HyperTerminal, SecureCRT, etc
• COM port

57
Logging into your Phone Bluetooth Sync

• Using file sync from PC to phone
• Written in Python for the PC
• Backend requires win32com and PySerial
• UI requires wxPython
• Uses PySerial on the PC side.
• http//people.csail.mit.edu/kapu/symbian/python.ht
  ml.
• No need to keep pushing .py scripts into your
  inbox!
• Has simple shell capabilities (ls, cat, rm)
• Screen snapshots for real-time demos!

58
Networking with GPRS
import urllib url "http//weather.gov/mdl/radar/
rcm1pix_b.gif"tempfile "c\\testimg.gif"urllib
.urlretrieve(url, tempfile)

• Downloading Media Content
• import urllib
• urllib.urlretrieve(url, tempfile)
• Handling the Content Playback
• content_handler.open(tempfile)
• rss, images, video, audio, etc
• Example
• urllib_example.py
• ex_video_download_descr.py

59
GRPS Uploading

• Why?
• Automatic Data Collection
• Mobile Blogging
• Database Interactions
• Modules
• urllib, httplib, ftplib (install)
• Examples
• ftp_example.py
• ex_image_upload_to_database.py

60
TCP/IP Pushing Data through Sockets

• Connect directly to your desktop computer

Phone Script import socket HOST
'217.30.180.11'desktop's IP PORT 12008 Port
number print "define socket" s
socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print "trying to connect to socket"
s.connect((HOST, PORT)) print "connected"
s.send('Hello, world') print "data send" data
s.recv(1024) s.close() print 'Received', data
Desktop Script HOST '' meaning the local
host PORT 12008 Arbitrary port print "define
the socket" s socket.socket(socket.AF_INET,
socket.SOCK_STREAM) print "bind the
socket" s.bind((HOST, PORT)) s.listen(1) print
"waiting of the client to connect" conn, addr
s.accept() print 'Connected by', addr while
1data conn.recv(1024) if not data break
conn.send(data) conn.close()
61
The Keyboard

• import keys
• from key_codes import
• Example ex_use_of_keys_descr.py

62
Keyboard Exercises

• Program an application that uses keyboard to
  trigger pop-up notes telling you what key has
  been pressed.
• while running
• if keyboard.pressed(EScancodeLeftArrow)       
   appuifw.note(u"Arrow left", "info")
• Create "Easter Egg" Codes in Your Application
  that trigger events if the user presses the right
  code
• Extra Credit Modify an application to accept
  keyboard commands to trigger functions rather
  than menus.