Stackless Python in EVE

1
Stackless Python in EVE

• Kristján Valur Jónsson
• kristjan_at_ccpgames.com
• CCP Games inc.

2
EVE

• MMORPG Space game
• Client / server
• Single shard massive server
• 120.000 active players, gt24.000 concurrent users
• World concurrency record on a shard
• Relies on Stackless Python

3
The Tranquility cluster

• 400 GHz CPU / 200 Gb RAM
• 2 Routers (CISCO Alteon)
• 14 Proxy servers (IBM Blade)
• 55 Sol servers (IBM x335)
• 2 DB servers (clustered, IBM Brick x445)
• FastT600 Fiber, 56 x FC 15k disks, DS4300
  3EXP700
• Windows 2000, MS SQL Server
• Currently being upgraded
• AMD x64

4
EVE Architecture

• COM-like basic architecture
• Python tighly integrated at an early stage
• Home-grown wrapping of BLUE objects

5
Stackless Python

• Tasklets
• Threads of execution. Not OS threads
• Lightweight
• No pre-emption
• Channels
• Tasklet rendezvous point
• Data passing
• Scheduling
• Synchronization

6
Stackless?

• No C stack
• Python stack in linked frame objects
• Tasklet switching by swapping frame chain
• Compromise
• stackless where possible.
• C stack whisked away if necessary

7
Channels
8
Channel semantics

• Send on a channel with no receiver blocks
  tasklet.
• Send on a channel with a (blocked) receiver,
  suspends tasklet and runs receiver immediately.
  Sender runs again in due course.
• Symmetric wrt. Send and Receive.
• balance, can have a queue of readers or
  writers.
• Conceptually similar to Unix pipes

9
Channel semantics, cont.

• Scheduling semantics are precise
• A blocked tasklet is run immediately
• Usable as a building block
• semaphores
• mutex
• critical section
• condition variables

10
Stackless in EVE

• BLUE foundation robust, but cumbersome
• RAD
• Stackless Python Python and so much more
• EVE is inconceivable without Stackless
• Everyone is a programmer

11
The main loop

• Establish stackless context

int WinMain(...) PyObject myApp new
EveApp() PyObject r PyStackless_CallMethod_Mai
n(MyApp, WinMain, 0) return
PyInt_AsLong( r )
12
The main loop cont.
PyObject EveAppWinMain(PyObject self,
PyObject args) PyOS-gtExecFile("script/sys/
autoexec.py") MSG msg
while(PeekMessage(msg, 0, 0, 0,
PM_REMOVE)) TranslateMessage(msg) DispatchMes
sage(msg) for (TickIt i
mTickers.begin( i ! mTickers.end()
i) i-gtmCb-gtOnTick(mTime, (void)taskname)

• Regular Windows message loop
• Runs in Stackless context
• The Main Tasklet

13
Autoexec.py

import blue def Startup() import service
srvMng service.ServiceManager() run
"dataconfig", "godma", ui",
srvMng.Run(run) Start up the client in a
tasklet! if CheckDXVersion() import blue
blue.pyos.CreateTasklet(Startup, (), )
14
Tickers

• Tickers are BLUE modules
• Trinity (the renderer)
• Netclient
• DB (on the server)
• Audio
• PyOS (special python services)

15
The PyOS tick

• Runs fresh tasklets
• (sleepers awoken elsewhere)

Tick() mSynchro-gtTick() PyObject
watchdogResult do watchdogResult
PyStackless_RunWatchdog(20000000) if
(!watchdogResult) PyFlushError("PumpPythonWatc
hdog") Py_XDECREF(watchdogresult) while
(!watchdogResult)
16
blue.pyos.synchro

• Synchro
• Provides Thread-like tasklet utilities
• Sleep(ms)
• Yield()
• BeNice()

17
blue.pyos.synchro cont.

• Sleep A python script makes the call
  blue.pyos.Sleep(200)
• C code runs
• Main tasklet check
• sleeper New Sleeper()mSleepers.insert(sleeper)
  PyObject r PyChannel_Receive(sleeper-gtmChanne
  l)
• Another tasklet runs

18
blue.pyos.synchro, ctd.

• Main tasklet in windows loop enters PyOSTick()
• mSleepers are examined for all that are due we
  do
• mSleepers.remove(sleeper)PyChannel_Send(sleeper
  s.mChannel, Py_NONE)
• Main tasklet is suspended (but runnable), sleeper
  runs.

19
Points to note

• A tasklet goes to sleep by calling
  PyChannel_Receive() on a channel which has no
  pending sender.
• It will sleep there (block) until someone sends
• Typically the main tasklet does this, doing
  PyChannel_Send() on a channel with a reader
• Ergo The main tasklet may not block

20
Socket Receive

• Use Windows asynchronous file API
• Provide a synchronous python API. A python
  script calls Read().
• Tasklet may be blocked for a long time, (many
  frames) other tasklets continue running.
• Do this using channels.

21
Receive, cont.

• Python script runsfoo, bar socket.Read()
• C code executes the request
• Request r new Request(this)WSAReceive(mSocke
  t, )mServe-gtinsert( r )PyChannel_Receive(r-gtm
  Channel)
• Tasklet is suspended

22
Receive, cont.

• Socket server is ticked from main loop
• For all requests that are marked completed, it
  transfers the data to the sleeping tasklets
• PyObject r PyString_FromStringAndSize(req-gtmDa
  ta, req-gtmDataLen)PyChannel_Send(req-gtmChannel,
  r)Py_DECREF(data)delete req
• The sleeping tasklet wakes up, main tasklet is
  suspended (but runnable)

23
Receive completed
24
Main Tasklet

• The one running the windows loop
• Can be suspended, allowing other tasklets to run
• Can be blocked, as long as there is another
  tasklet to unblock it (dangerous)
• Is responsible for waking up Sleepers, Yielders,
  IO tasklets, etc. therefore cannot be one of them
• Is flagged as non-blockable (stackless.get_current
  ().block_trap True)

25
Channel magic

• Channels perform the stackless context switch.
• If there is a C stack in the call chain, it will
  magically swap the stacks.
• Your entire C stack (with C and python
  invocations) is whisked away and stored, to be
  replaced with a new one.
• This allows stackless to simulate cooperative
  multi-threading

26
Co-operative multitasking

• Context is switched only at known points.
• In Stakcless, this is channel.send() and
  channel.receive()
• Also synchro.Yield(), synchro.Sleep(), BeNice(),
  socket and DB ops, etc.
• No unexpected context switches
• Almost no race conditions
• Program like you are single-threaded
• Very few exceptions.
• This extends to C state too!

27
Tasklets

• Tasklets are cheap
• Used liberally to reduce perceived lag
• UI events forked out to tasklets
• A click can have heavy consequences.
• Heavy logic
• DB Access
• Networks access
• special rendering tasks forked out to tasklets.
• controlling an audio track
• tasklet it out
• Use blue.pyos.synchro.BeNice() in large loops

28
Example UI Event

• Main tasklet receives window messages such as
  WM_CLICK
• Trinity invokes handler on UI elements or global
  handler
• Handler tasklets out any action to allow main
  thread to continue immediately.

def OnGlobalUp(self, args) if not self or
self.destroyed return mo
eve.triapp.uilib.mouseOver if mo in
self.children uthread.new(mo._OnClick)
class Action(xtriui.QuickDeco) def
_OnClick(self, args) pass
29
Thats all

• For more info
• http//www.ccpgames.com
• http//www.eve-online.com
• http//www.stackless.com
• kristjan_at_ccpgames.com