Code Intelligence in Komodo

1
Code Intelligence in Komodo

• Trent Mick
• TrentM_at_ActiveState.com

2
Code Intelligence is

• a Code Browser
• autocomplete and call tips

3
What in it for me?

• See how an IDE Code Browser can be built
• Learn about some useful Python packages
• Couldnt find Mitch Kapor and lunch isnt served
  yet, so

4
Code Intelligence System Overview
Komodo
Python CILE
Perl CILE
Manager API
PHP CILE
Scheduler
Tcl CILE
Manager
XSLT CILE
SQL
C/C CILE
CIDB
COM CILE
CIX (XML)
Java CILE
5
Application
Komodo
Python CILE
Python CILE
Perl CILE
Perl CILE
Manager API
PHP CILE
Scheduler
Tcl CILE
Manager
XSLT CILE
SQL
C/C CILE
CIDB
COM CILE
CIX (XML)
Java CILE
6
What were going for
Code Browser
Autocomplete
Call tips
7
Database
Komodo
Python CILE
Python CILE
Perl CILE
Perl CILE
Manager API
PHP CILE
Scheduler
Tcl CILE
Manager
XSLT CILE
SQL
C/C CILE
CIDB
COM CILE
CIX (XML)
Java CILE
8
Database

• SQLite for database
• Separate Files and Modules for Perl and Tcl
• Allow multiple scanners per file (e.g. Python,
  COM, and PyXPCOM)
• Tip include a Meta table

Meta
Files
Scans
Modules
Symbols
9
XML description format (CIX)
Komodo
Python CILE
Python CILE
Perl CILE
Perl CILE
Manager API
PHP CILE
Scheduler
Tcl CILE
Manager
XSLT CILE
C/C CILE
CIDB
COM CILE
CIX (XML)
Java CILE
10
XML description format (CIX)
       ltcodeintel version"0.1"gt
           ltfile path"foo.py" generator"Python"
                 language"Python"
                 md5""gt                ...
           lt/filegt        lt/codeintelgt

• language-agnostic
• storable and distributable

11
XML description format (CIX)
       ltcodeintel version"0.1"gt
           ltfile path"string.py" ...gt
               ltmodule name"string"gt
                   ltfunction line"199" name"ato
f"gt                        ltdocgtlt!CDATAatof(s) 
-gt float ...gtlt/docgt                        ltarg
ument name"s"/gt                    lt/functiongt
               lt/modulegt            lt/filegt   
    lt/codeintelgt

• ltmodulegt, ltfunctiongt, ltclassgt, ltinterfacegt,
  ltargumentgt, ltdocgt, ltvariablegt, etc. tags define
  structure of the code

12
Language Engines (CILEs)
Komodo
Python CILE
Python CILE
Perl CILE
Perl CILE
Manager API
PHP CILE
Scheduler
Tcl CILE
Manager
XSLT CILE
SQL
C/C CILE
CIDB
COM CILE
CIX (XML)
Java CILE
13
Python Language Engine
Python Source Code
 198     Convert string to float
 199    def atof(s)  200        """atof(s) -gt f
loat  201      202        Return the floating po
int   203      204        """
 205        return _float(s)
AST from compiler
Function('atof', 's', , 0, 'atof(s) -gt
float...', Stmt( Return(
CallFunc(Name('_float'), Name('s'), )))
CIX
ltmodule name"string"gt      ltfunction line"199" 
name"atof"gt           ltdocgtlt!CDATAatof(s) -gt f
loat ...gtlt/docgt           ltargument name"s"/gt
     lt/functiongt lt/modulegt
14
demo
15
PHP Language Engine
PHP Source Code
   2    lt?php    3     atof(s) -gt float
   4     ...    5    function atof(s) 
   6        return _float(s)    7    
   8    ?gt
parse with regular expressions
CIX
ltmodule name"string.php"gt      ltfunction line"1
99" name"atof"gt           ltdocgtlt!CDATAatof(s) 
-gt float ...gtlt/docgt           ltargument name"s
"/gt      lt/functiongt lt/modulegt
16
demo
17
Perl, Tcl, XSLT Language Engines

• The only thing that can parse Perl is Perl (but
  scintilla comes close)
• Tcl can parse Tcl
• Tclwrap can make an executable from Tcl
• libxml can parse XSLT

18
Scheduler
Komodo
Python CILE
Python CILE
Perl CILE
Perl CILE
Manager API
PHP CILE
Scheduler
Tcl CILE
Manager
XSLT CILE
SQL
C/C CILE
CIDB
COM CILE
CIX (XML)
Java CILE
19
Scheduler

• Info must update dynamically as the user types,
  but not interfere with editing i.e. asynchronous
• Scheduler is a thread that manages a small pool
  of Scan Runner threads to scan file content.

20
demo
21
Tip import Queue

• From the Python Library Reference

The Queue module implements a multi-producer,
multi-consumer FIFO queue. It is especially
useful in threads programming when information
must be exchanged safely between multiple threads.
Manager
Scheduler
s Scheduler() s.queue.put("scan
string.py") s.queue.put("scan string.php")
while 1 request self.queue.get()
process request...
22
Tip import Queue

• This is a Priority Queue

class PriorityQueue(Queue.Queue) def
_put(self, item) bisect.insort(self.queue
, item)
23
Autocomplete/Calltips

• Type inferencing
• c.f. Mike Salibs Starkiller talk tomorrow
• Automatic triggering (language dependent)
• Python foo.
• Perl foo-gt
• PHP pri

24
Possibilities

• Other language engines
• C/C, Java, COM, XML/XHTML
• Distributing CIX files for binary packages
• Custom user data in the database
• method frequency tracking

25
Thank you

• Questions?
• Trent Mick (TrentM_at_ActiveState.com)