Interactive Data Language IDL

1
Interactive Data Language (IDL)

• Margit Haberreiter
• (haberreiter_at_lasp.colorado.edu)
• LASP, room 135
• Acknowledgement Marty Snow

2
Outline

• What is IDL?
• Basic Syntax
• Functions, Procedures, and Library
• Reading Writing
• Plotting Printing
• Image Processing
• And Stuff.

3

• Pro test
• Save it as test.pro
• .compile filename
• filename

4
Interactive Data Language

• IDL was born here at LASP
• Very popular among scientists
• Handles arrays of data easily (spectra, time
  series, images, etc.)
• Easy to learn
• Portable
• Large existing user library
• Online Help type ? at prompt
• Type help at prompt for contents of variables
• Interactive or Compiled, your choice!

5
Data Types

• Integer (and Long Integer)
• Floating Point (and Double Precision)
• String
• Structure
• Pointer
• Also, complex, unsigned integers, etc.

IDL is dynamically typed which means a variable
can be defined as one type, and then be changed
to another within the same program unit.
6
Scalars, Vectors, Arrays Problem 1

• Scalar
• 42 a5 b0. c0.d
• help!
• Vector
• 1,2,3,5
• red,blue,happy,Burma Shave
• Array
• Up to 8 dimensions

Definition of arrays arr1intarr(3)
arr2fltarr(2,a) arr3dblarr(a,b,c,d) arr4str
arr(11)
7
Structures - Problem 2

• Can hold mixed data types
• Sday0,time0d0,label
• S.day5
• S.labelhappy time
• Can make arrays of structures
• S2replicate(time0d0,label,valuesfltarr(100)
  ,n_obs)
• S24.valuesfindgen(100)
• Names or Numbers
• S2.time is equivalent to S2.(0)

8
Rules of the Road

• Always start counting at zero.
• For help at any time, type help.
• Case Insensitive (EXC filenames).
• Odd is true, Even or Zero is false.
• An array of length 1 is not the same as a scalar.
  But you can reform it.

9
Squiggly or Square?

• Subscripting variables -- square
• Parameters of a function () round
• Definition of a structure curly

IDL is lax about enforcing these rules, but it is
to your benefit to follow them. Example
aexp5 6th element of variable
exp aexp(5) e5 You might have named a
variable the same name as a library function!
10
Subscripting - Problem 3

• za5
• za02
• za3
• mask5,6,7,8,9 zamask
• Rangewhere(mask gt 2 and mask le 4)
• Mask2mask(range)

aindgen(10) za5 za02 za3 mask5,6,7,
8,9 zamask Rangewhere((mask gt 6) and
(mask le 8)) Mask2mask(range)
11
The Size of Arrays

• Size(x)
• n_elements returns the number of elements in an
  array (vector, scalar).
• An undefined variable has zero elements.
• Often takes the place of size().
• Used to check if user has supplied parameters
• Used to index loops.

12
Basic IDL Syntax Problem 4

• aHello World
• print,a
• This is a comment
• a5dynamic data types
• b4.5findgen(100)
• za continuation
• sqrt(b)

13
Operators

• Arithmetic
• -
• /
• mod
• Min/Max
• lt (altb is the lesser)
• gt (agtb is the greater)
• Matrix Multiply

• Boolean Logic
• and
• not
• or
• xor
• Relational Operators
• eq (ab)
• ge (a?b)
• gt (a?b)
• le (a?b)
• lt (a?b)
• ne (a?b)

Note that you cant use any of these letter
combinations as variable names!
14
FOR loop

• for variableinit,limit,increment do statement
• for i0,100 do print,i,vectori
• for variableinit,limit,increment do begin
• statements
• endfor

Loops in IDL execute slowly (relatively
speaking), so try to use something else, like
where.
15
As IF. Problem 5

• if expression then statement
• if expression then statement else statement2
• if expression then begin
• statements
• endif else begin
• statements
• endelse

for x0,5 do begin if x gt 5 then
begin zdblarr(10,10) zz.1 z3,21. endif
else begin zdblarr(10) zx!pi endelse endfor
Good programming style ? avoid hardcoded numbers
in procedures
16
meanWHILE

• while expression do statement
• while expression do begin
• statements
• endwhile

17
just in CASE

• case expression of
• expression1 statement
• expression2 begin
• statements
• end
• expression3 statement
• else statement
• endcase

• Example
• x3
• CASE x OF
• 1 PRINT, 'one'
• 2 PRINT, 'two'
• 3 PRINT, 'three'
• 4 PRINT, 'four'
• ENDCASE

18
Other Flow Control Statements

• STOP excellent way to debug (.con or .c to
  continue)
• GOTO
• goto,label
• label
• SWITCH
• CONTINUE
• BREAK
• MESSAGE stops program and prints message

IDL Help This example illustrates how, unlike
CASE, SWITCH executes the first matching
statement and any following statements in the
SWITCH block x2 SWITCH x OF     1 PRINT,
'one'     2 PRINT, 'two'     3 PRINT, 'three'
    4 PRINT, 'four' ENDSWITCH
19
Procedures and Functions

• A list of IDL statements that could have been
  entered at the command prompt.
• Contained in separate file or files.
• Compiled automatically if files are properly
  named. Filename matches procedure with a .pro
  extension. USE LOWER CASE!
• All procedures compiled from start of file but
  stops when procedure name matches file name.
• Environmental variables define directories that
  IDL will automatically search. USER LIBRARY

20
Compile and Run

• Type procedure name at command prompt to run it.
  IDL will search for an appropriately named file
  and automatically compile and run the procedure.
• .r or .compile will compile the procedure, but
  not run it.
• IDL will compile all procedures in file until it
  hits the one matching the file name, then it will
  stop. Put the named procedure last in the file.

21
Simple Procedure

• pro simpleprocedure,a,b,c
• cab
• end

To run it, type simpleprocedure,var1,var2,result
print,result
22
Simple Function

• function eve,number
• if number mod 2 eq 0 then return,0
• return,1
• end

To run it, type resulteve(var) print,eve(var) pr
int, The result is , result
23
Whats the difference?

• Not much.
• Use a function to get a value that youll use
  right away (like an expression).
• EXAMPLE
• simpleprocedure,eve(0),eve(5),result
• Use a procedure when you return multiple results
  and dont want to just wrap them together in a
  structure or array.

24
Parameters and Keywords Problem 7

• pro proc2,input1,input2,result,doplotdoplot
• if n_elements(doplot) eq 0 then doplot0
• resultinput1sin(input2/!pi)
• if doplot gt 0 then begin
• set_plot,'win'
• !P.CHARSIZE3. system setting
• !P.MULTI0,1,3 system setting
• plot,input2,result,psym-4
• plot,input1
• plot,sin(input2/!pi)
• endif
• end

Order of Parameters is critical. Order of
Keywords is irrelevant. Keywords can normally be
abbreviated xtit instead of xtitle xs instead of
xstyle
25
Variables Global or Local? Problem 8

• Variables are local.sort of (Attention!!)
• A procedure can modify any of its parameters,
  which will change the variable in the calling
  procedure.

pro proc1 v0 print,v proc3,v print,v end
pro proc3,var2 var2var21 end
26
Error and stop

• If the IDL interpreter hits a STOP or encounters
  an error, flow halts at the local level. You can
  type help to view contents of variables and
  which line of code youre at.
• To return to the top level retall (return all
  the way)
• To clear everything, type .f (full reset) or
  .reset -gt Attention all variables deleted

27
Reading Writing Saving

• openr,openw,openu (Read, Write, Update)
• readf,readu (Formatted, Unformatted)
• close or free_lun
• IDL save files
• save,filedata.idl,var1,var2
• restore,data.idl

openr,lu,file.txt,/get_lun while not eof(lu) do
begin readf,lu,data endwhile free_lun,lu
28
Slight Gotcha
pro wont_work openr,lu,file.txt,/get_lun datafl
tarr(100) for i0,n_elements(data)-1 do
begin readf,lu,datai endfor free_lun,lu end
IDL passes by value, not by reference, so this
wont work.
29
Slight Gotcha
pro will_work openr,lu,file.txt,/get_lun datafl
tarr(100) i0 while not eof(lu) do
begin readf,lu,temporary dataitemporary ii1
counter endwhile free_lun,lu datadata0i-1 end
IDL passes by value, so this will make IDL happy.
30
Task Read data from file Problem 9

• pro willwork,filename,data
• openr,lu,filename,/get_lun get_lun sets
  file unit number
• datafltarr(2,1000)
• i0
• while not eof(lu) do begin
• readf,lu,temporary1,temporary2
• data0,itemporary1
• data1,itemporary2
• ii1 counter
• endwhile
• free_lun,lu
• datadata,0i-1
• set_plot,'win'
• !P.MULTI0
• plot,data(0,),data(1,),title'Data from file'
• end
• Run the procedure
• Add a plot using a parameter (see slide 22)

31
Homework 1LISIRD Data

• Go to http//lasp.colorado.edu/lisird and
  retrieve solar Lyman alpha (121.5 nm) data for at
  least two missions.
• Save data as text file.
• Write an IDL procedure to read data.

32
More on plottingBasic IDL Plotting Procedures

• Line plots
• plot
• oplot
• histogram
• Contour plots
• contour
• Surface plots
• surface
• shade_surf

33
Line em up Problem 10

• XFINDGEN(360)
• YSIN(X!DTOR)
• PLOT, X, Y, XRANGE0,360, /XSTYLE, XTITX,
  YTITY, TITSample Line Plot
• ZCOS(X!DTOR)
• OPLOT, X, Z, LINESTYLE2

34
Adding text to plot

• XFINDGEN(360)
• YSIN(X!DTOR)
• PLOT, X, Y, XRANGE0,360, /XSTYLE, XTIT'X',
  YTIT'Y', TIT'Sample Line Plot'
• ZCOS(X!DTOR)
• OPLOT, X, Z, LINES2
• XYOUTS, 100, 0, 'cos(x)'
• XYOUTS, 190, 0, 'sin(x)'

35
Histograms

• YRANDOMN(SEED, 100, /NORMAL)
• HHISTOGRAM(Y, BINSIZE0.2, LOCATIONSL)
• PLOT, L, H, PSYM10 histogram mode

36
Graphics Keywords

• change the style of the data
• add a title
• manipulate an axis/change tick marks
• change the format of text
• change coordinate systems
• etc.

37
Styles and Symbols Keywords

• Lines
• LINESTYLE0,1,2,3,4,5
• THICK change the thickness of the line (default
  is 1.0)

• Symbols
• PSYM1,2,3,10
• SYMSIZE change the size of the symbol
• USERSYM procedure to define PSYM8
• PSYM -1,2,3,8 solid line connecting the
  symbols
• N.B. PSYM3 (period) does NOT show up well in
  postscript plots and does not scale nicely with
  symsize. Better to use PSYM1, SYMSIZE.3 (or
  similar)
• ? Negative value of psym creates linesymbols

38
Titletown USA Keywords

• TITLE place a string of text at the top of a
  plot
• TITLEThis is my title
• TITLE_STRINGThis is my titleTITLETITLE_STRING
  
• SUBTITLE place a title below the x-axis
• XYZtitle places title on x, y, or z axis

39
Axes Tick Mark Keywords

• To manipulate individual axes
• Set the range
• XYZRANGEmin, max
• Change the axis style
• XYZSTYLEnumber
• Multiple effects can be achieved by adding values
  together
• Label an axis
• XYZTITLEstring

40
Axes Ticks Keywords 2

• To manipulate individual axes (cont.)
• Change the axis thickness
• XYZTHICKnumber (default is 1.0)
• Use axis procedure to draw axes with different
  scales
• XFINDGEN(360)
• YSIN(X!DTOR) system variable p/180
• PLOT, X, Y, XRANGE0,360, XSTYLE9, XTIT'X
  (Degrees)'
• AXIS, XAXIS1, XRANGE0,2!PI, XSTYLE1,
  XTITLE'(Radians)'

41
Clipped title
42
Fixed clipping
plot,x,y,xstyle9,xtit'X (degrees)',ymargin4,4
43
Text Formatting Keywords

• To change the size of text
• CHARSIZEnumber (default is 1.0)
• To change the thickness of the text
• CHARTHICKnumber (default is 1.0)
• Or set system variable
• !P.Charsize3

44
Coordinate systems

• IDL has 3 coordinate systems
• DATA (default)
• Uses the range of values of the data
• DEVICE
• Uses the device (X window or Postscript device)
  coordinates (i.e. pixels)
• NORMAL
• Normalized coordinates from 0 to 1 over the 3 axes

45
Adding color

• Color tables
• LOADCT, number
• XLOADCT (widget)
• Graphics keywords
• BACKGROUNDnumber
• COLORnumber
• Number from 0 to 255, corresponding to the color
  table
• colorfsc_color(red) using library routine
• On Mac, Windows DEVICE, DECOMPOSED0

46
Contour

• XFINDGEN(360)
• ZSIN(X!DTOR)COS(2X!DTOR)
• CONTOUR, Z, NLEVELS6, XRANGE0, 360,
  YRANGE0, 360, /XSTYLE, /YSTYLE, XTIT'X',
  YTIT'Y', TITSample Contour Plot

47
Surface plots (shaded surf)

• SHADE_SURF, Z, XRANGE0, 360, YRANGE0,
  360, /XSTYLE, /YSTYLE, XTIT'X', YTIT'Y',
  TIT'Sample Surface Plot 2'

48
Colored surface
loadct,3 shade_surf,z,tit'Sample Surface Plot
3',shadesbytscl(z1) Bytscl translates the
values of the array into a byte array from 0 to
255
49
Surface plots 2 (wire mesh)

• SURFACE, Z, XRANGE0, 360, YRANGE0, 360,
  /XSTYLE, /YSTYLE, XTIT'X', YTIT'Y',
  TIT'Sample Surface Plot 2'

50
Changing appearance of mesh
z2rebin(z,180,180) surface,z2
z3rebin(z,90,90) surface,z3
The possibilities are endless!
51
Multiple plots on a page

• !p.multia,b,c
• a number of plots left on page
• b number of columns
• c number of rows
• Example
• 6 plots per page (3x2), begin plotting in the top
  left
• !p.multi0,3,2
• To return to a single plot per window, !p.multi0

52
Making postscript files

• SET_PLOT, PS
• DEVICE, FILENAMEexample.ps
• , /LANDSCAPE
• , /PORTRAIT, YOFFSET1, YSIZE9, XSIZE6, /INCHES
• , /COLOR, BITS_PER_PIXEL8
• Plotting commands
• DEVICE, /CLOSE
• SET_PLOT, X
• For Windows, SET_PLOT, WIN

53
Detailed plotting...

• set_plot,'ps'
• tek_color
• !p.font1
• !p.color0
• !p.background1
• !p.MULTI0,2,4
• !P.MULTI0
• ang string("305B) Angstrom symbol
• device,filenamefileout'.ps',/color,ysize12,
  xsize17,xoffset1,yoffset10
• 
  
• plot,x1,y1,linestyle1,/ylog,thick6,
  /nodata does not plot the data
• tit'',CHARSIZE1.5, xtit'!Ml!N
  (nm)',XCHARSIZE1.5,
• ytit'intensity (erg s!E-1!Ncm!E-2!NHz!E-1!N)',YCH
  ARSIZE1.5, !E -gtexponent !I-gt index !N
  -gtnormal font
• xrangex0,x1,xs1, !M -gt mathfont, e.g.
  !Ml -gtlambda
• yrangey0,y1,ys1
• 
  
• overplot
• 
  
• oplot,x2,y2,linestyle0,thick3

54
Making other types of graphics files

• Use tvread function from David Fannings library
  www.dfanning.com
• plot,x,y
• imtvread(/png,filefilename,/nodialog)

55
Putting it all togetherHomework assignment 2

• Using LISIRD, create a composite Lyman-? time
  series using SORCE, TIMED SEE, UARS SOLSTICE, and
  SME data
• Plot each instrument as a different color
• Label each instrument
• X-axis yearY-axis W/m2/nm

56
Sample Solution
57
Imaging basics

• Images in IDL are just arrays. All standard
  mathematical operations are available.
• Images are usually saved in FITS format
• Header and array of integers (image)
• Use Goddard IDL Astronomy Library procedures to
  read both the header and image
• Raw images must be processed before they can be
  used for science. Example
• Remove the background counts (dark)
• Correct for variations across the CCD (flatfield)

58
Displaying images

• Basic IDL commands to display an array/image
• TV, image
• TVSCL, image
• Resize an image
• REBIN(image, xsize, ysize)
• xsize and ysize must be integer factors of the
  original size of the image
• Add color by loading a color table
• E.g. SOHO EIT has its own color table for each
  wavelength

59
and Stuff

• where
• size
• histogram
• string manipulation
• file_search
• map projections
• n_elements
• reading file formats (CDF, FITS, PNG, etc.)
• pointers

60
Where

• Use WHERE to find data values meeting some
  criterion.
• goodwhere(data gt 0.)
• betterwhere(data gt 1.5 and
• time gt julday(1,1,2005))
• data2datagood new variable data2
• datadatabetter redefine variable data

61
Size

• The size function returns lots of good
  information about your variable.
• (dimensions, n_elements, data type, etc.)
• ssize(data)
• An undefined variable is a valid type, so check
  for size or n_elements.

62
Histogram

• There are billions and billions of uses for
  histogram.
• hhistogram(data,binsize2.5,ominomin)
• binsfindgen(n_elements(h))omin

http//www.dfanning.com/tips/histogram_tutorial.ht
ml
63
Strings

• IDL understands regular expressions
• Concatenate strings with
• mynameyourname
• resultstrpos(myname,o',0)
• resultstrupcase(myname)
• resultstrlen(myname)
• strsplit, strmid, .

64
file_search

• Retrieves a list of filenames into a string
  array.
• listfile_search(path.idl)
• for i0,n_elements(list)-1 do begin
• restore,listi
• plot,time,irradiance,psym-4
• endfor

All your string manipulation skills can be used
to create the list of files, of course.
dialog_pickfile is an interactive way of
selecting files.
65
Plotting Data on a Map

• set up map projection
• map_set,/continents,/mercator
• (mercator cylindrical map projection)
• plot positional data on it
• MAP_SET, /MERCATOR, 0, -75, 90,
  CENTRAL_AZIMUTH90,
• /ISOTROPIC, LIMIT 32,-130, 70,-86, -5,-34,
  -58, -67,
• /GRID, LATDEL10, LONDEL10, /CONTINENTS,
• TITLE 'Transverse Mercator
• Other options /ROBINSON, /ORTHOGRAPHIC,
  ?map_set

66
Standard Data Formats

• IDL save files restore, filename
• FITS files dmrdfits(file,exten,head)
• netCDF read_netcdf,file,data,status

67
Pointers

• Variable contains a reference to data in memory,
  instead of containing the data.
• Needs to be de-referenced to get the data.

get_sorce_telemetry,data,info,jd1,jd2, externale
lementsolstice_a, itemnamegrat_pos,detec
tor_a,int_time gp(data.(0)).science
68
User Library Routines

• legend
• pause
• psymdot
• which or file_which
• fsc_color
• rainbow

69
Coyotes Guide to IDL

• www.dfanning.com The real source for tips and
  tricks

70
Colors and Plots

• xpalette to show color table
• tvrd (TV read)to grab graphics window and write
  to file
• add color definitions to startup file
• remember to make plots color-blind friendly if
  possible
• (linestyles --- and symbols xxx)

71
File Tips

• help IDLs automatic compiler by always naming
  your files with lower case
• IDL stops compiling when it hits the routine that
  matches the filename

72
IDL 7 Workbench

• New version of IDL uses Eclipse
• Backwards compatible as always

73
Tips Tricks for Workbench

• Hover help (and open declaration)
• Redefine key bindings to whatever youre used to
  ctrl-shift-L
• double click on history entries
• cntrl-i shifts focus to command line
• cntrl-space (or alt-slash) for command completion
• .edit filename
• search works like grepsearch for content in
  files

74
Just the beginning.

• Remember, theres a lot of IDL knowledge around
  here, so feel free to ask questions.
• Many user library routines might already do
  whatever you had in mind.
• Online help manual has lots of examples.
• SolarSoft, Widget programming, Object Oriented...