A Framework for Teaching Software Testing using FOSS Methodolo

1
Involving Students in the F/OSS Learning
Environment Encounters and Experiences
Sulayman K. Sowe sksowe_at_csd.auth.gr
FLOSSCOM Thessaloniki, Greece. November 13th
November 14th, 2006
2
Introduction

• Software Engineering Education (SE) Now
• Inspired by closed-source software development in
  lecture formats.
• Teaching specific SE skills in semesters.
• Project-based coursework. Students work in
  industry to gain practical experience.
• Challenges
• Expose students to the SE principles and
  techniques we teach them.
• Give students practice in large and long-term
  projects.
• Will industry scarify their software (code) to
  students?

3

• What we want to do
• Run a small pilot study in F/OSS mode.
• Students as volunteers in software testing in
  F/OSS projects.
• Give students opportunity to work on what they
  considered interesting.
• Give students real-world experience in dealing
  with F/OSS projects.
• Roles
• Students find and report bugs.
• Lecturer provide useful guidance and support.

4
F/OSS Teaching Learning Framework

• Implemented in the Introduction to Software
  Engineering (ISE) course.
• Duration 12.5 weeks in 5th Semester in 2005.
• 15 students volunteered
• 1 lecturer 1 adviser.

5
F/OSS Teaching Learning Framework Phase 1

• Introduction to F/OSS (8hrs)
• What is F/OSS?, projects, activities, testing,
  forums, etc.
• Exploratory process
• Students browse projects portals
  (sourceforge.net)
• Projects selection criteria
• Operating system, of developers, development
  status (a/ß), active mailing lists/forums.
• 1st class presentation
• Brief history of project, bug reporting
  procedure, testing tools used.

6
F/OSS Teaching Learning Framework Phase
2Students Involvement in Testing

• 2nd class presentation (Week 5)
• Types of bugs found, how they were found, what
  caused them, how they were reported, any
  responses?

7
F/OSS Teaching Learning Framework Phase 3

• 3rd class presentation (Week 12)
• student particulars (Project name, login id,
  website, history, screenshots)
• List of testing activities
• bugs found (bfn), bugs reported (brp), bugs
  fixed (bfx), replies (rep) received, Urls of
  brp, bfx, and rep.
• Likes, dislikes and future plans (if any) in the
  project.

8
F/OSS Teaching Learning Framework Phase 3

• Grading Coursework (50)
• Presentations 10
• 3 pts for each presentation in phase 1 2.
• 4 pts for presentation in phase 3.
• Project participation 12
• of emails exchanged with the student about his
  project.
• Working with testing tools 13
• using and understanding bug tracking systems/bug
  databases.
• Testing activities 15
• measured by 3 variables (bfn, brp, and rep)

9
F/OSS Framework Evaluation - 1
Evaluation Approach Testing activities Online
Surveys

• Testing activities. Students as bug seekers

• bfn gt brp, some bugs found were already reported.
• Not required but some students did well in bfx.

10
F/OSS Framework Evaluation - 2

• Correlation analysis (Bivariate)

.490

• bfn vs brp (r0.690, p0.001).

• brp vs rep (r0.490, p0.089). Interaction with
  community is moderate.

• bfx uncorrelated with bfn, brp, rep.

11
F/OSS Framework Evaluation - 3

• Factor Analysis (Principal Component Analysis
  Varimax rotation)

• Students behaviour explained in terms of 2
  latent factors.
• f1 bfn, brp and to some extent rep
• f2 bfx

The factor scores estimated by the Anderson-Rubin
method
12
F/OSS Framework Evaluation 4. Factor Analysis
Cont.

• Student 2 high score in f1 f2good
  performance
• Student 6 high score in f1 low score in f2
• Students 3, 5, 8, 9, 10, and 12moderate
  performance
• Students 4 and 11 high score in f2bug fixers

13
F/OSS Framework Evaluation - 5

• Online Surveys
• 2 online surveys (Week 6 13). N11 for both
  surveys
• 7 questions in Survey 1 were repeated in Survey 2

McNemer (NA) (100)
McNemer (p 0.039), Some significance
McNemer (p 0.754), Not Significant
McNemer (p 0.065), Significant
McNemer (p 0.002), Significant
McNemer (p 0.008), Not significant
McNemer (p 0.549), Not significant
14
Online Survey Results - 2
Friendly
Interesting project
Response
15
Experiences Lessons Learnt - 1
Face-to-face contact in phase 1 presentations
(brainstorming).

• Advantages
• Practical experience
• software testing in the bazaar.
• how large and complex F/OSS projects work.
• writing good bug reports.
• communicate ideas to virtual colleagues.
• working with tools they might use when they
  graduate.
• Flexible Schedules test whenever and wherever
  you want.
• Learning Opportunities learn new tools (e.g.
  Mantis Bug Tracker).
• Sample size 13, effective interaction with the
  students.

16
Experiences Lessons Learnt - 2

• Disadvantages
• Specialization one aspect of the ISE course
  (software testing).
• Collaboration encouraged but not required to
  work in groups/pairs.
• Evaluation problem Overachievers. How to
  reward them?
• Sample size 15 out of 150 volunteered.

Difficulties

• Lecturer as project manager
• more than preparing and delivering 2hrs lectures
• lecturer as projects manager

• Email/work overload keeping pace with
  students emails.

• When to stop scoring points students Now
  stop finding
• 
  and reporting bugs! ?

17
Experiences Lessons Learnt - 3
Possible Improvements

• Communication
• IM/Web-based discussion forum.
• Problems Timing, archiving, searching, etc.
• Contradict F/OSS Freedom Norm
• pre-select and present students diversity of
  projects.
• Group work/Community formation
• group students around specified projects .
• Identify task-leader based on his contribution.
• Grading/Evaluation
• views a bug report received from the projects
  community.
• Problems Available in few projects. Students
  view their own submission many times.

18
Conclusion

• Work in progress
• Conduct a full experiment involving larger sample
  of students and extend and use the statistical
  analysis method used in this pilot study.
• F/OSS SE Education- Projects as Bazaars of
  learning
• How can we blend the F/OSS teaching and learning
  environment with the formal SE teaching and
  learning context in colleges and universities?
• A broad understanding of the F/OSS pedagogy
• More research of F/OSS evaluation and assessment
  methodologies
• How F/OSS can improve the quality of teaching and
  learning.
• How to create a partnership between students and
  F/OSS developers, projects and industry?

19
SELECTED PUBLICATIONS

• Sowe, S. K., Karoulis, A., Stamelos I. (2005). A
  constructivist view of knowledge management in
  open source virtual communities. In Figueiredo,
  D. A., Paula, A. (eds), Managing Learning in
  Virtual Settings The Role of Context. Idea
  Group, Inc, pp.290-308.
• Sowe, S. K., Karoulis, A., Stamelos I., Bleris
  G.L. (2004). Free-Open Source learning community
  and web-based technologies. IEEE Learning
  Technology Newsletter, Vol. 6 (1), 2004.
  pp.26-29.
• Sowe, S.K., I. Stamelos. (2005). Identification
  of Knowledge Brokers in OSS Projects Mailing
  Lists through Social and Collaborative Networks.
  In Proceedings of 10th Panhellenic Conference in
  Informatics, Volos, Greece, pp. 132-141.
• Sowe, S.K., Stamelos, I., Angelis, I. 2006,
  Identifying knowledge brokers that yield software
  engineering knowledge in OSS projects,
  Information and Software Technology, Vol. 48,
  2006, pp 1025-1033
• Sowe, S.K., Stamelos, I., Deligiannis, I. 2006, A
  Framework for Teaching Software Testing using
  F/OSS Methodology, in IFIP International
  Federation for Information Processing, Vol. 203,
  Open Source Systems, eds. Damiani, E., Fitzerald,
  B., Scacchi, W., Scott, M., Succi, G., (Boston
  Springer), pp. 261-266.
• Sowe, S.K., I. Stamelos, L. Angelis. 2006, An
  Empirical Approach to Evaluate Students
  Participation in Open Source Software Projects.
  The IADIS CELDA 2006 conference, Barcelona,
  Spain, 8-10 December 2006.
• Sowe, S.K., Stamelos . 2006, Involving Software
  Engineering Students in Free and Open Source
  Software Projects Encounters and Experiences.
  Journal of Information Systems Education, To
  appear. This work was partially funded through
  the European Commission, DG Education and
  Culture, Socrates programme, Minerva action line,
  project ref 229405 - CP -1-2006-1- PT - MINERVA
  M (http//www.flosscom.net/).

20
Thanks!Questions?Comments?