Open Source and LargeScale Assessment Platforms: A True Winner

1
Open Source and Large-Scale Assessment Platforms
A True Winner?

• Thibaud Latour
• Matthieu Farcot
• CRP Henri Tudor

2
Agenda

• Open Source Business
• Computer-Based Assessment
• Computer-Based and Large-Scale Assessment Current
  Situations
• An Open-Source Platform

3
Open Source Business
4
Open Source Business Model

• Open source and profit margins
• Main aspect is RD
• RD ? externalisation consolidation
• An active open source community can provide
  significant benefits in cost to find and fix
  software defects after production release

5
Open Source Business Model

• How to generate more revenue?
• Economies of scale type revenue generation
• Sell product to more numerous customers
• Sell product in larger deals (longer customer
  relationships)
• Agility type revenue generation
• Introduce new product faster than others
• Create and exploit new markets faster than others

6
Open Source Business Model

• Open source and revenue generation
• Enables Agility type revenues extremely well
• Development model enables fast creation of atomic
  products in existing categories
• Innovation by integration
• Fast creation of new product by integration of
  existing components
• Enables new entrants to enter the market easily
  with specialized niche or customized solution,
  therefore
• Expand the ecosystem
• Attack new markets
• Create incremental opportunities
• Enable commoditize selected bits of functionality
• ? Focus on higher-value core competencies with
  less competition

7
Standard Software Business Model

• Produce a piece of software
• Decide on price unit measure user, server, CPU,
  time
• Charge as much as you think the traffic will bear
• If the software helps to control another market
• Costs do not occur at the same time sale occurs
• Possible to fix the business model later
• Lock in (if possible) make repeated
  customer-unfriendly behaviour possible
• Consequence of aggressively behaving badly is
  extreme profits
• 40 operating margins in mature companies with
  high customer base lock-in

8
Intangibles Are Part of the Business Model

• Good community behaviour is expected
• Measurable ? number of bug fixes,
• Bad behaviour becomes widely known and can end up
  support
• Customer benefit to everyone of good community
  behaviour is bigger ? more viable community
• Power of brand can offset perceived
  commoditization and other effect

9
Open Source Gives New Things to Charge for

• Not only being in the support-only business
  (all others are in to some extent)
• Pricing for support can be quite high depending
  on the market
• Support looks more value-added if there is a need
  for integrating lots of pieces
• Between pre-integrated suite and out-of-the-box
  parts that needs to be integrated from scratch by
  consultants
• Platform and standards create opportunity for
  add-in providers with deep expertise to gain
  volume from the platform

10
Computer-Based Assessment
11
Computer-Based Assessment

• Different purposes
• Surveys (sample)
• Monitoring (full population)
• Training (formative/certif.)
• High-Stake (pre/post)

Analysis of the past
System level
Individual level
Forecast
12
Computer-Based Assessment

• Different markets
• Education
• Policy makers and socio-economics
• Research in Psychology, Social and Educational
  Sciences
• HR business and human capital
• Regulation and legal certification

13
Computer-Based Assessment

• Different needs

• Recruitment
• Evaluation
• Function/responsibility grades
• New missions
• Restructuring
• Merger

• Individual (career plan, evaluation, coaching,
  internal mobility, )
• Teams
• Department
• Corporate entities
• Worldwide group level

• Social skills
• Business-related skills
• Management skills
• ICT literacy
• Language literacy

14
Computer-Based Assessment

• Different tools and processes
• As a result of this diversity
• Very diverse assessment processes
• Very diverse assessment instruments
• Very diverse delivery modes
• Very diverse metrics
• Diverse fauna of cats
• Computer-Assisted Testing ? assessing people
  using a computer
• Computer-Aided Testing ? support assessment with
  IT
• Computer Adaptive Testing ? adapt test to
  competency level

15
Computer-Based Assessment

• Not so disjoint
• Even if
• Different purposes
• Different markets
• Different needs
• Different cats
• There exist some commonalities
• ? Is it important to factor these commonalities
  in a platform fits all perspective?

16
Computer-Based Assessment

• Not so disjoint
• Even if
• Different purposes
• Different markets
• Different needs
• Different cats
• There exist some commonalities
• ? Is it important to factor these commonalities
  in a platform fits all perspective?

YES!
17
Platform fits all rationales
Test authors and administrators
Single isolated test
User-specific features
Specific Interface
Interface
Specific Interface
Plug-ins
Item/subject Admin.
Tests
Subjects

Shareable Common features
General platform
Exec. Items
Items
Exec. Test
Groups
Results
Delivery
User-specific features
Interface
Algorithms
Item templates
Plug-ins
Specific Interface
Reducing time-to-delivery
Evaluated users
18
Platform fits all rationales
Escaping from the test-by-test approach
Reducing mid- and long-term global cost
19
Computer-Based and Large-Scale Assessment Current
Situation
20
Market for Large Scale Assessment

• BIG market
• Lots of Lock-in effects
• Very HIGH Costs
• Lack of any REAL interoperability
• Oligopolistic nature
• few VERY big competitors,
• lots of very small companies.
• High level of market concentration

21
Market for Computer-Based Assessment

• Competition is atomic
• low concentration level of market competition
• Usually low level of investment needed to develop
  a CBA (or at least, far lower than for creating a
  large scale system)
• Market dominated by private companies
• Test-by-test model (tailor-made)
• Either transposition of paper-and-pencil
• Either concentration on very specialised niches
• Platform effort
• Mere transposition of paper-and-pencil / quizzes
• Focus on test, not on process
• Covers a limited portion of the space of
  assessment needs with no easy escape ( and
  techno limits)

22
Large Scale Computer-Based Assessment

• Large scale CBA is an oligopolistic market
• High costs for new entrants
• Lock-in effect and associated behaviours
• ? Has hindered so far the rise of TBA (at least
  when economic reasons are advocated)
• Need a disruptive move in the market
• Open Source approach
• Modify the market power to the benefit of users

23
Current Open Source Initiatives

• For Large Scale Assessment markets
• Very few
• For Computer-Based Assessment markets
• Yes, a lot
• But ? low interactivity and usability on the
  creative user side
• ? platform fits all a solution to these
  problems
• For the vendors good basis to
• Enable high profit margins
• Generate agility type revenues
• For the users
• Diversification of specialised components
• Standardisation effect
• Commoditization of common features
• ? Concentration of expenses on core competency
  needs and user problem
• Lower entry price

24
Open Source Platform for Large Scale Assessment

• Creation logic
• Obligation of results
• Scientific validation is important
• PEER REVIEW through universities
• System diverges but not by much from open
  innovation frameworks
• OPEN SOURCE Potential risk tackled by scaling
  effect and consolidation of quality-related
  action (many testers, many test cases)
• Very high logistic needed (consortiums)
• Federate different actors and investors
• OPEN SOURCE Potential winner against those
  barrier to entry

25
Open Source Platform for Computer-Based Assessment

• Lower the costs
• Creation costs
• Diffusion costs
• Free parking paradigm (low license costs)
• However Free software Open ? free
• See business model

26
An Open Source Platform
27
A Platform for the Future

• OS Licensed for instance Gnu GPL v.2
• An Open Source application
• Low cost and created by a strong team specialized
  in BOTH large scale and CBA (IT together with HS,
  not IT for HS)
• Potential advantage
• First mover
• Power of branding
• Federation of all actors with various needs (to
  avoid redudent investment and risks of forks)
• Versatility
• Importance of agility
• Openness
• You can add your new modules
• No (or very focussed) need for external
  development service companies
• Create a disruptive commodity market for Large
  scale CBA

28
Platform Scaling Effect
National
EU
INTL
29
Platform Scaling Effect

• Wide range of activities
• Strandardization
• IPR and brand managemant
• Distribution
• IT support, maintenance, evolution
• Services
• IT
• Domain specific
• Products (plug-ins, packages, )
• Community support
• Knowledge sharing

30
Platform Scaling Effect

• Focused organisation

31
Platform Scaling Effect

• Focused planning

32
Conclusion
33
Conclusion

• Large Scale Computer-Based Assessment
• Open Source
• True winner?
• YES

34
Credit

• Part of this presentation has been inspired (and
  sometimes reuses) slides from Brent C. Williams
• Brent C. Williams, Open Source Business Models
  A Wall Street Look at the Wild 2006 and the
  Prospects for Even More Fun in 2007 (Made
  available under the EPL v1.0)