Center for eBusiness@MIT

1
Towards Total Security Quality Management (TSQM)
Enterprise Perception Measurement and the
House of Security March 7, 2006 Professor
Stuart Madnick, Dr. Michael Siegel, Wee Horng Ang
(smadnick, msiegel, weeang_at_mit.edu)
2
MIT TEAM

• STUDENTS
• Wee Horng Ang
• Dinsha Mistree
• Venkataramana Thummisi

• FACULTY
• Yang Lee
• Stuart Madnick
• Michael Siegel
• Diane Strong
• Richard Wang
• Chrisy Yao

3
Overview of Project
Comprehensive List of Aspects of Security
Gap Analysis Instrument Validation and Refinement
Survey 1 and 2
Key Dimensions And Aspects
Academic Literature
Gap Analysis Instrument
Stakeholders And Roles
Industry Literature
Gap Analysis
Survey 3
Key Gap Findings
4
Brief Description of Surveys
Survey 1 Open-ended What does holistic Security
mean to you? Survey 2 Semi-structured What
does holistic Security mean to you? Similar to
Survey 1, but starts with 20 security
aspects. Survey 3 13 semi-structured questions
regarding Extended Enterprise security covering
issues such as Security Return on Investment,
Benefits of Security, and Extended Enterprise
Security.
5
Comprehensive List of Aspects of Security
Ability to effectively use data acceptance inspection Access access control mechanism access level access list Access modes access period access port access type Accountability accreditation accreditation authority add-on security administrative security Alert handling Antivirus Asset classification control assurance attack audit trail authenticate Authentication authenticator authorization automated information system (AIS) automated information system security automated security monitoring availability of data Availability of service back door backup plan Bell-La Padula model benign environment between-the-lines entry Brand equity is tied to customers perception about security Breach of confidentiality Breach of Security (BOS) Breach of integrity (BOI) browsing Buffer overflow Business loss Cache overflow call back capability category certification closed security environment communications security (COMSEC) Company preparedness compartment compartmented security mode Competitive edge Compliance compromise compromising emanations computer abuse computer cryptography computer fraud computer security subsystem concealment system confidentiality configuration control configuration management confinement confinement channel confinement property Connection contamination contingency plan control zone controlled access controlled sharing Controls Cookies cost-risk analysis countermeasure covert channel covert storage channel covert timing channel Credibility Criteria crypto-algorithm Cryptosecurity Customer confidence Customer loss Customers system Customized access Data control Data encryption Data Encryption Standard (DES) Data reliability dedicated security mode default classification Degausser Products List Denial of Service AND MANY MORE
6
Overview of Project Key Dimensions
Comprehensive List of Aspects of Security
Gap Analysis Instrument Validation and Refinement
Survey 1 and 2
Key Dimensions And Aspects
Academic Literature
Gap Analysis Instrument
Stakeholders And Roles
Industry Literature
Gap Analysis
Survey 3
Key Gap Findings
7
Dimensions of Security
House of Security
8
Good Security
Good Security provides Accessibility to data and
networks to appropriate users while
simultaneously protecting Confidentiality of data
and minimizing Vulnerabilities to attacks and
threats. Good Security Practice goes beyond
technical IT solutions. It is driven by a
Business Strategy with associated Security
Policies and Procedures implemented in a Culture
of Security. These are practices are supported by
IT Resources and Financial Resources dedicated to
Security.
9
Overview of Project Stakeholders and Roles
Comprehensive List of Aspects of Security
Gap Analysis Instrument Validation and Refinement
Survey 1 and 2
Key Dimensions And Aspects
Academic Literature
Gap Analysis Instrument
Stakeholders And Roles
Industry Literature
Gap Analysis
Survey 3
Key Gap Findings
10
Stakeholders
General Public
Extended Enterprise
Enterprise
Ring 1 Enterprise Ring 2 Extended
Enterprise Ring 3 General Public
11
Stakeholders Roles
Domain/Role
Level/Rank General business IT Organization General security/ physical security Partners (Extended Enterprise)
Top exec CEO, CFO, Top IT Mgt/CIO Top Security Mgt / CSO
Line/middle manager Business unit manager IT non-security managers ------------------------- IT security manager Security managers
Workers Business personnel IT non-security personnel ------------------------- IT security personnel Security personnel (e.g., guard)
12
Overview of Project Gap Analysis
Comprehensive List of Aspects of Security
Gap Analysis Instrument Validation and Refinement
Survey 1 and 2
Key Dimensions And Aspects
Academic Literature
Gap Analysis Instrument
Stakeholders And Roles
Industry Literature
Gap Analysis
Survey 3
Key Gap Findings
13
Differing Perceptions
Picture of old lady or young lady ?
Perceptions are as important as reality
14
Data Source(How do you cite in a Journal
article?)
15
Purpose of Gap Analysis
Purpose of Gap Analysis is to understand
Differences in Perceptions between factors such
(A) Security Status Assessment and Security
Importance (B) views of diverse Security
Stakeholders within the Enterprise and across
the Extended Enterprise
16
Purpose of Gap Analysis (cont.)

• Gaps represent Opportunities for Improvement
  within the Enterprise and across the Extended
  Enterprise
• When Status is below the Needs, these represent
  Areas for Improvement
• (B) When Status among Stakeholders show
  differences, these represent areas for
  Investigating sources of the differences
• Gaps may represent misunderstandings
• Gaps may represent differences in local knowledge
  and needs

17
Many Types of Gaps

• Performance Gaps Current Status v. Importance
• Role Gaps e.g., Business Managers v. IT staff
• Inter-Enterprise Gaps Internal Line Manager v.
  Supplier
• Initially, our focus is on Performance Gaps, much
  more data needed for analyzing Role and
  Enterprise Gaps
• Issue Gathering of enough data from same
  organization and partner data

18
Gap Analysis Questionnaire

• Questionnaire respondents are comprised of the
  diverse roles (IT, IT security, Users, Business
  managers, Executives, etc.) within the enterprise
  and across (suppliers, customers, collaborators,
  etc.) the extended enterprise.
• 2. Each respondent reports his/her view of actual
  assessment and importance of each aspect for both
  his/her organization and a partner organization.

19
Gap Analysis Questionnaire (cont.)

• Questions on the questionnaire cover the 8
  constructs of security
• Accessibility
• Vulnerability
• Confidentiality
• Financial resources for security
• Technology resources for security
• Business strategy for security
• Security policy and procedures
• Security culture
• 4. To ensure construct validity, (approx) 5
  questions are included for each construct.

20
Extended Enterprise Security Survey

• Form 01-23
• Towards Total Security Quality Management (TSQM)
• MITs Extended Enterprise Security Survey
• Introduction
• The following survey is part of a research
  project at MIT to develop a holistic framework to
  study enterprise security within and between
  organizations. Your responses to the following
  survey will provide us valuable insight about
  extended enterprise security. The extended
  enterprise includes an organization and its
  suppliers, customers, partners, and competitors.
  Extended enterprise security is concerned with
  security both within and between these
  organizations.
• The survey should take you about 20 minutes to
  fill out.
• Note about confidentiality Your responses to
  questionnaire items will not be revealed to your
  organization or to any other organization. Only
  aggregate results will be used in our analyses.
  If you would like to receive a copy of our
  research results, please provide your email
  address at the bottom of the survey.
• General Instructions
• 1. What does it mean by assessment and
  importance?
• The survey asks you to give your impression of
  the assessment and importance of various
  security issues.
• Assessment, means your view of how well your
  organization is doing on these issues.
• Importance means your view of how important
  this issue is to you.
• 2. There is no right or wrong answer to any
  question. We are asking for your view.
• You may not know exact details about your
  companys security. We are not asking for these
  details, but asking for your views. Please give
  your best estimate.

21
Your Organization Partner

• Extended Enterprise Security Survey
• Section 1 Your Organization
• Your Organization/Company
• Organization Name_________________________________
  _________________________
• Industry__________________________________________
  __________________________
• Approximate total number of employees in your
  entire organization ________________
• Your Job Title and Work Role _____________________
  ___________________________
• __________________________________________________
  _________________________
• Department/Division/Group_________________________
  __________________________
• In my organization, I am a
• _____(1) Executive (CEO,CFO, VP etc.)
• _____(2) Functional or Line Manager
• _____(3) Professional (Consultant, Engineer,
  In-house Expert, etc.)
• _____(4) Other Organizational Member
• In my organization, I work in the area of
• _____(1) Business Security Policy and Management
• _____(2) IT Security
• _____(2) IT but not in Security,
• _____(3) General/Physical Security,

22
Security Questions (40)
23
Survey Data Gathering

• Developed web-based survey
• Developed secure (https) web-based survey
  instrument
• Collected data
• Considerable partner company data, but need
  more
• Both miscellaneous and several company-wide
• Valuable for intra-company stakeholder gap
  analyses
• Preliminary analysis of increased pilot data
• Some sample analysis follows

24
Lots of Survey Data Gathered
25
Overview of Project Key Findings
Comprehensive List of Aspects of Security
Gap Analysis Instrument Validation and Refinement
Survey 1 and 2
Key Dimensions And Aspects
Academic Literature
Gap Analysis Instrument
Stakeholders And Roles
Industry Literature
Gap Analysis
Survey 3
Key Gap Findings
26
Gap Analysis Preliminary Findings
Mostly Performance Gaps Some Role and
Inter-Enterprise Gaps Explore at item level
(yet not construct level) - Data recently
received - Only very limited analysis so far -
All Findings that follow are preliminary
27
Some speculationSample Security Culture
Question 39 People are aware of good security
practices.

• Assessment vs Importance?
• Assessment Importance About same (10)
• ?? ?? ??
• Assessment Your Organization vs Partner
• Your Org Partner About Same (10)
• ?? ?? ??
• Assessment Different Roles/Functions
• IT Security IT, not Security Genl Mgt
• ?? ?? ??
• ?? ?? ?? (lowest?)

28
Evaluating Statistical Significance
MA vs MI Gaps
Significant at 99.99 level 28
Significant at 99 level 11
Significant at 95 level 0
Significant at 90 level 1
Less than 90 0
Total 40
Gap significance notation Significant at the
99.99 level Significant at the 99 level
Significant at the 95 level Significant
at the 90 level.
29
Gap Analysis FindingsSecurity Culture
Question 18 People in the organization
carefully follow good security practices.
Question 26 People in the organization can be
trusted not to tamper with data and networks.
Question 39 In the organization, people are
aware of good security practices. Qn 18 Gap
1.24 Qn 26 Gap 1.01 Qn 39 Gap
1.28
30
Gap Analysis Findings Different Organizations
Question 39 People are aware of good
security practices.

• Gap between Assessment and Importance
• for your company
• Overall 1.28
• (5.04 vs. 6.32)
• Miscellaneous 1 2.40 (4.20 vs. 6.60)
• Company X 2 1.83
• (5.00 vs. 6.83)
• Company W 2 1.89 (4.61 vs. 6.50)
• Company I 3 0.44
• (5.33 vs. 5.78)

1 Original pilot sample diverse array of
companies many middle-managers
2 High-tech organizations 3 Non-USA company
31
Gap Analysis Findings Compared with Partner
Organization
Question 39 People are aware of good security
practices.

• Gap between Assessment and
• Importance for your company
• Overall 1.28
• (5.04 vs. 6.32)
• Gap between Assessment and
• Importance for partner company
• Overall 0.70
• (5.25 vs. 5.95)

General conclusion - View partner as better
in assessment - But it is also less
important -gt So Gap is much less
But not exactly true for all organizations
32
Gap Analysis Findings Compared with Partner
Organization
Question 39 People are aware of good security
practices.

• Your Organization Partner Organization

Some observations Gaps all smaller, but
Assessment /- Importance /-
33
Gap Analysis Findings Different Roles/Areas
Question 39 People are aware of good security
practices.

• Your Organization Partner Organization

• Some observations
• Not huge difference in gaps for your
  organization
• - More significant gaps in views of partner
  organization

• IT Security people perceive much less gap in
  partner
• - And much lower importance for partner

34
Overview of Project Instrument Validation
Comprehensive List of Aspects of Security
Gap Analysis Instrument Validation and Refinement
Survey 1 and 2
Key Dimensions And Aspects
Academic Literature
Gap Analysis Instrument
Stakeholders And Roles
Industry Literature
Gap Analysis
Survey 3
Key Gap Findings
35
Phase 2 Underway (mostly completed)

• Collect more data especially for intra-company
  (partner) stakeholder analysis
• Complete analysis of pilot data
• Complete construct analysis
• Refine stakeholder and dimensions
• Refine questionnaire items
• Revise gap analysis instrument

36
Instrument Analysis for Construct Reliability
and Validity

• Reliability means produces consistent results
• The multiple questions (components) for each
  construct produce strongly correlated responses
• Determined by computing Cronbach Alphas
• Validity means components are more closely
  correlated with the others of that construct than
  they are with components of another construct
• - Convergent Validity form a single construct
• Evaluated using Average Variance Extracted (AVE)
• - Discriminant Validity not part of another
  construct
• Evaluated by requirement that squared multiple
  correlation between two constructs less than AVE
  of each construct

37
Analysis of Construct Reliability and Validity
38
Revised Instrument
39
Average Construct Values
40
Constructs Average Values Standard Deviations
41
Average Construct Variances
42
Absolute Construct Variances
43
Some Preliminary Insights

• Highest assessments in accessibility indicates
  that businesses are still primarily concerned
  with information access and use. Low assessment
  in security culture, further confirms that
  security management have yet to mature to the
  same level of security awareness and depth.
• Low Gaps in overall Accessibility levels states
  that accessibility is very well-established,
  perhaps to the point of saturation.
• 3 High standard deviations in Security Policy
  indicates there is a disparity between the
  various companies/ industries.
• 4 The large MI-MA gap, and PI -PA gap in security
  culture, shows companies are beginning to
  understand the need to achieve further
  improvement, highlighting an important area of
  potential growth.
• 6 Partners assessment lower than self assessment
  indicates the aura of "invincibility" is
  present,that companies believe they are safer
  than their partners. Of course, everyone is
  someone elses partner.
• 7 Partners importance of security lower than self
  security reiterates the point that they believe
  their own companies rate these qualities more
  importantly on their agenda than would their
  partners.

44
Company Assessment Values
45
Company Assessment Gaps
46
Role Assessment
47
Area Assessment
48
Next steps Phase 3

• Large-scale Gap Analysis Study
• IBM
• Nortel
• RSA Security Conference
• (mailing post-conference)
• 25-50 responses from 3 or more members of
  eBusiness Center (e.g., BT, UPS)
• Cisco
• Two rounds 500 responses
• 5000 responses
• Extensive Gap Analysis Results

49
Next steps Phase 4 (longer-term)

• Longer-term Pursue other related security
  measurement activities
• Other Survey Instruments
• Case Studies
• Best Practices
• Benchmarking
• Security Methodology

50
What is Good Security? It can be a matter of
opinion (perception)
51
Thank you
Stuart Madnick T 617-253-6671 E
smadnick_at_mit.edu URL http//web.mit.edu/smadnick/w
ww/Projects/I-SEE20CeB.pdf
52
Extra Slides

53
Gap Analysis FindingsAccessibility
Question 40 The organizations data and
networks are usually available when needed Gap
0.40 6.72 (My Importance) vs. 6.32
(My Assessment) Note indicates significant
at the 99 level
54
Gap Analysis FindingsVulnerability
Question 1 The organizations data and networks
are rarely tampered with by unauthorized
access. Gap 1.22 6.60 (My
Importance) vs. 5.38 (My Assessment) Note
indicates significant at the 99.99 level
55
Gap Analysis FindingsConfidentiality
Question 38 The organization provides good
protection of confidential corporate data.
Gap 0.58 6.50 (My Importance) vs. 5.
92 (My Assessment)
56
Gap Analysis FindingsFinancial Resource for
Security
Question 2 In the organization, security is
adequately funded. Gap 0.78 6.39 (My
Importance) vs. 5.61 (My Assessment)
57
Gap Analysis Findings IT Resource for Security
Question 5 Business managers are involved with
IT security policies. Question 17 The
organization has adequate technology for
supporting security. Qn 5 Gap 1.08
5.96 (My Importance) vs. 4.88 (My
Assessment) Qn 17 Gap 0.51 6.37
(My Importance) vs. 5.86 (My
Assessment)
58
Gap Analysis FindingsBusiness Strategy for
Security
Question 4 The organizations security
strategy sets directions for its security
practices. Question 19 The organization has a
well-defined and communicated security
strategy. Qn 4 Gap 0.64 6.33 (My
Importance) vs. 5.69 (My Assessment)
Qn 19 Gap 1.07 6.14 (My
Importance) vs. 5.07 (My Assessment)

59
Gap Analysis FindingsPolicy and Procedures for
Security
Question 25 The organization has adequate
procedures for ensuring the physical security of
buildings and equipment. Question 30 The
organization has procedures for detecting and
punishing security violations. Qn 25 Gap
1.07 6.42 (My Importance) vs.
5.38 (My Assessment) Qn 30 Gap 0.94
6.25 (My Importance) vs. 5.31 (My
Assessment)