Empiricism%20and%20Guessing%20Among%20Oracle%20Professionals

1
Empiricism and GuessingAmong Oracle Professionals

• Jeremiah Wilton
• http//www.ora-600.net

2
ORA-600 Consulting

• Independent Consulting
• Multiple clients
• Emergencies
• Seminars
• Recruiting

3
Me and Guessing

• Started out guessing a lot
• Felt my work was sound
• Insufficient mentoring/oversight
• No overall method to draw from
• No tradition of excellence

4
Oracle and Guessing

• 7.3 Tuning Guide is a manual in guessology
• Where are wait events described?
• Emphasis on long lists of stuff
• What sounds good?
• BCHR OK
• Spin Count - Cool!
• Seemed more art than science

5
Erroneous Literature

• Literature was full of nonsense written by
  mischievous elves
• Still 80 correct
• Would have been helpful but I could get by
  without it so I mainly did

6
Compensation Issue

• One of my problems was compensation
• Late '90s steep increase in DBA salaries
• Low pay low motivation

7
Amazon and Me

• Wild guesses and misinformation fell flat
• High expectations for accuracy and knowledge
• Gifted colleagues set level of excellence
• Better compensation
• Afraid to let colleagues down

8
Sad state of our profession

• Poor quality is common
• Unqualified DBAs are often well compensated
• Driven by law of supply and demand
• Insufficient resources for poor DBAs get better
• Oracle is specialized and complex and it is easy
  to fool people

9
Role of hiring in low quality

• Non-technical/non-DBAs hiring DBAs
• Unqualified DBAs get hired at good salaries
• Creates resentment among colleagues
• Waste of money
• No valid review process without senior technical
  leadership
• Hire someone to help do this right!

10
You do you know you are unqualified?

• Poor DBAs don't know there's a problem
• Manual? What manual?
• No mentoring
• Only exceptional people can self-develop in these
  circumstances

11
The boys' club

• Personal circumstances play into work hours and
  availability
• 80-hour DBAs can be really good
• 40-hour should be equally acceptable
• Freebie for company
• Encourages low DBA headcount
• Emphasis should be on quality
• Reasonable work levels ? unqualified

12
More opportunities for improvement

• IOUG.org
• OTN
• Oracle-L
• comp.databases.oracle.server

• Has candidate ever posted to a forum?
• Goes to motivation, interest in learning or
  interest in sharing knowledge

13
State of DBA candidates

• So many unqualified candidates
• Actually working!
• Unbelievable!

14
TABOS

• Ask any DBA candidate the slow database
  question
• Answer is usually a punchlist in no particular
  order
• TABOS Try A Bunch Of Stuff
• Oracle Support uses TABOS

15
Origins of TABOS in Oracle docs

• Oracle 7 Tuning Guide
• Insufficient CPU
• Insufficient memory
• Insufficient I/O
• Network constraints
• Software constraints
• Pointed DBAs at vsysstat
• Scant and muddled references to wait events

16
Candidates echo the manual

• Candidate espousing TABOS sounds like the Oracle7
  Tuning Guide
• Check CPU usage
• Check memory usage
• Check for high BCHR
• Check for some other hit/miss ratios you found on
  the Internet
• Check for full disks
• Check for too many extents
• Check for fragmentation
• Check for disk sorts
• Check for locks
• Check for swapping
• Check for FT scans
• Check for bad plans
• etc...

17
TABOS... works?

• You might stumble upon the problem
• Works enough to just keep practitioners employed
• Not fast or clever, but...
• Sounds technical - lots of jargon
• Makes you look busy!
• But might never find the problem if not on your
  list
• Waste of time

18
Is TABOS for real?

• Just a figment of the job interview
• Even underqualified DBAs use some experience to
  order their list of checks
• Experience trumps a wild guess
• I wouldn't use even experience-driven TABOS
• Using waits and ASH still faster and more reliable

19
Good Guesses

• Some educated guesses are effective
• All DBAs guess
• Educated guess is as good as the education
• Empirical from eµpe???sµ?? experience
• Experience is a valid form of evidence/data

20
Pure Empiricism

• Are there purely empirical DBAs?
• Yes BAAG Battle Against Any Guess
• http//www.battleagainstanyguess.com
• Eliminate guesswork from the decision-making
  process
• BAAG in the tradition of BAARF

21
BAAG has a point

• Oracle's instrumentation is the best way to solve
  most problems
• Slow database question is the poster child for
  ASH
• Starting with ASH/waits assumes that Oracle is
  the problem
• For reality-based troubleshooting, we need
  equally good instrumentation on app server

22
How empirical can you really be?

• Time and resources limit empiricism
• Many decisions don't merit exhaustive means
• A good DBA knows when to make an educated guess

23
The Axis of Guessing
Most DBAs
Empiricism
TABOS
Educated Guessing
24
There are no pure empiricists

• So why BAAG?
• An effort to distance from messy thought
• Large number of unqualified and undertrained DBAs
• Polarizing effect on skilled DBAs
• Real axis is learners vs. slackers

25
A meaningless axis

• Separatism is destructive
• Us vs. Them
• Imaginary division
• Anti-guessing ? Science
• The key to a scientific approach is verification
• Oracle's instrumentation is ample

26
Issue of incentive

• Most DBAs won't improve without incentive
• High DBA demand reduces incentive
• Non-DBAs can't tell a DBA is unqualified

27
Manuals are the first step

• Oracle7 Tuning Guide notwithstanding
• Manuals are pretty good now
• Require patience and persistence
• Some people are not book learners
• Experiential learners not compatible with
  production DBA work

28
Incentives for Excellence

• I was motivated by high expectations and
  corporate culture
• Can also be naturally bright and curious
• Most need organizational incentive

29
State of Oracle Education

• No critical thinking taught
• Emphasis on syntax and rote facts
• Which top Oracle gurus ever came out of OCP?
• OCP and non-OCP indistinguishable in interviews

30
Senior review and mentoring

• Underqualified DBAs often have no role models
• Pro-mentoring is cost-effective
• Junior DBAs are half price
• Not primadonnas
• Smart junior DBAs become senior in just few years

31
Role of technology

• Middle tiers and applications need
  instrumentation
• Opportunities at
• Design time
• Purchase time
• Enables analytical methods

32
Conclusions

• Problem is not empiricism vs. guessing
• Problem is sad state of excellence in DBA work
• Solutions
• High expectations
• Critical thinking skills
• Meaningful education (mentoring, NoCOUG)?
• Instrumentation at the app layer
• Don't hire unqualified people