??????? Sphinx-?

1
??????? Sphinx-?

• ????????????? ????? ?????????? ??????

2
??? ?? ??????

• Sphinx ?????????????? ?????????

3
??? ?? ??????

• Sphinx ?????????????? ?????????
• ????? ??????

4
??? ?? ??????

• Sphinx ?????????????? ?????????
• ????? ??????
• ????? ?? ??????

5
??? ?? ??????

• Sphinx ?????????????? ?????????
• ????? ??????
• ????? ?? ??????
• ????? ???????? ?????? ??????

6
??? ?? ??????

• Sphinx ?????????????? ?????????
• ????? ??????
• ????? ?? ??????
• ????? ???????? ?????? ??????
• ????? ????? ?????
• ?????????? ?????, ????????, ???????? ?????????,
  ?????????????, IO throttling, ? ??? 10-20 ??????
  ?????????? ????????

7
????? ??? ????????

• ???? ?? ????? ? ????????
• ???????? ?????? ??? ??? ? ????? ???
• ??????? ????? ?? ????????????
• ?????????? ??????? C ???????????

8
????? ??? ????????

• ???? ?? ????? ? ????????
• ???????? ?????? ??? ??? ? ????? ???
• ??????? ????? ?? ????????????
• ?????????? ??????? C ???????????
• ??? ??????
• ??? ??? ? ????? ???????? ??????
• ??? ????? ?????????????? ??????
• ??? ????? ???????????????? ??????

9
????? 1. ??????? ??????
10
????????? workflow

• ??????? ???????????
• ????? ????

11
????????? workflow

• ??????? ???????????
• ????? ????
• ???? ????????? ?????? (?????? ? ??? ?????)
• ???? ???????
• ????? ????????? ?????
• ??? ???????????? ?????
• ???? ?????????? ?????????

12
??? ???????? ??????????

• ? ???? ?????, ? ?????????
• ???? 1 ???? ??????????
• ???????? ????????? (???? ?? ??????????)
• ????????? ?????? ???????? ?? ?????
• ???????????? ????? (??????????, ?????)
• ???????? ????? ?? wordid (CRC32/64)
• ????? ??? ????????? ??????

13
??? ???????? ??????????

• ???? 2 ?????????? ?????
• ??? (hit, ?????????) ?????? (docid,wordid,wordpo
  s)
• ?? ????? ???????? ????????????? (?? wordid)
  ??????
• ????????? ???????? ?????? ?????
• ?? ?????? ????????? ????????????? ??????
• ??????????
• ???????? ? ????????? ???????? MVA
• ????????? ordinals
• ????????? extern ????????

14
????? ? ??? ?????

• ?????? ??????? ?? ?????? ????????
• ????????? ????????????? ????????
• ??????? (?????? wordid)
• ???????? (?????? ???? docinfoextern)
• ?????? ?????????? (??? ??????? ?????)
• ?????? ??????? (??? ??????? ?????)
• ??? ??????? ???????, ?????? ??? IO

15
??? ???????? ?????

• ??? ??????? ?????????? ???????
• ?????? ?????? ?????????? (??????????,
  ??????????????? ???????)
• ????????? (?????? WHERE)
• ????????? (??????? ???? ??????????)
• ????????? (?????? ORDER BY)
• ?????????? (?????? GROUP BY)
• ????????? ?????????? ?? ???? ????????

16
1. ???? ??????

• ?????????? ?????? ??????????
• 1 ???????? ????? 1 IO (?????? ??????????)
• ?????? ???????? ??? ???????? ??????????
• ????????? ??????????????? () ????? ???????
• ?? ????, ????? ?????? ???? ???????? ????
• ??? ?????? ???? ???, ??? ? ???????? ??? ????????
  ??????? ???? ???????? 2x IO/CPU

17
1. ???? ??????

• ?????????? ?????? ??????????
• 1 ???????? ????? 1 IO (?????? ??????????)
• ?????? ???????? ??? ???????? ??????????
• ????????? ??????????????? () ????? ???????
• ?? ????, ????? ?????? ???? ???????? ????
• ??? ?????? ???? ???, ??? ? ???????? ??? ????????
  ??????? ???? ???????? 2x IO/CPU
• ?????? The Who ????? ?????? ??????

18
2. ???? ??????????

• docinfoinline
• ???????? ???????? ? ?????? ??????????
• ??? ???????? ??????????? ????? ???!
• ???????? ????? ????? ?????? ? ?????
• docinfoextern
• ???????? ???????? ? ????????? ?????? (?????)
• ????????? ?????????? ? RAM
• ??? ?? docid ???????? ?????
• ??????? ????????
• C???????? ????? ?????????? ? ????????

19
3. ???? ????????????

• ?????? ??????? ?? ranker-?
• ????????? ??????? ????
• ??????? ??? ?????????????
• ?? ????? ???????? ??????? ????!
• ????????? ????? ???????????
• ????? ??????? phrase proximity BM25
• ????? ??????? none (weight1)
• ????????? ????? ?????????? ? ??????????

20
4. ???? ??????????

• ????????? ????? ???????????
• ??? ??????? ?? ???????? ?????????? (?????????
  ?????? ? ??????? _at_id asc)
• ??? ??????? ?? max_matches
• ??? ?????? max, ??? ???? ???????
• 1-10K ?????????, 100K ???????
• 10-20 ???????

21
5. ???? ???????????

• ??????????? ?????? ?????? ??????????
• ???? ????? ???????????
• ???? max_matches
• ????????, ?? max_matches ??????? ???????? _at_count
  ? _at_distinct

22
????? 2. ???????????? ??????
23
??? ?????????????? ???????

• ????????? ?????? (partitioning)
• ?????? ???????????? ? ??????????
• ??????? ?????? ???????? ????
• ??????? ? MTF ???????
• ????????????? (multi queries)

24
??? ?????????????? ???????

• ????????? ?????? (partitioning)
• ?????? ???????????? ? ??????????
• ??????? ?????? ???????? ????
• ??????? ? MTF ???????
• ????????????? (multi queries)
• ????????? ????? ?????? ??? ??????? ??????

25
1. ????????? ??????

• ????? ??????????? ???, ??? ?????? ?????
• ???????? ? ???????????????
• ?????????, ??????????????? ?????? ?????????
• ???????? ? ???????????
• ?????????, ???? ?????? ?? ?????? ????????
• ???????? ? CPU/HDD?
• ?????????, ???????? ?? ?????? cores/HDDs/boxes

26
1a. ????????? ??? ??????????

• ?????????? ??????? ??????
• ?? ??????? ????? ????????? ??????????
• ????????? ????? ????????? ?????
• 1-10 ???????? ???????? ???????
• ????????? ?????????? ? 50 (3024...)
• ????????? ?????????? ? 2000 (!!!)

27
1b. ????????? ??? ??????????

• ?????????, ?? 100 ??????? ?? ?????????? ??????
  ????????
• ???????????? ???? ?????? ?????? ????
• ?????????? ??????????? (3? ???????? Query())
• ????????? ?????? ??? ???????????? ??????????
  ?????????????? ??????
• ??? ?????????? ?? ????????? ?????? ??
• ??? ???????????? ???????? ??? (!)

28
1c. ????????? ??? CPU/HDD

• ?????????????? ??????, ????? ???? ?????? ??
  ?????????? ???????????
• ??????????? searchd ??? ?? ????

index dist1 type distributed local
chunk01 agent localhost3312chunk02 agent
localhost3312chunk03 agent
localhost3312chunk04
29
1c. ??? ?????? CPU/HDD ??????

• ??? ??????????? ????????
• vmstat ??? ? ????????? ????? CPU?
• oprofile ??? ????????? ????? CPU?
• iostat ????????? ????? HDD?
• ???? ????, ???? ????? searchd --iostats
• ?????? ??? ???????? (us/sy/bi/bo), ??!
• ??????? HDD ????? ????????? ? iops
• ??????? CPU ????? ????????? ? sy

30
2. ????????????

• ?????? ?????? ?????? (?.?. ??????? ? ??????
  ?????? extended2)
• ?????? ?? ????????? phraseBM25, ???????????
  ??????? ???? ?? ?????????
• ?????? ?????????? ????? ????????
• ?????? _at_weight ???????????? ?????? (???? ipod,
  ????????? ?? ????)
• ?????? ????? ??????????

31
3. ??????? ?????? ????????

• ????????? ????
• ??? ??????????, ????????? ??????????? ????????
  ????? ? ???????? (_authorid123)
• ??? ??????, ????????? ??? ? ??????
• ???????? ??????
• ??? ???????, ??? ??????
• ???????? ?????
• ???????? ??????, ?? ?????? ?? ?????

32
3. ??????? ?????? ????????

• ???? ?????? ?????? ???????? ????
• ???? ?????????? ????? ??????????
• ????? CPUIO, ?????? ?????? CPU
• ??????? ????-????? ????????????? ????????
  ???????
• ?????? ???????? ???? ?????????? ? ?????!
• ?????? ???????? ????? ?????????? ? ??????!

33
4. ??????? ? MTF ???????

• ??????? ???????????? ???????????????
• ? ???????, ????????? ???????????!
• ????? ??????? ????? ??????? ? ??????
• ????? ????????? ????? ??????? ? ?????
• ??? ?????? ????-??????? ?? ?????
• ???????? ?????? ? ???????

34
5. ?????????????

• ????? ??????? ????? ???????? ??????
• ?????? ???????? network roundtrip
• ?????? ????? ????????? ???????????
• ????? ?????? ? ?????? ?????? ?????? ??????
  ??????????-???????????
• 2x ??????????? ??????????? ??????

35
5. ?????????????
client new SphinxClient () q laptop //
coming from website user client-gtSetSortMode (
SPH_SORT_EXTENDED, _at_weight desc) client-gtAddQu
ery ( q, products ) client-gtSetGroupBy (
SPH_GROUPBY_ATTR, vendor_id ) client-gtAddQuery
( q, products ) client-gtResetGroupBy
() client-gtSetSortMode ( SPH_SORT_EXTENDED,
price asc ) client-gtSetLimit ( 0, 10
) result client-gtRunQueries ()
36
6. ??? ??????? ??????

• ???? ????? ?????? ?? ????????
• Cutoff (??. SetLimits())
• ??????? ?????? ????? N ?????? ??????????
• ? ?????? ???????, ?? ????????
• MaxQueryTime (??. SetMaxQueryTime())
• ??????? ?????? ????? M ???????????
• ? ?????? ???????, ?? ????????

37
6. ??? ??????? ??????

• ???? ????? ?????? ?? ????????
• Consulting ?
• ????? ???????? ????????????
• ????? ???????? ????????????

38
????? 3. ?????? ?????????????????
39
?????? ??????

• ???? 160M ?????-??????
• ????? ??? ??????? (?? ????????groupby)

1. row
domain_id
440682 link_id 15 url_from
http//www.insidegamer.nl/forum/viewtopic.php?t40
750 url_to http//xbox360achievements.
org/content/view/101/114/ anchor
NULL from_site_id 9835 from_forum_id
1818 from_author_id 282 from_message_id
2586 message_published 2006-09-30 000000
...
40
?????? ???

• ?????? ??????
• 8 ?????, 4x CPU, 5M ?????? ?? CPU
• ?????????? Sphinx
• ???????????? ????? MySQL
• ??????????? ??????
• ?????? 15-20M rows/CPU
• ????????? ???????? ?????? ???????

41
?????? ???

• ???????? ?????????? ????? URL ??? ??????????
  UDF-???
• ??????? ????????? ????????? ????????

1. row
url_from
http//www.insidegamer.nl/forum/viewtopic.php?t40
750 urlize(url_from,0) wwwinsidegamernl
insidegamernl
insidegamernlforum
insidegamernlforumviewtopic.php
insidegamernlforumviewtopic.phpt40750 ur
lize(url_from,1) wwwinsidegamernl
insidegamernl
insidegamernlforum
insidegamernlforumviewtopic.php
42
?????? ???

• 64 ???????
• 4 ????? searchd ?? ??????, ?? ????? CPU/HDD
• 2 ??????? (maindelta) ?? CPU
• ???????????? ???????????
• Web box ?????????? ??????? ????? searchd
• ??????? ?????????? ???? ? ????????? 3 ?????
• ?????????? 4 ?????, ?????? ??? startup/update
• ?????????? plain HDD, ?????? ??? IO stepping

43
??????????

• ???????? ?????????
• ?????? ?????? ??? ??????
• ?????? ?????? ? ???????? 0.5
• ??????? ????? ??????? 0.125 sec
• 90 ???????? ????? 0.227 sec
• 95 ???????? ????? 0.352 sec
• 99 ???????? ????? 2.888 sec

44
?????