A Storage Engine for Amazon S3

1
A Storage Engine for Amazon S3

• How to store a hundred billion BLOBs without
  buying a single disk.

2
Who is this guy?
3
http//fallenpegasus.com/code
4
Storage Engine
5
Protocol Translator
6
TraditionalStorage Enginesuse the local disk
7
Network Storage Engines
8

• Federated
• ODBC
• HTTP
• MemCacheD
• and...

9
S3
10
What is S3?
11
Petabytes?!
12
This can't be free
13
Why do I want to use it?
14
An empty disk costs the same as a full one.
15
Can I move myexisting database over?
16
Translating S3 to MySQL

• AWS Account ? CREATE SERVER
• S3 Bucket ? Table
• S3 Item ? Row
• S3 Item Key ? VARCHAR PRIMARY KEY
• S3 Item Contents ? BLOB or VARCHAR

17

• CREATE SERVER 'MyAWSAcct'
• FOREIGN DATA WRAPPER 'AWS'
• OPTIONS
• (USER 'aws id string',
• PASSWORD 'aws secret string')

18

• CREATE TABLE 'bierce' (
• 'word' VARCHAR(255) NOT NULL PRIMARY KEY,
• 'defn' BLOB)
• CHARSETutf-8
• ENGINEAWSS3
• CONNECTION'awss3 DevilDictionary server
  MyAWSAcct'

19

• SELECT defn FROM bierce WHERE word'WIT'

20

• SELECT defn FROM bierce WHERE word'WIT'
• WIT, n. The salt with which the American humorist
  spoils his intellectual cookery by leaving it out.

21

• INSERT INTO bierce (word, defn) VALUES
• ('AUTHOR',
• 'One noted for confusing bitterness with humor.')

22

• DELETE FROM bierce WHERE word'AUTHOR'

23
So, what's it good for?
24
If you build it,they will come.
25

• Saving EC2 work
• SQL CMS for S3 virtual hosting
• Huge list of persistent primary keys
• Big slow BLOBs, to join against fast local
  tables.
• The Image Server Problem
• ... and more

26
Transactions?
27
What do you mean,No temporal guarantees?!
28
No WHERE clausemeansMAKE MONEY FASTfor Amazon
29
The Future(watch me wave my hands)
30

• Code Improvement
• S3 HTTP Metadata
• Multiple Data Columns
• Information Schema
• Security Authentication
• Transfer Storage Compression
• Streaming BLOB
• More Storage Engines

31
Code Improvement
32
S3 HTTPMetadata
33
Multiple Data Columns
34
Sharing a solution withHTTP and MemCacheDEngines
35
Information Schema
36
Security Authentication
37
Transfer StorageCompression
38
(new slide)Streaming BLOBslike PBXT
39
More storage engines
40
More storage engines

• A Storage Engine for AWS SQS
• There are more AWS services rolling out
• Replication distribution via S3
• Cluster persistent storage via S3
• ... and the EC2 holy grail

41
A Generic SchemaStorage Engine for S3
42
Back to the present...
43
Where do I get it?
44
Where do I get it?

• MySQL 5.1
• GPL
• http//fallenpegasus.com/code

45
You never write,you never call...
46

• Suggestions
• Use Cases
• Bug Reports
• Patches
• Feature Requests
• Support Requests
• Gigs Proposals
• mark_at_fallenpegasus.com

47
Questions?