Peace S/W utilities - 100 row inserts per power bill in CO
PostgreSQL9 has more unstructured data features
up-front schema design important
ACID, BASE -- eventual consistency -- concensus problem
can't do shared nothing with bank balance e.g.
oops - Guy left!
hybrid solutions - relational plus noSQL -- by hand
Guy's back! :)
FOO - NYC? show us your stack
gone off MongoDB
if mySQL solves problem, use it
PostgreSQL h-store engine? new
high-write vs high-read/query reqs
how to develop
SQL not relational (?) - R O'B
flume central server
OLTP + analytical - conflicting needs
solution: replicate (denormalize)?
Guy : what about huge records (but not many of them necessarily?
MongoDB grid file system
e.g. Radio Telescope etc.
CFD struct mech
hard to nail down rigid schema for metadata
(link from Mark Derricutt to Guy)
TB per record
x-ray @ cellular level 30GB per record / blob
Provenance - algorithm version etc. - where data came from and how processed
Guy involved with EU grid provenance project til 2006
use cases organ transplant privacy
airflow / CFD/ CAD meshes - pressure on surface - FEM mech deformation
100 CPU hours per run - don't want to rerun when forgot little thing like algo version
Rob O'B no longer complete self-contained record in one place - use queues more
Gear-man - Tim ?? session
graph database - transactions, actor state
XML db's Termino? (e)Xist? Berkeley (db) ?
Stonebraker - destroying his reputation / selling soul?
all redundancy in H/W - everything in RAM
Mark D Spring data - hibernate
Tokyo? Kyoto? Sky were using Tokyo?
Hadoop / RIACK / Cassandra
Mongo most popular by far ?
promoted self by speed claims (but will lose data sometimes)
Guy teaching in Honors paper
PIG query based langs
couch targeting mobile
big data can be quite small
Data Science - O'Reilly - Data Size?
u4j New4j? scientific data (who?) neo4j
not transactional use case
=> graphs analysis
millions of nodes - look for patterns
perturbations - see whether nwk evolves
neo4j GPL cut down - AfferoGPL license for full ACID-compliant version
designed to work around 'the cloud hole'
AGPL - must make code available even if not distributing modified project
torquebase? (Tim ?)
gremlin? query language on top of Neo4J
HF trading = FPGA GPU etc coming in - big $ for sub-ms ping times in same datacentre as market servers
need to be first
Oracle license $1M US - Peace
2days on paper - call centre