Changes between Version 82 and Version 83 of SatelliteBigData

Show
Ignore:
Timestamp:
2009/04/06 20:29:40 (15 years ago)
Author:
mn
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • SatelliteBigData

    v82 v83  
    6565Really big objects such as the data from simulations [IS THIS CORRECT?] require specialized storage systems such as [http://oss.sgi.com/projects/xfs/ XFS],[http://opensolaris.org/os/community/zfs/ ZFS], [http://wiki.lustre.org/index.php?title=Main_Page Lustre],  [http://www.pvfs.org/ PVFS2] or future Linux filesystem [http://btrfs.wiki.kernel.org/index.php/Main_Page Brtfs]. 
    6666 
    67 In contrast to the above, diffraction results, microarray results or next-gen sequencing reads involve a largish number of objects which become more difficult to query. They are typically still stored in RDBMS but might require some tweaking that digresses from a normalized relational database model, for example databases based on a key/value model (e.g. [http://www.oracle.com/technology/products/berkeley-db/index.html BerkeleyDB], [http://tokyocabinet.sourceforge.net/index.html Tokyo Cabinet],  BigTable, [http://hadoop.apache.org/core/ Hadoop] ). 
     67In contrast to the above, diffraction results, microarray results or next-gen sequencing reads involve a largish number of objects which become more difficult to query. They are typically still stored in RDBMS but might require some tweaking that digresses from a normalized relational database model, for example databases based on a key/value model (e.g. [http://www.oracle.com/technology/products/berkeley-db/index.html BerkeleyDB], [http://tokyocabinet.sourceforge.net/index.html Tokyo Cabinet],  [http://en.wikipedia.org/wiki/BigTable BigTable], [http://hadoop.apache.org/core/ Hadoop] ). 
    6868Apart from obvious things to do such as creating good indices, further optimization can be found by using as few joins as possible and therefore organizing the data so that it can be stored in 2 or 3 tables/indexes (e.g. eeDB). Another alternative could be the usage of specialized storage systems, like the ones used in high energy physics experiments or astronomy (for instance [http://www.hdfgroup.org/HDF5/ HDF5]). 
    6969