Context Navigation

Version 49 (modified by plindenbaum, 16 years ago)
--

Satellite meeting for handling next-generation datasets

Topics

As more and more projects are spewing out datasets we might have to look at new ways of working with them. Really big datasets (e.g. SRA) require technical expertise other than what we can provide. But we can still try and learn from each other on how to handle our own datasets.

The object of this meeting is to exchange ideas on this topic and discuss possible solutions or practices.

Topics:

Next-gen sequencing
Protein-interacting networks?
Cloud storage/compute?

Chairperson

Jan Aerts
Jessica Severin

Date

18th March, 2009

Room

Meeting Room (1F)

Attendees

Jan Aerts
Jessica Severin
José María Fernández
Todd Harris
Yunsun Nam
Pierre Lindenbaum
Keun-Joon Park
Raoul Jean Pierre Bonnal
Yasukazu "yaskaz" Nakamura (yaskaz@…)
Takeshi Kawashima
Shuichi Kawashima
Fumikazu Konishi

Notes

Data storage will not be a problem (if you have enogh money). But downloading and/or manipulating Big sequence data must be problem for data producers (Sanger, RIKEN OSC), data distribution providers (NCBI, EBI/SRA, DDBJ) and end users after the data is made public. Is there a possible solution?

And I have a big question: Do you really need and/or use "Short Read Archive?" -- yaskaz

yes, there is interesting biology in there and comparing our new experiments against existing knowledge is needed, but currently this requires us to download data out of SRA and into our local integration systems - jessica
yes and no: No, for the short reads where the position on a reference seq is known you could just store the position rather than the entire read. Yes, imagine a user find a new unknown SNP at a given position: he will want to get the sequences/quality(?) of all the reads overlapping the mutation for this assembly as well as for the other individuals' genome. - Pierre
- this is only true for genome resquencing. for RNA->genome mapping data you do want the read since it can capture RNA modifications which are not the same as a SNP -- jessica

Discussion

The purpose of this workshop was to discuss the approaches people use to handle their data. We could obviously not go into the very technical approaches as are being applied to very large datasets such as the Short Read Archive because that requires a different expertise.

Three different levels in the data handling pipeline were discussed.

Storage

We listed the different types of storage solution used in different fields according to the amount and type of data. A distinction was made between storing a lot of small objects versus storing a few huge objects. Issues like latency and variety of data influence this distinction as well.

Protein-protein interaction datasets typically consist of a relatively small number of small objects. This type of data requires no advanced storage systems or tweaks to common systems; a simple RDBMS will do. Data like genome sequences and assembly data involve larger objects (1 genome sequence -> 3Gb), but are still easily manageable on a standard filesystem. Really big objects such as the data from simulations [IS THIS CORRECT?] require specialized storage systems such as ZFS, Lustre, XFS, PVFS2 or future Linux filesystem Brtfs.

In contrast to the above, diffraction results, microarray results or next-gen sequencing reads involve a largish number of objects which become more difficult to query. They are typically still stored in RDBMS but might require some tweaking that digresses from a normalized relational database model, for example databases based on a key/value model (e.g. BerkeleyDB, Tokyo Cabinet, BigTable?, Hadoop ). Apart from obvious things to do such as creating good indices, further optimization can be found by using as few joins as possible and therefore organizing the data so that it can be stored in 2 or 3 tables/indexes (e.g. eeDB). Another alternative could be the usage of specialized storage systems, like the ones used in high energy physics experiments or astronomy (for instance HDF5).

Several attendees are looking into new ways to store their data because they are hitting the ceiling of their storage capacity. Several technologies were mentioned, including OGSADAI which is a grid-based solution where you can setup several databases that then can be queried as one. Other technologies like the cloud might also provide part of a solution. Amazon S3 and GoogleBase allow for storing very large amounts of data (large numbers of large objects) and are relatively cheap. The upload to these systems is however very slow which must be taken into account. In addition, these services are commercial and it might be dangerous to store non-public data there. A possible solution mentioned involves creating an encrypted data image to be uploaded instead of the original dataset. Another issue is that these companies might in the future decide to increase their prices or even stop their activities. Using the cloud for data storage therefore means that you still need a local backup in case this happens.

In addition to the storage problem there is the issue of actually submitting our new data to public resources when publishing. Our datasets are getting very big and FTP often is not good enough. This forces many of us to have to submit data on DVD via FedEx? for example. This is not optimal.

[Jessica: can you add your vertical red line to this as well?]

Querying

Several approaches to make the data accessible to others were discussed. For smaller datasets regular SQL can be used to get to the data (example: the Ensembl MySQL server on ensembldb.ensembl.org). If the database schema has to be tweaked to allow larger datasets (e.g. using a minimal number of tables such as eeDB) an API becomes necessary. A webservice API where a text-representation of an object can be retrieved by URL was mentioned as useful in this case (e.g. http://www.example.com/genes/BRCA2;format=bed). For even larger datasets you want to try and limit the types of queries people can run. This way you can build a toolkit to ask this limited number of questions, but optimize that toolkit so that latency is reduced.

Streaming

  [SQL, File indexes] -> API -> toolkit -> [web, clients, mashup-analyzers]
   ^                      |
   |                      |
   +----------------------+

Currently the available public resources like SRA, GEO, ArrayExpress? are only providing query facilities on the metadata of the experiments surrounding the data. The data is available as files to download (often in the original format) but they do not provide facilities to externally explore the data and ask biological questions on the data. This then forces anyone who wants to explore the dataset to download this data into local integration systems before they can ask their biological questions.

Working with existing big data

SRA, GEO, ArrayExpress?: today they just provide the metadata of the dataset, not an ability to explore the actual data
now most of us pull the whole thing down and then work with it
sometimes it is even hard to send for submission, sometimes DVD to move it around. not optimal.
what are the queries we want to do?
- what are the reads in this region, definitely
but also SRA maybe does not want do everything, but they will not turn data down and want everyone to send them the published data
maybe not all data will end up in ONE archive (because it is so big). maybe need to query multiple centers to find all data (DDBJ, SRA, GEO, ArrayExpress?, korea? China?)

Processing

Apart from storage and getting the data out again we briefly discussed how to actually work with these data, i.e. running scripts on them. It is clear that shipment of large datasets should be avoided; it is often much simpler to bring the software to the data instead.

The main message from this discussion was that each database should be optimized for its own data and will therefore be completely different from the next. However, it is the API that can provide interoperability if we can come up with a common set of terms for querying. We believe that current web-technologies like mashups based on very simple APIs like Flickr's can be an alternative to other grid-based approached for integrating data.

Work group discussion topics

next gen sequencer WG (big data and seq analysis)

applications and use cases:

all processing must be run on a server, can not be run on a laptop, how to control and monitor such processes from something like a laptop
local vs webservice. web service convenient but maybe not so secure which means people need to install our systems locally. but also collaboration needs access to data and tools
how to define the quality of sequence? contamination, auto screen tool?
moving to a data-driven research, not so much hypothesis driven. how to FIND biology in big datasets?
searching / comparing now mostly happening on the metadata (time series, tissue, environments)
metagenomics or novel genomes, no reference, can our tools do this? How?
maybe we get a strange sequence out of our instrument, how do we know? is it error? is it interesting biology? CG content off, maybe technical errors (concatonated primers)
how to bring the computation to the data because bring the data to the computation (1TB streaming) maybe not best approach
as sequencer companies build more and more sequence processing into the instruments, more and more processing becomes routine. it is a moving target. how do we aid in new biological discovery? moving beyond simple processing and simple work flows.

what is big data?

read level, contigs, assembly, scaffold each different level
amount vs complexity!! maybe amount not so hard, but complexity ....
variation, snp
data is big when the applications can not handle it. If 1GB of data crashes a viewer program than that is BIG. If 100k elements takes a day to upload, this upload must happen every time we want to work with the data, and it takes another day to filter for a service, then that is probably BIG data.
is it really needed to store everything ? -- Raoul
- Compressing data in some way?
  - collapse redundant informations ( possible for resequencing and de-novo sequencing )

How to collaborate and communicate?

emailing files or providing files on secure FTP sites works for now
reducing dataset complexity by processing can make BIG data small, but does this throw away biology?
secure webservices are maybe a much better solution where more advanced tools can be built which can work with sections of the dataset, but this requires the ability to provide sub-access queries into these datasets

Data production centers

RIKEN OSC-LSA http://www.osc.riken.jp/ is producing lots of data, but this data must be managed, manipulated, and mined for biology before it can be released. EdgeExpressDB (eeDB) was developed during FANTOM4 project and is now being used for in-house big data management and visualization of big datasets. eeDB is effectively an object-database which is implemented as an API and webservices. The system will be ported to C and file indexes this summer which will give us at least a 100x performance boost. Currently this API toolkit and webservices are written in perl with a narrow/deep mysql snowflake schema. This generation1 system of the API can manipulate short-read data for our internal research purposes and is proving to scale very well. eeDB works with node and network, sequence tag, mapping, and expression data at the level of billions of elements very easily. Queries can access individual objects, edges, and work with streams or sets of objects queried by regions, node, or networks.
SRA is still evaluating technology to do region based access of short-reads