Satellite meeting for handling next-generation datasets

Topics

As more and more projects are spewing out datasets we might have to look at new paradigms of working with them. The conventional way of storing everything in a database and querying with SQL has become challenging or just impossible. Data are now more often stored in their original data formats.

The object of this meeting is to exchange ideas on this topic and discuss possible solutions or practices.

Topics:

• Next-gen sequencing
• Protein-interacting networks?
• Cloud storage/compute?

Chairperson

• Jan Aerts
• Jessica Severin

Date

• 18th March, 2009

Room

• Meeting Room (1F)

Attendees

• Jan Aerts
• Jessica Severin
• José María Fernández
• Todd Harris
• Yunsun Nam
• Pierre Lindenbaum
• Keun-Joon Park
• Raoul Jean Pierre Bonnal
• Yasukazu "yaskaz" Nakamura (yaskaz@…)

Notes

Data storage will not be a problem (if you have enogh money). But downloading and/or manipulating Big sequence data must be problem for data producers (Sanger, RIKEN OSC), data distribution providers (NCBI, EBI/SRA, DDBJ) and end users after the data is made public. Is there a possible solution?

And I have a big question: Do you really need and/or use "Short Read Archive?" -- yaskaz

• yes, there is interesting biology in there and comparing our new experiments against existing knowledge is needed, but currently this requires us to download data out of SRA and into our local integration systems - jessica

Discussion

The purpose of this workshop was to discuss the approaches people use to handle their data. We could obviously not go into the very technical approaches as are being applied to very large datasets such as the Short Read Archive because that requires a different expertise.

Three different levels in the data handling pipeline were discussed.

Storage

We listed the different types of storage solution used in different fields according to the amount and type of data. A distinction was made between storing a lot of small objects versus storing a few huge objects. Issues like latency and variety of data influence this distinction as well.

Protein-protein interaction datasets typically consist of a relatively small number of small objects. This type of data requires no advanced storage systems or tweaks to common systems; a simple RDBMS will do. Data like genome sequences and assembly data involve larger objects (1 genome sequence -> 3Gb), but are still easily manageable on a standard filesystem. Really big objects such as the data from simulations [IS THIS CORRECT?] require specialized storage systems such as ZFS, lustre or PVFS.

In contrast to the above, diffraction results, microarray results or next-gen sequencing reads involve a largish number of objects which become more difficult to query. They are typically still stored in RDBMS but might require some tweaking that digresses from a normalized relational database model. Apart from obvious things to do such as creating good indices, further optimization can be found by using as few joins as possible and therefore organizing the data so that it can be stored in 2 or 3 tables/indexes (e.g. eeDB).

Several attendees are looking into new ways to store their data because they are hitting the ceiling of their storage capacity. Several technologies were mentioned, including OGSADAI which is a grid-based solution where you can setup several databases that then can be queried as one. Other technologies like the cloud might also provide part of a solution. Amazon S3 and GoogleBase? allow for storing very large amounts of data (large numbers of large objects) and are relatively cheap. The upload is however very slow to these systems which must be taken into account. In addition, these services are commercial and it might be dangerous to store non-public data there. A possible solution mentioned involves creating an encrypted data image to be uploaded instead of the original dataset. Another issue is that these companies might in the future decide to increase their prices or even stop their activities. Using the cloud for data storage therefore means that you still need a local backup in case this happens.

[Jessica: can you add your vertical red line to this as well?]

Querying

Several approaches to make the data accessible to others were discussed. For smaller datasets regular SQL can be used to get to the data (example: the Ensembl MySQL server on ensembldb.ensembl.org). If the database schema has to be tweaked to allow larger datasets (e.g. using a minimal number of tables such as eeDB) an API becomes necessary. A RESTful API where an object can be retrieved by URL was mentioned as useful in this case. For even larger datasets you want to try and limit the types of queries people can run. This way you can build a toolkit to ask this limited number of questions, but optimize that toolkit so that latency is reduced.

Streaming

[SQL, File indexes] -> API -> toolkit -> [web, clients, mashup-analyzers]

^{| | | +----------------------+}

Processing

Apart from storage and getting the data out again we briefly discussed how to actually work with these data, i.e. running scripts on them. It is clear that shipment of large datasets should be avoided; it is often much simpler to bring the software to the data instead.

The main message from this discussion was that each database should be optimized for its own data and will therefore be completely different from the next. However, it is the API that can provide interoperability if we can come up with a common set of terms for querying. We believe that current web-technologies like mashups based on very simple APIs like Flickr's can be an alternative to other grid-based approached for integrating data.

Work goup discussion topics

next gen sequencer WG (big data and seq analysis)

applications and use cases:

• all processing must be run on a server, can not be run on a laptop, how to control and monitor such processes from something like a laptop
• local vs webservice. web service convenient but maybe not so secure which means people need to install our systems locally. but also collaboration needs access to data and tools
• how to define the quality of sequence? contamination, auto screen tool?
• moving to a data-driven research, not so much hypothesis driven. how to FIND biology in big datasets?
• searching / comparing now mostly happening on the metadata (time series, tissue, environments)
• metagenomics or novel genomes, no reference, can our tools do this? How?
• maybe we get a strange sequence out of our instrument, how do we know? is it error? is it interesting biology? CG content off, maybe technical errors (concatonated primers)
• how to bring the computation to the data because bring the data to the computation (1TB streaming) maybe not best approach
• as sequencer companies build more and more sequence processing into the instruments, more and more processing becomes routine. it is a moving target. how do we aid in new biological discovery? moving beyond simple processing and simple work flows.

what is big data?

• read level, contigs, assembly, scaffold each different level
• amount vs complexity!! maybe amount not so hard, but complexity ....
• variation, snp
• data is big when the applications can not handle it. If 1GB of data crashes a viewer program than that is BIG. If 100k elements takes a day to upload, this upload must happen every time we want to work with the data, and it takes another day to filter for a service, then that is probably BIG data.

How to collaborate and communicate?

• emailing files or providing files on secure FTP sites works for now
• reducing dataset complexity by processing can make BIG data small, but does this throw away biology?
• secure webservices are maybe a much better solution where more advanced tools can be built which can work with sections of the dataset, but this requires the ability to provide sub-access queries into these datasets

Data production centers

• RIKEN OSC-LSA  http://www.osc.riken.jp/ is producing lots of data, but this data must be managed, manipulated, and mined for biology before it can be released. EdgeExpressDB (eeDB) was developed during FANTOM4 project and is now being used for in-house big data management and visualization of big datasets. eeDB is effectively an object-database which is implemented as an API and webservices. The system will be ported to C and file indexes this summer which will give us at least a 100x performance boost. Currently this API toolkit and webservices are written in perl with a narrow/deep mysql snowflake schema. This generation1 system of the API can manipulate short-read data for our internal research purposes and is proving to scale very well. eeDB works with node and network, sequence tag, mapping, and expression data at the level of billions of elements very easily. Queries can access individual objects, edges, and work with streams or sets of objects queried by regions, node, or networks.
• SRA is still evaluating technology to do region based access of short-reads

Working with existing big data

• SRA, GEO, ArrayExpress?
• now most of us pull the whole thing down and then work with it
• sometimes it is even hard to send for submission, sometimes DVD to move it around. not optimal.
• what are the queries we want to do?
  • what are the reads in this region, definitely
• but also SRA maybe does not want do everything, but they will not turn data down and want everyone to send them the published data
• maybe not all data will end up in ONE archive (because it is so big). maybe need to query multiple centers to find all data (DDBJ, SRA, GEO, ArrayExpress?, korea? China?)

What are the common ways we would want to query?

• should we try to define? yes
• who should or can provide the data?
  • maybe the the data production centers can also provide access to data
  • eventually the DataCenters? will also provide fine-grained access
  • the new high data production rates are going to make it expensive for central data providers
• region query again we need! means mapping needed by the services
• but also want the read Sequence since we can extract SNP, need quality
• NEED data providers to create complete meta-data descriptions.

IN THE END the goal is to find biology. Having access to the individual data elements is critical and this can not be just locked away inside files that can not be internally accessed.

TODOs