| 43 | | == Work goup discussions == |
| | 46 | The purpose of this workshop was to discuss the approaches people use to handle their data. We could obviously not go into the very technical approaches as are being applied to very large datasets such as the Short Read Archive because that requires a different expertise. |
| | 47 | |
| | 48 | Three different levels in the data handling pipeline were discussed. |
| | 49 | |
| | 50 | ==== Storage ==== |
| | 51 | |
| | 52 | We listed the different types of storage solution used in different fields according to the amount and type of data. A distinction was made between storing a lot of small objects versus storing a few huge objects. Issues like latency and variety of data influence this distinction as well. |
| | 53 | |
| | 54 | Protein-protein interaction datasets typically consist of a relatively small number of small objects. This type of data requires no advanced storage systems or tweaks to common systems; a simple RDBMS will do. |
| | 55 | Data like genome sequences and assembly data involve larger objects (1 genome sequence -> 3Gb), but are still easily manageable on a standard filesystem. |
| | 56 | Really big objects such as the data from simulations [IS THIS CORRECT?] require specialized storage systems such as ZFS, lustre or PVFS. |
| | 57 | |
| | 58 | In contrast to the above, diffraction results, microarray results or next-gen sequencing reads involve a largish number of objects which become more difficult to query. They are typically still stored in RDBMS but might require some tweaking that digresses from a normalized relational database model. Apart from obvious things to do such as creating good indices, further optimization can be found by using as few joins as possible and therefore organizing the data so that it can be stored in 2 or 3 tables (e.g. eeDB). |
| | 59 | |
| | 60 | Several attendees are looking into new ways to store their data because they are hitting the ceiling of their storage capacity. Several technologies were mentioned, including OGSADAI which is a grid-based solution where you can setup several databases that then can be queried as one. Other technologies like the cloud might also provide part of a solution. Amazon S3 and GoogleBase allow for storing very large amounts of data (large numbers of large objects) and are relatively cheap. The upload is however very slow to these systems which must be taken into account. In addition, these services are commercial and it might be dangerous to store non-public data there. A possible solution mentioned involves creating an encrypted data image to be uploaded instead of the original dataset. Another issue is that these companies might in the future decide to increase their prices or even stop their activities. Using the cloud for data storage therefore means that you still need a local backup in case this happens. |
| | 61 | |
| | 62 | [Jessica: can you add your vertical red line to this as well?] |
| | 63 | |
| | 64 | ==== Querying ==== |
| | 65 | |
| | 66 | Several approaches to make the data accessible to others were discussed. For smaller datasets regular SQL can be used to get to the data (example: the Ensembl MySQL server on ensembldb.ensembl.org). If the database schema has to be tweaked to allow larger datasets (e.g. using a minimal number of tables such as eeDB) an API becomes necessary. A RESTful API where an object can be retrieved by URL was mentioned as useful in this case. For even larger datasets you want to try and limit the types of queries people can run. This way you can build a toolkit to ask this limited number of questions, but optimize that toolkit so that latency is reduced. |
| | 67 | |
| | 68 | Streaming |
| | 69 | |
| | 70 | SQL -> API -> toolkit |
| | 71 | ^ | |
| | 72 | | | |
| | 73 | +------+ |
| | 74 | |
| | 75 | ==== Processing ==== |
| | 76 | |
| | 77 | Apart from storage and getting the data out again we briefly discussed how to actually work with these data, i.e. running scripts on them. It is clear that shipment of large datasets should be avoided; it is often much simpler to bring the software to the data instead. |
| | 78 | |
| | 79 | The main message from this discussion was that each database should be optimized for its own data and will therefore be completely different from the next. However, it is the API that can provide interoperability if we can come up with a common set of terms for querying. We believe that current web-technologies like mashups based on very simple APIs like Flickr's can be an alternative to other grid-based approached for integrating data. |
| | 80 | |
| | 81 | == Work goup discussion topics == |
