Are there canonical books, resources, or readings for how to design data structures that will be primarily read and written to a disk rather than memory? Most of what I learned in school about big-O assumes that, for example, random access is O(1). However, random disk reads are really slow due to spacial locality.
People who write databases obviously have solutions to this problem - for example, DuckDB is based on a number of papers that have come out over the years on this topic.
If I wanted to design, ie, a tree structure which was intended to be read/written from a disk, are there general principles or patterns the have been developed to take advantage of locality of reference, minimize random reads, or decrease the overhead of writes, that I could familiarize myself with?
What is the CLRS for disk?
Comments URL: https://news.ycombinator.com/item?id=32965075
Points: 8
# Comments: 5
Continue reading...
People who write databases obviously have solutions to this problem - for example, DuckDB is based on a number of papers that have come out over the years on this topic.
If I wanted to design, ie, a tree structure which was intended to be read/written from a disk, are there general principles or patterns the have been developed to take advantage of locality of reference, minimize random reads, or decrease the overhead of writes, that I could familiarize myself with?
What is the CLRS for disk?
Comments URL: https://news.ycombinator.com/item?id=32965075
Points: 8
# Comments: 5
Continue reading...