Samba on Scale: Difference between revisions
From SambaWiki
Slowfranklin (talk | contribs) (Created page with "= Introduction = Samba on Scale is the place to collect ideas, research and prototypes about the future of Scale-out Samba, both on-prem and in the cloud. = List of scalable...") |
Slowfranklin (talk | contribs) |
||
(124 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
⚫ | |||
= Introduction = |
|||
For an overview of Consistency Models see [https://jepsen.io/consistency Jepsen] or [https://en.wikipedia.org/wiki/Consistency_model Wikipedia]. Note: LWT are defined as multiple operations on a single key. |
|||
Samba on Scale is the place to collect ideas, research and prototypes about the future of Scale-out Samba, both on-prem and in the cloud. |
|||
{| class="wikitable sortable" |
|||
⚫ | |||
|- |
|||
! Name !! Type || Consistency Model || Transactions || C Client || Notes |
|||
|- |
|||
| [https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba ctdb] || K/V || Volatile DBs: [https://arxiv.org/pdf/1512.00168.pdf Prefix-Linearizable?]<br />Persistent DBs: [https://jepsen.io/consistency/models/linearizable Strict Serializability] || No<br/>Yes || yes || |
|||
|- |
|||
| [https://etcd.io/ etcd] || K/V || [https://etcd.io/docs/v3.3/learning/api_guarantees/ Linearizability] || LWT ("mini") || yes || [https://etcd.io/docs/v3.3/learning/why/ Why?] |
|||
|- |
|||
| [https://docs.ceph.com/en/quincy/rados/api/librados-intro/ Ceph Rados] || K/V ||[http://www.cs.nthu.edu.tw/~ychung/conference/2016_ICPADS.pdf Linearizability] || LWT via Object Operations || yes || |
|||
|- |
|||
| [https://www.foundationdb.org/ FoundationDB] || K/V || [https://jepsen.io/consistency/models/strict-serializable Strict Serializability] || yes || yes || [http://charap.co/reading-group-foundationdb-a-distributed-unbundled-transactional-key-value-store/ Architecture review] |
|||
|- |
|||
| [https://www.yugabyte.com/ YugabyteDB] || SQL || [https://youtu.be/w_zYYF3-iSo?t=2001 Single-key Linearizability with LWT] || yes || yes || |
|||
|- |
|||
| [https://cassandra.apache.org/_/index.html Apache Cassandra 1-4] || CQL || Configurable, highest:<br />[https://cassandra.apache.org/doc/latest/cassandra/architecture/guarantees.html#lightweight-transactions-with-linearizable-consistency Single-key Linearizability with LWT]|| yes || [https://docs.datastax.com/en/developer/cpp-driver/2.16/topics/ yes] || |
|||
|- |
|||
| [https://cassandra.apache.org/_/index.html Apache Cassandra 5] || CQL || Upcoming:<br />[https://thenewstack.io/an-apache-cassandra-breakthrough-acid-transactions-at-scale/ Strict Serializability] [https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions?preview=/188744725/188744736/Accord.pdf]|| yes || [https://docs.datastax.com/en/developer/cpp-driver/2.16/topics/ yes] || |
|||
|- |
|||
| [https://www.cockroachlabs.com/ CockroachDB] || SQL || [https://www.cockroachlabs.com/blog/consistency-model/#cockroachdbs-consistency-model-more-than-serializable-less-than-strict-serializability Serializable+], [https://www.cockroachlabs.com/blog/db-consistency-isolation-terminology/ Strong Partition Serializability] || LWT || yes || [https://jepsen.io/analyses/cockroachdb-beta-20160829 Jepsen] |
|||
|- |
|||
| [https://www.scylladb.com/ ScyllaDB] || CQL || [https://docs.scylladb.com/stable/using-scylla/lwt.html Single-key Linearizability with LWT] || LWT || yes || [https://docs.scylladb.com/stable/kb/dpdk-hardware.html Intel DPDK via Seastar] |
|||
|- |
|||
| [https://tikv.org/ tikv] || K/V || [https://tikv.org/docs/3.0/reference/faq/ Strong Consistency][https://github.com/tikv/tikv/issues/13956] || yes (Percolator based) || no (C++) || |
|||
|- |
|||
| [https://pegasus.apache.org/ Apache Pegasus] || ? || ? || ? || no (C++) || |
|||
|- |
|||
| [https://hbase.apache.org/ Apache HBase] || ? || ? || ? || ? || |
|||
|- |
|||
| [https://ignite.apache.org/ Apache Ignite] || ? || ? || ? || ? || |
|||
|- |
|||
| [https://riak.com/ Riak] || K/V || [https://docs.riak.com/riak/kv/2.2.3/learn/concepts/strong-consistency.1.html Linearizability] || No || ? || |
|||
|} |
|||
= Rapid Samba dbwrap backend prototyping = |
|||
A new dbwrap backend called [https://git.samba.org/?p=slow/samba.git;a=shortlog;h=refs/heads/dbwrap_py dbwrap_py] that calls into Python for the database operations. Available backends: |
|||
* [https://git.samba.org/?p=slow/samba.git;a=blob;f=python/samba/samba3/dbwrap_py_fdb.py;h=bc16143a97844f88225b57c52462bc1e008642b2;hb=refs/heads/dbwrap_py FoundationDB] |
|||
* [https://git.samba.org/?p=slow/samba.git;a=blob;f=python/samba/samba3/dbwrap_py_tdb.py;h=1931ff2f43a3a522d402d20dfc3306b184b037d8;hb=refs/heads/dbwrap_py TDB] |
|||
= Strict-serializability, can we get away with something weaker? = |
|||
* [https://www.cockroachlabs.com/blog/db-consistency-isolation-terminology/ How to talk about consistency and isolation in distributed DBs] |
|||
* [https://dbmsmusings.blogspot.com/2019/06/correctness-anomalies-under.html Correctness Anomalies Under Serializable Isolation] |
|||
* [https://kousiknath.medium.com/consistency-guarantees-in-distributed-systems-explained-simply-720caa034116 Consistency Guarantees in Distributed Systems Explained Simply] |
|||
* [http://muratbuffalo.blogspot.com/2022/08/strict-serializability-but-at-what-cost.html Strict-serializability, but at what cost, for what purpose?] |
|||
* [https://www.cockroachlabs.com/blog/consistency-model/ CockroachDB's consistency model] |
|||
= List of Opensource Clustered Filesystems = |
= List of Opensource Clustered Filesystems = |
||
{| class="wikitable sortable" |
|||
|- |
|||
! Name !! Built on-top of Object Storage !! Notes |
|||
|- |
|||
| [https://ceph.com Ceph] || yes || |
|||
|- |
|||
| [https://gluster.org GlusterFS] || no || |
|||
|- |
|||
| [https://juicefs.com/en/ JuiceFS] || yes || |
|||
|- |
|||
| [https://moosefs.com/ MooseFS] || no || |
|||
|- |
|||
| [https://objectivefs.com/ ObjectiveFS] || yes || |
|||
|- |
|||
| [https://www.lustre.org/ Lustre] || no || |
|||
|- |
|||
| [https://www.beegfs.io/ BeegFS] || no || |
|||
|} |
|||
= Links = |
|||
* [https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf Replicated Data Consistency Explained Through Baseball] |
|||
* [https://fauna.com/blog/comparison-of-scalable-database-isolation-levels A Comparison of Scalable Database Isolation Levels] |
|||
* [https://fauna.com/blog/demystifying-database-systems-correctness-anomalies-under-serializable-isolation Correctness Anomalies Under Serializable Isolation] |
|||
* [https://accelazh.github.io/storage/Linearizability-Vs-Serializability-And-Distributed-Transactions Linearizability vs Serializability, and Distributed Transactions] |
|||
* [https://docs.yugabyte.com/preview/faq/comparisons/ Comparison of a few SQL and no-SQL DBs] |
|||
* [https://li-boxuan.medium.com/distributed-transaction-in-database-from-epaxos-to-accord-6de7999ad08e Summary of Paxos, Multipaxos, Raft, Epaxos and ACCORD] |
|||
* [https://www.cockroachlabs.com/blog/consistency-model/ Nice Summary of Consistency Models and Issues in CockroachDB] |
|||
* [https://forums.foundationdb.org/t/proof-of-concept-filesystem-layer/1605 FUSE Filesystem on-top of FoundationDB] |
|||
* [https://github.com/brianfrankcooper/YCSB YCSB] |
|||
* [https://benchant.com/de/ranking/datenbank-ranking Benchant YCSB Ranking] |
|||
* [https://www.mydistributed.systems/2020/08/foundationdb.html How FoundationDB works] |
|||
* [https://www.youtube.com/watch?v=xeQHa3Z-d-A Fast General Purpose Transactions in Apache Cassandra] |
|||
* [https://www.youtube.com/watch?v=w_zYYF3-iSo Comparing Distributed Transaction Architectures for the Cloud Era] |
|||
* [https://pdos.csail.mit.edu/6.824/index.html MIT Distributed Systems Course] |
|||
* [http://www.vldb.org/pvldb/vol7/p181-bailis.pdf Highly Available Transactions: Virtues and Limitations] |
|||
* [https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45855.pdf Spanner, TrueTime and The CAP Theorem] |
|||
* [https://paxos.systems/ Paxos Made Moderately Complex] |
Latest revision as of 13:03, 6 May 2023
List of scalable Key/Value Stores
For an overview of Consistency Models see Jepsen or Wikipedia. Note: LWT are defined as multiple operations on a single key.
Name | Type | Consistency Model | Transactions | C Client | Notes |
---|---|---|---|---|---|
ctdb | K/V | Volatile DBs: Prefix-Linearizable? Persistent DBs: Strict Serializability |
No Yes |
yes | |
etcd | K/V | Linearizability | LWT ("mini") | yes | Why? |
Ceph Rados | K/V | Linearizability | LWT via Object Operations | yes | |
FoundationDB | K/V | Strict Serializability | yes | yes | Architecture review |
YugabyteDB | SQL | Single-key Linearizability with LWT | yes | yes | |
Apache Cassandra 1-4 | CQL | Configurable, highest: Single-key Linearizability with LWT |
yes | yes | |
Apache Cassandra 5 | CQL | Upcoming: Strict Serializability [1] |
yes | yes | |
CockroachDB | SQL | Serializable+, Strong Partition Serializability | LWT | yes | Jepsen |
ScyllaDB | CQL | Single-key Linearizability with LWT | LWT | yes | Intel DPDK via Seastar |
tikv | K/V | Strong Consistency[2] | yes (Percolator based) | no (C++) | |
Apache Pegasus | ? | ? | ? | no (C++) | |
Apache HBase | ? | ? | ? | ? | |
Apache Ignite | ? | ? | ? | ? | |
Riak | K/V | Linearizability | No | ? |
Rapid Samba dbwrap backend prototyping
A new dbwrap backend called dbwrap_py that calls into Python for the database operations. Available backends:
Strict-serializability, can we get away with something weaker?
- How to talk about consistency and isolation in distributed DBs
- Correctness Anomalies Under Serializable Isolation
- Consistency Guarantees in Distributed Systems Explained Simply
- Strict-serializability, but at what cost, for what purpose?
- CockroachDB's consistency model
List of Opensource Clustered Filesystems
Name | Built on-top of Object Storage | Notes |
---|---|---|
Ceph | yes | |
GlusterFS | no | |
JuiceFS | yes | |
MooseFS | no | |
ObjectiveFS | yes | |
Lustre | no | |
BeegFS | no |
Links
- Replicated Data Consistency Explained Through Baseball
- A Comparison of Scalable Database Isolation Levels
- Correctness Anomalies Under Serializable Isolation
- Linearizability vs Serializability, and Distributed Transactions
- Comparison of a few SQL and no-SQL DBs
- Summary of Paxos, Multipaxos, Raft, Epaxos and ACCORD
- Nice Summary of Consistency Models and Issues in CockroachDB
- FUSE Filesystem on-top of FoundationDB
- YCSB
- Benchant YCSB Ranking
- How FoundationDB works
- Fast General Purpose Transactions in Apache Cassandra
- Comparing Distributed Transaction Architectures for the Cloud Era
- MIT Distributed Systems Course
- Highly Available Transactions: Virtues and Limitations
- Spanner, TrueTime and The CAP Theorem
- Paxos Made Moderately Complex