etcd is an open source and highly available distributed key-value storage system and is commonly used in critical data storage and service discovery and registration use cases. It is focusing on:
- Simple: well-defined, user-facing API (gRPC)
- Secure: automatic TLS with optional client cert authentication
- Fast: benchmarked 10,000 writes/sec
- Reliable: properly distributed using Raft
etcd and Redis both support key-value storage and can be set up in distributed systems. Also Redis supporst more key value types. But they are used in different use cases and have different focus areas given below differences:
- Redis doesn't provide strong data consistency in distributed environment, also it cannot guarantee one can get the latest data
- Redis doesn't offer a good watch mechanism for data change
- etcd ensures strong data consistency but with poorer performance compared to Redis when low latency is needed
Below is a guide on installing etcd and using it.
- Go to etcd Github release page and download the available stable release.
- Unzip the downloaded package and you would find two binary: etcd and etcdctl
- Add the package location to OS Path and then can run below command to verify etcd server cam be started or not.
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead 2021-03-07 15:56:07.142971 I | etcdmain: etcd Version: 3.4.15 2021-03-07 15:56:07.826568 I | etcdmain: Git SHA: aa7126864 2021-03-07 15:56:07.831452 I | etcdmain: Go Version: go1.12.17 2021-03-07 15:56:07.831452 I | etcdmain: Go OS/Arch: windows/amd64 2021-03-07 15:56:07.832427 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4 2021-03-07 15:56:07.832427 N | etcdmain: failed to detect default host (default host not supported on windows_amd64) 2021-03-07 15:56:07.833404 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead 2021-03-07 15:56:07.867560 I | embed: name = default 2021-03-07 15:56:07.869512 I | embed: data dir = default.etcd 2021-03-07 15:56:07.869512 I | embed: member dir = default.etcd\member 2021-03-07 15:56:07.869512 I | embed: heartbeat = 100ms 2021-03-07 15:56:07.869512 I | embed: election = 1000ms 2021-03-07 15:56:07.869512 I | embed: snapshot count = 100000 2021-03-07 15:56:07.870492 I | embed: advertise client URLs = http://localhost:2379 2021-03-07 15:56:07.931517 I | wal: releasing file lock to rename "default.etcd\\member\\wal.tmp" to "default.etcd\\member\\wal" 2021-03-07 15:56:07.974514 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32 raft2021/03/07 15:56:07 INFO: 8e9e05c52164694d switched to configuration voters=() raft2021/03/07 15:56:07 INFO: 8e9e05c52164694d became follower at term 0 raft2021/03/07 15:56:07 INFO: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0] raft2021/03/07 15:56:07 INFO: 8e9e05c52164694d became follower at term 1 raft2021/03/07 15:56:07 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437) 2021-03-07 15:56:07.992597 W | auth: simple token is not cryptographically signed 2021-03-07 15:56:08.081499 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided] 2021-03-07 15:56:08.088330 E | etcdserver: cannot monitor file descriptor usage (cannot get FDUsage on windows) 2021-03-07 15:56:08.102970 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10) 2021-03-07 15:56:08.105975 I | embed: listening for peers on 127.0.0.1:2380 raft2021/03/07 15:56:08 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437) 2021-03-07 15:56:08.117690 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 raft2021/03/07 15:56:08 INFO: 8e9e05c52164694d is starting a new election at term 1 raft2021/03/07 15:56:08 INFO: 8e9e05c52164694d became candidate at term 2 raft2021/03/07 15:56:08 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2 raft2021/03/07 15:56:08 INFO: 8e9e05c52164694d became leader at term 2 raft2021/03/07 15:56:08 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2 2021-03-07 15:56:08.337163 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32 2021-03-07 15:56:08.353843 I | etcdserver: setting up the initial cluster version to 3.4 2021-03-07 15:56:08.376287 I | embed: ready to serve client requests 2021-03-07 15:56:08.387999 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged! 2021-03-07 15:56:08.414353 N | etcdserver/membership: set the initial cluster version to 3.4 2021-03-07 15:56:08.425087 I | etcdserver/api: enabled capabilities for version 3.4​If you see above, it means the server is started successfully.
- Now can run the client command etcdctl to verify the version.
C:\Users\pike6>etcdctl version etcdctl version: 3.4.15 API version: 3.4​It shows the client version and also the API version.
There are v2 and v3 versions of API supported, ETCDCTL_API can be set to choose which version of API to use. By default the version of API being used would be v3 for release 3.x and above.
Any key created with v2 API cannot be accessed with v3 API and vice versa.
Now given the etcd server is started with command etcd. Let's start a new terminal to run etcdctl to play around with it.
Write key
Since it's a key value pair, so can run below to set key foo and value bar.
C:\Users\pike6>etcdctl put foo bar
OKGet key
Get a key value pair is also easy, just run etcdctl get.
C:\Users\pike6>etcdctl get foo
foo
barIf only want to get the value corresponding to a key, can add option --print-value-only.
C:\Users\pike6>etcdctl get foo --print-value-only
barIf the key doesn't exist, it will print nothing.
C:\Users\pike6>etcdctl get foo1If wanna get some keys within some range, can run etcdctl get start end. Note, end index is excluded.
C:\Users\pike6>etcdctl put foo1 bar1
OK
C:\Users\pike6>etcdctl put foo2 bar2
OK
C:\Users\pike6>etcdctl put foo3 bar3
OK
C:\Users\pike6>etcdctl get foo1 foo3
foo1
bar1
foo2
bar2
C:\Users\pike6>etcdctl get foo1 foo3 --print-value-only
bar1
bar2If wanna get all with prefix foo, can run below.
C:\Users\pike6>etcdctl get --prefix foo --print-value-only
bar
bar1
bar2
bar3If wanna limit the number of keys to return, can add --limit option.
C:\Users\pike6>etcdctl get --prefix --limit=2 foo --print-value-only
bar
bar1Delete key
etcdctl del can be used to delete a key
C:\Users\pike6>etcdctl del foo
1The output will be the number of keys deleted.
If wanna delete a range, can do etcdctl del start end.
C:\Users\pike6>etcdctl del foo1 foo3
2If wanna delete key with prefix, add --prefix option.
C:\Users\pike6>etcdctl del --prefix foo
1Watch key
Sometimes a key value may have been updated by other client and there might be need to watch the change of a key so that a proper action can be taken by the corresponding client caring about the change. etcdctl watch can be used in such cases.
C:\Users\pike6>etcdctl watch foo
PUT
foo
barIf wanna watch multiple keys at the same time, can add option -i.
C:\Users\pike6>etcdctl watch -i
watch foo
watch zoo
PUT
foo
bar1
PUT
zoo
testGrant lease
etcd can have lease set to basically handle expiration of keys. If a key is granted with a lease, it will be deleted once the lease is reached.
A lease can be created by running etcdctl lease grant TTL. And the output will indicate the id of the granted lease.
C:\Users\pike6>etcdctl lease grant 60
lease 694d780bd7fccb0c granted with TTL(60s)Thereafter a key can be set with the granted lease so that it would be deleted automatically when the TTL reaches.
C:\Users\pike6>etcdctl put --lease=694d780bd7fccb0c foo bar
OK
C:\Users\pike6>etcdctl get foo
foo
bar
C:\Users\pike6>etcdctl get foo
foo
bar
C:\Users\pike6>etcdctl get foo
Revoke lease
Once a lease is granted and if in case wanna revoke the lease, it can be done with etcdctl revoke lease ID. When the lease is revoked, all associated key will be deleted.
C:\Users\pike6>etcdctl lease grant 60
lease 694d780bd7fccb13 granted with TTL(60s)
C:\Users\pike6>etcdctl put --lease=694d780bd7fccb13 foo bar
OK
C:\Users\pike6>etcdctl get foo
foo
bar
C:\Users\pike6>etcdctl lease revoke 694d780bd7fccb13
lease 694d780bd7fccb13 revoked
C:\Users\pike6>etcdctl get fooKeep lease alive
In case a lease TTL is not enough and wanna keep the key alive for more time, the lease can be renewed by keeping it alive.
C:\Users\pike6>etcdctl lease grant 60
lease 694d780bd7fccb19 granted with TTL(60s)
C:\Users\pike6>etcdctl put --lease=694d780bd7fccb19 foo bar
OK
C:\Users\pike6>etcdctl get foo
foo
bar
C:\Users\pike6>etcdctl lease keep-alive 694d780bd7fccb19
lease 694d780bd7fccb19 keepalived with TTL(60)
lease 694d780bd7fccb19 keepalived with TTL(60)
lease 694d780bd7fccb19 keepalived with TTL(60)
lease 694d780bd7fccb19 keepalived with TTL(60)Get lease information
Once a lease is granted, can check the lease status with etcdctl lease timetolive.
C:\Users\pike6>etcdctl lease grant 60
lease 694d780bd7fccb1e granted with TTL(60s)
C:\Users\pike6>etcdctl put --lease=694d780bd7fccb1e foo bar
OK
C:\Users\pike6>etcdctl lease timetolive --keys 694d780bd7fccb1e
lease 694d780bd7fccb1e granted with TTL(60s), remaining(40s), attached keys([foo])
A lease can be associated with multiple keys and all keys will be deleted if the lease is expired.
C:\Users\pike6>etcdctl lease grant 600
lease 694d780bd7fccb22 granted with TTL(600s)
C:\Users\pike6>etcdctl put --lease=694d780bd7fccb22 foo bar
OK
C:\Users\pike6>etcdctl put --lease=694d780bd7fccb22 foo1 bar1
OK
C:\Users\pike6>etcdctl lease timetolive --keys 694d780bd7fccb22
lease 694d780bd7fccb22 granted with TTL(600s), remaining(578s), attached keys([foo foo1])etcd can be used in below common scenarios.
- Service discovery and registration such as in gRPC micro services
- Critical data configuration storage such as in Kubernetes clusters
- Load balancer(mainly for data storage access)
- Distributed lock
- Cluster monitoring and distributed system leader election
