115 lines
2.9 KiB
Markdown
115 lines
2.9 KiB
Markdown
# domain64
|
|
|
|
`domain64` is a BIGINT (or 64-bit) type that can be used to encode all domains we are likely to encounter. It represents well as JSonnet/JSON, and can be used in partitioning database tables easily.
|
|
|
|
## what is it
|
|
|
|
To encode all of the TLDs, domains, and subdomains we will encounter, we'll use a `domain64` encoding. It maps the entire URL space into a single, 64-bit number (or, `BIGSERIAL` in Postgres).
|
|
|
|
|
|
```mermaid
|
|
packet-beta
|
|
0-7: "FF | TLD"
|
|
8-31: "FFFFFF | Domain"
|
|
32-39: "FF | Subdomain"
|
|
40-63: "FFFFFF | Path"
|
|
```
|
|
|
|
```
|
|
FF:FFFFFF:FF:FFFFFF
|
|
```
|
|
|
|
or
|
|
|
|
```
|
|
tld:domain:subdomain:path
|
|
```
|
|
|
|
or
|
|
|
|
```
|
|
com:jadud:www:teaching:berea
|
|
```
|
|
|
|
can be indexed/partitioned uniquely.
|
|
|
|
This lets us track
|
|
|
|
* 255 (#FF) TLDs
|
|
* 16,777,216 (#FFFFFF) domains under each TLD
|
|
* 255 (#FF) subdomains under each domain
|
|
* 16,777,216 (#FFFFFF) paths on a given domain
|
|
|
|
## what that means
|
|
|
|
There are only around 10 TLDs that make up the majority of all sites on the internet. The search engine maxes out at tracking 256 unique TLDs (#00-#FF).
|
|
|
|
Each TLD can hold up to 16M unique sites. There are 302M `.com` domains, meaning , 36M `.cn`, and 20M `.org`. Again, this is for a "personal" search engine, and it is not intended to scale to handling all of the internet. Handling ~ 5% of `.com` (or 75% of `.org`) is *just fine*.
|
|
|
|
Under a domain, it is possible to uniquely partition off 255 subdomains (where `00` is "no subdomain").
|
|
|
|
Paths can be indexed uniquely, up to 16M per subdomain.
|
|
|
|
## example
|
|
|
|
```
|
|
01:000001:00:000000 com.jadud
|
|
01:000001:01:000000 gov.jadud.research
|
|
01:000001:02:000000 gov.jadud.teaching
|
|
01:000001:02:000001 gov.jadud.teaching/olin
|
|
01:000001:02:000002 gov.jadud.teaching/berea
|
|
```
|
|
|
|
|
|
| tld | domain | sub | path | hex | dec |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| com | jadud | _ | _ | #x0100000100000000 | 72057598332895232 |
|
|
| com | jadud | research | _ | #x0100000101000000 | 72057598332895488 |
|
|
| com | jadud | teaching | _ | #x0100000102000000 | 72057598366449664 |
|
|
| com | jadud | teaching | olin | #x0100000102000001 | 72057598366449665 |
|
|
| com | jadud | teaching | berea | #x0100000102000002 | 72057598366449666 |
|
|
|
|
## for partitioning
|
|
|
|
On a table that contains a `domain64` value, we can partition based on numeric ranges very efficiently.
|
|
|
|
|
|
```sql
|
|
CREATE TABLE comjadud PARTITION OF com
|
|
FOR VALUES FROM (0x0100000100000000) TO (0x01000001FFFFFFFF);
|
|
```
|
|
|
|
Or
|
|
|
|
```sql
|
|
CREATE TABLE comjadudresearch PARTITION OF com
|
|
FOR VALUES FROM (0x0100000101000000) TO (0xx0100000101FFFFFF);
|
|
```
|
|
|
|
## As Jsonnet/JSON
|
|
|
|
Jsonnet will naturally sort by the hex key values.
|
|
|
|
```
|
|
{
|
|
"01": {
|
|
"name": "com",
|
|
"children": {
|
|
"00000001": {
|
|
"name": "jadud",
|
|
"children": {
|
|
"01": "research",
|
|
"02": "teaching",
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"02": {
|
|
"name": "org",
|
|
"children": {
|
|
...
|
|
}
|
|
}
|
|
}
|
|
```
|