Posted on 2022-02-01

Intro

In this text, I am going to point to how, and why I got an Self sustaining Device
Number and some
IPv6 addresses.

Getting some IPv6 cravings again

In stupid 2020, I learn
https://blog.dave.tf/put up/current-kubernetes/.
In this put up, the author stated within the occasion that they were to form something current, they would
take care of “IPv6 solely, largely”. This put up got me to focused on having some
IPv6 connectivity again.

Before I got to Montreal, I passe to grasp earn entry to to IPv6 Web. I will’t
be conscious for definite, however I convey it change into as soon as via a Hurricane
Electric tunnel.

No Native IPv6

My Web Service Supplier (ISP) is a limited ISP. They put no longer in discovering the final
mile. They offer native IPv6 for
some loads of subscription of theirs, the set apart they will. However, on the carrier I’m
subscribed to, the final mile owner is nonetheless within the course of of deploying IPv6
(constantly-grasp-been-meme.png).

No native IPv6 arrangement I am going to deserve to setup some tunnels, one skill or one other.

The exhaust of a tunnel suppliers

The first thing I appeared at change into as soon as Hurrincane Electric, since it change into as soon as the solely
provider I knew at the time. Sadly, they solely provide
GRE tunnels,
which arrangement no encryption. One would possibly argue that in 2020+, with HTTPS and
DoT/DoH,
there is restricted unencrypted site visitors, however to that I am going to acknowledge “meh”.

Developing my in discovering tunnel

I believed I would possibly rent a virtual machine (preferably, since tunnels require
limited assets and VMs are skill much less dear than devoted servers) and scurry my in discovering
tunnel with the IPv6 it provides.

As mentioned in my infrastructure blog put up, I
grasp more than one networks (VLAN) at home. On memoir of I did no longer deserve to exercise out some
unholy things, I desired to grasp a /64 per community, which a ways more than one /64s for
my home.

I went on the hunt for a provider that affords something like a /56 (or the
chance to earn more than one /64s). Sadly, I did no longer in discovering the rest cheap.
I lastly found some excessive discontinue servers that came with a /48 however since they
worth virtually about as primary as my rent, I am going to pass. Most suppliers give at best a /64,
nonetheless it is truly a /128 (lol) or nothing (yeah who cares about IPv6).

It’d be more tense

I requested a community engineer buddy if he knew any web hosting providers and products providing
bigger than a /64 with a low worth machine and -properly- he gave a community-engineer form
of acknowledge “right earn your IPv6 addresses and teach them”.

After inquiring more factor, he generous answered and I definite to proceed with
this.

Some context though

Whereas I assemble no longer qualify as a community engineer, I’m no longer entirely ignorant
community-wise. I passe to work for a community operator (so
I’m no stranger to BGP) and I passe to be a volunteer for a
couple of non-profit
ISPs reduction in France.

Getting some assets

Disclaimer: Have point to of what follows is my in discovering interpretation. Ride learn the
connected parties’ web sites and agreements to grasp your in discovering belief.

Following my buddy’s recommendation, I dwelling out to earn some IPv6 addresses and an ASN
to teach them. I would possibly then scheme my in discovering (encrypted needless to claim) tunnels to
earn IPv6 at home.

I would also be in a feature to exercise out what I had wanted for years: play with
anycast.

Deciding on a RIR

IP addresses and an ASN would possibly moreover be got via a
RIR.

Thanks to my non-public reveal (which I would possibly no longer earn into), there are two RIRs I
would possibly ask: ARIN and the RIPE.

ARIN

ARIN is
the RIR for Corporatist The US. At the same time because it is doubtless you’ll be no longer a company, properly it is doubtless you’ll be no longer
going to head very a ways.

I believed of as creating my in discovering, however the worth exceeded what I change into as soon as ready to exhaust
on the mission. As cheap because it would possibly were for a company, it would possibly
no longer be for me.

RIPE

RIPE is the RIR for Socialist Europe.
You would possibly presumably be an particular person and you wish some assets? That’s entirely pleasing, breeze ask
for some. Properly, now in a roundabout arrangement. RIPE would no longer search the recommendation of with peasants, that you just can well deserve to ask
a
LIR.
In the occasion that they will provide it directly, they operate. Otherwise, they act as a proxy
between you and the RIPE.

I went for this chance. From my time volunteering, I know fairly many of us
in fairly loads of LIRs.

Grifon

I selected Grifon for no explicit reason.

Obtaining the assets

My preliminary thought change into as soon as to earn a /48 to earn IPv6 at home and a /48 to play with
anycast (attributable to it is the smallest community that you just can well teach on the Web).
I would possibly no longer operate the rest else with the /48 I would anycast, by grasp.

So after completing my membership, I requested a /48 IPv6 from the RIPE
(via my LIR, as defined). A pair of days after the ask and with some
educate-up questions, I got my first
prefix. Now that I had some contend with
set apart, I would possibly elaborate the necessity for an ASN. I made the ask and got
it.

So I requested a /48 to my LIR from its in discovering assets.
Alarig generous carved my 2d
/48 out of the LIR reserved contend with
set apart attributable to this.

(For the readers no longer versed within the RIPE-world technicalities, the first /48 is
a PI, the 2d is a
PA).

Getting a third /48

Quickly after I setup IPv6 at home, I seen Google believed I change into as soon as in France.
Provided that even broad networks battle to repair
complications, I had
no hope for myself. I believed that presumably the utilization of a netblock from ARIN would resolve
my reveal.

First and main, I went to ask a non-profit I make contributions to, however
it did no longer work attributable to we hit a technical limitation from a general provider.

Then, I discovered the Nato Web Service. They
provide a /48 (or more need to that you just can well elaborate the necessity) out of a netblock known as
feda (attributable to it comes from 2602:feda::/36).

Sadly, this did no longer resolve my geolocation reveal with Google. I even had
a brand current reveal, my FEDA block change into as soon as geolocated in China, however I without complications fastened it in
maxmind db, and it appears to be like to were enough.

However, as the quote says “Everybody has a discovering out netblock. Any other folks are
fortunate enough enough to grasp a unconditionally separate netblock to scurry production
in.”, I had now a /48 I
would possibly exhaust to take a look at stuff for anycast.

The exhaust of an IPAM

Are you into IPAM porn? On memoir of need to it is doubtless you’ll be into IPAM porn, it is doubtless you’ll be in for a
treat!

Now that I had 3 netblocks that I change into as soon as going to reduce into smaller networks, I
would want a tool to tune
utilization. This display masks day, most
other folks exhaust NetBox. I believed I
change into as soon as going to make exhaust of it, however I learn a
couple of
times the
author of sidekiq and it made me ticket I did no longer need the type of complex tool.

IPAM v1

For shits and giggles, I before the entirety belief “would no longer or no longer it be fine to make exhaust of tree(1)
to see the entirety??”. I created directories for blocks, and recordsdata for
addresses. Here is what it gave the impression of:

~/git/git.chown.me/ipam/ipv6 (master=)$ tree
.
├── 2001: 67c: 291c::-48
│   └── 2001: 67c: 291c: : 1
└── 2a0e:f43::-48
    ├── 2a0e:f43: 0: 100:-56
    │   └── NEXT-ONE
    ├── 2a0e:f43: 0:fd00::-56
    │   ├── 2a0e:f43: 0:fd00:: 1
    │   └── INTERCO-WG1
    ├── 2a0e:f43: 0:fe00::-56
    │   ├── 2a0e:f43: 0:fe00:: 254
    │   └── INTERCO-WG0
    ├── 2a0e:f43: 0:ff00::-56
    │   └── 2a0e:f43: 0:ff00:: 1
    └── 2a0e:f43::-56
        ├── 2a0e:f43: 0: 10::-64
        │   └── 2a0e:f43: 0: 10:: 1
        ├── 2a0e:f43: 0: 40::-64
        │   └── 2a0e:f43: 0: 40:: 1
        ├── 2a0e:f43: 0: 60::-64
        │   └── 2a0e:f43: 0: 60:: 1
        ├── 2a0e:f43: 0: 70::-64
        │   └── 2a0e:f43: 0: 70:: 1
        └── 2a0e:f43: 0: 80::-64
            └── 2a0e:f43: 0: 80:: 1

Show: This predates the switch to the feda netblock.

However within the tip, modifying recordsdata change into as soon as no longer easy attributable to I needed to flee the final
: in my shell. I had loads of fun creating this arborescence, nonetheless it change into as soon as time
to switch on to something more practical.

IPAM v2

I went for a single textual whisper material file in a json-impressed structure. Here is what it appears to be like to be like
like:

$ head -n 30 ipam.txt
ANNOUNCED BY BGP-YYZ
2001: 67c: 291c::/48 {
    2001: 67c: 291c::1 {
        anycast.chown.me
    }
}

ANNOUNCED BY BGP-YYZ, NS4
2602:feda:b8e::/48 {
    ANNOUNCED BY pancake
    2602:feda:b8e::/56 {
        2602:feda:b8e: 10::/64 {
            LAN
            2602:feda:b8e: 10::1 { pancake:vlan10 }
        }
        2602:feda:b8e: 40::/64 {
            PHONE
            2602:feda:b8e: 40::1 { pancake:vlan40 }
        }
        2602:feda:b8e: 60::/64 {
            WINDOWS
            2602:feda:b8e: 60::1 { pancake:vlan60 }
        }
        2602:feda:b8e: 80::/64 {
            RTBH
            2602:feda:b8e: 80::1 { pancake:vlan80 }
        }
    }
[...]

Show that right here RTBH is solely how I named the community, it be no longer connected to
right RTBH.

I dwelling up the file with vim and I will without complications (un)fold any stage whether I need
an outline or a detailed gaze. Also this is maybe no longer entirely updated haha.

The infrastructure

For anycast

My preliminary thought change into as soon as to earn some VMs all the arrangement in which via the enviornment and teach the /48 on
each. More uncomplicated stated than carried out, attributable to my necessities are to search out a provider
which:

• affords a low worth and limited VM (i.e. 1cpu, 1G of ram, 20G of disk)
• is willing to setup a BGP session so I will teach my IP addresses
• lets in me to install OpenBSD
• provides frequent web hosting stuff, like atmosphere a PTR on supplied IP addresses

I believed “anycast is easy, you right teach your IP in every single place, and carried out”.
Properly, certain, however in point of fact no. A minimal of need to you don’t desire to abide by RFC
7511. Honest routing requires a
lot of work.

I currently grasp 4 VMs in this anycast community:

• bgp-dus in Düsseldorf, Germany
• bgp-mrs in Marseille, France
• bgp-yyz in Toronto, Canada
• ns4 also in Toronto, Canada

Here’s a work in development that doubtlessly deserves its in discovering blog put up when it be
entirely carried out, so I would possibly no longer breeze additional into particulars.

For my IPv6 at home

As you right learn, I in point of fact grasp two VMs in Toronto. I need I will grasp a provider in
Montreal to reduce latency, unfortunately I’ve no longer been in a feature to search out one fairly
yet.

I needed to resolve some tunnelling skills. I picked up WireGuard® attributable to it
had no longer too long ago made it into OpenBSD kernels (search
wg(4)) and my skills with ipsec is as “right”
as the following person.

My contemporary setup is:

~/git/git.chown.me/ipam (master=)$ cat schema.txt 
Upstream 1        Upstream 2
   |                 |
   |                 |
   R1------ wg ------R2
   |                 |
   wg                wg
   |                 |
   -------- R3 -------

R1 and R2 are my VMs in Toronto, and R3 is my router at home. Sure, my router at
home makes exhaust of BGP, each to teach its in discovering netblock over BGP and to resolve the
best route between R1+Upstream 1 and R2+Upstream 2. Will not be that fine chilly??! 😀

R1 and R2 each teach my /48 to their provider. They operate so with my public
ASN.

They’ve a wg link between each loads of. The diagram is twofold:

1. if the session with their upstream fails, the site visitors will circulate to the loads of
2. if wg between R3 and R1 or R2 dies. Web whisper online visitors will circulate via the remaining
   wg link

Case 1 is no longer in point of fact an argument. As soon as the session with the upstream fails, it
would possibly no longer earn the elephantine
gaze
anymore, which arrangement R3 would possibly no longer earn the elephantine gaze from that router, and it will
send site visitors solely to the loads of. Web whisper online visitors to me will swap robotically
supplied the upstream stops announcing my route (it would possibly nonetheless, however in most cases it
would no longer)

I prepend that path with my ASN 15 times (picked by “wants to be right enough
lol”) to retain a ways from the utilization of it in traditional situation.

This straightforward link change into as soon as in point of fact fairly a tall exchange attributable to except then, R1 and R2
passe to exercise out some stateful firewalling (to boot to the one carried out on R3).
However, this exchange intended site visitors would possibly circulate asymmetrically, so I needed to
swap to stateless firewall (which I restricted to the impart community, the
relaxation of the site visitors is nonetheless checked by pf(4)
with stateful principles).

R3 proclaims the /56 I in point of fact grasp at home over BGP to R1 and R2. “But this is inter
AS, why did no longer you exhaust an IGP???”. Properly wg(4) would no longer give a own to multicast, and
ospf6d (and even
eigrpd) wants it. You are going to be in a feature to operate without
buuuut… I tried and struggled with ospf6d, so sticking with
bgpd change into as soon as skill more uncomplicated.

Fun fact: I even began to put in writing my in discovering igpd, however I lickety-split realized I change into as soon as right
reimplementing bgpd poorly so I aborted.

I in point of fact exhaust a non-public ASN to teach the /56. I picked 4200211935, so it be
obviously each “it be my ASN”, and “it be no longer my ASN“:

danj@bgp-yyz:~$ bgpctl sh
Neighbor                   AS    MsgRcvd    MsgSent  OutQ Up/Down  Advise/PrfRcvd
pancake-6          4200211935      17289    2134334     0 5d22h44m      1
ns4-6                  211935    1213381    1930550     0 5d23h30m 134718
xenyth-6                62513    1945805      17297     0 6d00h06m 138770

Pointless to claim since I teach a /56 and a non-public ASN, I desired to forestall checking
RPKI for this explicit host. Happily, bgpd’s principles machine is extraordinarily easy
to work with.

Machine

Pointless to claim the entirety runs OpenBSD! It has a sexy
bgpd in immoral. OpenBSD ships
rpki-client which one can exhaust to validate ROA
(“give a own to the routing security” in layman’s terms).

OpenBSD builders changed OpenBGPD config since final I passe it. The article I
distress essentially the most about is messing what I teach to my peers. They need to grasp
filters, however I assemble no longer deserve to be that guy. OpenBGPD’s config file is determined in a
skill that it be laborious to clutter up, thanks to sane defaults and an out of this world good judgment.

It ships with an gorgeous instance config file making easy to begin the utilization of it!
For that reason, I’m no longer going to factor mine.

OpenBGPD makes exhaust of limited memory:

danj@ns4:~$ bgpctl display masks rib nei vultr-6 in | wc -l
  135254
danj@ns4:~$ bgpctl display masks rib nei bgp-yyz-6 in | wc -l
  139312
danj@ns4:~$ bgpctl display masks rib memory
RDE memory statistics
    139583 IPv6 unicast community entries the utilization of 7.5M of memory
    279161 rib entries the utilization of 17.0M of memory
    823926 prefix entries the utilization of 101M of memory
    156446 BGP path attribute entries the utilization of 10.7M of memory
       and retaining 823926 references
    138180 BGP AS-PATH attribute entries the utilization of 11.6M of memory
       and retaining 156446 references
       819 entries for 6470 BGP communities the utilization of 178Okay of memory
       and retaining 823926 references
      6803 BGP attributes entries the utilization of 266Okay of memory
       and retaining 41980 references
      6802 BGP attributes the utilization of 54.1K of memory
    306537 as-dwelling aspects in 280152 tables the utilization of 10.9M of memory
    511038 prefix-dwelling aspects the utilization of 21.6M of memory
RIB the utilization of 148M of memory
Sets the utilization of 32.5M of memory

RDE hash statistics
    path hash: size 131072, 156446 entries
        min 0 max 8 avg/std-dev = 1.194/0.759
    aspath hash: size 131072, 138180 entries
        min 0 max 8 avg/std-dev = 1.054/0.943
    comm hash: size 16384, 819 entries
        min 0 max 3 avg/std-dev = 0.050/0.000
    attr hash: size 16384, 6803 entries
        min 0 max 5 avg/std-dev = 0.415/0.000

Most VMs grasp solely 1G of ram and 1 cpu.

danj@ns4:~$ top -b -ores
load averages:  0.01,  0.05,  0.02    ns4.chown.me 20: 35: 57
65 processes: 1 working, 63 lazy, 1 on processor  up 13 days,  3: 58
CPU states:  2.9% user,  0.0% fine,  2.1% sys,  0.0% poke,  0.1% intr, 94.9% lazy
Memory: Genuine: 411M/713M act/tot Free: 256M Cache: 152M Swap: 192M/512M

PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU COMMAND
90287 _bgpd      2    0  231M  238M sleep     ballot     36: 22  0.00% bgpd
88230 _bgpd      2    0   26M   30M lazy      ballot      8: 29  0.00% bgpd
61228 root       2    0   20M   22M sleep     ballot     16: 59  0.00% bgpd
[...]

RPKI

I did no longer deserve to scurry rpki-client on each
and each router. I would possibly no longer either attributable to it makes exhaust of a truckload of inodes and
my /var/ partitions would possibly no longer provide you with the money for it.

I believed of as the utilization of RTR,
however it intended working more software program (e.g.
gortr/stayrtr).

Also bgpd would no longer give a own to (yet?) encrypted RTR so it would possibly grasp intended either
doing RTR unecrypted (yuck), or scurry even more software program.

What I done up doing is working rpki-client on my web server (on which I added
a explicit partion with skill more inodes).

42 -n rpki-client -v && 
    cp /var/db/rpki-client/openbgpd /var/www/static.chown.me/pub/rpki/openbgpd && 
    gzip -f /var/www/static.chown.me/pub/rpki/openbgpd

And on my bgpd routers

57 -n ftp -o /var/db/rpki-client/openbgpd.gz https://static.chown.me/pub/rpki/openbgpd.gz && 
    gunzip -f /var/db/rpki-client/openbgpd.gz && 
    bgpd -n && bgpctl reload

15 minutes must be enough, it passe to scurry in 5 minutes, nonetheless it appears to be like it
now runs in round 8 minutes, I assume I would possibly nonetheless setup some monitoring haha.

OpenBSD Contributions

Pointless to claim, I discovered some improvements for the software program I exhaust via this
mission. Here are some fixes that made it into the OpenBSD trees attributable to of my
taking part in round:

• fastened a frequent beauty reveal
• improved the bgpd.conf instance
• fastened a man online page
• fastened one other man online page
• updated afresh1’s bgpd nrpe take a look at
• got anyone to repair a horror when destroying a wg(4) interface

Cost

Pointless to claim this odd passion of mine prices money. I’m however more than pleased of how
low I would possibly reduction my expenses.

Administrative prices

Here is what I paid Grifon:

• 15€/y for membership prices
• 90€/y for administrative prices to earn the ASN/IPv6 assets

Hosting prices

Out of 4 VMs I scurry BGP on, I have been the utilization of 1 for various things, so I’m no longer
counting it since I would pay for it no matter this mission.

Here is what I pay for the host:

• bgp-mrs: 0€ (thanks Evolix! <3)
• bgp-dus: 9€/month
• bgp-yyz: 8.50US$/month

Misc assets

Web whisper online visitors engineering

Although I messed round with BGP before, I hadn’t in point of fact long gone deeper than the
ground. Since I had a l