Category: Pool status updates

Categories
General Hardware Media Pool status updates

Update on ANP Servers

We had a few rough periods and needed to rethink our approach. In that regard we made a new server structure as can be seen below in the image.

This has improved our stability and now our main pool server has had an uptime over a day without any problems. We are converting now around 82% of blocks an improvement of 75% from previous settings.

Finally an update on network equipment: Router has arrived and network cards as well. But we are waiting for the transceivers / cable equipment. Probably by next week we have everything and ready to install. The installation is likely to take half a day or with maximum preparations I could probably manage to get the pool up and running again in 1-2 hours. I will try to select a point in between blocks that is optimal – probably during next weekend.

Categories
General Hardware Pool status updates

ADA North Pool invests in new router hardware

ADA North Pool invests in the Mikrotik RB4011 for more powerful router hardware, this gives us capability to utilize some of the amazing performance of the Mellanox Connect-X 4 cards we have purchased. The router can handle roughly 5-5.5 million packets per second worth of routing small packages of 64 bytes (typical for staking pools who only need to transmit small info to each IP address)

Categories
General Hardware Pool status updates

ADA North Pool invest in network equipment

As a first step to a potential data center class capability ADA North Pool as purchased both a Mellanox ConnectX-4 MCX4111A-XCAT 10Gigabit Ethernet Card and a Mellanox ConnectX-4 MCX4121A-ACAT 25Gigabit Ethernet Card. In theory these cards can in optimal situations handle 75 million TCP packets of information per second.

Categories
General Media Pool status updates

ANP produces its 500th block for delegators during ITN testnet

Categories
General Pool status updates

A glance into ANP Server

I tought I would share my current screen setup on the server. I am using terminator bash terminals and hardware monitoring is your standard stuff like Glances, Htop and Iotop. But also I use Chrony for making sure my system clock is up to date and tuned to make sure my network is up to date, both running as services on the computer. I had my own custom made scripts for forking / bootstrap checking but replaced them with the great script Redoracle made. I also run Prometheus, Grafana and Nginx for monitoring and websites and a few security measurements, one of them that I think its fine to mention is I run Fail2ban. I have removed some information that could affect security and replaced with a red bar.

Some of the hardware monitoring. I have a default view and a view also with background scripts (for example I check how well Chrony is doing in making sure my system clock is as accurate as possible)
Currently the hardware is running a main server, 2 nodes and one extra server that I am helping IOHK out testing with. I run an error parser script and I also run the great script from Redoracle mentioned above and with Github link to its repository if anyone else wants to use the same. As you can see from the above pictures we have plenty of hardware resource room to grow utilizing somewhere around 10-15% CPU and around 10% of memory. Connections vary between 2000-4000 and tracked connections are around 13000-15000. In the “do stuff” box I run scripts from time to time like checking how many files open on the system (usually 18000-20000 at any given moment) or watt usage of my Nvidia card (sadly around 18 watt so more than what I expected but luckily since CPU is only at 10% utilization the Ryzen draws less power than what I expected as well.)
Categories
General Media Pool status updates

ADA North Pool will produce its 400th block at roughly 1PM UTC

Celebrating a milestone for ADA North pool we have come a long way since epoch 1 and 2 where we produce one block in each epoch! We have tweaked the numbers even more when it comes to connection settings and believe we have found a sweet spot that will give us even more optimal operations.

Categories
General Pool status updates

ADA North Pool Upgrades Nginx / PHP and adds system maintainance tools

To keep everything running smoothly we upgrade to latest packages for our webserver technology and we also added chronyd to make sure our server keeps running at correct time (in millisecond measurement ranges) while also adding maintenance software that automates upgrades and keeps the system tuned and performing over time.

Categories
General Pool status updates

ADA North Pool upgrades to Jormungandr 0.8.5

I compiled 0.8.5 because the pre-compiled options do not offer Journald log support and you can get this if you compile the binaries yourself.

Categories
General Pool status updates

ADA North Pool adds software based tuning of network performance in real time

Having read several TCP networking guides and doing tweaks adjusting to any – at the moment – type of challenges I realized this would be very time consuming. I am trying to set up as much automation as possible so the server is robust. In that regard we found https://tuned-project.org/ and it has had already dramatic results cutting cpu usage in half and allowing us to double our network connectivity without any networking errors.

Categories
General Media Pool status updates

ADA North Pool adds jormungandr monitoring

Had another long night where I tweaked the node cluster (less nodes but more quality to each nodes with more connections and file limits for each) and also optimized the server painstakingly reducing one by one some metrics I know affects the quality the serer has to read and process incoming connections fast until I found an optimal point. After this the server seems again more stable (hoping this time it is going to last for a while!)

Now with the server more stable I have had some time to add more “bells and whistles” type of features so I have now not only a hardware monitoring but also a Jormungandr server monitoring that user can look how the server is doing right here on the web page, I have added these as menu links. As always feel free to contact me for encouragement / suggestions.