With a very high input/output for our files I have decided that its more important with NVME SSD performance for input output file operations than 4Terrabyte of RAID and will likely switch over to full NVME hardware if we are successful in the Testnet period. It also has the side benefit of a 90% less watt usage per disk.
Category: Pool status updates
To improve system performance we have added several tweaks and also some software that will help us better handle large connection loads that is needed to keep having an “overview” of the network that will help our server not following forks. One such measure is the google BBR algorithm that has been tried and tested by a major corporation in high stress situations. https://cloud.google.com/blog/products/gcp/tcp-bbr-congestion-control-comes-to-gcp-your-internet-just-got-faster
Hopefully epoch 7 will be a good one! To be sure we are still following with dedication and keep improving the network performance looking under any stone we can find.
I had to pull an all nighter as even while the new cluster of nodes is helping out a lot in stability it did not help when there where suddenly a strong propagation in the network for several forks. To help with this issue we also increased the network capacity of the main server so now it sees most of the active nodes in the network. This allows the server to look at what is the longest chain and correctly identify what path to take regarding forks.
Also when you look at the numbers we had an amazing performance. We lost some blocks during the early time period of forks but around 4 AM (as I told I did an all nighter for this) after also bootrapping to the nearest IOHK servers (3 servers are located in EU) and implementing these changes the pool performed admirable. Lets look at the math:
We know that 4320 blocks are on offer each epoch (10% of the total 43200 slots) We also know that currently only around half of this is actually produced by pools (around 2600). In our case we have a stake of around 66 million delegated to us and 0.95% of the total stake. With 4320 blocks we should with 0.95% recieve 41 blocks. Guess how many blocks ADA North Pool produced during the epoch? 41! So overall given all that happened epoch 6 I am actually very pleased with the performance and keep in mind that performance matric is a short term performance metric, you should also ask yourself the question how many blocks is this pool producing compared with its stake?
Epoch 5 was rough on ADA North Pool who only managed to produce around 60% of blocks. Granted it was rough for many other pool operators but ANP prides itself in trying to the best of its ability to produce blocks. To that extent we have utilized our strong server hardware and added a cluster of nodes that will help keep the network stable and performing well. We have also done several more tweaks to the nginx html server to improve its performance so the site is available.
First of all I would like to apologize that the webserver has been down for 24 hours. The site was simply not ready for the kind of load that came with the Test Net 0.8.2 ITN when all users of the Deadelus wallet wanted to access it. I have done a lot of optimization these last hours and hope to come back much stronger because of this. Due to DNS propagation it will take 24-48 hours before users see these changes. As I said earlier I am learning this as we go.
In that regard we also had some trouble with bootstrapping simply getting stuck due to the massive amount of bootstrapping calls to IOHK servers. To make sure this is not a problem in the feature we have added both our own backup nodes, private nodes in a pool alliance and added more features to error script logging for automation of handling any incidents.
In short I am still learning but the pool and the website is getting stronger and stronger day by day.
As the 0.8.2 ITN had its first problem with block not moving up ADA North Pool has added a temporary fail safe in an automated script to make sure server resets if the block is not moving up again. More longer term we are working into more advance automation of logging tools (probably trough Greylog) but for the time being this is an easy workaround.
ADA North Pool had its first error that caused the service to have to reset. This took somewhat less than 1 second so did not in any way affect the pools ability to produce blocks but in line with transparency and since we are in TestNet phase this is reported here on server status updates.
Another historic moment ADA North Pool has now been lucky to be selected for a block to be produced in two epochs in a row. Also this epoch our block was produced successfully and took 0.0033 seconds from wake up time until finished time.
ADA North Pool joined the official 0.8.2 Incentivized Test Net (ITN) Friday the 13th December 2019. We look forward to providing staking to all of you and a glance at pooltools.io shows the competition will be stiff with the delegator the one to benefit. I hope you would still consider our pool among the other great pools on offer and hopefully we can all together make this ecosystem thrive.
Youtube link to historic moment when we went from Epoch 0 to Epoch 1.
On another note I made a typo for the ITN description and the pool is described with only 1 TB of RAID 1. In actuality the pool has 2 X 4 TB Seagate Enterprise Exos disks together with two Kingston A2000 500GB NVMe’s. Will update it at some point later in the test phase.