Previous incidents

VPS 5 - EPYC recovered.

This has now been resolved. Cosmic have for now disabled GTT in London.

This has been resolved now.

Issue has been resolved and the system is back online.

I have identified the cause for the 2 drops that occurred: This was because of dual BGP announcements on this router causing conflicting routes, this has been fixed now.

This machine is now back online. All power has been restored and services are running as normal again.

We're now marking this as resolved - The issue yesterday according to path was a issue with GTT in Europe. They've brought another link live with RETN which should resolve the issues moving forwards.

Our germany location is back online now. Awaiting a debrief from our DC about what happened regarding that outage.

This has been resolved now.

We're aware of the downtime/drop that occurred on germany core router 2.
We're looking into why this happened but based on the information we can see currently it was a attack leaking.

Issue was resolved, We've also added Java 21 support.

This was resolved, had a small issue with network configurations on this machine.

Issue has been resolved, was a brief issue with the route servers on cosmic side and a small configuration issue our end that has been fixed aswell.

Very sorry about the downtime.

Remote hands have resolved this issue for us. Was a issue with the windows licence key.

We're aware of the downtime on the germany core router 1, mostly effecting 95.214.180.0/24 range. We've asked path for assistance in locating the IP's under attack and have for now resolved this. We will continue to monitor this closely.

We're aware of the downtime in our Germany location, this is a DC Side issue and is unrelated to our side of the network. This isn't a path issue either. Our DC is working on fixing this.

The datacenter team is still working on this.

Network issues are now resolved we apologise for this issue.

This should be resolved now. If you're still having issues with login please do let us know.

This was resolved.

Upon monitoring everything seems stable. We've been unable to locate the exact cause of this downtime but are actively monitoring to see if anything flags up that would point towards why.

We're still actively working on ensuring this is sorted. But currently most users are able to connect again to the database.

This was resolved.

All clients on this node (VPS 4) have been moved to VPS 6, 7 and 8 (New NvMe Nodes) so will benefit from faster speeds. UK 4 is now being taken out of service until it is upgraded in the coming month.

Issue in the end was related to a high load VM causing some issues with resource management within virt. we've now load balanced this system out and it's performing as expected now.

Confirmed with another host they have seen the same drop. Have asked for a reason as to why this happened from path.

After digging around in some system logs, it looks to have been a stall within the kernel of some kind. page_pool having a stalled pool shutdown seems to have been part of the issue (Resource management).

We will closely monitor this router for a couple days to ensure it keeps running ok.

System memory was upgraded.

UK VPS 4 is now back online - Was a dead port on the ethernet card as we predicted. this has been swapped out and the system is running okay now.

This is resolved for now, was multiple IP's with large attacks leaking. Upwards of 15Gbps was hitting our network and path recorded over 121Gbps total.

We've asked again for further assistance with this.