On October 11, a failure occurred in Yandex Cloud that affected the Layero infrastructure. The control panel, deployments and sites on Layero have become unavailable.
On October 18, the incident was closed. The control panel, builds, deployments and sites are working. Recovery history - below.
What happened
By Yandex Cloud message, as a result of a UAV attack, the infrastructure of the data center was damaged. The incident affected the services on which Layero was running.
What does this mean for Layero users
The operation of the panel, new deployments, sites, payments and support has been restored. Current component availability is posted on status page.
What changed after the incident
Management services, server applications and object storage have been moved to Selectel. Public website addresses are saved; There is no need to change DNS due to this transfer.
Free period and subscriptions
For new users, the free period is again 14 days. Free period extensions and payment transfers already granted will remain in place. The term of your period and the date of the next payment can be checked in your account dashboard.
Old Shared database data
The old Shared database data has not yet been restored; their safety and restoration period have not yet been confirmed. Closing an incident relates to the operation of the service. If you have questions about your database data, please contact support.
Latest updates at top. Time is indicated in Moscow.
Operation restored
Completing disaster recovery
PanelDeploysSitesPaymentsSupport
Emergency API restrictions have been lifted, background tasks have been resumed. The 14-day free period has been restored for new users. Previously issued extensions are retained. The issue of restoring old Shared data remains open.
Migration
Object storage and distribution sources
S3EdgePanelDocumentation
The panel and documentation have been transferred to Selectel S3 storage. We checked HTML, JavaScript, styles, deep panel paths and 404 responses. The transfer of custom deployment artifacts continues.
Restored
Payment API and charge schedule
APISubscriptionsScheduler
Linking a payment method and canceling auto-renewal are available again.
Restored
Delivery of support messages
TelegramEmailChat in your dashboard
Reception of requests, delivery of responses to the office and mail notifications have been restored. We checked both directions of exchange: from the account to support and back.
Checked
Restoring PostgreSQL from backups
PostgreSQLpg_restoreBackups
We checked the restoration of the platform database and copies of the Shared databases of the new circuit in a separate environment. The check applies to new infrastructure and does not confirm the return of old Shared data from the emergency data center.
Restored
Access to the account and dedicated databases
APIAuthorizationPostgreSQL
Opened a account dashboard for all users. Access to dedicated databases has been restored; working with old Shared data remains a separate step.
Restored
Authorization, Git integration and project APIs
Yandex OAuthEmail codeGit-webhooksCLI / MCP
We checked login via Yandex and email, import of repositories, processing of Git-webhooks and auto-deployment. We checked the basic API operations through the panel, CLI and MCP.
Migration
Application Image Registry
OCI RegistryRuntimeDocker / Podman
Prepared the Selectel registry for application images. We checked the loading of images using the checksum and the launch of containers. Previously published images were migrated separately from new builds.
Checked
Full path of the new deployment
CLIBuild QueueCollectorRuntime
The end-to-end test tested the creation of a deployment, queue processing, build and launch of a fullstack application. The frontend and backend API responded to the public HTTPS address.
Switch
Incoming traffic and routing between providers
NLBWireGuardEdge / TLS
Directed incoming web traffic to new Selectel nodes. After loading errors, we rolled back the switch, adjusted the network parameters and re-checked HTML, JavaScript, styles and HTTPS. Public addresses have been preserved.
Checked
Server applications on new runtime nodes
SSRFullstackRuntime
The test fullstack application was built and launched on the new infrastructure. We checked the frontend, server API and separately SSR.
Restored
Build connected to new circuit
RunnerBuild QueuePostgreSQL
Connected the existing subassembly to the rebuilt platform. It began receiving tasks from the queue of the new control loop.
Restored
Control circuit and platform base
APIPostgreSQLEdge
Restored the platform base and prepared the main services in Selectel. Recording operations and background tasks were opened in stages after checks to prevent the old and new circuits from running at the same time.
Restored
Distribution of static and access to the panel
Static sitesPanelS3
Restored distribution of static sites and loading of the control panel. The availability of the panel did not mean that the API, new deployments and server applications were fully operational.
Diagnostics
Impact of incident on Layero confirmed
EdgeAPINLB
External checks showed timeouts for the panel, API and user sites. 200 responses from individual S3 sources and balancer checks did not confirm platform availability for users. Disaster recovery has begun.
Accident
Beginning of the Yandex Cloud incident
Provider infrastructure
The start time is indicated by Yandex Cloud status. The provider reported that the zones were unavailable. The incident affected the infrastructure on which Layero's management services, websites and user databases worked.