How to rescue ec2 linux when upgrade the kernel failed

rescue steps

  • Force stop the failed ec2 instance
  • Detach the root ebs volume
  • Launch new ec2 instance need support Nitro System the instance type and attach the previous EBS volume
  • Add new user in new ec2 instance and get the /etc/shadows encrypt password copy to old device same file administrator user overide the line
  • Detach the ebs volume from new instance and stop it
  • Retach the ebs volume to first ec2 and start it use serial console connect to it and use overide the user credetials to fix some issue
  • Or immediately fix some problem in new ec2

useful command or script

 1# list the installed kernel
 2dpkg --list | grep linux-image 
 3apt list --installed | egrep '^linux' | grep $(uname -r)
 4# view some logs
 5grep $(uname -r)  /var/log/dpkg.log*
 6# in centos 
 7rpm -qa --last | grep kernel
 8
 9# try to change grub to old boot
10vi /etc/default/grub
11# to modify some value
12GRUB_DEFAULT=0
13GRUB_TIMEOUT_STYLE=menu
14GRUB_TIMEOUT=10
15GRUB_TERMINAL="console serial"
16GRUB_SERIAL_COMMAND="serial —speed=115200“
17
18# remember to update grub make it work when reboot 
19sudo update-grub
20
21# to view grub version may needs different command
22sudo grub2-mkconfig -o /boot/grub2/grub.cfg
23sudo grub-mkconfig -o /boot/grub/grub.cfg
24
25# to view kernel module status
26systemctl status systemd-modules-load.service
27
28# may be finally this command let u surprise
29dpkg --configure -a

The amazon linux 2 upgrade the kernel

 1# list the kernel
 2amazon-linux-extras list | grep kernel
 3# disable the kernel
 4amazon-linux-extras disable kernel-5.10 -y
 5# install the new kernel
 6amazon-linux-extras install kernel-5.15 -y
 7# reboot
 8reboot
 9# confirmation
10rpm -qa | grep kernel
11uname -a

Reference