ssh not working any more after today update
Gennar1 1 Nov 2011
ssh root@192.168.XXX.XXX ssh: connect to host 192.168.XXX.XXX port 22: Connection refused
The box is responding as usual to telnet:
telnet 192.168.XXX.XXX Trying 192.168.XXX.XXX... Connected to 192.168.XXX.XXX. Escape character is '^]'. OpenEmbedded Linux et9x00 openpli 2.1 et9x00 et9x00 login: root Password:
Maybe the problem is related to this update:
http://openpli.git.s...53c1b47e2932b95
Edited by Gennar1, 1 November 2011 - 14:17.
Gennar1 1 Nov 2011
There is something wrong in the startup script:
/etc/init.d/dropbear
as the script is no more able to start the server correctly.
dropbear works perfectly when started manually:
/usr/sbin/dropbear -E
and then it's possible to ssh into the box again.
But the startup script don't work anymore, as if you type:
root@et9x00:~# /etc/init.d/dropbear stop Stopping Dropbear SSH server: stopped /usr/sbin/dropbear (pid 739) [739] Nov 01 15:21:42 premature exit: Terminated by signal dropbear. root@et9x00:~# /etc/init.d/dropbear start Starting Dropbear SSH server: dropbear.
then again it's not possible to log into the box through ssh.
Edited by Gennar1, 1 November 2011 - 15:23.
blzr 1 Nov 2011
brt@brt:~$ ssh root@192.168.2.111 root@192.168.2.111's password: root@vuduo:~#//and tested after stopping/starting dropbear too
Gennar1 1 Nov 2011
If I start dropbear with:
/usr/sbin/dropbear -E
it works OK (-E is an option to send logs to stderr instead of syslog).
But if I start it normally:
/usr/sbin/dropbear
then it does not start! That's why the startup script is not working.
Maybe I've messed up syslog...
Gennar1 1 Nov 2011
To fix it I had to reflash, because reinstalling busybox killed the box (you can not opkg install remote files because there is no wget, nor local files because another binary is missing).
After reflash with a recent backup, I restored the original /etc/default/busybox-syslog and then I run opkg update again: this time everything was fine and now syslog and dropbear work again.
Gennar1 1 Nov 2011
After a reboot, the same problem was still here.
On syslog I can see:
Nov 1 17:36:19 et9x00 authpriv.warn dropbear[518]: Failed reading '/etc/dropbear/dropbear_dss_host_key', disabling DSS Nov 1 17:36:19 et9x00 authpriv.info dropbear[519]: premature exit: Failed to daemonize: No such device
The first line is expected (we are using rsa and not DSS), but the second line points to the error that prevents dropbear from running. A device i missing and it is "/dev/null"!
Or better, /dev/null is still there but it is replaced by a regular file instead of a device:
root@et9x00:~# ls -al /dev/null -rw-rw-rw- 1 root root 0 Jan 1 1970 /dev/null root@et9x00:~# echo test > /dev/null root@et9x00:~# cat /dev/null test
To fix the problem just remove the file and replace it with a new device:
root@et9x00:~# rm /dev/null root@et9x00:~# mknod /dev/null c 1 3
and now everything is fine again:
root@et9x00:~# ls -al /dev/null crw-r--r-- 1 root root 1, 3 Nov 1 17:44 /dev/null root@et9x00:~# echo test > /dev/null root@et9x00:~# cat /dev/null
and in fact dropbear and ssh work again!
Probably the update of busybox replaced erroneously /dev/null with a normal file
Edited by Gennar1, 1 November 2011 - 17:50.
Gennar1 1 Nov 2011
What's wrong?
hemispherical1 1 Nov 2011
I forced reinstall of busybox-syslog, (I don't seem to have any trouble downloading thru opkg & wget works fine), but that didn't help either.
--
hemi
Ps. Man I took too long to post this, I started before the previous 2 posts...
Pps. Ditto on /dev/null being a normal file?
Edited by hemispherical1, 1 November 2011 - 18:07.
MiLo 1 Nov 2011
Which is unlikely, mdev starts at S04...
Gennar1 1 Nov 2011
root@et9x00:~# rm /dev/null root@et9x00:~# mknod /dev/null c 1 3 root@et9x00:~# chmod 666 /dev/null root@et9x00:~# ls -al /dev/null crw-rw-rw- 1 root root 1, 3 Jan 1 1970 /dev/null root@et9x00:~# /etc/init.d/dropbear start Starting Dropbear SSH server: dropbear.
But after very reboot /dev/null is replaced again with the plain file, and dropbear stops working.
Having a normal file in place of /dev/null may cause big problems, as the file will grow indefinitely and will take all the free flash memory!
So this is going to be a big issue. We better fix it soon.
Gennar1 1 Nov 2011
My guess would be that something that writes to /dev/null is started before mdev (mdev arranges for most /dev/ things to exist).
Which is unlikely, mdev starts at S04...
root@et9x00:~# ls -l /etc/rcS.d/ lrwxrwxrwx 1 root root 16 Oct 19 02:02 S02banner -> ../init.d/banner lrwxrwxrwx 1 root root 18 Oct 19 02:02 S03sysfs -> ../init.d/sysfs.sh lrwxrwxrwx 1 root root 14 Oct 19 02:02 S04mdev -> ../init.d/mdev lrwxrwxrwx 1 root root 17 Oct 19 02:02 S05devices -> ../init.d/devices lrwxrwxrwx 1 root root 19 Oct 19 02:02 S10checkroot -> ../init.d/checkroot lrwxrwxrwx 1 root root 21 Oct 19 02:02 S20modutils.sh -> ../init.d/modutils.sh lrwxrwxrwx 1 root root 18 Oct 19 02:02 S21bootlogo -> ../init.d/bootlogo lrwxrwxrwx 1 root root 18 Oct 19 02:02 S22fpupdate -> ../init.d/fpupdate lrwxrwxrwx 1 root root 17 Oct 19 02:02 S30ramdisk -> ../init.d/ramdisk lrwxrwxrwx 1 root root 21 Oct 19 02:02 S35mountall.sh -> ../init.d/mountall.sh lrwxrwxrwx 1 root root 30 Oct 19 02:02 S37populate-volatile.sh -> ../init.d/populate-volatile.sh lrwxrwxrwx 1 root root 19 Oct 19 02:02 S38devpts.sh -> ../init.d/devpts.sh lrwxrwxrwx 1 root root 20 Oct 19 02:02 S39alsa-state -> ../init.d/alsa-state lrwxrwxrwx 1 root root 21 Oct 19 02:02 S39hostname.sh -> ../init.d/hostname.sh lrwxrwxrwx 1 root root 20 Oct 19 02:02 S41networking -> ../init.d/networking lrwxrwxrwx 1 root root 17 Oct 19 02:02 S43portmap -> ../init.d/portmap lrwxrwxrwx 1 root root 21 Oct 19 02:02 S45mountnfs.sh -> ../init.d/mountnfs.sh lrwxrwxrwx 1 root root 21 Oct 19 02:02 S55bootmisc.sh -> ../init.d/bootmisc.sh lrwxrwxrwx 1 root root 19 Oct 19 02:02 S99finish.sh -> ../init.d/finish.sh
pieterg 1 Nov 2011
That commit wasn't doing anything yet till we bumped busybox PR yesterday.
MiLo 1 Nov 2011
But after very reboot /dev/null is replaced again with the plain file, and dropbear stops working.
Having a normal file in place of /dev/null may cause big problems, as the file will grow indefinitely and will take all the free flash memory!
Luckily /dev/ is in RAM (that's why your changes disappear at boot) and limited to only 64k, so it fills up really quick and won't kill your flash or anything. Which is still very bad, because some things need to write there...
Gennar1 1 Nov 2011
Probably my fault, in the mdev-mount.sh script I piped the 'mount' output to /dev/null. I must have made a typo there I guess.
That commit wasn't doing anything yet till we bumped busybox PR yesterday.
Ok, so it's init.d/sysfs.sh that is calling mount at S03 before mdev is started.
Gennar1 1 Nov 2011
But after very reboot /dev/null is replaced again with the plain file, and dropbear stops working.
Having a normal file in place of /dev/null may cause big problems, as the file will grow indefinitely and will take all the free flash memory!
Luckily /dev/ is in RAM (that's why your changes disappear at boot) and limited to only 64k, so it fills up really quick and won't kill your flash or anything. Which is still very bad, because some things need to write there...
I see, thanks.
Gennar1 1 Nov 2011
are you sure it's not just /etc/mdev/mdev-mount.sh instead?
You must be right, but when is called mdev-mount.sh?
MiLo 1 Nov 2011
theparasol 1 Nov 2011
Going to use telnet for the time being