We have several RUT956 devices running on remote locations. Each of them are equipped with USB sticks and are configured to expand it’s root file system which enabled us to install Python on it, which we need to run our software on it. When I updated our software I noticed later the new files were gone and looking for it with the ‘ls’-command, I got the message “ls: script.py: No error information”. When looking in the system logs, the cronjobs that run some of the script state: “comm ls: deleted inode referenced”. So I get the impression the file system has become corrupt. Is there a way to enable file system checking/auto error correction? And if I enable the uci-option “fstab.@global[0].check_fs” value to ‘1’, does it automatically correct the data? Best regards, Eyso
No automatic recovery procedures are implemented into RutOS.
If the devices are still reachable, I’d suggest removing memory expansion, formatting the drives, and re-uploading the necessary files onto the device.
To avoid these issues in the future, I’d suggest writing all temporary data into the volatile storage (/tmp) to avoid damage to the flash chip of the device or the USB drive, as often write operations can damage both USB drive and the router itself.
Thank you for your swift answer!
All my writes are indeed primarily to subfolders of /tmp, but we have the USB drive not only to be able to store the Python library on it, but also for buffering purposes, in case the device loses its internet connection and/or power - which happens so every now and then. Technically, the data is being zipped and stored on the usb stick every minute.
The devices are located in factories and installations and are therefore difficult to get access to, so I try to configure the system to be as resilient as possible. Being assured the filesystem won’t get corrupted is essential to us. Best regards, Eyso
In this case, I’m afraid that not much can be done to prevent the corruption, other than writing the data less often. In the future, I’d suggest considering devices from the RUTX lineup, and using a USB → Serial adapter. As these devices have much more Flash storage available for the user, Python could be installed on the device itself without expanding the /overlay partition onto a Flash drive.
I’d also suggest making sure that there are no ongoing write operations to the flash drive when the device is being powered off, as that could also cause the file system to corrupt.
Finally, make sure that the flash drives are formatted in exFAT FS, as it’s more reliable for operations like these.
Hello Daumantas,
Thank you for your suggestion!
The RUT956 ticks many of our boxes (though we stil do miss a real time clock option), so I’ll have a look into the RUTX lineup.
I’m a collegue of Eyso. Thank you for the help so far!
The specific error we’re getting is this one:
“deleted inode referenced”
We do write some data to the USB drive every minute, we could minimize this; however I’m not sure if the issue we’re facing is due to too wear on the disk. Minimizing during shutdown is a good tip, however we’re not fully in control about when a customer gives a power cycle sadly.
We’ve seen this on two occasions, on two different routers:
a file that we write ourself in /mnt/sda1/
a system file: /etc/config/dnssec
The second one worries us the most since this is part of the distro/firmware.
Your tip about exFAT is for the RUTX alternative? Or also for the overlay partition?
The second one worries us the most since this is part of the distro/firmware.
If I understand correctly, you are using our provided SDK, and have included additional packages, correct? As we do not include DNSSEC into our firmware.
If that is the case, it seems like fsck utility is available for OpenWRT, but needs to be added to our SDK to use it. This utility can help restore corrupted file systems. More information on this forum thread:
As for the exFAT, this is a filesystem that is widely used in Linux distributions. Judging from the errors it is already in use on your devices, but just wanted to double-check, as NTFS can cause other issues.
We are not using the SDK; we’re just running our scripts either as a cron job or as a service. Regarding the missing inode “/etc/config/dnssec” I need to correct my colleague; the inode missing is actually is /etc/ipsec.conf, a file which we did not (directly) modify or create. I suspected the option “Synchronous write” under USB mount settings, which I did not modify so it was disabled. I now have it enabled, hoping this helps. I also altered our scripts to keep the writing to an absolute minimum.