Difference between revisions of "Linux and Your USB Flash Drive"

Revision as of 10:57, 27 May 2019

The Linux Penguin just never fully understood USB attached storage.

Flash memory: An EPROM, or erasable programmable read-only memory, is a type of computer memory chip that retains its data when its power supply is switched off. From this technology we have the modern solid state storage devices known as flash memory or memory cards, including thumb drives and flash drives.

The Thumb Drive

Most popular among the types of EPROM removable memory is the thumb drive, a flash drive that you plug directly into the computer USB port. Replacing the floppy disk from a time long ago, the modern thumb drive can hold magnitudes more data. The floppy drive is no longer practical for most applications. A thumb drive can be made to be bootable and is often used to install an operating system on a computer.

Installing Linux from a thumb drive onto a PC is probably the only time that the USB memory seems to function correctly. While working in a Linux desktop environment on a daily basis I have found the use of USB flash memory very problematic. Problems include:

• File copy progress misreported, progress indicator in the file manager (all of them) shows completed in seconds while you wait for many minutes for the process to actually complete and the dialog to go away.
• File drive rarely cleanly ummounts
• Process frequently hangs for over a minute after the i/o activity completes. What is it doing? We have to wait for the dialog to go away even though its done!
• Process fails on copying large files. Files of 1GB or greater often result in the progress interface showing complete but hanging up and not releasing the drive indefinitely. How long? I once let it sit overnight and it was still hung up in the morning. Linux would not even property shut down!

Why is USB flash memory support so flaky in Linux? It is difficult to determine the problem and cooperate with developers on a resolution due to factors including:

• Linux zealots defend the kernel and deny the existence of a problem rather blaming the end user
• Microsoft poorly documents FAT elements such as bit flags relating to corruption of the file system making it difficult for kernel developers to ensure compatibility
• Linux desktop file managers failing to cope with the USB file system and relying on buggy kernel support
• A new industry focus on so called cloud storage has demoted focus on attached / portable mass storage
• Something different happens every time I connect my thumb drive: sometimes the file manger opens, sometimes it asks if I should open it in the file manager, sometimes nothing there is an audio sound, sometimes not --- all with the exact same thumb drive!

There are a number of lovely error messages. I think this one gets to the point...

"The files have been correctly transferred to the USB drive, however the program could not communicate to the kernel that the writing is completed."

Microsoft Windows Says Linux Corrupted My USB Drive

In ordinary cases and under circumstances where the file copy process went (as smooth as possible) on the Linux system, the flash drive being removed and attached to the Windows system may complain that it is necessary to "Repair this drive." Typically the drive does NOT need to be repaired even though Windows complains. The problem has to do with lack of Linux Kernel support for specific flags of a FAT32, FAT16, or NTFS filesystem. The Microsoft people won't provide clear documentation on specific flags and the Linux kernel developers are too stubborn to take the time and properly reverse engineer until full compatibility is realized.

Yes, it is a fact that Microsoft won't share, and even when they do, they provide bad information. However, flags such as ClnShutBitMask and HrdErrBitMask are not that difficult to reverse engineer based on simple observation of behavior. Linux kernel developers are too busy adding candy and crap to the kernel rather than stepping back and fixing long standing irritation problems such as USB drive support.

According to Microsoft:

• ClnShutBitMask: If bit is 1, the volume is “clean”. The volume can be mounted for access. If bit is 0, the volume is “dirty” indicating that a FAT file system driver was unable to dismount the volume properly (during a prior mount operation). The volume contents should be scanned for any damage to file system metadata.
• HrdErrBitMask: If this bit is 1, no disk read/write errors were encountered. If this bit is 0, the file system driver implementation encountered a disk I/O error on the volume the last time it was mounted, which is an indicator that some sectors may have gone bad. The volume contents should be scanned with a disk repair utility that does surface analysis on it looking for new bad sectors.

Bottom line - ignore the error from Microsoft Windows and keep using the drive, your files are probably fine. I say probably because there are a number of other things like can do to butcher your files beyond the kernel not understanding a couple bit flags.

Linux doesn't see my thumb drive

You can refer to USB Device Diagnostics in Linux for help using commands such as lsusb to diagnose why your thumb drive is not coming up when you connect it. These are console commands. There should be a graphical way to accomplish these tasks in Linux, however, that would require someone taking the time to create a good desktop environment tool for Linux users. I am a big fan of the command line, however, I am an anachronism and the modern Linux user demands graphical tools to do these things. Good desktop utilities do not seem to exist!

Terminal Mount

The command lsblk or LIST BLOCK DEVICES will help you determine if and where your Thumb drive is mounted. On Ubuntu and Mint the system likes to put your flash drive under /media/username - which is kinda lame. We can use grep to cut out clutter:

lsblk|grep media

Ah there's our USB thumb drive (based an an actual system example under Linux Mint)

nicole@cat$ lsblk|grep media
└─sdc1                8:33   1  14.6G  0 part /media/nicole/STORE N GO

Automount will mount the thumb drive when inserted as long as it has an acceptable recognized file system. You can manually mount it.

udisksctl mount -b /dev/sdc1 

When you're done you should make sure you have nothing running in the path of the file drive, including your own console

cd ~

And dismount the thumb drive

udisksctl unmount -b /dev/sdc1

Gnome, nautilus copy files to USB stops at 100% or near

The recommendations below relate to how much system RAM you have. Some of these fixes I believe are specific to people with 16MB of RAM. If you have a different amount you may have to make adjustments.

source (StackExchange) : http://unix.stackexchange.com/questions/180818/gnome-nautilus-copy-files-to-usb-stops-at-100-or-near

The reason it happens that way is that the program says "write this data" and the linux kernel copies it into a memory buffer that is queued to go to disk, and then says "ok, done". So the program thinks it has copied everything. Then the program closes the file, but suddenly the kernel makes it wait while that buffer is pushed out to disk.

So, unfortunately the program can't tell you how long it will take to flush the buffer because it doesn't know.

If you want to try some power-user tricks, you can reduce the size of the buffer that Linux uses by setting the kernel parameter vm.dirty_bytes to something like 15000000 (15 MB). This means the application can't get more than 15MB ahead of its actual progress. (You can change kernel parameters on the fly with sudo sysctl vm.dirty_bytes=15000000 but making them stay across a reboot requires changing a config file like /etc/sysctl.conf which might be specific to your distro.)

A side effect is that your computer might have lower data-writing throughput with this setting, but on the whole, I find it helpful to see that a program is running a long time while it writes lots of data vs. the confusion of having a program appear to be done with its job but the system lagging badly as the kernel does the actual work. Setting dirty_bytes to a reasonably small value can also help prevent your system from becoming unresponsive when you're low on free memory and run a program that suddenly writes lots of data.

But, don't set it too small! I use 15MB as a rough estimate that the kernel can flush the buffer to a normal hard drive in 1/4 of a second or less. It keeps my system from feeling "laggy".

source (AskUbuntu) : http://askubuntu.com/questions/397249/system-freezes-unresponsive-unusable-when-copying-large-file-to-usb/685293#685293

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

into /etc/sysctl.conf and running

sudo sysctl -p

source (AskUbuntu) : https://askubuntu.com/questions/734827/transfer-freezes-when-writing-big-files-on-usb-drive

If your running a x64 bit Ubuntu or other Linux and find USB transfers hang at the end apply this fix:

sudo echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
sudo echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes

I suggest you edit your /etc/rc.local file to make this change persistent across reboots.

sudo vi /etc/rc.local

Go to the bottom of the file and leave a space then paste in those two lines.

Don't like the changes??? To revert the changes enter this in console and remove the lines in /etc/rc.local

sudo echo 0 > /proc/sys/vm/dirty_background_bytes
sudo echo 0 > /proc/sys/vm/dirty_bytes

Related Pages

• USB Device Diagnostics in Linux
• Solid State Removable Storage
• Sandisk U3 Flash Drive Virus