Formatting USB Problem

[Radicool]

So I bought a USB 2.0 the other day. It was a cheap one from Woolworths. I used it and it seemed to transfer at a reasonable rate (cant remember what). My brother however formatted it for the Xbox 360 to copy his accounts across.

So now that I have formatted back to FAT32, with the highest allocation size, my USB seems really slow. I never get a transfer rate higher than 4MB/sec now. It seems to take a little longer for the computer to recongnise it. I unticked the "quick format" box as well.

Is formatting a USB bad? Why has it slowed my USB so much? Or maybe it hasn't, it just seems a lot slower since when I bought it and after I formatted it. Is 3.5-4MB/sec slow??

 

[Maker]

Are you usin Win7 or Vista? I found this by googling.

As for the 5 - 6mb/s transfer rate being " normal " it's not, that's slow for USB 2.0 or better. the actual cause of the problem is vinsta and win 7's approach to making USB devices safer to remove (in case of error during transfer i'm sure)

here's the way to get around that HUGE bottle-neck:

In Device Manager, right-click the USB drive in Disk drives folder, then select Properties, switch to Policies tab, and choose Optimize for performance. Click OK to keep it.

NOTE: If you use this method, make SURE you use the sub removal tool that pops up in your system tray when you first plugged in the device (system tray is the little area near your clock on the task bar)


After performing this fix for myself, my transfer rate on a cheapo cruizer micro went from 4.5 - 5mb/s to 20 - 30mb/s on small files, and average of 10 - 15mb/s on huge file transfers

This is because this method utelizes windows write-behind system of file management.

 

[Radicool]

Interesting. But this still doesnt explain why it seems slower after I formatted my USB.

However, 2 questions about what you said earlier.

1.) If I don't safety remove the USB, will data still be lost, even when not transferring data? Exactly how unsafe it is then.

2.) This option is saved onto the USB, or on the computer? So will I have to change this setting again on my friend's computer if I decide to plug my USB into his computer?

 

[Furiion52]

I can answer your first question
I have never or rarely safe removed a USB from my laptop. I just close the window and unplug it. Never lost any data, however when I put it into my friends Linux pc it got formatted to NSCII or something and I lost all the data :/ I guess you just have to be careful. Oh and sometime unplugging during transfer or even if something on the USB is in use it can cause file corruption.

 

[Xonok]

Interesting. But this still doesnt explain why it seems slower after I formatted my USB.

However, 2 questions about what you said earlier.

1.) If I don't safety remove the USB, will data still be lost, even when not transferring data? Exactly how unsafe it is then.

2.) This option is saved onto the USB, or on the computer? So will I have to change this setting again on my friend's computer if I decide to plug my USB into his computer?

1) The rule of thumb is that when something is being written, then it will be corrupted. Windows doesn't always tell you when it's writing though.
2) Quite certainly on the computer.

 

[Dr Super Good]

What you are asking makes no sense. USB 2.0 is a serial communication stand, what on earth does it have to do with file systems?!

Did you mean a USB memory stick?
The problem is likely because you formated it as FAT32, a file system aimed at compatibility rather than performance. There is a reason it is not commonly used on HDDs in this day and age.

Most HDDs are formated as NTFS, a dated but much more efficient file format. Formating such drives as NTFS will probably yeild considerable performance increases. The disadvantage is that Mac and Linux systems might have problems recognizing the drive.

1.) If I don't safety remove the USB, will data still be lost, even when not transferring data? Exactly how unsafe it is then.

It is impossible to tell. Windows might buffer any amount of data to be written for any amount of time after a transfer appears complete. Safely removing the memory stick is the only way to be sure against data loss as it forces the operating system to flush all file caches and make sure everything is written. Memory mapped files in write mode are especially prone to this type of loss as the OS will avoid writing if the page is undergoing constant change.

Be aware that the drive will always read faster than it can write.

 

[Radicool]

What you are asking makes no sense. USB 2.0 is a serial communication stand, what on earth does it have to do with file systems?!

Did you mean a USB memory stick?

Yes I have a memory stick. Sorry, Everybody just calls it a USB around here. It uses 2.0 technology (as apose to the 3.0 memory sticks that have recently starting showing up).

The problem is likely because you formated it as FAT32, a file system aimed at compatibility rather than performance. There is a reason it is not commonly used on HDDs in this day and age.

I am aware that NTFS is much faster. I only stick with FAT32 because of it's compatability with my DVD player, etc. FAT32 is the default format a memory stick comes in here (in Australia anyway).

However my original question is yet to be answered. My USB (memory stick) seemed to perform fast before the formatting. After my brother formatted it for the Xbox, then I formatted it back to FAT32, the write speeds vary from 3.5-4MB/sec. It just seems very slow. A friend also commented on how slow my USB is compared to his when we were waiting on a movie to copy onto it. I cant actually remember what speeds it acheived before the formatting process, however it seems like it is slower now (and my friend thinks so too).

I am wondering if formatting can affect the performance of a USB (memory stick). Or maybe I did something wrong? This might be just a silly suspicion due to the uncertainty of my question, but is 3.5MB/sec a slow write speed for a USB? Thats probebly going to answer my question

 

[Dr Super Good]

I am wondering if formatting can affect the performance of a USB (memory stick). Or maybe I did something wrong? This might be just a silly suspicion due to the uncertainty of my question, but is 3.5MB/sec a slow write speed for a USB? Thats probebly going to answer my question

If the file format has a block size (which I believe they do?) then it can very easilly. Solid State memory comes in chips made of blocks of storage space. The smallest change that can be made to solid state memory is that of a single block, meaning that even changing 1 bit will require the re-writing of an entire block. If you do not format the drive to use the correct block size you will end up with mal-aligned blocks. This will cause a lot of writing overhead potentially as it may have to rewrite a block twice for every 2 logical blocks given. As long as logical blocks are a whole multiple of physical blocks then the performance should remain optimum. You will want to make sure that logical blocks are the same size as physical blocks as this allows the drive to more effeiciently manage block wear (allowing it to last longer).

 

[Radicool]

Very Interesting. For clarification, Windows 7 format tool asks an input of a "allocation unit size". I'm guesing this is what your talking about when you say "logical block size".

When I selected the allocation unit size, I chose the larget block size. From my research, this would cause the most wasted space (which wasnt a concern to me) but have the fastest speeds. However your saying thats not neccicarily the case. I need to find the "optimum" logical block size which depends on the physical block size.

Is there any way of finding out what the "optimum" logical block size would be? Or will windows 7 automatically choose the best block size by default?


Edit: Here is an image of the format screen. I assume the "default allocation size" will detect the optimum block size then? (I'm sorry I forgot about that when I posted this).

Format

 

[Dr Super Good]

Mechanical hard disks are a platter of bytes. How you group these causes them to become blocks. Blocks are used by file system formats to make managing easier as they are a standard size and the smallest unit that the file system can manipulate. As a mechanical drive is an array of bytes, how you choose to partition it into blocks has no influence on performance outside of the nature of such block sizes.

Solid state has a very different problem. As they need a controler to manage wear (speard pysical usage as each cell can only be erased a finite number of times) they are pre-divided into blocks of a certain size. When you request a block be updated in a file, the actual physical location of that block will change as it will rather write to a new block than erase an existing block. Further more, it can only write data in block sizes, not individual bytes. This is very different from a mechanical drive where changes to blocks change the area of storage allocated to that block.

If you format the device to use logical block sizes (the block sizes the file system format uses) that are different from the physical block sizes (the block sizes the controler uses to manage the SSD) then there is a possiblity of degrade performance. The file system will use the logical block size as efficiently as possible to read and write but this may cause unomptimal physical block interaction. An example could be changing a 1 byte file resulting in multiple block erase or block writes putting unnescescary wear on the drive. Especially when using a lot of very small files, the drive will have to do a lot more work than it should which results in a slower file write rate. If you set the logical block size smaller than the physical block size then you get a very ugly situation where writing two blocks causes the first block to be written, read and written again due to the second block sharing a physical block with it. This not only greatly degrades performance but also puts extra wear on the drive.

The internet being full of computer illeterates does not explain the optimum block size well. Try with a variety of block sizes and you may see performance increases.

 

[Radicool]

Thanks. Your efforts are appreicated as always

 

[Ralle]

I have lost data on multiple occasions when unplugging without safe removal. On linux you can say the "sync" command and then unplug it without dismounting. "sync" flushes the write buffers to ensure that all data is written.
Copying speeds to any device seem to also depend on file size. The speed is much lower if you are copying 1000x of 1 KB files in comparison with 1x 1000 KB file. It could be that when you first got it you copied big files to it and now you're copying small files.
The performance should not degrade significantly over time, but one thing that comes to mind is the following: On any storage device when deleting a file, it is not physically removed. This means it's still there. When using flash based storage (USB sticks or SSDs) it needs to delete the data before overwriting it. This is why modern SSDs have the TRIM command. I suppose the "non-quick" formatting would clean it up and make it faster, but only until it is full of deleted stuff again.
I guess you just have to live with it.

 

[Dr Super Good]

Acording to that wikipedia page, the optimum block size should be 4 KB, which is the size of SSD internal pages.

 

[Radicool]

When using flash based storage (USB sticks or SSDs) it needs to delete the data before overwriting it.

Interesting. Thanks for your answer.

Acording to that wikipedia page, the optimum block size should be 4 KB, which is the size of SSD internal pages.

Thanks again. Ill try formatting with 4 KB when I get home from work.