You are here

Defragment Thumb Drive

27 posts / 0 new
Last post
BErG123
Offline
Last seen: 15 years 6 months ago
Joined: 2008-09-29 06:16
Defragment Thumb Drive

Just to help people and bring up the subject, and maybe to suggest that it should be written on the support pages for helping portable apps (at least Firefox) run faster, I suggest using a defragmenting program to defragment your thumb drive. I don't know which ones can and can't, but I tried Diskeeper Lite 9 (available in a huge download of accessories (89MB) at http://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmir... or wherever else you can find it, if you can), and it did just fine. It has an option to analyze the selected drive to show you how defragmented it is, and defragment it. Although I've never tried making Diskeeper portable.

Simeon
Simeon's picture
Offline
Last seen: 9 years 5 months ago
DeveloperTranslator
Joined: 2006-09-25 15:15
But

most current flash drives use wear levelling to prolong their life span. On these drives, defragmenting isn't useful, it just makes them die faster.

"What about Love?" - "Overrated. Biochemically no different than eating large quantities of chocolate." - Al Pacino in The Devils Advocate

RMB Fixed
Offline
Last seen: 14 years 2 months ago
Joined: 2006-10-24 10:30
...

RAM means "Random Access Memory" and has no moving parts .
This means that fragmentation is not an issue with flash-drives,
wear-levelling or not .
You should NOT defragment your flash-drive, it doesn't help anything
and requires A LOT of write-operations ..

Simeon
Simeon's picture
Offline
Last seen: 9 years 5 months ago
DeveloperTranslator
Joined: 2006-09-25 15:15
I know

but wasn't the original poster talking about flash drives?

"What about Love?" - "Overrated. Biochemically no different than eating large quantities of chocolate." - Al Pacino in The Devils Advocate

m2
Offline
Last seen: 13 years 1 month ago
Joined: 2006-12-12 12:00
I guess he was. I guess RMB

I guess he was.
I guess RMB Fixed thinks that flash is random access memory. It's not, but almost...it's performance doesn't suffer from random access, but addressing unit is too big to call it RAM.

Anyway:
Formatting flash gives nothing, wastes your time and reduces lifespan. I guess that it's a good thing to write in the usage guide.

"Those people who think they know everything are a great annoyance to those of us who do." Asimov

Simeon
Simeon's picture
Offline
Last seen: 9 years 5 months ago
DeveloperTranslator
Joined: 2006-09-25 15:15
Ah

I get it now.

"What about Love?" - "Overrated. Biochemically no different than eating large quantities of chocolate." - Al Pacino in The Devils Advocate

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
flsh file system

Some people had first quite right idea to defrg the flsh, since the wear leveling does normally not happen over the whole chip, but rather in smaller islands, their number depending on the kind and size of the chips involved.
So it first looks promissing to get at least some files together, so some cycles of addressing could be saved.
This would be very hard to measure anyway, but the problem furthermore is that the sticks use flash files system, which is the manufacturers idea how to let the included controller read and write to all those chips and islands on them. So in fact, what we see is kind of virtual stuff, the formating and size of the flash etc are often just some parameters stored in the controller.
So one can buy 16gb stick, nice , new , very cheap!! It will display 16gb space on windows, one can write to it, read it. On linux with dd, something starts often go fishy, since the number of sectors it can read out seem to be suddenly much smaller then needed for the 16gb.
Then data written to the drive under win start disappear somehow randomly when trying to write more then 4gb to it. The files are still listed, they have the right size, but when it comes to retrieve them ....

Happened to me recently, lucky the 16gb was just a well meant gift from someone.

OK, the geometry of the real physical storage on a rotating disk is also not so as we can see it on our screens, the controllers are getting so smart and can tell also anything we like rather then the truth. So even here, the things will become somewhat theoretical in near future. Since we can only defragg what our windows or what does tell us, we still do so often some defragging, we have been told so since dos times.
But we have to be avare of the fact, that more and more we defragg only some virtual volumes and not the harddrive itself. This is defragged allready by the controller of the hard drive without we have any controll over it.

Otto Sykora
Basel, Switzerland

m2
Offline
Last seen: 13 years 1 month ago
Joined: 2006-12-12 12:00
Wrong. There are no

Wrong. There are no continuous regions on a flash drive. Everything is being dynamically remapped, you're not changing islands but sets of points.
If you feel like helping your wear leveling you'd do much better by moving everything from your flash and back. Effect way better and because unlike with defrag, you make mostly big writes this way, you'll likely write less.
I doubt it's worth it though.

"Those people who think they know everything are a great annoyance to those of us who do." Asimov

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
yes, but

zones are defined before all is going to be used. They are not physically insulated or so, but defined from-to.
And wear leveling will happen only inside such zone or island or simply an assigned space of sectors, not over whole chip or array of chips, this would be too complex and probably slow, I dont know.
The question remains how big those sets of sectors are. Some old small flash like I have in some embeded units have some 8, other apparently 16 other more. But those are small compared to present day usbsticks. I have here some of 12mg , some 32mb, some 64mg. from outside all looks similar, inside all can differ depending somehow on some specific ideas of the manufacturer it seems.

Otto Sykora
Basel, Switzerland

jamvaru
Offline
Last seen: 2 years 5 months ago
Joined: 2009-08-20 15:39
sorting

I reformatted my 8gb stick (super talent) as NTFS @ 64k/cluster

it is much faster...

it seems to be even faster (at some things, like transferring whole folders) if I use jkdefrag to sort by file name (folder/file)

however, this means a certain portion (the "front") of the drive is used more frequently

would this "kill" my flash drive, or just the parts that get used more frequently?

also, what about when you have it nearly or all full? what if it is mostly fragmented?

are there any benchmarks for this? or online information regarding benchmark testing with fragmentation and without? with different sorting algorythms?

thanks

good thread, a little angry, though

;>jamvaru

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
you did nothing good

to your drive.

NTFS writes a lot extra, so it uses the drive more, however this is nowadays more and more less important , the drives and the firmware inside is getting better and better.
Non-journaling file systems as FAT are more suitable unless you need to transport big files (bigger then 4GB).
If something writes to front or not, you will never discover. You have absolutely no way to access any part of your drive from your file system. The file system you format the drive is just virtual, the filesystem of the drive internally is different and depends on manufacturer etc.

There is no way defragment your drive either. All you can do is defrag the virtual file system and this is completely pointless.

Otto Sykora
Basel, Switzerland

LOGAN-Portable
LOGAN-Portable's picture
Offline
Last seen: 11 years 2 months ago
Developer
Joined: 2007-09-11 12:24
Please do never defragment a

Please do never defragment a USB flash thumbdrive. You get no speed gain this way.

Bensawsome
Offline
Last seen: 2 years 1 month ago
Joined: 2006-04-22 19:27
Also...

Also like people said up at the top it REALLY shortens the life of ur usb drive. The first time I got one I was stupid and defraged it every week and by the end of the month it was dead -_-

 iLike Macs, iPwn, However you put it... Apple is better ^_^ 
"Claiming that your operating system is the best in the world because more people use it is like saying McDonalds makes the best food in the world..."

m-p-3
m-p-3's picture
Offline
Last seen: 3 months 3 weeks ago
Joined: 2006-06-17 21:25
Best way to kill your

Best way to kill your thumbdrive. Fragmentation cause no speed loss on these devices.

Stevoisiak
Stevoisiak's picture
Offline
Last seen: 4 years 7 months ago
Joined: 2008-02-05 11:22
A quote from MyDefrag (AKA JKDefrag)

According to the MyDefrag website, (Which used to be JKdefrag,)

Yes. Flash memory disks (such as USB memory sticks and Solid State Disks (SSD)) have a limited number of erase-write cycles. The MyDefrag defragmentation and optimization will move files to new locations, which involves erasing and writing, so it will reduce the lifespan of your flash memory.

But there is no cause for alarm. Modern flash memory disks have at least 10,000 write cycles, more expensive types use different hardware that is guaranteed for a minimum of 100,000 cycles. All flash memory disks use a technique called wear-leveling. The controller in the memory disk will automatically reassign blocks in the memory so that all the memory is worn down evenly. For a good explanation of how this works see the * Corsair USB Flash Wear-Leveling and Life Span article on the Corsair website. In order to wear out a cheap 10,000 cycle flash memory disk in ten years, you would have to write to EVERY BLOCK in the device about 2.7 times per day, every single day. This does not take into account error correction, which will extend the life even further, and the fact that the 10,000 cycles is a guaranteed minimum, typical flash memory will handle an order of magnitude more write cycles.

The MyDefrag script to defragment and optimize Flash memory is specially designed to move as little data as possible. Fragmented files are defragmented (this takes just a single write cycle), unfragmented files are not touched at all. Gaps are filled by moving all the files together, if there are no gaps then MyDefrag will do nothing.

Nevertheless, my advice is to use some discretion and not defragment/optimize flash memory disks every day, but only incidentally, for example once per month.

Simplifying daily life through technology

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
all sounds great

but:

>Fragmented files are defragmented (this takes just a single write cycle), unfragmented files are not touched at all. Gaps are filled by moving all the files together, if there are no gaps then MyDefrag will do nothing.

Otto Sykora
Basel, Switzerland

jamvaru
Offline
Last seen: 2 years 5 months ago
Joined: 2009-08-20 15:39
well...

nevertheless,

it seems to make a bit of a difference in general for there to be a reorganization of the drive in favor of files/folders alpha/numeric

I have decided to do this and go one (or two) steps further

using a simple batch file and regular jkdefrag (not sure about the portable version)

do a sort by file/folder

then:

whenever you feel like it, as often as you desire, do:

1. a defrag only
2. a move up (to end of drive)

when this is complete and redone, there is no more movement, the files are optimized, there is free space at the "beginning" of the drive (or virtual drive)...

the result is faster access and less fragmentation.

when you do decide to run the batch again (just steps 1,2 above), there will be some minor defragmentation and everything loose will be collected at the top "" of the drive.

If there is "wear leveling", then there should be an internal record of what has been written to and what needs to be written to. When your drive is full only a few bits can be written to, period. So, eventually some bits will go and others will remain.

The question is: does this mean the drive will just die, or just a few bits at a time?

thanks,

j

;>jamvaru

Bahamut
Bahamut's picture
Offline
Last seen: 12 years 3 months ago
Joined: 2006-04-07 08:44
No no no

There is no performance difference whether the data is fragmented or not on flash drives. Unlike a hard drive, nothing has to move to get to another sector.
Also, internal wear leveling makes it impossible to write the bits in a specific order physically, which means you can't truly defrag it anyway. The OS cannot choose where data is stored physically on a flash drive.

You will kill your flash drive without any benefit. It really is as simple as that. It is absolutely unnecessary and nearly impossible to defrag a flash drive. Any attempts will only shorten the drive's lifespan.

Vintage!

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
not so simple

>when this is complete and redone, there is no more movement, the files are optimized, there is free space at the "beginning" of the drive (or virtual drive)...the result is faster access and less fragmentation.If there is "wear leveling", then there should be an internal record of what has been written to and what needs to be written to. The question is: does this mean the drive will just die, or just a few bits at a time?

Otto Sykora
Basel, Switzerland

jamvaru
Offline
Last seen: 2 years 5 months ago
Joined: 2009-08-20 15:39
thank you

I appreciate your professional approach to this discussion.

I think 10 to 100 thousand writes (or is it reads and writes?) is enough to consider the point moot. I still believe consolidating free space and making files contiguous is worth the small extra wear involved in the process. Maybe the benefit is a small one, or perhaps nothing at all. I'd like to see some scientific studies under controlled conditions. Speed testing before and after, with lots of fragmentation and little free space, etc.

very interesting discussion

{oh, I am referring to the virtual file system, not the random one)

;>jamvaru

Ed_P
Offline
Last seen: 5 years 5 months ago
Joined: 2007-02-19 09:09
Definite benefits

One member found there to be significant improvement in app speed when he defrag'ed an app's files. See his comments here:

https://portableapps.com/node/20385

Doubtful this would hold true for all apps but it may for those that are large and/or perform a lot of IO.

As for the wear and tear on a flash drive, with the prices decreasing every day they are a commodity. When they wear out replace them with new ones, which is most cases will be bigger and faster and less expensive.

Ed

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
there is one important point

in the theread 20385.

No defragg was done! it was cleared and then copied back. This is very big difference. This kind of defragg is included in the low level file system of the nand flash.
Removing big bunch of files and pasting them back to now half empty space on the flash will more or less write the files in adjacent eraseblocks if possible, thus kind of defragging them.

Otto Sykora
Basel, Switzerland

Ed_P
Offline
Last seen: 5 years 5 months ago
Joined: 2007-02-19 09:09
So it can be done

So defragging a USB drive can be done, and it can yield improvements in accessing the files on it. It just can't be done in the same manner as Windows uses in defragging a hard drive. Rather a script to xcopy the USB files and folders to the hd, erase the files from the USB drive and then xcopy everything back to it. A piece of cake to set up.

Ed

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
yes and no

thought it will work only on those very cheap once partially and also it can work only if large part or all will be removed and rewritten.
It will still not do any positioning of anything, it will simply write in adjacent eraseblocks if they are marked as free as long as data stream is coming in. Also our file allocation table will be rewritten during that procedure and moved from one eraseblock to other all the time and disturbing the consecutive data write to the free space. The same will do the compulsory copy of the file allocation table.
(we can not write over something, eraseblock has to be cleared completely first before it can be reused again).
So if many small files will be written in that way, still same or eeven bigger fragmenting can result. If large files are written, this might be more useful.

***It will all make no help to the data written on the flash, the access times will be same still. No improvements to access of files will result at all, no improvements are possible***.

The only thing will be marginally speedup is later write of large data files, since the empty space might be more made of connected eraseblocks then otherwise.

More sophisticated once use a table, where they write down how much was every erase block used in the past and will allocate the writes to eraseblocks which had the least usage up to now, so there all tricks will not work.

Otto Sykora
Basel, Switzerland

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
depends on

>10 to 100 thousand writes (or is it reads and writes?)I still believe consolidating free space and making files contiguous is worth the small extra wear involved in the process. Maybe the benefit is a small one, or perhaps nothing at all. I'd like to see some scientific studies under controlled conditions.

Otto Sykora
Basel, Switzerland

jamvaru
Offline
Last seen: 2 years 5 months ago
Joined: 2009-08-20 15:39
what happens then?

when you do a defrag, several small chunks are copied to a separate location on the drive, contiguously.

so, isn't this satisfying the concept of adjacent erase blocks? Or, does the drive split the file up into separate, non-adjacent erase blocks randomly, even though you are telling it to write adjacently?

so, the flash drive, if the second is true, does not care what the 'virtual' file system says/thinks, it is always random...?

i think i get it now...

is that right?

so, the virtual fs is only an indicator of the true fs, something windows can manage

the true (random) fs couldn't care less what windows thinks

whatever time it takes to translate from the random to the virtual to windows must be infinitesimally small (wow i spelled that right)

so,

1. the random fs cannot be manipulated at all, because it is random, even in entire move/move operations.
2. no real time is spent in the virtual operations, so none is saved by any modification of the virtual fs, nor any space

i do notice a lag in my firefox, for example, typing this... i'm going to try the move/move test and see if it works.

later,

j

;>jamvaru

ottosykora
Offline
Last seen: 23 hours 59 min ago
Joined: 2007-10-11 17:48
yes

>when you do a defrag, several small chunks are copied to a separate location on the drive, contiguously.so, isn't this satisfying the concept of adjacent erase blocks? Or, does the drive split the file up into separate, non-adjacent erase blocks randomly, even though you are telling it to write adjacently?so, the virtual fs is only an indicator of the true fs, something windows can managei do notice a lag in my firefox, for example, typing this... i'm going to try the move/move test and see if it works.

Otto Sykora
Basel, Switzerland

Log in or register to post comments