Disklace's products scientific background

The 3rd Annual Haifa Experimental Systems Conference has accepted our scientific research poster

========================================================================

Chaos, Entropy and Disk-Fragmentation.

Disklace products' scientific background.

1. The background

Disk-Fragmentation was invented over 30 years ago, as a brilliant solution to the disk-management problem. Most of today's computer users are not aware of the fact that disk storage had to be managed manually, and failing to maintain this was the primary reason for failure of daily tasks in the past. After fragmentation was implemented, operating-systems' users no longer needed to bother where the computer chose to store the data, as it was always available when required.

The problems started rearing their heads when cheap disks with high capacity came onto the market and data on disks began to be scattered so that it took longer to access. At first, the difference is only micro seconds, increasing to seconds, then to many seconds until it comes noticeable, and the user starts being worried

The reason for this cumulative slowdown effect on the performance is a design limitation, and the forthcoming changes in technology are probably not going to change it.

Another historical fact is that disk defragmentation software appeared only few years after the disk itself and, until then, in order to rearrange the disk, it was performed in conjunction with a backup procedure. The disk was copied to another location, the new disk replaced the old one which was then archived with the backups. Introducing the fragmentation concept made this option redundant and is thus rarely used now.

From the user point of view, it looks as though a sudden change in the computer's response time has occurred. The bad news is that in many cases, even after running the defragmentation utility, the user is still not satisfied, and begins using terms like: "My computer is old.", "I have to buy a new computer" etc. Many users accept the verdict, while others tend to replace their computer.

To this state of uncertainty and confusion, salvation may come from Disklace products.

2. Chaos has its own rules

Using the word "Chaos" is generally meant to show an unpredictable outcome, but with today's processing power it is no longer fatal. Chaotic systems are systems that look random but in fact are not. Using computers we can follow the complexity of the system, thereby avoiding the uncertain behavior that characterized yesterday's systems. The storage world is one of the examples in which chaotic rules have been recruited to assist the industry.

Using the Chaos Theory, we can explain why data on the disk becomes messy even before the user notices that it has gone out of control.

In the data life-cycle, when a file is deleted, it is erased from all the places where it was held, but when a file is created, the computer cannot know its eventual size, so it is allocated to the first available empty space. When this space is insufficient the computer continues with the next available free space and so on. The result is that when a disk is already heavily fragmented, new files are fragmented from the moment they are created.

When the computer is new, the chaos has its own rules which allow us to follow and accommodate it. When the number of files increases and the situation becomes more complex involving more parameters than can be handled, it is still a chaotic system, but we begin to see the "After Mess" effect of the system.

To this situation another parameter is added: The user experience. Up to that point fragmentation was chaotic but the user could handle it. When the user starts noticing a change in the response time of the system and experiencing longer waiting times, this is the moment the "red line" has been crossed. At this stage it is still possible to attempt a defragmentation utility to reduce fragmentation, but in many cases it is too late, and restoring the computer's original performance is impossible.

3. Entropy and disk fragmentation

Entropy is defined in the Second Law of Thermodynamics and measures the inability of the system to function. Entropy is measured in temperature units.

In many aspects, an over-fragmented disk resembles an entropy system. From the user point of view, at this point there is apparently nothing wrong with the system, but is nevertheless unable to function. Running any defragmentation utility will prove futile and will never restore the disk to the original state.

To dramatize the situation, we can compare an over-fragmented disk to a "Black Hole". It's not visible, but once you enter it, there is no way back.

All computer users are familiar with the above case. Managers responsible for the infrastructure have no way of controlling the situation apart from listening to users' complaints regarding this state.

The situation turns into a puzzle without a solution. It cannot be solved because too many parameters have become static and cannot participate in the game any longer. At this point the user becomes dissatisfied and, as previously mentioned, begins using terms like: "My computer is old." or, "I have to buy a new computer".

Upgrading to a new computer and copying the data on to a new disk seems to solve all the problems. The only possible remaining problem is the price.

4. The Theory: Looking for the threshold

"Chaos" is a term used by mathematicians, while "Entropy" is related to Physics. We hypothesized that there might be an adhesion between the two that could be measured and enable implementation of preventive maintenance procedures adaptable to the storage area.

We have begun looking for the definitive point at which "Entropy" starts because, till that point, even with the existing chaotic situation, the user is satisfied and does not suffer from any performance degradation. The aim was to investigate the change in chaotic behavior along the data life-time of any disk, and to try to calculate, based on aggregate measurements and statistical distributions, at which point the situation is getting dangerously close to an uncontrolled entropy, thus permitting warning well in advance

The measurement unit suggested for this purpose is calculated as the combined second statistical moment of all the files and free space on the disk.

The first "Study" took over 2 years and involved 250,000 users of whom 30% have returned with results of their disk measurement.

The purpose of this study was to test the proposed technique that would determine the fragmentation situation in all disks regardless of size. In these tests, when a disk was only partially used, the effective disk size (The highest water mark used) was taken into consideration rather than the physical disk size which may have a higher capacity but is irrelevant for fragmentation calculation.

The results of the measurement technique used at this stage (Patent Pending) showed a normal statistical distribution, which proved that the measurement technique behaved in a way that enabled conclusions to be drawn. The samples are being continuously and dynamically recalculated. The number of samples used was much higher than that required to reach scientific conclusions, and further results that arrive continuously make the graph "smoother" without changing its basic characteristics.

The statistical Gauss "bell-shaped" distribution enables the determination of the fragmentation level of every disk, and warns the user of the proximity to the inflection point of the distribution graph at which the chaotic system might become entropic. That point can be considered as the entropy threshold. Minor distortions in the Gauss shape show that there is still research to be done.

LaceWatcher PRO was developed recently as the software solution for large accounts. It uses the same measurement technique as the previous stage and enables unattended checking of the fragmentation level from a central point in the organization by sending a Level I alert or by activating a standard remote defragmentation utility at the user's station. The threshold level at which the alert is produced, or the defragmentation activated, can be modified according to specific needs, and is based on the user's own experience and requirements. A second level alert tells the operator which station or server has already passed from "Stage I" (Chaos) into "Stage II" (Entropy). The alert can be sent to any Asset Management or Problem Management software. The results are recorded on a disk level, providing history-data for long-term management.

5. Conclusions

1: Disk fragmentation is too expensive to ignore: Every disk has its own unique characteristics caused by the mix of file-size and file-types but every disk will eventually reach the point where fragmentation starts affecting the response time.

2: Using the obtained Lacelevel factor, it is possible to optimize "Preventive Maintenance" procedures in computers. We have succeeded in modeling disks regardless of size. A proper maintenance which is performed on time can be used to increase the lifespan of computers thereby reducing upgrade budgets.

3: Implementation by stages: We suggest implementing first the monitoring capabilities of LaceWatcher PRO to get a comprehensive view of the fragmentation in your installation, gradually adding computers to the "Remote Defragmentation" process.

4, Calculate your own Inflection point:

Yours vs. Global.

Every organization can fine-tune the parameters based on the data-characteristics on the managed disks, accompanied by the specific graph for the relevant population.

As calculated by independent research, the saving for users of the system will be 30% of their annual budget for upgrading old servers, workstations and portable computers.

* * *

Please visit a walkthrough and DEMO of LaceWatcher PRO at http://www.disklace.com/LaceWatcherPro.html

Our next step research is planned in collaboration with :