Data validation

Validation - what is it?

Due to the reasons described below, it is not reasonable to expect "100% correct" validation report even on a perfectly good volume. 95-98% is more reasonable expectation.

Important note

Limitations of the validation process

• The most significant is that the validation process can not substitute human review, because there is no way a program can distinguish e.g. between the latest (useful) and older (worthless) versions of the file.
• On top of that, typical validator does not "truly" read the file. When possible, validator will skim along the file touching and verifying several key points in the file. This is presumed the most fruitful approach because of the following two reasons:
  • Validation of some file formats requires excessive computation effort, which would be impractical unless your other computer is Cray.
  • There are way too many file formats out there. Some of these file formats are fairly complicated. Full-blown reader implementations for all these file formats would just be cost-prohibitive.
• Some files are in fact a combination of two separate file formats. For example, a self-extracting archive consists of the decompression module executable file immediately followed by the compressed archive data. For such files, the format is checked as specified by the file extension (in the above example the executable module will be checked, rather than compressed data).
• Some files would be different format from what their extensions suggest. For example, some files left from the older days have .DOC extension but are in fact plain text. The validator may try to verify these files as Microsoft Word format, which is obviously not the case. This will trigger false "validation failure" indication.
• Not all file formats are supported. Refer to the full list of the file formats supported by the validation engine for details.