Are there any Best Practices to prevent/reduce the occurance of UV file Blinks

Blinks. Our product uses the Universe database with Pick Flavored account running on various Windows OS platforms. What, if anything, can be done to reduce the number of file  blinks that we are seeing? Three causes that we've identified so far are: Power failure, No more disk space, Anti-virus. 

Are any UV versions (or Windows versions) more susceptible than others to Blinks?

Thanks in advance for your inputs.

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

Hi Gil!

Without asking for a bit more detail, here are some really quick off-the-cuff suggestions - none of which are specific to the multi-value world:
Power Failures - Any system running a database should be running on a UPS of one sort or another (generator would be even better). Some UPS models can even be configured to send a signal to the server to initiate a clean shutdown if the power is out for an extended period and their batteries start running low.
Running out of disk space - if this is a gradual growth, the systems should be initially spec'd with more disk (not "just enough" for the install) and scheduled tasks should be set up to monitor disk usage. If this is a sudden, unexpected growth (like a runaway process of some sort), do what you can to eliminate those (based on what may have happened in the past) and again, put some monitors in place.
Anti-virus - any anti-virus/anti-malware that scans a database server should be configured to ignore the files/folders known to be database files.

Hope that helps some.

Brian

Hi Gil!

Without asking for a bit more detail, here are some really quick off-the-cuff suggestions - none of which are specific to the multi-value world:
Power Failures - Any system running a database should be running on a UPS of one sort or another (generator would be even better). Some UPS models can even be configured to send a signal to the server to initiate a clean shutdown if the power is out for an extended period and their batteries start running low.
Running out of disk space - if this is a gradual growth, the systems should be initially spec'd with more disk (not "just enough" for the install) and scheduled tasks should be set up to monitor disk usage. If this is a sudden, unexpected growth (like a runaway process of some sort), do what you can to eliminate those (based on what may have happened in the past) and again, put some monitors in place.
Anti-virus - any anti-virus/anti-malware that scans a database server should be configured to ignore the files/folders known to be database files.

Hope that helps some.

Brian

What are the other common causes of blinks and how can we minimize the risks? Will poorly sized files increase the chances of getting a blink on the file?

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

What are the other common causes of blinks and how can we minimize the risks? Will poorly sized files increase the chances of getting a blink on the file?

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

With some of the items already mentioned, a poorly sized file can be a contributing factor. For example, if the file is heavily into overflow groups, a record update can involve writing multiple groups in the overflow chain. The number of overflow buffers that need to be updated can increase the chance that an interrupted write operation will result in an issue with the overflow buffer pointers. 

------------------------------
Neil Morris
Universe Advanced Technical Support
Rocket Software
------------------------------

Hi Gil!

Without asking for a bit more detail, here are some really quick off-the-cuff suggestions - none of which are specific to the multi-value world:
Power Failures - Any system running a database should be running on a UPS of one sort or another (generator would be even better). Some UPS models can even be configured to send a signal to the server to initiate a clean shutdown if the power is out for an extended period and their batteries start running low.
Running out of disk space - if this is a gradual growth, the systems should be initially spec'd with more disk (not "just enough" for the install) and scheduled tasks should be set up to monitor disk usage. If this is a sudden, unexpected growth (like a runaway process of some sort), do what you can to eliminate those (based on what may have happened in the past) and again, put some monitors in place.
Anti-virus - any anti-virus/anti-malware that scans a database server should be configured to ignore the files/folders known to be database files.

Hope that helps some.

Brian

Gil,

Adding to Brian's execellent post on using a UPS, please opt for an 'online' UPS.

• An 'online' UPS is always supplying power to the system and incoming electrical power goes to the UPS, not directly to the protected system. The advantage is that if and when main power glitches or fails there is no switchover to battery which can itself cause a dip as the switchover takes place..They should also protect against overvoltages, brownouts and spikes.
  
  
• A final word on the UPS - get a good well-known brand name and track record - do not skimp. They are very specifically intended for mission-critical systems.

Regards

JJ

------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

Hi Gil!

Without asking for a bit more detail, here are some really quick off-the-cuff suggestions - none of which are specific to the multi-value world:
Power Failures - Any system running a database should be running on a UPS of one sort or another (generator would be even better). Some UPS models can even be configured to send a signal to the server to initiate a clean shutdown if the power is out for an extended period and their batteries start running low.
Running out of disk space - if this is a gradual growth, the systems should be initially spec'd with more disk (not "just enough" for the install) and scheduled tasks should be set up to monitor disk usage. If this is a sudden, unexpected growth (like a runaway process of some sort), do what you can to eliminate those (based on what may have happened in the past) and again, put some monitors in place.
Anti-virus - any anti-virus/anti-malware that scans a database server should be configured to ignore the files/folders known to be database files.

Hope that helps some.

Brian

This is correct.  We here have a task in our Universe scheduler, that every week tells us how much disk space has been used and left.  And emails that to the sysadmin.

------------------------------
Will Johnson
Systems Analyst
Rocket Forum Shared Account
------------------------------

This is correct.  We here have a task in our Universe scheduler, that every week tells us how much disk space has been used and left.  And emails that to the sysadmin.

------------------------------
Will Johnson
Systems Analyst
Rocket Forum Shared Account
------------------------------

What is a reliable method to identify files with blinks proactively? Will a simple SELECT or COUNT be effective? I'm thinking to make a scheduled job to check our files for corruption.

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

What is a reliable method to identify files with blinks proactively? Will a simple SELECT or COUNT be effective? I'm thinking to make a scheduled job to check our files for corruption.

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

If you want to check for file corruption on a proactive basis, using the 'fixtool' command might be best approach. 

------------------------------
Neil Morris
Universe Advanced Technical Support
Rocket Software
------------------------------

Blinks. Our product uses the Universe database with Pick Flavored account running on various Windows OS platforms. What, if anything, can be done to reduce the number of file  blinks that we are seeing? Three causes that we've identified so far are: Power failure, No more disk space, Anti-virus. 

Are any UV versions (or Windows versions) more susceptible than others to Blinks?

Thanks in advance for your inputs.

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

Hi Gil, 

I suggest to performs daily backup with uvbackup tool and check log. 

Then, perform, on regulary basis, a HASH. HELP and anticipate overflow with resizing. 

I hope this help

------------------------------
Manu Fernandes
------------------------------

What is a reliable method to identify files with blinks proactively? Will a simple SELECT or COUNT be effective? I'm thinking to make a scheduled job to check our files for corruption.

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

If you wish to use the SELECT or COUNT approach, include a WITH clause that will not select any records. Example: WITH PositiveNumericField < 0.

This ensures the record is read, including any oversize record blocks. Otherwise, this just reads the primary group chain record blocks and ID. 

The most robust approach would use fixtool.

------------------------------
Mark A Baldridge
Principal Consultant
Thought Mirror
Nacogdoches, Texas United States
------------------------------

Blinks. Our product uses the Universe database with Pick Flavored account running on various Windows OS platforms. What, if anything, can be done to reduce the number of file  blinks that we are seeing? Three causes that we've identified so far are: Power failure, No more disk space, Anti-virus. 

Are any UV versions (or Windows versions) more susceptible than others to Blinks?

Thanks in advance for your inputs.

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

Hi Gil, re the no more disk space.

Does your windows installation have separate file systems (partitions?) for things like your main UV database(s), logs, software? Do processes that run your application write logs to the UV database file system or to a separate file system? If you're running/storing everything in a single file system then that increases the likelihood of filling up your disk and borking(tm) your data with blinks. Especially if your files are grossly overflowed.

If your system(s) isn't already setup with depurate database/log/software partitions then perhaps a reorganisation might be needed to minimise impacts of runaway processes filling up logs etc. I quick and dirty way of achieving this (in unix at least) is to use symbolic links to relocate local logs in your DB out to a separate disk or partition. I'm sure windows has a similar feature but I'm not familiar with it.

Hope the above is of assistance.

Cheers,
Peter

------------------------------
Peter Cheney
Developer and Systems Superstar
Firstmac
Brisbane Qld Australia
------------------------------

If you wish to use the SELECT or COUNT approach, include a WITH clause that will not select any records. Example: WITH PositiveNumericField < 0.

This ensures the record is read, including any oversize record blocks. Otherwise, this just reads the primary group chain record blocks and ID. 

The most robust approach would use fixtool.

------------------------------
Mark A Baldridge
Principal Consultant
Thought Mirror
Nacogdoches, Texas United States
------------------------------

LIST with DET.SUP also works.

------------------------------
Tyrel Marak
Technical Support Manager
Aptron Corporation
Florham Park NJ US
------------------------------

What is a reliable method to identify files with blinks proactively? Will a simple SELECT or COUNT be effective? I'm thinking to make a scheduled job to check our files for corruption.

------------------------------
Gil Steidle
DEV
DDI System Inc
brick NJ US
------------------------------

Hi Gil,

For us badly sized files has always been the biggest culprit for UniVerse file corruption, that and dynamic files which seem to be susceptible to corruption following system failures.

We have developed our own daily FILE.STAT suite that performs the FILE.STAT on each file on a daily basis overnight, we log and report any Blink errors identified, but with the robustness of UNIX system nowadays we rarely see them. We maintain a history of each file so that we can try to predict forward growth of each file a have the ability to automatically resize files or manually manage the process.

Saying that, there is the Rocket provided ACCOUNT.FILE.STATS which logs each time it's run to a file called STAT.FILE with an internal date/time stamp ID, examining the last run for each file would identify both if the FILE.STAT failed (Blink error) by checking the FILE.STAT attributes (they will not be populated) and as a bonus this also enables you to check that your files are sized correctly so you can remediate as appropriate.

Sample table below below shows the omission of the number of Records (Items) within the file, the first two entries were whilst a Blink error existed on a test file.

SORT STAT.FILE BY @ID FILENAME.LONG FILETYPE FILEMOD FILESEP FILESIZE RECORDS 17:37:06  21-05-24  PAGE 1
ID..................... File Name..................... Type Mod..... Sep File....... Record.
                                                                         Size....... Count..

20596*59877*0012        DUFF                             18      101   2      104448
20596*60074*0012        DUFF                             18      101   2      104448
20596*60200*0012        DUFF                             18      101   2      104448     415
20596*60361*0012        DUFF                             18      101   2      104448     414
20596*60489*0012        DUFF                             18      101   2      104448     414
20596*60881*0012        DUFF                             18      101   2     7135232   58913
20596*61027*0012        DUFF                             18      101   2     7135232   58913
20596*63155*0012        DUFF                             18      101   2      104448       0

Regards

MArk 

------------------------------
Mark Copp
Infrastructure Design
Travis Perkins Trading Company Ltd
Northampton GB
------------------------------

Hi Gil,

For us badly sized files has always been the biggest culprit for UniVerse file corruption, that and dynamic files which seem to be susceptible to corruption following system failures.

We have developed our own daily FILE.STAT suite that performs the FILE.STAT on each file on a daily basis overnight, we log and report any Blink errors identified, but with the robustness of UNIX system nowadays we rarely see them. We maintain a history of each file so that we can try to predict forward growth of each file a have the ability to automatically resize files or manually manage the process.

Saying that, there is the Rocket provided ACCOUNT.FILE.STATS which logs each time it's run to a file called STAT.FILE with an internal date/time stamp ID, examining the last run for each file would identify both if the FILE.STAT failed (Blink error) by checking the FILE.STAT attributes (they will not be populated) and as a bonus this also enables you to check that your files are sized correctly so you can remediate as appropriate.

Sample table below below shows the omission of the number of Records (Items) within the file, the first two entries were whilst a Blink error existed on a test file.

SORT STAT.FILE BY @ID FILENAME.LONG FILETYPE FILEMOD FILESEP FILESIZE RECORDS 17:37:06  21-05-24  PAGE 1
ID..................... File Name..................... Type Mod..... Sep File....... Record.
                                                                         Size....... Count..

20596*59877*0012        DUFF                             18      101   2      104448
20596*60074*0012        DUFF                             18      101   2      104448
20596*60200*0012        DUFF                             18      101   2      104448     415
20596*60361*0012        DUFF                             18      101   2      104448     414
20596*60489*0012        DUFF                             18      101   2      104448     414
20596*60881*0012        DUFF                             18      101   2     7135232   58913
20596*61027*0012        DUFF                             18      101   2     7135232   58913
20596*63155*0012        DUFF                             18      101   2      104448       0

Regards

MArk 

------------------------------
Mark Copp
Infrastructure Design
Travis Perkins Trading Company Ltd
Northampton GB
------------------------------

As you observe, this is the occurrence of a system failure. UniVerse will NOT break files just by using them. There are no holes in the locking mechanism. If files break, there is an underlying cause which you can address. 

Both the overflow chain of group blocks holding record headers, and the linkage of oversize record blocks can be corrupted. The exposure time of active updates differ for each linkage structure related to system crashes, and a disk full condition will coincide when the file is extended for overflow. Oversize record handling - records that do not fit in a group including the record header and ID (and the LARGE.RECORD factor in dynamic files) - is different. UniVerse frees the original disk blocks for the oversize record, then allocates the oversize group blocks it needs for the updated or new record, then writes the data. If the write fails because it cannot allocate enough blocks for the new record, the write will fail (without corruption), however it will fail after it has removed the original oversize record.

------------------------------
Mark A Baldridge
Principal Consultant
Thought Mirror
Nacogdoches, Texas United States
------------------------------

Hi Gil, re the no more disk space.

Does your windows installation have separate file systems (partitions?) for things like your main UV database(s), logs, software? Do processes that run your application write logs to the UV database file system or to a separate file system? If you're running/storing everything in a single file system then that increases the likelihood of filling up your disk and borking(tm) your data with blinks. Especially if your files are grossly overflowed.

If your system(s) isn't already setup with depurate database/log/software partitions then perhaps a reorganisation might be needed to minimise impacts of runaway processes filling up logs etc. I quick and dirty way of achieving this (in unix at least) is to use symbolic links to relocate local logs in your DB out to a separate disk or partition. I'm sure windows has a similar feature but I'm not familiar with it.

Hope the above is of assistance.

Cheers,
Peter

------------------------------
Peter Cheney
Developer and Systems Superstar
Firstmac
Brisbane Qld Australia
------------------------------

If files are grossly overflowed, switch to dynamic hashed, type 30.

I use type 30 for files that tend to grow.

For static tables, I'll size them correctly to static hashed, & monitor weekly with a scheduled report of anything out of whack.

I use ACCOUNT.FILE.STATS weekly & save each weeks data in my own FSTAT.HIST so I can sort by file then date.  It's dictionary is basically same as STAT.FILE, with a few fancy I-descriptors that compare the previous FSTAT.HIST record to the current one I  can get things like BytesAddedPerDay & RecsAddedPerDay  & averages & see when things become unstable.

 DICT FSTAT.HIST DATABYTES.PERDAY

0001: I works on lists sorted BY[.DSND] FILENAME BY[.DSND] RUNDATE
0002: @2;FILENAME; @4;RUNDATE; @6;DATABYTES; IF @1=@2 & @3#@4 THEN (@5-@6)/(@3-@4) ELSE ''
0003:
0004: DataBytesý / Day
0005: 9R0Z
0006: S

------------------------------
Chuck Stevenson
DBA / SW Developer
Pomeroy
US
------------------------------

If files are grossly overflowed, switch to dynamic hashed, type 30.

I use type 30 for files that tend to grow.

For static tables, I'll size them correctly to static hashed, & monitor weekly with a scheduled report of anything out of whack.

I use ACCOUNT.FILE.STATS weekly & save each weeks data in my own FSTAT.HIST so I can sort by file then date.  It's dictionary is basically same as STAT.FILE, with a few fancy I-descriptors that compare the previous FSTAT.HIST record to the current one I  can get things like BytesAddedPerDay & RecsAddedPerDay  & averages & see when things become unstable.

 DICT FSTAT.HIST DATABYTES.PERDAY

0001: I works on lists sorted BY[.DSND] FILENAME BY[.DSND] RUNDATE
0002: @2;FILENAME; @4;RUNDATE; @6;DATABYTES; IF @1=@2 & @3#@4 THEN (@5-@6)/(@3-@4) ELSE ''
0003:
0004: DataBytesý / Day
0005: 9R0Z
0006: S

------------------------------
Chuck Stevenson
DBA / SW Developer
Pomeroy
US
------------------------------

I like the idea of a FSTAT.HIST file.  I had never thought of doing it that way, but it would certainly allow better reporting for trends.  Now I just need to come up with some I-Descriptors to generate the reports.

------------------------------
Joseph von Arx
Software Developer
Data Management Associates Inc DMA
Cincinnati OH US
------------------------------

I like the idea of a FSTAT.HIST file.  I had never thought of doing it that way, but it would certainly allow better reporting for trends.  Now I just need to come up with some I-Descriptors to generate the reports.

------------------------------
Joseph von Arx
Software Developer
Data Management Associates Inc DMA
Cincinnati OH US
------------------------------

Does that  sample i-descriptor make sense?  I can explain it if anyone wants.

------------------------------
Chuck Stevenson
DBA / SW Developer
Pomeroy
US
------------------------------

Does that  sample i-descriptor make sense?  I can explain it if anyone wants.

------------------------------
Chuck Stevenson
DBA / SW Developer
Pomeroy
US
------------------------------

I think your sample I-Descriptor is a little confusing to me because I cannot understand how you can reference @ positions 2, 4, and 6 before they are even defined in the I-Descriptor.  I think that may require additional explanation for those less familiar with constructing these dictionaries.  I think we understand the intent of the sample based on the name, but the execution needs a little clarification.

The trick for me is to develop some new I-Descriptors that will allow trending and forecasting to more accurately determine what values to use in a RESIZE command to account for estimated growth plus additional padding.  There is a lot to chew on and understand before I get that far, though.

------------------------------
Joseph von Arx
Software Developer
Data Management Associates Inc DMA
Cincinnati OH US
------------------------------

I think your sample I-Descriptor is a little confusing to me because I cannot understand how you can reference @ positions 2, 4, and 6 before they are even defined in the I-Descriptor.  I think that may require additional explanation for those less familiar with constructing these dictionaries.  I think we understand the intent of the sample based on the name, but the execution needs a little clarification.

The trick for me is to develop some new I-Descriptors that will allow trending and forecasting to more accurately determine what values to use in a RESIZE command to account for estimated growth plus additional padding.  There is a lot to chew on and understand before I get that far, though.

------------------------------
Joseph von Arx
Software Developer
Data Management Associates Inc DMA
Cincinnati OH US
------------------------------

re. I-Descriptor is a little confusing to me because I cannot understand how you can reference @ positions 2, 4, and 6 before they are even defined in the I-Descriptor

By referencing @2 from 1st position, you get the value as it was stored in the previous execution of the i-descriptor, i.e., the previous record's value.   That's why I said it only works from a presorted select list. For the very 1st record @2 comes back null.  

 It has always worked that way in UV and PI before. But it is undocumented so they've never actually committed to it, but I can't imagine them taking it away.   If they did, at very worst the logic could be replaced with a subroutine that caches the record to common for comparison on subsequent executions.

UD works differently.  In UV these buffers are i-descriptor-specific.    In UD they are shared across all I-descriptors.  At least that's how it worked when I last looked, lo, these many years ago.

------------------------------
Chuck Stevenson
DBA / SW Developer
Pomeroy
US
------------------------------