Wrote a Monte Carlo simulation for interest rates and FX rates. It generally performed well but there was something off in the results. During testing two portfolios containing the exact opposite positions should have netted to zero. But that wasn't happening. We kept getting a small residual exposure.

Kept paring down the portfolios, then pared down the number of simulations, added LOTS of tracing data.

After weeks I isolated it to a bug in the DEC double precision floating point library.

In certain circumstances adding a very small positive number to the very small negative number of equal magnitude did not yield zero. (1e-09) + (-1e-09) would leave a small positive result.

Isolated the bug, created test cases to verify the problem, and we submitted them to DEC.

They were able to recreate the problem & issued a software fix to handle what was a hardware defect.

_________

In order to make progress in our development while waiting for DEC to identify and fix the problem we defined a variable EPSILON as 1e-08 and changed our code from

if ( (a + b) ==0) to if ((a + b) <EPSILON )

This might qualify as unforeseen by the designer but definitely foreseen by me.

A customer did a project to build a distributed FX trading system. The system itself ran on a local Novell network with a dedicated machine supporting an X.25 connection to the outside world. In addition there was another dedicated machine that acted as the local database for static information like the names, addresses, and network addresses of the different counterparties.

However the vendor decided not to use a 3rd party database like ORACLE but wrote their own storage system based on linked lists.

In reviewing the design I called out that the database was unlikely to support the type and volume of queries & that they should consider getting a 3rd party database rather than writing their own.

I was soundly voted down and quite rudely to "stay in my lane" which was production support and training. ( It was a contract gig I'd taken to fill some dead time. )

The vendor delivered the system and everybody was congratulating themselves after a successful demo. I noted that the vendor only ever picked from the first ten counterparties ( all starting with "A" or "B".). I asked the presenter to pull up the counterparty list & typed in 'Zaire Bank' which I knew was one of the banks supported. The system ran for a minute, then five minutes, then 15 minutes; all the time flashing a little light when the next record was retrieved.

There were about 300 counterparties in the database and each record required 5-10 seconds. So to get to the Zaire Bank was going to take somewhere between 1500 and 3000 seconds ( i.e. 25 and 50 minutes ).

There were other instances of retrieving data that demonstrated the same problem - for example, asking for a trade history that was greater than a few days - which was a pretty normal request.

By the time the customer and the vendor agreed on a redesign and reimplementation the market opportunity was gone.

Here's another one:

We wrote a Videotex system that ran on DEC PDP-11's and then subsequently on DEC VAX equipment.

We used one machine as a dedicated database machine.

Periodically when load was high a query from the database would come back as incomplete but only when it was running on one of the machines.

There would be no indication of error. However the end user application would report a slightly incorrect result.

We tested the query running on each machine by logging onto it directly. Everything worked perfectly.

Finally after many different attempts at diagnosis failed we pulled up the data center tiles.

And there, connecting the suspect machine, was a very long, coiled internet cable. We took it out and measured it and it was about 10% longer than the maximum allowable cable length. The installer either mismeasured or decided that the cable was close enough in length.

Replaced the cable with one that was met the specifications and the problem went away.