If you have never heard the term black box, it is essentially a system where inputs/actions are sent, “something” happens, and an output/result is received. Generally speaking, the user of a black box doesn’t need to know the “something” that creates the intended result. In other words, you don’t need to know “how” the water is being produced from the faucet. You just know that if you turn the knob, “something” happens and then water will come out.
With the onslaught of Artificial Intelligence (a black box on steroids) hitting the market, business owners should pay close attention to not only how these new methods of working can help them provide better results for their customers but also how they can harm the business/customer relationship when you don’t understand how they work.
I’ve highlighted two instances below where companies have used systems making automated decisions which had a negative impact on the customer experience. One was with Fidelity Investments and the other with PayPal.
Fidelity Investments
While trying to establish an EFT link to a new bank account, Fidelity’s systems continued to reject the bank connection even though all supplied information was correct.
Normally I don’t mind the occasional “flagging” of activity, especially with financial systems. It shows the company is serious about security. The problem arises when no company representative can tell you “why” the system is rejecting the connection so you can take corrective measures.
Even after elevating the issue to my account manager, the best explanation they had was that they subscribed to a service and that the service will only give them a ‘yes’ or a ‘no’ and never a ‘why’. Essentially a black box.
The black box in question was from Early Warning, the company behind Zelle, which consumes and manipulates bank supplied customer transaction data to provide the tool used by Fidelity. Get your consumer report here: https://www.earlywarning.com/consumer-information
At the end of the day, Fidelity’s decision to use this “intelligent” system without knowing what it was doing forced their customer to:
- Dig through boxes to find a check book.
- Void a check.
- Download, print out, and fill in a paper form.
- Find an envelope and address it.
- Dig through boxes again to find a stamp.
- Walk the letter to the mailbox.
- Wait for a postal employee to drive to my house and retrieve the letter.
- Wait for the letter be transported via car, truck, and air to Fidelity.
- Wait for an employee Fidelity is paying to open the letter and process the same data I entered initially.
PayPal
With PayPal, a non-profit client had a large sum of money they needed to withdraw from their business PayPal account to fund an upcoming event.
Despite having many years of historical transaction data to review, PayPal would not release* the donations in the account as the activity was deemed ‘suspicious’.
After days and days of back and forth with PayPal representatives, no one could (or would) give a firm explanation as to “why” the funds were being held. The best explanation PayPal would provide was that their systems determined it was a risk and that telling my client “why” would pose a security “risk”.
We will never be able to find out whether or not PayPal customer service knew why their system decided to lock up funds. The opaqueness of PayPal’s systems, policies, and responses to their customer ultimately resulted in PayPal (and its subsidiary Venmo) losing all of my business, as well as never being referred again to my clients needing payment processing.
Additional read: A PayPal Policy at Scale
* It is important to note here that when PayPal takes this kind of action, PayPal can lock away your funds anywhere from 21 days up to six months, per their user agreement.