Jun 28, 2013

Confidence Level Crisis

When you're - like me - a born professional optimist, but nevertheless sometimes worry about the unavoidable misery in the world, you ask yourself this question:

Why does God not act? 

Think about this question and try to answer it, before reading any further..



The answer to this question is very simple:

God does not act because he's conscious of everything  

The moral of this anecdote is that when you're fully aware of all the risks and their possible impact, chances are high you'll not be able to take any well-argued decision at all, as any decision will eventually fail when your objective is to rule out all possible risks.

You see, a question has come up that we can't agree on,
perhaps because we've read too many books.


Bertolt Brecht, Life of Galileo (Leben des Galilei)

On the other hand, if you're not risk-conscious at all regarding a decision to be taken, most probably you'll take the wrong decision.

'Mathematical Confident'
So this leaves us with the inevitable conclusion that in our eager to take risk-based decisions, a reasoned decision is nothing more than the somehow optimized outcome of a weighted sum of a limited number of subjective perceived risks. 'Perceived' and 'Weighted', thanks to the fact that we're unaware of certain risks, or 'filter', 'manipulate' or 'model' risks in such a way that we can be 'mathematical confident'. In other words, we've become victims of the "My calculator tells me I'm right! - Effect".

Risk Consciousness Fallacy
This way of taking risk based decisions has the 'advantage' that practice will prove it's never quite right. Implying you can gradually 'adjust' and 'improve' or 'optimize' your decision model endlessly.
Endlessly, up to the point where you've included so much new or adjusted risk sources and possible impacts, that the degrees in freedom of being able to take a 'confident' decision have become zero.


Risk & Investment Management Crisis
After a number of crises - in particular the 2008 systemic crisis - we've come to the point that we realize:
  • There are much more types of risk than we thought there would be
  • Most type of risks are nonlinear instead of linear
  • New risks are constantly 'born'
  • We'll not ever be able to identify or significantly control every possible kind of risk
  • Our current (outdated) investment model can't capture nonlinear risk
  • Most (investment) risks depend heavily on political measures and policy
  • Investment risks are more artificial and political based and driven, than statistical
  • Market Values are 'manipulable' and therefore 'artificial'
  • Risk free rates are volatile, unsure and decreasing
  • Traditional mathematical calculated 'confidence levels' fall short (model risk)
  • As Confidence Levels rise, Confidence Intervals and Value at Risk increase

Fallacy
One of the most basic implicit fallacies in investment modeling, is that mathematical confidence levels based on historical data are seen as 'trusted' confidence levels regarding future projections. Key point is that a confidence level (itself) is a conditional (Bayesian) probability .

Let's illustrate this in short.
A calculated model confidence level (CL) is only valid under the 'condition' that the 'Risk Structure' (e.g. mean, standard deviation, moments, etc.) of our analysed historical data set (H) that is used for modeling, is also valid in the future (F). This implies that our traditional confidence level is in fact a conditional probability : P(confidence level = x% | F=H ).

Example
  • The (increasing) Basel III confidence level is set at P( x ∈ VaR-Confidence-Interval | F=H) = 99.9% in accordance with a one year default level of 0.1% (= 1-99,9%).
  • Now please estimate roughly the probability P(F=H), that the risk structure of the historical (asset classes and obligations) data set (H) that is used for Basel III calculations, will also be 100% valid in the near future (F).
  • Let's assume you rate this probability based on the enormous economic shifts in our economy (optimistic and independent) at P(F=H)=95% for the next year.
  • The actual unconditional confidence level now becomes P( x ∈ VaR-Confidence-Interval) = P( x ∈ VaR-Confidence-Interval | F=H) × P(F=H) = 99.9% × 95% = 94.905%
Although a lot of remarks could be made whether the above method is scientifically 100% correct, one thing is sure: traditional risk methods in combination with sky high confidence levels fall short in times of economic shifts (currency wars, economic stagnation, etc). Or in other words:

Unconditional Financial Institutions Confidence Levels will be in line with our own poor economic forecast confidence levels. 



A detailed Societe Generale (SG) report tells us that not only economic forecasts like GDP growth, but also stocks can not be forecasted by analysts.


Over the period 2000-2006 the US average 24-month forecast error is 93% (12-month: 47%). With an average 24-month forecast error of 95% (12-month: 43%), Europe doesn't do any better. Forecasts with this kind of scale of error are totally worthless.

Confidence Level Crisis
Just focusing on sky high risk confidence levels of 99.9% or more is prohibiting financial institutions to take risks that are fundamental to their existence. 'Taking Risk' is part of the core business of a financial institution. Elimination of risk will therefore kill financial institutions on the long run. One way or the other, we have to deal with this Confidence Level Crisis.

The way out
The way for financial institutions to get out of this risk paradox is to recognize, identify and examine nonlinear and systemic risks and to structure not only capital, but also assets and obligations in such a (dynamic) way that they are financial and economic 'crisis proof'. All this without being blinded by a 'one point' theoretical Confidence Level..

Actuaries, econometricians and economists can help by developing nonlinear interactive asset models that demonstrate how (much) returns and risks and strategies are interrelated in a dynamic economic environment of continuing crises.

This way boards, management and investment advisory committees are supported in their continuous decision process to add value to all stakeholders and across all assets, obligations and capital.

Calculating small default probabilities in the order of the Planck Constant (6.626 069 57 x 10-34 J.s) are useless. Only creating strategies that prevent defaults, make sense.

Let's get more confident! ;-)

Sources/Links
- SG-Report: Mind Matters (Forecasting fails)
Are Men Overconfident Users?

Jun 4, 2013

Europe: Who Pays Who?

Where are we heading with Europe? Let's examine some main topics that illustrate Europe's future direction.

EU Budget Development
The European budget comprises roughly 130,000 Million Euros yearly. The ongoing (2013) budget process in terms of 'net contributions' (= Benefits -Contribution) per country is not transparent. Only the figures until 2011 are available.

Let's take a look a the net contribution development from the start of Europe (2000) until the last available data in 2011. Countries that (net) receive money are 'placed above the x-axis, countries that (net) have to pay money are placed below the x-axis.


Clear is that the amount of money that is transferred from the North&West European countries to the South&East European countries is growing fast from around 15,000 Million Euro in 2000 to around 34,000 Million Euro in 2011.

In a stable Europe the net contributions should decline and vary. However, the opposite is the case!

Who Pays, Who Receives?
It's interesting to look at which countries in Europe are financing other countries.

Take a look at the 2011 transfer of money between all European countries. To create a clear overview, the numbers are rounded in Millions. As a consequence some rows or columns  might not exactly sum up to their total.


To illustrate this table:
  • In 2011 Poland received 10,975 Million Euro, of which The Netherlands payed 711 Million Euro of the total of 2,214 Million Euro The Netherlands (net) payed
  • Germany payed a net total of 9,003 Million Euros, of which 1,217 Million euro was payed to Greece

It really gets frightening when we analyse the total amount of money transferred between 2000 and 2011:


This table tells us inter alia:
  • Over the period 2000-2011 a total amount of 253,119 Million Euro was transferred from North&West to South&East Europe. The end of this imbalanced flow of money is not yest in sight. The future looks dark, due to the current crises in Greece and Cyprus, as well as the worrying situation in Spain and Portugal. 
  • Spain has already received 63,563 Million Euro of which 22,115 Million Euro was payed by Germany.
  • Germany pay 88,067 Million Euro to support other structural failing European countries, and will continue to do so...... 

Declining Support
On top of the bad financial perspective as presented above, PEW Research recently published, the next survey results:


Only in Germany a small majority still 'believes' in Europe... Other European countries have given up on Europe. Overall, the support for an 'Integral Europe' is declining...

Let's end with the trust European countries have in each other. I leave the trivial conclusions up to the intelligent readers of this blog.. Please don't laugh, while examining the outcomes.


Conclusion
Europe..., how long will it last....???????

 Sources/Links
- Spreadsheets with data used in this blog
- EU Budget 2011
- Folketinget: Net EU Contributions
- PEW Research: The New Sick Man of Europe: the European Union

Apr 30, 2013

Willem-Alexander, the New Dutch King

Today - April 30, 2013 - is a special day for The Netherlands.

After 123 years of Dutch queens and a 33-year reign of Queen Beatrix, Willem-Alexander (46) has become the new king in the Netherlands. He's also the youngest monarch in Europe.



A Modern King
Willem-Alexander, a modern King who - together with his wife Maxima and their three lovely daughters - makes his first 'statement' by demonstrating he stands with and for the people that he represents, as his complete family spontaneously appears on stage together with the world's No. 1 DJ Armin van Buuren and the the Dutch Royal Concertgebouw Orchestra.




New Challenges
In his inauguration speech Willem-Alexander stated he's taking the job at a time when many people in the Netherlands feel vulnerable and uncertain. Vulnerable in their work or health. Uncertain about their income or home environment.

Unemployment
And indeed, according to Eurostat unemployment rates in The Netherlands (6.4%) and Europe (10.9%)  are spiking. In Spain even 26.7% of the population is jobless....








Moreover, younger people (under age 25) suffer most from unemployment:



A new generation with good intentions
Willem-Alexander stated that he's concerned about these developments and will contribute to a better world through cooperation, by strengthening the bond of mutual trust between the people and their government, maintain our democracy and serve the public interest.

We need a positive new generation with people like Willem-Alexander.
Let's hope he succeeds! 

Sources 
- Picture: Volkskrant

Apr 5, 2013

S&P-500 or Bonds?

On March 28 2013 the S&P 500 hit a new record
1,569.19 Up 6.34(0.41%) Mar 28

Key question is of course will 'L'histoire se répète".....???


Now, take a short look at the (above) S&P 500 last decades development.

Let nature do its work by drowning your brain in the unstoppable growth of debt and considering the fiscal cliffs and endless Quantitative Easing  (QE) programs.

Ask yourself... will new QE-X programs really offer any help....

Without any doubt or any complex investment analyses it's clear that we're heading for a Jungfrau's downfall.

The question is not if, but when exactly and how deep?


S&P 500 
Every year Aswath Damodaran, professor  of Finance at the Stern School of Business at NYU, updates the S&P 500 yearly total return and compares it with the yearly performance of 10Y U.S. Treasury Bonds and Treasury Bills.

Last year's (2012) performance comes down to:

- S&P 500 : 15.83%  ('stocks')
- T. Bills               : 0.05%
- 10Y T. Bonds : 2.97%

Let's take a look at a more general summary of his conclusions: 



Difficult Choice
Most financial institutions (pension funds, insurers, investment funds, banks) are at the crossroad of taking difficult decisions. Investing in 10Y Bonds with an artificial and historical low interest rate of around 2-3% with the risk of depreciation in case of raising interest rates, due to inflation or otherwise. Or going for 'risk' by investing in S&P 500 like funds with relatively high risk.......

In order to get more sight at this 'risk' issue, let's take a look at the 10 and 5 years development:



From this quick investigation it becomes painfully clear that - despite whatever the risk free rate may be - all risk indicators (sharpe, Sortino) point out that the risk on s&P 500 stocks is not adequately rewarded. The M2 (Modigliani risk-adjusted performance) indicator expresses that same fact more intuitively by showing a  10y S&P 500 fictive return of 4.2% if we correct the 10Y average performance of 7.9% for the additional risk level of S&P 500 stocks (against the risk level of 10Y bonds).

Sharpe, wider and in detail
As becomes clear from the next historical sharpe chart, the last decade is not really convincing that an S&P

 500 strategy will pay out......



Take a long Breath...
To confidentially execute a S&P 500 investment strategy it takes a period of 17 years (or more) to avoid an average negative return, as the next charts shows.



In practice this implies that mainly pension funds - with long investment horizons of 15 years and longer - can benefit more or less long-term riskless  from a S&P 500 investment strategy.

However, even from a 17-year cycle perspective it's clear we're still in a  long-term downward trend.

Don't worry, if 'math' shows we're out of options, we can always pray!

Links
-  Damodaran Blog: A Sweet Spot for US Equities: Opportunity and Dangers
- Yahoo S&P 500
- Spreadsheet: historical returns S&P 500 - Bonds
- Spreadsheet: S&P-500 Analysis

Mar 25, 2013

Compliance Plus Check

Most often and at a basic level, compliance is perceived and defined as 'regulatory compliance'.

If we comply (act in accordance) with federal or local or local authorities and their requirements, making sure that our company is following all the necessary rules and regulations, we seem done.

Compliance Plus...

However, the regulatory compliance level is only a start. 

It's simply not enough to be 'Supervisory Compliant'. Compliance is more! Much more.....

Let's find out what Compliance Plus represents...


Compliance, a Competitive Tool
In practice, compliance can be a distinctive competitive tool​. Being ahead of regulatory compliance, means that a company defines its own compliance level that not only includes regulatory compliance, but also its own professional compliance level, based on its own specific risk structure. 


Compliance, Part of Risk Management
Moreover, one compliance level higher, compliance also embeds the risk of future change of regulations rules and the not-yet-defined compliance rules of future products and investment strategies that are necessary for a sustainable successful development of your company.

As  regulation changes when the economic of financial circumstances change, regulatory compliance is also a substantial part of risk management. That's why regulatory compliance has to be included in our risk models. 

Compliance Check
To check if your company is compliant at a Compliance Plus Level, simply 


at Symetrics. It takes only two minutes to take this test, as all compliance check boxes are already 'checked' in advance. So if you like 'unticking boxes' instead of 'ticking boxes', here's a splendid opportunity.

Mar 17, 2013

AIFMD Fun of Funds

To prevent future crises, a new European law, the Alternative Investment Fund Managers Directive (AIFMD)  came into force on 22 July 2011.

The new directive has to be implemented before 22 July 2013 and will also apply to non-EU fund managers if they ares managing or marketing an AIF to investors in the EU.

It is believed that the directive will reduce the number of non-EU managers operating within the EU.

AIF's assets and risk management
Although the AIFM-Directive has many new demands (appoint independent valuer, custodian, disclosure) we'll focus here on the requirement to ensure an independent evaluation of the AIF's assets and risk management.

AIFMD Risk Management Obligations
  • Every AIFM needs to have an adequate documented risk management policy, covering all possible risks faced by the AIFs
  • Every AIFM has to set quantitative and qualitative risk limits for each AIF for all possible kind of risks 
  • An AIFM's Risk Measurement Procedure should include requirements for: backtesting, stress-testing, scenario analyses and the rules should describe remedial action plans when limits are breached. 

So far so good, peace of a cake, you would think. Unfortunately: NO!

1. In-depth market risk assessment: too complex and not adequate
An adequate in-depth market risk assessment of AIFMs AIFs actually requires a full 'fund of funds' transparency of the portfolio of the (AIF) funds.

The problem is that full 'fund of funds' transparency does not exist yet, nor can it finally be fully obtained. It's simply too complex:
  • undefined systemic risks are often beneath the analyse surface
  • (re)hedged risks could be part of a fatal unknown or unapparent self-reference hedge cycle
  • in-depth 'fund of funds' management is time consuming and presumes that risk profiles of sub-funds are available, when in practice they are often not

To illustrate the desperate, funny and useless efforts that are made to tame the 'fund of funds' issues within the AIFMD, just take a look at the next quote from the AIF Handbook draft 2013 :

Section 5-iii-1, Alias 'Fun' of Funds
"Any proposed investment by a Qualifying Investor AIF into another investment fund must be clearly disclosed.
Disclosure must focus on the implications of this policy regarding 
increased costs to unitholders (i.e. the fact that fees will arise at two or, in the cases where the underlying fund it itself a fund of funds, three levels – the Qualifying Investor AIF, the underlying fund of funds and the underlying funds in which the underlying fund of funds invests) and the resultant lack of transparency in investments."

I hope you're still with me after all this fund of funds of funds of funds fun..... ;-)

Thus, in-depth market risk assessments in a non transparent market are inadequate and may potentially result in ill-founded or even erroneous conclusions (e.g.' false safety').

Market Risk Assessment
The adequacy of an AIFMD's Market Risk Assessment could be roughly defined as:

MRA-Adequacy = ADTQ x RPQ x RMQ


With: ADTQ= Asset Data Transparency Quality, RPQ= Risk Policy Quality and RMQ = Risk Model Quality.

Just let your colleague rate your ADTQ, RPQ and RMQ on a ten point scale. If the outcome MRA-Adequacy is lower than 800, consider your test as inadequate.

As transparency also includes full sub-cycle  'fund of funds' transparency, often ADTQ will not score high enough for an adequate test outcome.

Example
Suppose an AIF consists of 30% 'fund of funds' with minor risk information regarding the sub-funds.All other scores of the AIF score well (10). In this case the test adequacy score is 700 (= 7 x 10 x 10) . Conclusion: the quality of your risk assessment is insufficient for drawing robust conclusions.

2. Alternative: Strategic Market Risk Assessment
Instead of - come what may - trying to get to the endless bottom of a 'fund to fund' construction, a more strategic risk assessment approach -  as an alternative -could work out much more effective. A strategic market risk assessment that assesses the nature, risk and policy of a AIF and its investments and that implicitly takes into account non-linear risks, the presence of systemic risk, a large number of weighted and not-weighted economic scenarios, stress tests and fat tail risks.

The Secret of the Chef
Many (hedge) funds have only a limited transparent investment policy or an investment policy that  - for whatever reason -is regarded as 'The Secret of the Chef'.

In these kind of funds 'full disclosure' will end in a lot of degrees of freedom in 'risk policy' and corresponding mandates.

It's important to realize that the more degrees of  freedom in 'risk policy' a manager of a fund has, the more risk will emerge in the above formulated alternative assessment.

New alternative market risk models?
Key question is: are there new models that can assess investment strategies and portfolios in a systemic risk environment and on basis of non-linear modeling.

The answer to this question is : Yes, very soon!

Symetrics, a brand new company in the Netherlands is developing an investment decision support and assessment system called SyMath, that is based on nonlinear modeling, grasps systemic risk and includes future crises. SyMath will be on the market mid 2013.



Until then will have to assess AIFMs with pen and pencil... ;-)


Links, Used Sources


Feb 26, 2013

NOT Discriminating is NOT possible

Tomorrow I'll be discussing the borders of solidarity as a panel member at an actuarial congress (VSAE)  for econometricians in The Hague (The Netherlands).

In a Dutch interview preceding the congress, two students asked me:

"The  Court of Justice of the European Union (CJEU) has decided that the use of gender as a risk factor by insurers should not lead to individual differences in premiums and benefits.
What is your opinion?"

My short answer was :

NOT Discriminating is 
NOT Possible

Examples
Let me illustrate this 'quantum quote' with two examples.

Example I: Gender Neutral Car Insurance 
  • It's scientifically proven that women are better drivers, have just as much car accidents as men, but cause less damage. That's a fact and that's why car insurance for women is cheaper than for men. 
  • As from December 21, 2012, European insurers are not allowed to 'discriminate' anymore by gender, implying equal car insurance premiums for men and women. 
  • If insurers calculate this premium as the weighted average of their portfolio, women are obliged to pay (much) more premium than before and also more than actually and actuarially necessary regarding their gender group. 
  • Therefore women are de facto discriminated, although the genuine intention was NOT to discriminate!
  • Not only women, but also insurers are discriminated as they now will be faced with anti-selection: Relatively more men will choose an insurance cover, as car insurance premiums for men have become less than the premium corresponding with the expected damage for their gender group. Insurers will therefore face a loss on car insurance. 
  • Based on solvency legislation, the insurer will (next) be 'forced' to increase the average weighted premium. This - in turn - is at odds with the measures envisaged by the European Court. 
  • A similar kind of reasoning applies also for unisex rates for pension and life insurance.
  • The upcoming (2014) US health care law will also prohibit “gender rating”. However, gaps persist in most states. There seem to be no signs of insurers that have taken steps to reduce them.

The conclusion must be that discrimination regulation is carried too far.

 'Over-Solidarity' as in this case has nothing to do with real solidarity and is in nobody's interest; it has become 'Anti-solidarity'.

The proposed measures - no matter how well intended - have a opposite effect and should be reconsidered on basis of the question: are the discriminating effects before the new legislation more or less than after?

We've got to stop discrimination due to over-discrimination and anti-discrimination!

Insurance Rating Fallacy: Gender anti-discrimination laws are superseded 
Prohibiting "gender", "marital status" and "age" as rating elements doesn't solve anything.

Modern rating systems based on data mining (Google history), social media (premium quoting on basis of: your smart-phone that captures and shares your drivingstyle with the insurer) and neural networks are "black boxes" that quote insurance premiums in such a way that every client can get individually quoted on bases of his 'profile'.


That 'profile' doesn't have to contain any of the forbidden discriminating elements (nor direct related) to get satisfactory results for clients as well as insurers. Although there are also simple (e.g. Bayesian-Classification) techniques to derive a clients gender from other non-discriminant related variables (e.g. height, weight and foot-size determine gender quite accurately) in an insurers direct or indirect related data base, insurers and their actuaries would end up in an unwanted ethical dilemma by using these direct-related techniques.

Another illustrative and strong example of determining your gender on bases of - at first sight - non-gender-related information is Hacker Factor's "Gender Guesser"  that attempts to determine an author's gender based on the words used. Try  "Gender Guesser" for yourself HERE. Take a part of an email you've written (more than 300 words), copy-paste it to Gender Guesser and notice how gender Guesser will probably determine your gender without any problem in a split of a second!


These simple techniques show that the developed anti discrimination legislation is superseded and has become irrelevant for insurers and their clients to come anyhow to an adequate and ethical responsible rating policy on basis of neural networks or social media related information, such as information from smartphones that transmit your driving style information to the insurer (why not, if you have nothing to hide?).

Example 2:.Women on Boards: Commission proposes 40% objective
The European Commission has proposed legislation with the aim of attaining a 40% women presence objective in non-executive board-member positions of publicly listed large companies.
Currently, large boards are dominated by men (85% non-executive, 91% executive).

No matter how welcome and needful women are on board level, forcing such a development makes no sense and will have an adverse effect.

From experience I can tell that women who really qualify for board level positions, are very unhappy if they are appointed under the vigor of gender legislation and not on basis of their acknowledged competences.

This is perhaps a sign that women who really qualify feel discriminated by this new proposal. Proposals should better emphasize on stimulating women presence on board level and take away old boys network principles.

Conclusion
Anti discriminating legislation often results in the exact opposite of what is intended. Legislation is often superseded, should be carefully evaluated on its effects and certainly reconsidered if the discriminating effects after applying the new legislation increase.


Used Sources and Links

Humor: Actuarial Creativity

As actuaries we've studied a lot in life. And to keep up with actuarial science we'll probably keep studying until our personal mortality rate hits us finally in the back.

Although study brought us to the top of financial and statistic modeling, there's a small but fatal risk that we become so engrossed in our work that we loose our creativity or ability to solve things in a simple way.

Test

To test whether you're still a creative 'simplist', let's do a short 3 question test. Here it is:

Question 1
 "Show how it is possible to determine the height of a tall building with the aid of a barometer."



If you think you've solved this high school level problem, go to the next question

Question 2
 "Solve question 1 with another method."

If you think you've solved this problem as wel, go to the final question

Question 3
 "Solve question 1 with 4 other methods."

Evaluation
Although actuaries never give up, there's a slight chance you had to surrender and are longing for the answer.
In that case (only), read further for the answer.

Answer: The Barometer Fable
Bob Pease (Nat.Semi.) records the story of the Physics student who got the following question in an exam: "You are given an accurate barometer, how would you use it to determine the height of a skyscraper ?"

  1. He answered: "Go to the top floor, tie a long piece of string to the barometer, let it down 'till it touches the ground and measure the length of the string".

    The examiner wasn't satisfied, so they decided to interview the guy: "Can you give us another method, one which demonstrates your knowledge of Physics ?"
     
  2.  "Sure, go to the top floor, drop the barometer off, and measure how long before it hits the ground……"

    "Not, quite what we wanted, care to try again ?"
     
  3. "Make a pendulum of the barometer, measure its period at the bottom, then measure its period at the top……"

    "..another try ?…."
     
  4. "Measure the length of the barometer, then mount it vertically on the ground on a sunny day and measure its shadow, measure the shadow of the skyscraper….."

    "….and again ?…."
     
  5. "walk up the stairs and use the barometer as a ruler to measure the height of the walls in the stairwells."

    "…One more try ?"
     
  6. "Find where the janitor lives, knock on his door and say
    'Please, Mr Janitor, if I give you this nice Barometer, will you tell me the height of this building ?"


Find more than 140 solutions and read the original famous Barometer Fable, as published in 1968 in an article  by Alexander Calandra.

Warning!
Keep in mind that not every method leads to satisfactory results.
An uncertainty analysis of determining a building height using a barometer, developed by Israel Urieli, shows that this method is not accurate at all!

So the surprising news is that the first two alternative methods mentioned above are more accurate than the method you learn at high school.

Finally
It's always best when you can solve an (actuarial) problem in more than one way and the outcomes point in the same direction. The more a specific solution comes to front by applying different methods and/or data, the more confident you can be that the outcome is robust.

Used Sources