So when a Fed governor, Jeremy Stein, takes time away from all those big picture things as he did in a recent speech to highlight an obscure sub-corner of a sub-corner of the financial markets, in this case mortgage REITs, it should make you stop and think. And when he throws in charts like this one it might even cause you to say you’re terrified:

Indeed, as the chart shows, mortgage real estate investment trusts (REITs) have grown a lot, particularly since the credit crisis. And they are odd and seemingly complicated financial things.

Mortgage REITs don’t actually do any home lending, at least not directly to borrowers, and they don’t service any loans. In Wall Street terms, they are the very definition of a carry trade. They borrow money at low short-term rates and use that money to buy up longer-term assets paying higher interest, in this case mortgages, collecting the difference between those as profits.

And that, perhaps surprisingly in the wake of the housing bust and financial crisis, has worked out very well. The pull-back of Fannie Mae and Freddie Mac after the financial crisis, at least initially, created a void of buyers for mortgage bonds. What’s more, the Fed has made borrowing very cheap. On top of that, at a time of low interest rates, mortgage REITs have attracted a lot of investors because they pay high dividends, pushing up their stock prices, which as a result has made it even easier for them to go on their mortgage bond buying binge.

MORE: Why some homeowners are turning down free money

And that’s where we are now. The good news is despite the scary chart, mortgage REITs are still a very small portion of the mortgage bond market, just 10% of the $5.6 trillion market for government-insured mortgages. And the Fed is buying $40 billion in mortgage bonds a month. So it wouldn’t take the Fed that long to suck up all the extra volume if somehow mortgage REITs ran into problems.

The bad news is that mortgage REITs do appear to be running into trouble. In a recent report on the sector, Nomura research analyst Bill Caracache included a chart on what he calls economic return. For nearly all the mortgage REITs it was either zero or negative. Annaly Capital Management (NLY), the largest of the mortgage REITs, had a negative economic return of 8.6%.

The problem: for the last year or so, the spread between short-term rates and mortgage rates has been shrinking, as mortgage rates have come down. The interest rate spread – the difference between what it has to pay to borrow and what the company makes in income from mortgages – at Annaly Capital, has shrunk to just 0.95 of a percentage point.

But Annaly still has to make its hefty dividends. The REIT has a yield of 12.65%. So Annaly needs to boost its returns. And there are a number of ways to do that, one of them being taking on more leverage, which is what Annaly has done. In the past year, its leverage ratio has grown to 6.5 from 5.4. That means it’s holding less capital.

MORE: We still have renters to thank for healthier housing market

Financial firms need capital to absorb losses. And Annaly could soon have to swallow some significant losses. The company has $125 billion in plain-vanilla mortgage bonds that would lose value if interest rates were to rise, which they have started to do recently, and many expect will continue this year.

Annaly’s management has contended that it has plenty of capital. They point out that their current capital ration of about 11% is higher than most banks, and more than three times as high as metric was at many of the nation’s largest banks, some of which were levered 30-to-1, going into the financial crisis.

But that doesn’t mean Annaly is totally safe. About two-thirds of its mortgage bonds don’t have to be repaid in full until 2042. The longer the time period until a bond is paid back, the more it will lose when interest rates rise. But most mortgage loans don’t last 30 years. The average life of a mortgage is more like 5-to-7 years. Still, if rates rise 2%, Annaly could lose $13 billion of its $17 billion in capital, nearly wiping out the firm’s capital.

Annaly has hedges in place that could limit its losses to a few billion dollars if interest rates were to rise. But there is a long history in financial markets of well-planned hedges failing.

And while Annaly is still small compared to the rest of the mortgage market, the fear is that it and other mortgage REITs would have to retreat at the same time the Fed is winding down its mortgage bond buying program, which some have said could happen as soon as later this year. That could cause a problem for banks and others that hold a lot of mortgage loans and bonds. But with government-backed mortgage bonds paying around 2.5%, is that really a risk you would take?