A friend of mine and I canoed a local river last late winter/spring. The spring rain had not come so the water was gin clear and on the low side. I saw really nice bows screaming around the water. The temperature was bitter cold. So I don't buy into the lethargic theory. The rainbows we saw were in fast riffles a lot and groups of 3-4. Boy were they fast and big. I think they get lethargic when it's too warm not cold. it's more difficult for them to breath when the temp is high
I do think the fish move around a lot. Depending on the rivers makeup that might dictate where they are and how often. There is a spot locally on a big river. It's an ideal run. If the fish are there you'll do great, big fish a plenty. When they aren't you can stick around for a few hours and eventually something will come by because they have to eat at some point.
On the other hand, when it rains I am sure the local smaller rivers/tribs rise and the fish explore up them to get out of the big rivers flow pressure and mud.
Like you I only swing. When you only swing then you might not have as much success as some other techniques or so those I know who nymph suggest. But that's OK.
If you want to catch a trout or steelhead on a dry bomber, then you fish a dry bomber until you do even if you go bust for months. If you want to catch trout in the winter swinging streamer patters, then do it up and know there will be days you walk away scratching your head. Its still great to be out there and I love casting the 2H as much as hooking up.
The best for me is when you look over the water and pretend you know a fishes'' mind and say, I'd be right there if it were me. Then you make a perfect cast, it all comes together and BAM you hook up with a beauty.
When that happens to me, I pack my gear and just hangout on the river bank watching the river flow