There is normally a big difference between 50 and 25 yards because the bullet is rising to the point of aim. It is also true that if you sight in dead on at 23 yards you will be very close to dead on at 100 yards.
If you have a high mounted scope with the scope center-line more than 1.7" above the bore center-line you get the biggest differences of POI at different ranges.
Generally big bores sighted 1" high at 50 are on at 100 if you have a good low scope mounting.
I suggest you sight in again and try what I do. Sight in with a small 1" bull target at 50 yards and adjust so your POI at 50 yards places the group center 1" above the center of the bull. You should be on at 100 and a little low at 25 while the highest POI is at ~75 yards This is normal for a big bore..and keeps you in the vital out 125 yards just by aiming at the center of the vital broadside.
Also note: the Handi is a very light rifle for the caliber and you may experience recoil having a serious impact on POI, especially from a LeadSled or a low rest if you lean into it. Shoot sitting straight up and not leaning into the rifle and use a high rest so you can sit up straight. I always avoid leaning into a big bore rifle when shooting for this very reason. It is NOT a shotgun at the skeet range.
If you really have a high scope mount that you can see under the scope at the open sights, Lose that nightmare for big bores and get a normal one with the scope bell no farther than 1/4" from the barrel and remove the rear sight if you need to do that to get the scope close to the barrel with the right scope mount.