You do not need to remove the seat just to remove the backrest
If you have the "original" you must remove the pivot bolt at the base of the backrest that secures it to the permament bracket that is screwed to a mounting bracket from under the seat.
If you have the newer QUICK OUT option a sturdy jerk upward on the backrest will release it.
The back rest proper can be removed by un-screwing the hinge pin. (I replaced that with a square wire lock pin.) To remove the rest of the back rest, you have to take the seat off and un-bolt it from the bottom seat.