If it's acceptable to require button press/release, then the user presses the button before starting the swipe, then releases it at the end of the swipe. Then you compare the start and end positions. The axis with the largest change in position, and its direction, is your swipe. Example, from 1,4 to 20,100 is an up swipe.

This would be like swiping on a virtual touch screen. the button press being like your finger contacting the screen, and the release being like your finger lifting off the screen.

Otherwise, you could try filtering out "swipes" shorter than some threshold, then "decode" the remaining (large) swipes the same as above.

You will need to experiment to determine the threshold.

It is possible to devise an adaptive algorithm for distinguishing random hand movement from intentional swipes by tracking the largest "small" movement over a sliding wondow window of a few minutes, then accepting a movement significantly larger as a control swipe. Again, you will have to experiment.