roboticus explained it from the implementation side, but you can also look to the specification:
A frame has an associated type. Each frame belonging to the same
message contains the same type of data. Broadly speaking, there are
types for textual data (which is interpreted as UTF-8 [RFC3629]
text), binary data (whose interpretation is left up to the
application), and control frames ...
So Net::WebSocket::Server allows you to specify handlers for the two kinds of data frames.
That would be a JS issue; you haven't shown your JS code and I don't see this "type='binary'" in the WebSocket API. This works fine for me:
var socket = new WebSocket("ws://localhost:8080");
socket.send("Hello"); // sent as text (UTF-8)
socket.send( new Blob(["World"]) ); // sent as binary