Now there's a migration task from TOMCAT to SAP WAS, no EJB stuff, all under J2EE specs except the connection between form-based authentication and database.
A Realm is a "database" of usernames and passwords that identify valid users of a web application (or set of web applications), plus an enumeration of the list of roles associated with each valid user.
In many cases, it is desireable to "connect" a servlet container to some existing authentication database or mechanism that already exists in the production environment.
In Tomcat, you can configure Realm in server.xml to enable this function with its own implementation, a <Realm> element and an implementation of org.apache.catalina.realm.Realm which is an interface.
I would like to know how's this situation realized in SAP WAS enviroment, a pluggable login modules or something else
have a look at the documentation about login modules in SAP NetWeaver.
It describes how to create and integrate your own login module. There's a example login module as well.
In addition have a look at how to configure login module stacks and policy configurations for an application on SAP Web AS.
Hope that helps,